Replies: 81 comments 33 replies
-
|
This is a limitation on the systemd side. They will only accept notifications, or PID files, that are created by or sent by root, for security reasons - even if the User and Group of the unit file are explicitly set to start the process as a non-root user. Their recommendation was to start the container as a user service of the user in question via |
Beta Was this translation helpful? Give feedback.
-
|
Previous discussion: #9642 |
Beta Was this translation helpful? Give feedback.
-
|
Thank you both. For now I've worked around it by managing the service under the user's systemd which is clunky to say the least. I don't understand systemd's security argument - if the process is run as a given user, why would systemd not allow that user's process to send sd_notify? Who else could? But I guess this is no flaw of podman. #9642 mentions some code changes that need to happen to podman for sd_notify, what are those? And have they progressed since March? I guess you could close this issue or use it to track progress. |
Beta Was this translation helpful? Give feedback.
-
Yes, there is some progress. The main PID is now communicated via sd notify but there are still some remaining issues. For instance, |
Beta Was this translation helpful? Give feedback.
-
|
I think the next big thing to tackle is finding a way how to lift the |
Beta Was this translation helpful? Give feedback.
-
But even that is rejected by systemd, as seen in the logs above. |
Beta Was this translation helpful? Give feedback.
-
|
I fear there's not much Podman can do at the moment. |
Beta Was this translation helpful? Give feedback.
-
|
Only after solving this problem can become truly rootless. So I have to keep using the root account for now. |
Beta Was this translation helpful? Give feedback.
-
|
Is there a quick overview what, at the moment, the best approach / workaround is for starting podman containers with systemd as a specific non-root user? Furthermore, if a container is run as root, is there a workaround how to change the ownership of files and directories created inside the container (in a bound volume) to a specific host user? |
Beta Was this translation helpful? Give feedback.
-
use use |
Beta Was this translation helpful? Give feedback.
-
The services need to be started and managed as the specific non-root user. Using the |
Beta Was this translation helpful? Give feedback.
-
For the moment my workaround is to run such containers in a systemd --user. This means that for every system service I want to run as a rootless container, I need to create a separate system user, enable linger, and run a separate systemd --user instance for that user. It works but it's clunky, e.g. restarting Nginx is Inside these rootless containers root is mapped to the system user, which is a different uid for each service. If something inside the containers runs as non-root, that gets mapped to a high-numbered host uid by default. However with some magic on the host you can map a specific non-root uid in the container to a host uid of your choice, which can then be mapped to a different non-root uid in a different container running under a different user. I should probably document my setup one of these days... |
Beta Was this translation helpful? Give feedback.
-
|
@Gchbg If you are running a recent systemd version (for instance by running Fedora 35), I think you could run No need to set |
Beta Was this translation helpful? Give feedback.
-
Is that
How does that relate to what @Gchbg and @eriksjolund wrote above? Do I have to run several instances of systemd or is there another way? For systemd beginners like me, it is quite difficult to understand the various layers of abstraction and user permission between systemd, host processes and containers. After all, I would assume that this is the use case for 80 % of the users: run some container service that gets restarted automatically when the machine boots and that is as restricted as possible (by means of user permissions). |
Beta Was this translation helpful? Give feedback.
-
I got it wrong, modifying UID and GID via env requires entrypoint.sh。 https://docs.docker.com/engine/security/userns-remap/ |
Beta Was this translation helpful? Give feedback.
-
|
@markstos Not really, just documenting the fact that you need to use |
Beta Was this translation helpful? Give feedback.
-
This is also what I need, and what seems to be a pretty valid use case. Additionally, for peristent state one can still use I actually almost got it working but ran against a brick wall with this issue, i.e. Hacking away with things like (I got the normal |
Beta Was this translation helpful? Give feedback.
-
Since I don't think it was mentioned yet, you should probably not do that since it can happen that the Podman process is killed while the container keeps running (see #9642 (reply in thread)). |
Beta Was this translation helpful? Give feedback.
-
|
I'm still confused why we're all having problems with this; clearly using User= is not the recommended/supported approach. So the recommended/supported approach really is to run containers as root? Am I missing something and people generally think that's ok? Non-containerised services wouldn't be running as root right? So why is it ok to run containerised services as root? |
Beta Was this translation helpful? Give feedback.
-
Personally, I have taken to just running it as a user service with lingering enabled. It still lets me start it on boot, and manage/observe it via systemctl and journalctl. |
Beta Was this translation helpful? Give feedback.
-
|
I think at this point we should change this to a discussion. User= causes lots of issues with running podman and rootless support is fairly easy. I also recomend that people look at using rootful with --userns=auto, which will run your containers each in a unigue user nemespace. |
Beta Was this translation helpful? Give feedback.
-
|
IDMap is only available in Rootful mode, Kernel does not support this for rootless mode. If you want to share the directory between containers running in different User Namespaces, then you either need to map the same group into each or setup the directory as setgroup and leak that group into each container and make the directory writable by group access, of make the directory world writable. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for the answer! I have just come across #17753, stating essentially the same. |
Beta Was this translation helpful? Give feedback.
-
@rhatdan I would be very grateful if you could elaborate on this approach a little further. If I understand correctly, you propose creating a new group which owns the directory of interest (directory writable by group and with
I am happy about every pointer! |
Beta Was this translation helpful? Give feedback.
-
|
I'm also interested in this. Each container will have their own data directory from the host's perspective, but the host wants the files created there to have same owner/group across all of them. For my threat model, it's fine if all the containers run as the same non-root host user. I seem to recall a podman option that allowed all user ids within a container to be squashed to a single user id from the host's perspective. I can't find an option like that now, but it might help for me if it exists. I've added the book to my "to read" list. So far I'm having luck using |
Beta Was this translation helpful? Give feedback.
-
|
Podman allows you to squash all of the UIDs/GIDs in an image to a single UID. But if the process within the container attempts to setuid to a different User, then Podman does nothing, the kernel would take over and either deny or allow it based on the user namespace the container is running in. The primary use case of squashing the UIDs/GIDs is for users with only their UID, IE no entries in /etc/subuid and /etc/subgid. |
Beta Was this translation helpful? Give feedback.
-
|
If you setup a directory that is setgid to a GID (foobar), and the user running podman is in the foobar group, then you can leak the foobar group into the container with --group-add keep-groups. This should allow any containers written this way to write to the group. |
Beta Was this translation helpful? Give feedback.
-
|
@giuseppe WDYT of allowing something like. podman run --userns=auto --gidmap=1000:1000:1 alpine cat /proc/self/uid_mapError: --userns and --uidmap/--gidmap/--subuidname/--subgidname are mutually exclusive |
Beta Was this translation helpful? Give feedback.
-
|
if you want to force a specific mapping with |
Beta Was this translation helpful? Give feedback.
-
|
Nice so you do this for two different containers as well with both sharing the same GID. |
Beta Was this translation helpful? Give feedback.
-
|
I tried some more with I don't know how robust the solution is but at least something is working. /etc/systemd/system/example3.service /etc/systemd/system/example3.socket Note that rootless podman runs the nginx container with socket activation (port 80) without being blocked by the I added this as Example 3 in |
Beta Was this translation helpful? Give feedback.
-
|
In case it helps anyone else trying to use rootless podman to run systemd user units, configured by ansible, on Rocky8 or Rocky9 I did a bit of experimenting here: https://github.com/sjpb/systemd-podman-experiments. The conclusion I drew is that actually this is still pretty user-unfriendly for my use-case and I think it'd still be really nice to just be able to set |
Beta Was this translation helpful? Give feedback.
-
|
I also considered trying to get this to work and gave up. My attempts to use --userns=auto only worked for a first container that had that, but not a second. I guess the automatically selected ranges conflicted. My next attempt to try might be more explicitly set non-overlapping ranges for the containers, or to get --userns=auto to work for mulitple containers (Maybe I need to set a much larger pool of ids in the first place? The default range had something like 65,000 Ids in it, and I only need less 10 per container, so it seems like there should be plenty. |
Beta Was this translation helpful? Give feedback.
-
|
I've been working on setting up a bare-metal environment that runs exclusively in ram on fcos and boots over pxe with an ignition file. My goal is to go from zero -> running github actions container / terraform cloud agent container securely without manual intervention. Admittedly I am pretty unfamiliar with linux internals and systemd, but I am trying to familiarize myself. Ignition only supports systemd for launching services, so understanding this is pretty crucial. At this point I've been able to figure out every step of my config, but getting rootless containers working has been a thorn in my side. Trying to set the I'm a bit stumped here and am reverting back to running as root for now, but I'm open to any pointers (including this ignition process as a whole):
|
Beta Was this translation helpful? Give feedback.
-
|
+1 As quadlets were created with the idea of "integrating better with systemd", not supporting the systemd way of managing rootless services does not make sense. |
Beta Was this translation helpful? Give feedback.
-
|
What's the next-best-thing approach for running a container as systemd service, but as a non-root user that's easy and repeatable?
I would happy if all uids in the container were collapsed to a single user and I could specific that user with |
Beta Was this translation helpful? Give feedback.
-
|
Yes it does --userns=auto will look at all containers created and pick a range of UIDs/GIDs that are not in use. If you run with --userns=auto in rootless mode, then you can quickly run out of UIDs/GIDs. In rootful mode you have around 4 billion UIDs to share. In rootless by default you only have 65k. |
Beta Was this translation helpful? Give feedback.
-
rootless podman is "running podman in a user namespace" + "running workload in a user namespace". If you use --userns with root, you only lose the first half of rootless. Once the container runs there are no differences between root with userns and rootless. The confusion here seems to be around User=. That doesn't set the environment for rootless to work correctly: it does not set the user session so there are no tmp dirs, as well as no journal for logs. |
Beta Was this translation helpful? Give feedback.
-
|
I will note that the issues @giuseppe raised are fundamentally systemd problems - if those were fixed, rootless Podman would work with User= without issue. If you'd like this to work, please talk to the systemd maintainers; there's nothing the Podman team can do about it. For reference, we have had this conversation with the systemd team in the past, and the recommended solution back then was to use |
Beta Was this translation helpful? Give feedback.
-
@rhatdan Is there a way to work around this by e.g. changing the overflowuid value for a user namespace? Or is that a system-wide setting? |
Beta Was this translation helpful? Give feedback.
-
|
No way that I know of . |
Beta Was this translation helpful? Give feedback.
-
|
I was able to run rootless podman containers using systemd units with "User=" without using I just use normal service commands like Podman was run with the following options, where 1111 is the serviceuser's uid: Systemd unit file has the following lines (I posted only the most important lines): Subuids was configured for the serviceuser: Also linger state was enabled for servicesuser account: The key option is
@eriksjolund posted this option in his message. |
Beta Was this translation helpful? Give feedback.
-
|
Interesting, thank you. We use something similar except without |
Beta Was this translation helpful? Give feedback.
-
|
This seems to be the magic combination that makes
|
Beta Was this translation helpful? Give feedback.
-
|
An attempt to provide clarity to this method, here are the docs for all the mentioned options. I have not yet tested this method myself. Let's start with with podman options: podman run options
systemd unit options.
Interpretation
SummaryIt's amazing that someone found a combination of flags and directives which seem to enable User=, because even from reading all their docs together, it still not clear how they could work together to accomplish that! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the overview! Shallow and pedantic, but:
conmon
conmon Edit: Some clarifications, as far as I know of them:
The CID file is used as a sentinel so the service doesn't leak resources while stopping. Cf. #13236
Yes.
It's not the conmon process that handles service readiness, but the podman process. Since systemd looks at the conmon process to determine service uptime,
No, delegation here refers to cgroup handling. Normally only systemd is allowed to create and remove cgroups, but with |
Beta Was this translation helpful? Give feedback.
and others
-
Is this a BUG REPORT or FEATURE REQUEST?
/kind bug
Description
I want to have a systemd system service that runs a rootless container under an isolated user, but systemd rejects the sd_notify call and terminates the service.
A similar problem was menitoned but not resolved in #5572, which seems to have been closed without a resolution.
Happy to help tracking this down.
Steps to reproduce the issue:
podman generate systemd --new:Describe the results you received:
Describe the results you expected:
Nginx runs until the end of time.
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
apt list podman):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
Yes and yes.
Additional environment details (AWS, VirtualBox, physical, etc.):
Machine is a VM.
Beta Was this translation helpful? Give feedback.
All reactions