Podman volume mounts, rootless container, and non-root user in container

[gaoshancha]

I am working with the official ISC Bind9 container image (which appears to be based on Debian Linux though it says Alpine), and I'm having trouble with permissions and mounted voumes:

I start the container with mounts in my user-mode account as follows:

```
podman run --name bind9 \
-p 11153:53/udp \
-p 11153:53/tcp \
-v /home/devops/work/dns/primary/etc:/etc/bind/:rw,Z \
-v /home/devops/work/dns/primary/cache:/var/cache/bind:rw,Z, \
-v /home/devops/work/dns/primary/lib:/var/lib/bind:rw,Z, \
-v /home/devops/work/dns/primary/log:/var/log:rw,Z, \
internetsystemsconsortium/bind9:9.20 -4 -g -c /etc/bind/named.conf
```

There are no podman logs to reference, so I set the -g flag on named binary so that errors go to stderr.  The container starts but exits with error code 1 after about 3 seconds, and I get the following error messages that appear to be access issues between my host and the container???

```
09-Nov-2024 21:26:44.329 the working directory is not writable
09-Nov-2024 21:26:44.330 loading configuration: permission denied
09-Nov-2024 21:26:44.330 exiting (due to fatal error)
```

Digging into the container layers, I see that in the container a `bind` user is used to run `named` in the ENTRYPOINT part of the container.

```
...
23 VOLUME [/etc/bind /var/cache/bind /var/lib/bind /var/log] 0 B
24 EXPOSE map[443/tcp:{} 53/tcp:{} 53/udp:{} 853/tcp:{} 0 B
25 ENTRYPOINT ["/usr/sbin/named" "-u" "bind"] 0 B
26 CMD ["-f" "-c" "/etc/bind/named.conf" "-L" 0 B
...
```

After much searching around the Internet, I understand this is because from the perspective of execution environment inside the container, these mounted volumes have root:root permissions, but the container was built but ISC for use with a bind user.  I suppose if I was running Docker rather than Podman this would just work?

So after some more digging, I was able to get the UID and GID of the bind user inside the container by not mounting any volumes and then spawning a shell in the container execution environment.

```
podman run -d --name=bind9 -p 11153:53/udp -p 11153:53/tcp internetsystemsconsortium/bind9:9.20
podman exec -it bind9 /bin/sh
/ # cat /etc/passwd...
bind:x:100:101:Linux User,,,:/var/cache/bind:/sbin/nologin
```

After more Internet research and RedHat reading, it seems I should use the --userns option on podman to map the mounted volumes with the bind user in the container. I tried adding `--userns=keep-id:uid=100,gid=101` to the podman run command but I still get the same exact errors.

At this point I tried a "hail mary" and on the host machine I made the volumes chmod 777, but I still get the same exact errors.

So:

• I'm not sure what to try next...do I have to create a useless bind user account on the host and somehow map the UID and GID to the container?  These seems like an extrenuous step and that there should be something I can do with podman to handle this.
• Since I didn't create this container myself, this is serving as a good learning point.  With podman, what is the best practice for rootless containers?  Should rootless containers not use a non-root user inside the container?  Or should this container be run as "root" container and then use a non-root user inside the container?

[gaoshancha]

OK, this gets interesting, after some more experimenting, it looks like the bind user can indeed read/write using --userns flags.  Here is the troubleshooting process:

1) No --userns flag, what are the perms on the mounted directory?

podman run -d --name=bind9 -p 11153:53/udp -p 11153:53/tcp -v /home/devops/work/lpic2/dns/primary:/primary:Z internetsystemsconsortium/bind9:9.20
podman exec -it bind9 /bin/sh
/ $ whoami
bind
/ $ ls -l /
...
drwxr-xr-x 1 root root 28 Nov 10 05:41 primary
...

OK, this confirms that the owernship of the mounted volume is seen as root:root. 

2) Now lets add --userns=auto:

podman run -d --name=bind9 -p 11153:53/udp -p 11153:53/tcp --userns=auto -v /home/devops/work/lpic2/dns/primary:/primary:Z internetsystemsconsortium/bind9:9.20
podman exec -it bind9 /bin/sh
/ $ ls -l /
...
drwxr-xr-x 1 nobody nobody 28 Nov 10 05:41 primary
...

Now the ownership seen in the container is nobody:nobody.

3) Let's try specifically mapping the UID and GID of the bind user with --userns=keep-id:uid=100,gid=101:

podman run -d --name=bind9 -p 11153:53/udp -p 11153:53/tcp --userns=keep-id:uid=100,gid=101 -v /home/devops/work/lpic2/dns/primary:/primary:Z internetsystemsconsortium/bind9:9.20
podman exec -it bind9 /bin/sh
/ $ ls -l /
...
drwxr-xr-x 1 bind bind 28 Nov 10 05:41 primary
...

Now the ownership is bind:bind!  And the bind user can now read/write to the mounted volume:

/ $ cat /primary/etc/named.conf
zone "example.com" IN {
type master;
file "example.com.zone";
};
/ $ echo "can I write?" > /primary/lib/file_created_in_container.txt
/ $ cat /primary/lib/file_created_in_container.txt
can I write?

Ok this is looking good!  Finally, can I start named?

/ $ whoami
bind
/ $ /usr/sbin/named -u bind -4 -g -c /primary/etc/named.conf
...
10-Nov-2024 16:26:39.809 the working directory is not writable
10-Nov-2024 16:26:39.809 loading configuration: permission denied
10-Nov-2024 16:26:39.809 exiting (due to fatal error)

But if I don't reference the mounted volume at all:

/usr/sbin/named -u bind -4 -g

The binary named runs and no errors are reported.  So it looks like --userns, while it seems to have fixed perms, there is something else going on...and I confirmed on another machine that it just works with Docker.  So...this is a podman thing I suppose.

(sigh) Well...I don't know what that means now

[gaoshancha · Accepted solution]

OK, I was able to get the soultion with some help on Ask Fedora.  Basically, instead of doing thigns the Docker way and simply mapping a user folder to the container using "-v", you create volumes for the rootless container as follows.

podman volume create bind-config
podman volume create bind-lib
podman volume create bind-log
podman volume create bind-cache

Then map these to the proper directories with the podman run command:

podman run --name bind9 \
-p 11153:53/udp \
-p 11153:53/tcp \
-v bind-config:/etc/bind:rw,Z \
-v bind-lib:/var/lib/bind:rw,Z \
-v bind-log:/var/log:rw,Z \
-v bind-cache:/var/cache/bind:rw,Z \
internetsystemsconsortium/bind9:9.20 -4 -g