While podman pull with non-root: Error: lchown /run/systemd/netif: operation not permitted

[Yeni84]

Hello,

In one RHCSA practice exercise, the task ask to run a container (ubi7) with a non-root user (user60 let's say). This non-root user has the home directory in an autofs share in another VM (some previous practice exam task). The home user auto fs task I say I configured it correctly. I tested with df -hT, mount | grep nfsmountpoint, created files under both paths nfsserver:/home/user60 and /nfsmountpoint/home/user60 and I can files in both paths. Also su - username on the autofs client goes to the home directory /nfsmountpoint/home/user60. user60 has same UID and password on nfsserver and autofs client.

The problem is that when I try to podman pull image with this user, user60, I get the error bellow. It is not a problem for registry or image as with another user (user1 whose home directory is locally, not NFS share) I can pull the same images successfully.

[user60@rhcsa3 ~]$ podman pull docker://docker.io/avivataqua/ubi7
Trying to pull docker://docker.io/avivataqua/ubi7...
Getting image source signatures
Copying blob 22b1a8fa1ec0 done
Copying blob bdb299bbb0dd done
Copying config ff6c628742 done
Writing manifest to image destination
Storing signatures
Error processing tar file(exit status 1): lchown /run/systemd/netif: operation not permitted
Error: unable to pull docker://docker.io/avivataqua/ubi7: Error committing the finished image: error adding layer with blob "sha256:bdb299bbb0dd3e702dfbbb75ac15ded4d971f0e669ebd5fddf0d0a70d750fa5f": Error processing tar file(exit status 1): lchown /run/systemd/netif: operation not permitted

Any hints about it?

Thank you.

[ricardodacosta]

Please share the output of /etc/redhat-release

[Yeni84]

[root@server-base ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.0 (Ootpa)
[root@server-base ~]#

[root@server-base ~]# podman version
Version: 2.0.5
API Version: 1
Go Version: go1.14.7
Built: Wed Sep 23 19:18:02 2020
OS/Arch: linux/amd64
[root@server-base ~]# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.0 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.0"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.0 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.0:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

[charith1994]

might be interesting:

"Does not work on NFS or parallel filesystem homedirs"

Quote from

https://github.com/containers/libpod/blob/master/rootless.md

[flozano]

Hi Yeni,

You need a newer release of RHEL. Your output shows that you're using RHEL 8.0 and that release didn't have complete rootless support for podman. An alternative would be prepending sudo to all your podman, skopeo, and buildah commands.

Looks like somehow you got a newer podman than provided by RHEL 8.0 -- which could render your system unsupported in a production scenario -- but you would still miss other packages and kernel updates to support network namespaces and other bits for rootless podman.

[Yeni84]

I tried to install the latest RHEL kernel (8.3) from the RHSM repository. The steps/commands/output are in the attached txt file below.

[root@server-base ~]# dnf install kernel-4.18.0-240.10.1.el8_3 kernel-headers-4.18.0-240.10.1.el8_3 kernel-tools-4.18.0-240.10.1.el8_3 kernel-tools-libs-4.18.0-240.10.1.el8_3 kernel-devel-4.18.0-240.10.1.el8_3

....

Upgraded:
dracut-network-049-27.git20190906.el8_1.1.x86_64 dracut-049-27.git20190906.el8_1.1.x86_64 dracut-config-rescue-049-27.git20190906.el8_1.1.x86_64 dracut-squash-049-27.git20190906.el8_1.1.x86_64
kexec-tools-2.0.20-34.el8_3.1.x86_64 linux-firmware-20200619-101.git3890db36.el8_3.noarch kernel-tools-4.18.0-240.10.1.el8_3.x86_64 kernel-tools-libs-4.18.0-240.10.1.el8_3.x86_64
kernel-headers-4.18.0-240.10.1.el8_3.x86_64

Installed:
kernel-devel-4.18.0-240.10.1.el8_3.x86_64 kernel-4.18.0-240.10.1.el8_3.x86_64 kernel-modules-4.18.0-240.10.1.el8_3.x86_64 kernel-core-4.18.0-240.10.1.el8_3.x86_64

Complete!
[root@server-base ~]#

The installation seems completed successfully but when I reboot (the new kernel is the default entry now at the boot menu) it does not boot successfully. No messages appear, only a black screen and a blinking cursor. I waited for more than 15 minutes and nothing happens. I have to power off the VM. I tried on two VMs and it is the same behaviuor. Any hints? Thank you.

[Yeni84]

Update: Yesterday I downloaded the RHEL8.2 iso  and installed RHEL8.2 in a new VM. I will try the exercise above (container run with non-root user who has home directory on autofs) this week and update here the outcome. 

[flozano]

I don't know the details but there are issues with rootless containers and NFS homes. I assume you're using NFS because you mentioned autofs.

Rootless containers require uid namespaces and NFS is not happy about it.