Rootless Podman NFS Storage (Nextcloud/MariaDB/Redis on Rocky 9 with Quadlet)

[Shadow8472]

I am rebuilding my homelab with Quadlets.
EDIT: Very important, I'm using a Synology NAS, which I believe isn't quite 1:1 with normal NFS implementations, having four squash options: [root|everyone] to [admin|guest]. At the moment of edit, I am using Everyone to Guest.

Working Quadlets:

• homelab-services.network
• caddy.container (Reverse proxy) [Domains from router]
• caddy-config.volume
• caddy-data.volume
• Vaultwarden.container (HTTPS from Caddy)
• vaultwarden-data.volume

Nextcloud Project (Notes in parentheses):

• nextcloud.pod.disabled (Podman v5.0.0 feature; Rocky 9 has v4.9.4)
• nextcloud.network
• nextcloud-fastvolume.volume (SSD storage)
• nextcloud-photovolume.volume (HDD storage)
• nextcloud-mariaDB.volume (NFS)
• nextcloud-nextcloud.container
• nextcloud-mariadb.container
• nextcloud-redis.container

Researched, but tentatively rejected:

• Ansible (Feels overweight for application)
• Podlet (Couldn't get to work quickly)

7/8 nextcloud* quadlets generate systemd --user unit files, except nextcloud.pod.
Nextcloud and MariaDB do not start.

$ systemctl --user start nextcloud-mariadb.service 
Job for nextcloud-mariadb.service failed because the control process exited with error code.
See "systemctl --user status nextcloud-mariadb.service" and "journalctl --user -xeu nextcloud-mariadb.service" for details.

$ systemctl --user status nextcloud-nextcloud.service
× nextcloud-nextcloud.service - Nextcloud Personal Cloud
     Loaded: loaded (/home/podmanuser/.config/containers/systemd/nextcloud/nextcloud-nextcloud.container; generated)
     Active: failed (Result: exit-code) since Wed 2024-10-02 01:37:08 PDT; 1min 36s ago
    Process: 102825 ExecStart=/usr/bin/podman run  [...] docker.io/library/nextcloud:latest (code=exited, status=125)
    Process: 102834 ExecStopPost=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1001/nextcloud-nextcloud.cid (code=exited, status=0/SUCCESS)
   Main PID: 102825 (code=exited, status=125)
        CPU: 73ms

journalctl --user -xeu nextcloud-mariadb.service refuses to talk to underprivelaged users. Workaround: journalctl -xe in a privileged account.

Looks like I goofed with one error each from MariaDB and Nextcloud, but their counterparts are the same.

In a post from last year, I learned it was a permissions issue with NFS. #20519 After many suggestions, I was nudged toward Quadlet and using a file overlay when I got stuck trying to fix uid/gid numbers. I'm on Linux kernel version 5.14.0, which I read is supposed to have native file system overlay to help https://www.redhat.com/sysadmin/podman-rootless-overlay.

#nextcloud-nextcloud.container
[Unit]
Description=Nextcloud Personal Cloud
After=local-fs.target nextcloud-mariadb.service nextcloud-redis.service

[Container]
Label=app=nextcloud
ContainerName=NextCloud Home
#Pod=nextcloud.pod
Image=docker.io/library/nextcloud:latest
AutoUpdate=registry
Network=homelab-services.network
Network=nextcloud.network
Volume=nextcloud-fastvolume:/var/www/html
Volume=nextcloud-photovolume:/var/www/html/data/Photos
Environment=MARIADB_USER=nextclouddbuser
#Secret=password
#Environment=MARIADB_PASSWORD=/run/secrets/password
Environment=MARIADB_PASSWORD=mdbpass
#Secret=rootPassword
#Environment=MARIADB_ROOT_PASSWORD=/run/secrets/rootPassword
Environment=MARIADB_ROOT_PASSWORD=mdbrootpass

[Service]
Restart=always

[Install]
# Start by default on boot
WantedBy=default.target

#nextcloud.network
[Unit]
Description=NextCloud non-Pod Network
After=network-online.target

[Network]
Label=app=nextcloud
NetworkName=nextcloud
Subnet=10.98.75.0/24
Gateway=10.98.75.1

[Install]
WantedBy=default.target

Too long; didn't read?
I want rootless Nextcloud/MariaDB Podman containers doing their heavy weight data storage over NFS without.

Thanks!

[ygalblum]

Thanks for the detailed comment.

From what I can see, nextcloud-nextcloud.container does not have the NFS based volume. So, let's first focus on the MariaDB one. Can you share all its Quadlet files (i.e. .container and the rest of its dependencies)

[eriksjolund]

Regarding the error

 /entrypoint.sh: 111: cannot create /usr/local/etc/php/conf.d/redis-session.ini: Permission denied

I circumvented that error by creating an empty file ~/nextcloud-data/redis-session.ini and then bind-mounting it

Volume=%h/nextcloud-data/redis-session.ini:/usr/local/etc/php/conf.d/redis-session.ini:z

Reference:
https://github.com/eriksjolund/nextcloud-podman/blob/9b01bdfa282f84d6a5b4d18cd9df5d62ae9e0ea2/nextcloud.container#L22

In your setup probably remove :z

[Shadow8472]

Synology tech support got back and basically said they don't support extended permissions over NFS. ever. even with LDAP/Kerberos (If I understand him correctly). Try SMB if at all possible.

Truth be told, I'll need a day or two to sort through what either this or Synology official support said, but I did pick up an idea to try. I am going to try un-flattening permissions and making actual accounts for the different users MariaDB and Nextcloud... No, no Nextcloud's entrypoint.sh will still need the extended permissions. I think my best bet is to bind mount a modified copy of entrypoint.sh that doesn't care about extended permissions.

[Shadow8472]

I'm not ready to return to this project, but I found something that might be relevant.
https://github.com/derpibooru/philomena

If you have SELinux enforcing (Fedora, Arch, others; manifests as a Could not find a Mix.Project error), you should run the following in the application directory on the host before proceeding:

chcon -Rt svirt_sandbox_file_t .

This allows Docker or Podman to bind mount the application directory into the containers.
CHange CONtext.

[Shadow8472]

10 months later...
I am 99% sure I found my answer today.

The problem wasn't ever Rocky Linux -- it was never about Rocky Linux. It was Synology's DSM all along. In short: DSM is on Linux kernel 4.4.x, a version now maintained under Civil Infrastructure Platform (CIP) support -- think an LTS for LTS versions. From what I've gathered, Podman relies on Extended Attributes (xattrs) to create namespaces. This feature was added in Linux kernel 5.9. Importantly for me, Linux kernel 5.10 is set to have a CIP support period, so when 4.4 is finally sunset in a couple years, maybe this stunt will be possible. That is assuming DSM skips a CIP version and doesn't strip out that functionality.