I would like to enhance the AIO compatibility with podman. Are the some development docs about the mastercontainer?

[aanno]

Hello,

I looked into the problems of AIO with podman a bit. I've got the impression that adding --add-host <your-nextcloud-aio-fq-domain>:host-gateway would improve things substantial.

However I'm new to the AIO mastercontainer and its PHP code. AFAIK, the mastercontainer controls starting the other containers by using docker unix socket.

I would like to interfere here and test my idea. Is there some docs about mastercontainer development? Where is the place where the PHP code starts new containers?

Kind regards, aanno

[szaimen]

Hi,

However I'm new to the AIO mastercontainer and its PHP code. AFAIK, the mastercontainer controls starting the other containers by using docker unix socket.

Yes, this is how AIO works.

Is there some docs about mastercontainer development? Where is the place where the PHP code starts new containers?

The code is here: https://github.com/nextcloud/all-in-one/tree/main/php#php-docker-controller. There you can also find some info on how to run this from source.

I would like to enhance the AIO compatibility with podman.

However, one of the main problems with podman currently is, that the included watchtower container which updates the mastercontainer is not compatible with podman. So this is one of the things that would need to be fixed. See containrrr/watchtower#1060.

And even after fixing this, I am personally against adding support for podman, since, as stated in the docs, this would add another platform that we would need to test against. So you still have the option to either use docker rootless, use the manual-install https://github.com/nextcloud/all-in-one/tree/main/manual-install or follow #3487 which is not supported by us though.

[aanno]

Dear @szaimen ,

thank you for prompt answer. I completely understand that there a no plans to get podman official supported. However, I really think that AIO is just very near to unofficially run on podman right now.

Podman has evolved substantially over the years. And it certainly supports the docker engine API out of the box, even in rootless mode. The following lines start the socket:

# start socket in rootless mode
systemctl --user restart podman.socket
# check if socket is up
curl -H "Content-Type: application/json" \
        --unix-socket /run/user/$UID/podman/podman.sock \
    http://localhost/_ping
export DOCKER_SOCKET=/run/user/$UID/podman/podman.sock
export DOCKER_HOST=unix://$DOCKER_SOCKET
echo "export DOCKER_SOCKET=$DOCKER_SOCKET"
echo "export DOCKER_HOST=$DOCKER_HOST"

The socket is not at /var/run/docker.sock. But apart from that it is up and kicking. In my experiments the watchtower container has not been a problem.

However 'self reference' is still a bit difficult.

[apparle]

@aanno I'd like to help too and have some minor AIO development experience.

I recently migrated my server to a newer OS and also migrated all my containers to podman user-space instead of docker. So far, with the user's docker.sock mapped, it seems to just work at a high level for basic functionality. So far I haven't reached the point of testing the backups or updates etc.
Few things:

1. While restoring from a backup, the redis volume is restored as empty volume but as owned by the root user. Post restore when I start using nextcloud, the redis container launches with uid 999 which doesn't have write access to that volume. Consequently, the nextcloud's main container spikes CPU usage to the moon. I had to manually chown the directory to correct uid after which it just worked. @szaimen how is this not a problem for a regular docker save and restore (on a different machine) ?
2. After machine reboot, podman doesn't restart the user-space containers. So I've manually create a systemd unit to start the containers. This makes sense for the mastercontainer since I wrote the compose file, but then it only starts the mastercontainer. I've to open the AIO UI and click on Start containers to actually start Nextcloud. It'd be much nicer to have some way to just start containers directly.
3. I don't use Collabara or Talk, so haven't yet noticed the issue you highlighted in how to run aio image using podman (docker replacement) #5090 (reply in thread) . Does the container try to reach itself over the network by its own name? Can you explain what's the expected behavior in docker?

@szaimen I know your hesitation about supporting one more platform, but podman has been making steady progress and I think it's getting to a point where it can completely replace docker, which IMO is a good thing given it's architecture. How about we take a stab at identifying & fixing the issues as long as they don't break anything with regular docker or require dramatically different code? At that point it can remain a "it works but isn't officially supported state" until someone officially steps in to maintain it. Since I've migrated my personal setup to podman, I'm highly motivated to fix the basic functionality for myself at least :-)

[apparle]

Ah. I can check on that too with:

podman exec -it --env START_CONTAINERS=1 nextcloud-aio-mastercontainer bash -c "[ -f /mnt/docker-aio-config/data/configuration.json ] && grep -q 'wasStartButtonClicked.*1' /mnt/docker-aio-config/data/configuration.json && /daily-backup.sh || echo Initial setup not yet done."

Should this condition check be right at top of daily-backup.sh since that whole script shouldn't run if the check fails anyway?
Or should I keep it as a part of the service file?

PS: It's not a perfect parsing of json, but that's the reasonable design with linux CLI tools. Ideal json parsing can also be done with jq, but for that it'll have to be installed as part of the mastercontainer's image.

[szaimen]

Should this condition check be right at top of daily-backup.sh since that whole script shouldn't run if the check fails anyway? Or should I keep it as a part of the service file?

Not sure honestly since this was at least until now never a use case it was built for.

BTW, you will probably need to use AUTOMATIC_UPDATES=1 instead of START_CONTAINERS=1 since otherwise it will not pull the images on the first run. After the first run is done, you can indeed use START_CONTAINERS=1 as the images are pulled then already

[szaimen]

Another and probably ths best option might be to adjust the podman-restart.service to also consider containers that have unless-stopped set as restart policy. See containers/podman#20418

[apparle]

BTW, you will probably need to use AUTOMATIC_UPDATES=1 instead of START_CONTAINERS=1 since otherwise it will not pull the images on the first run. After the first run is done, you can indeed use START_CONTAINERS=1 as the images are pulled then already

I don't understand this scenario: If someone has done the initial setup (guarded by aforementioned wasStartButtonClicked check), then that would have pulled the images anyway, and containers should start fine post reboot. On the other hand, if they've not done the initial setup, then the wasStartButtonClicked check fails and daily-backup.sh is not run. And this is fine, since it only starts the mastercontainer and user should do the initial set up. Am I missing some corner case?

I don't want to enable AUTOMATIC_UPDATES=1 because I don't want the containers to be updated at machine reboot. They should only be updated by mastercontainer / aio-watchtower after usual backups, not on spurious reboots for unrelated reasons.

Another and probably ths best option might be to adjust the podman-restart.service to also consider containers that have unless-stopped set as restart policy. See containers/podman#20418

The rootless user-space podman containers don't have a podman-restart.service by default.
I can create one manually, and I did in fact try to do so, but ran into similar issues highlighted on the thread. I also attempted to do a manual script that inspects all stopped containers' RestartPolicy and StoppedByUser to implement the right restart behavior but it was too flaky regarding the unless-stopped behavior. On the other hand, using the daily-backup.sh script is quite rock solid. I've rebooted my machine many times in last 2 weeks and the set up has been quite stable!

Also, for all my other services, the systemd mindset of a service file per container service (eg: traefik, immich etc.) is actually quite intuitive -- it fits naturally with linux's view of starting/stopping/enabling/disabling services, and also allows for ordering with other service (like networking, reverse proxy etc.). If we do decide to support podman first-class down the road, I'd advocate for providing a service file (simlar to docker's compose file) with this start up behavior built in. And just for that down-the-road vision, it makes sense to make daily-backup.sh be robust with more checks baked in.

[szaimen]

BTW, you will probably need to use AUTOMATIC_UPDATES=1 instead of START_CONTAINERS=1 since otherwise it will not pull the images on the first run. After the first run is done, you can indeed use START_CONTAINERS=1 as the images are pulled then already

I don't understand this scenario: If someone has done the initial setup (guarded by aforementioned wasStartButtonClicked check), then that would have pulled the images anyway, and containers should start fine post reboot. On the other hand, if they've not done the initial setup, then the wasStartButtonClicked check fails and daily-backup.sh is not run. And this is fine, since it only starts the mastercontainer and user should do the initial set up. Am I missing some corner case?

I don't want to enable AUTOMATIC_UPDATES=1 because I don't want the containers to be updated at machine reboot. They should only be updated by mastercontainer / aio-watchtower after usual backups, not on spurious reboots for unrelated reasons.

You are right. START_CONTAINERS=1 should suffice for your use case.

On the other hand, using the daily-backup.sh script is quite rock solid. I've rebooted my machine many times in last 2 weeks and the set up has been quite stable!

Okay cool!

[apparle]

It looks like containrrr/watchtower#1060 does have a proposed fix: containrrr/watchtower#2072 but looking at the release cadence of watchtower, I'm not hopeful :-(

[apparle]

I worked around the issue of watchtower by creating an image for watchtower with that PR merged (and set to default true) - apparle/watchtower:1.7.1_podmanCompatible , and then also building an aio watchtower image on top of that - docker.io/apparle/nextcloud-aio-watchtower:v10.6.1_podmanCompatible. I did test this once by manually triggering the watchtower container to see it behaves correctly with podman and it does.

Now, I'd like to test this over a longer period. @szaimen is there some way to use a custom image tag for any container like watchtower instead of pulling it from nextcloud's official images? Or alternatively, some way to disable the pull for the watchtower image if it exists already -- then I could locally fake the official tag to this other image?
If not, what's your thoughts on adding an environment variable to either allow for custom image and/or disable the pull?

[szaimen]

Honestly I am not sure about the watchtower change as adding this change to aio means basically that we would need to maintain a fork, no?

[apparle]

I was thinking AIO only provides something along the lines of environment variable WATCHTOWER_IMAGE_TAG which has a default value of docker.io/nextcloud/aio-watchtower:latest but someone can override it to a different repo:tag like docker.io/apparle/nextcloud-aio-watchtower:latest. This way, it's not AIO's testing/maintenance responsibility for this path other than providing the env variable. And it allows for someone else (like me) to maintain a podman compatible image for the watchtower on the side.

BUT now I think even that's not needed. I've found a different solution to achieve the same effect as above without modifying the AIO codebase at all.
Create a ~/.config/containers/registries.conf with these contents:

[[registry]]
prefix="docker.io/containrrr/watchtower"
location = "docker.io/apparle/watchtower"

[[registry]]
prefix = "docker.io/nextcloud/aio-watchtower"
location = "docker.io/apparle/nextcloud-aio-watchtower"

This redirects podman into pulling images for watch tower from completely different source. Arguably this is advanced trickery, so someone naive should absolutely not do this. But at least lets me and other podman users get AIO working, and then I can test/discover/fix other issues with podman.
Note, I only tested this with manual docker pull so far; yet to be tested from within mastercontainer's php code, waiting for next update.

Eventually when either containrrr/watchtower#2072 or containers/podman#23824 gets merged, then I don't need to maintain a parallel watchtower image.

[szaimen]

BUT now I think even that's not needed. I've found a different solution to achieve the same effect as above without modifying the AIO codebase at all. Create a ~/.config/containers/registries.conf with these contents:

[[registry]]
prefix="docker.io/containrrr/watchtower"
location = "docker.io/apparle/watchtower"

[[registry]]
prefix = "docker.io/nextcloud/aio-watchtower"
location = "docker.io/apparle/nextcloud-aio-watchtower"

This redirects podman into pulling images for watch tower from completely different source.

LGTM :)

[aanno]

Hello,

thank you very much for all your comments. However, I'm able to run all-in-one on podman now without any changes to code. For all the glory details see #5090 (comment)

Kind regards,
aanno

[apparle]

Did you do anything to get watchtower working, so you get containers to update correctly? Or alternatively, how are you managing updates?

The solution I've so far is listed in earlier comments above -- are you using the same? From I read on internet, there isn't any other workaround.

[aanno]

Dear @apparle ,

I'm not aware of any watchtower problem with podman and I have already update twice without any problems.

[apparle]

I'm surprised. I wonder if you're on cgroupsv1 and that's why you're not running into issue containrrr/watchtower#1060 yet.
Do you mind sharing the output of podman info ?

[aanno]

Hm, see my old comment for why I'm thinking that containrrr/watchtower#1060 is no longer relevant. I just copied the remark into the 1060 ticket as well.

$ podman info
host:
  arch: arm64
  buildahVersion: 1.39.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 98
    systemPercent: 0.85
    userPercent: 1.14
  cpus: 10
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "41"
  eventLogger: journald
  freeLocks: 2020
  hostname: netzgeneration
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1003
      size: 1
    - container_id: 1
      host_id: 720896
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1003
      size: 1
    - container_id: 1
      host_id: 720896
      size: 65536
  kernel: 6.13.5-200.fc41.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 8054398976
  memTotal: 16715886592
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc41.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.0-1.fc41.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.0
  ociRuntime:
    name: crun
    package: crun-1.20-2.fc41.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/user/1003/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250217.ga1e48a0-2.fc41.aarch64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1003/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1.fc41.aarch64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 4294701056
  swapTotal: 4294963200
  uptime: 89h 51m 18.00s (Approximately 3.71 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/nc/.config/containers/storage.conf
  containerStore:
    number: 13
    paused: 0
    running: 13
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/nc/.local/share/containers/storage
  graphRootAllocated: 810201686016
  graphRootUsed: 120170962944
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 23
  runRoot: /run/user/1003/containers
  transientStore: false
  volumePath: /var/home/nc/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.0
  BuildOrigin: Fedora Project
  Built: 1739232000
  BuiltTime: Tue Feb 11 00:00:00 2025
  GitCommit: ""
  GoVersion: go1.23.5
  Os: linux
  OsArch: linux/arm64
  Version: 5.4.0
``

[apparle]

No, this isn't just about just using the right socket. I'm already using the latest podman.sock on /run/user/$UID/podman/podman.sock.

What happens is watchtower queries the details about the container using podman inspect <...>, stores them, updates the images, and then tries to recreate that container to be exactly same by plugging in all the stored values.
At least on my podman (version 4.9.3), when you do podman inspect nextcloud-aio-mastercontainer it shows "MemorySwappiness": 0, when it should have been null instead of 0.
Watchtower then stores this value and when it tries to recreate the container by setting MemorySwappiness to 0, it errors out because as cgroupsv2 doesn't support that (as intentionally implemented by containers/podman#13916)

It could be the case that your latest version of podman (5.4.0) has indeed improved by returning MemorySwappiness: null but as far as I can read, that's not part of podman release until 6.0 -- see containers/podman#23824 and containers/podman#24126. Maybe Fedora 41 has pulled bleeding-edge patches that fix it for you, or maybe you've some sort of custom bleeding edge setup ?
Can you share the output of podman inspect nextcloud-aio-mastercontainer ?

[aanno]

$ podman inspect nextcloud-aio-mastercontainer
[
     {
          "Id": "0caa3f29d089c9d01fd011acb5823b4061dd6c716a093c8ae583e26bdfceb4fa",
          "Created": "2025-04-06T12:45:21.298199003Z",
          "Path": "/run/podman-init",
          "Args": [
               "--",
               "/start.sh"
          ],
          "State": {
               "OciVersion": "1.2.0",
               "Status": "running",
               "Running": true,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 10782,
               "ConmonPid": 10780,
               "ExitCode": 0,
               "Error": "container 0caa3f29d089c9d01fd011acb5823b4061dd6c716a093c8ae583e26bdfceb4fa: container is running",
               "StartedAt": "2025-04-06T12:45:21.73392596Z",
               "FinishedAt": "0001-01-01T00:00:00Z",
               "Health": {
                    "Status": "healthy",
                    "FailingStreak": 0,
                    "Log": [
                         {
                              "Start": "2025-04-10T08:56:20.623645064Z",
                              "End": "2025-04-10T08:56:20.753995244Z",
                              "ExitCode": 0,
                              "Output": "Connection to 127.0.0.1 80 port [tcp/http] succeeded!\nConnection to 127.0.0.1 8000 port [tcp/*] succeeded!\nConnection to 127.0.0.1 8080 port [tcp/http-alt] succeeded!\nConnection to 127.0.0.1 8443 port [tcp/*] succeeded!\nConnection to 127.0.0.1 9000 port [tcp/*] succeeded!\nConnection to 127.0.0.1 9876 port [tcp/*] succeeded!\n"
                         },
                         {
                              "Start": "2025-04-10T08:56:51.611764453Z",
                              "End": "2025-04-10T08:56:51.778339808Z",
                              "ExitCode": 0,
                              "Output": "Connection to 127.0.0.1 80 port [tcp/http] succeeded!\nConnection to 127.0.0.1 8000 port [tcp/*] succeeded!\nConnection to 127.0.0.1 8080 port [tcp/http-alt] succeeded!\nConnection to 127.0.0.1 8443 port [tcp/*] succeeded!\nConnection to 127.0.0.1 9000 port [tcp/*] succeeded!\nConnection to 127.0.0.1 9876 port [tcp/*] succeeded!\n"
                         },
                         {
                              "Start": "2025-04-10T08:57:22.613406838Z",
                              "End": "2025-04-10T08:57:22.891650364Z",
                              "ExitCode": 0,
                              "Output": "Connection to 127.0.0.1 80 port [tcp/http] succeeded!\nConnection to 127.0.0.1 8000 port [tcp/*] succeeded!\nConnection to 127.0.0.1 8080 port [tcp/http-alt] succeeded!\nConnection to 127.0.0.1 8443 port [tcp/*] succeeded!\nConnection to 127.0.0.1 9000 port [tcp/*] succeeded!\nConnection to 127.0.0.1 9876 port [tcp/*] succeeded!\n"
                         },
                         {
                              "Start": "2025-04-10T08:57:53.642606774Z",
                              "End": "2025-04-10T08:57:53.940052831Z",
                              "ExitCode": 0,
                              "Output": "Connection to 127.0.0.1 80 port [tcp/http] succeeded!\nConnection to 127.0.0.1 8000 port [tcp/*] succeeded!\nConnection to 127.0.0.1 8080 port [tcp/http-alt] succeeded!\nConnection to 127.0.0.1 8443 port [tcp/*] succeeded!\nConnection to 127.0.0.1 9000 port [tcp/*] succeeded!\nConnection to 127.0.0.1 9876 port [tcp/*] succeeded!\n"
                         },
                         {
                              "Start": "2025-04-10T08:58:24.893311809Z",
                              "End": "2025-04-10T08:58:24.973561659Z",
                              "ExitCode": 0,
                              "Output": "Connection to 127.0.0.1 80 port [tcp/http] succeeded!\nConnection to 127.0.0.1 8000 port [tcp/*] succeeded!\nConnection to 127.0.0.1 8080 port [tcp/http-alt] succeeded!\nConnection to 127.0.0.1 8443 port [tcp/*] succeeded!\nConnection to 127.0.0.1 9000 port [tcp/*] succeeded!\nConnection to 127.0.0.1 9876 port [tcp/*] succeeded!\n"
                         }
                    ]
               },
               "CgroupPath": "/user.slice/user-1003.slice/[email protected]/user.slice/user-libpod_pod_1a29f3a0b922bd18316b02526f761ab63c3f5de78a0739a2a5d6d96edebae7f5.slice/libpod-0caa3f29d089c9d01fd011acb5823b4061dd6c716a093c8ae583e26bdfceb4fa.scope",
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          },
          "Image": "2d79a59a95d8f20efbe39921e57e3f7e9b70dd1d47c597afa4d4f8b2b89b6f0c",
          "ImageDigest": "sha256:53d4a4ec39cfb602a54037dfe99ee7777b4357c9c6f41b23d351245a0ee4ef43",
          "ImageName": "docker.io/nextcloud/all-in-one:latest",
          "Rootfs": "",
          "Pod": "1a29f3a0b922bd18316b02526f761ab63c3f5de78a0739a2a5d6d96edebae7f5",
          "ResolvConfPath": "/run/user/1003/containers/overlay-containers/0caa3f29d089c9d01fd011acb5823b4061dd6c716a093c8ae583e26bdfceb4fa/userdata/resolv.conf",
          "HostnamePath": "/run/user/1003/containers/overlay-containers/0caa3f29d089c9d01fd011acb5823b4061dd6c716a093c8ae583e26bdfceb4fa/userdata/hostname",
          "HostsPath": "/run/user/1003/containers/overlay-containers/0caa3f29d089c9d01fd011acb5823b4061dd6c716a093c8ae583e26bdfceb4fa/userdata/hosts",
          "StaticDir": "/var/home/nc/.local/share/containers/storage/overlay-containers/0caa3f29d089c9d01fd011acb5823b4061dd6c716a093c8ae583e26bdfceb4fa/userdata",
          "OCIConfigPath": "/var/home/nc/.local/share/containers/storage/overlay-containers/0caa3f29d089c9d01fd011acb5823b4061dd6c716a093c8ae583e26bdfceb4fa/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/user/1003/containers/overlay-containers/0caa3f29d089c9d01fd011acb5823b4061dd6c716a093c8ae583e26bdfceb4fa/userdata/conmon.pid",
          "PidFile": "/run/user/1003/containers/overlay-containers/0caa3f29d089c9d01fd011acb5823b4061dd6c716a093c8ae583e26bdfceb4fa/userdata/pidfile",
          "Name": "nextcloud-aio-mastercontainer",
          "RestartCount": 0,
          "Driver": "overlay",
          "MountLabel": "system_u:object_r:container_file_t:s0:c1022,c1023",
          "ProcessLabel": "",
          "AppArmorProfile": "",
          "EffectiveCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "BoundingCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "ExecIDs": [],
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/home/nc/.local/share/containers/storage/overlay/2de17071e4568edc6823f2fcbe62839f224cf5fe179dc2a0f305cfd964b5b009/diff:/var/home/nc/.local/share/containers/storage/overlay/7239317a8a4bf930ee38d13f62664f2c3e509530b4d7cec91392f503df24d750/diff:/var/home/nc/.local/share/containers/storage/overlay/1984dd6a790dcd08d01c3c8419b905d656eae991e7fede5d15c340f864afb5a3/diff:/var/home/nc/.local/share/containers/storage/overlay/fb7991f6c3ee94c23e71c121fa1c0e00db53b219aee232c1cea74794018748a1/diff:/var/home/nc/.local/share/containers/storage/overlay/7767fcc453f3053f23c3706475c2d793cb91482a27d11968a9efe6c013205b11/diff:/var/home/nc/.local/share/containers/storage/overlay/fa27682e53ac65fe4fae8718842cd75aff664031d55fd9936faaee8bc299451d/diff:/var/home/nc/.local/share/containers/storage/overlay/b88d199cb66abb2318a5f0c4d270ae0a231946dd64ba52ca68091a24f591a815/diff:/var/home/nc/.local/share/containers/storage/overlay/c23ea3a3c433b7faa061885e5f921e6b624131a3e024de9f1809da695318432c/diff:/var/home/nc/.local/share/containers/storage/overlay/4b8c25d9bbe472854db35ac961f1dd91df05749e1de2431e795b17e4d283f06e/diff:/var/home/nc/.local/share/containers/storage/overlay/01359c4a1fdd85d8db0a26f4f5a740d80103a2577292a3592e4ccce24e90219e/diff:/var/home/nc/.local/share/containers/storage/overlay/ceb9d510d976d692f6a1a88d037172a87b9ca9f67cec46dbbcae1eb344a14d1f/diff:/var/home/nc/.local/share/containers/storage/overlay/cc542502cbf6643b7c8ce848d577bedfebb954b481604fa1d73bc638a8f81086/diff:/var/home/nc/.local/share/containers/storage/overlay/cfe5d154dd02251444e96d21506835cf753ce44a017bf3603d68dcd9b791884a/diff:/var/home/nc/.local/share/containers/storage/overlay/a3b01dfa6b70fa01a660ecbc08142db9c6f3511527939f60257291f90dbb133e/diff:/var/home/nc/.local/share/containers/storage/overlay/1a53ddab60b9d5afa4c54ae234763487796ede91e0dccdba4d6b0694ee8ae3a7/diff:/var/home/nc/.local/share/containers/storage/overlay/52e54c7a567e3bbca5dc57f65f66e422af7f7debd3dd57b4edc0783f31e291c1/diff:/var/home/nc/.local/share/containers/storage/overlay/166a7067eaf0b6263e3d64f43dcedf71980cbf142169ecadf904511080100e2d/diff:/var/home/nc/.local/share/containers/storage/overlay/cb972388ca4250db0a469da893a979d6879db571d07e29f2ade451303a24b68e/diff:/var/home/nc/.local/share/containers/storage/overlay/a16e98724c05975ee8c40d8fe389c3481373d34ab20a1cf52ea2accc43f71f4c/diff",
                    "MergedDir": "/var/home/nc/.local/share/containers/storage/overlay/43df2590ffddd90d83252e47d58cb8cddd38eceda23c4d0934d883a3360fb384/merged",
                    "UpperDir": "/var/home/nc/.local/share/containers/storage/overlay/43df2590ffddd90d83252e47d58cb8cddd38eceda23c4d0934d883a3360fb384/diff",
                    "WorkDir": "/var/home/nc/.local/share/containers/storage/overlay/43df2590ffddd90d83252e47d58cb8cddd38eceda23c4d0934d883a3360fb384/work"
               }
          },
          "Mounts": [
               {
                    "Type": "volume",
                    "Name": "nextcloud_aio_mastercontainer",
                    "Source": "/var/home/nc/.local/share/containers/storage/volumes/nextcloud_aio_mastercontainer/_data",
                    "Destination": "/mnt/docker-aio-config",
                    "Driver": "local",
                    "Mode": "z",
                    "Options": [
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "bind",
                    "Source": "/var/home/nc/scm/all-in-one/aio-on-fcos/backup",
                    "Destination": "/mnt/backup",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "bind",
                    "Source": "/run/user/1003/podman/podman.sock",
                    "Destination": "/var/run/docker.sock",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": false,
                    "Propagation": "rprivate"
               }
          ],
          "Dependencies": [],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "",
               "IPAddress": "",
               "IPPrefixLen": 0,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {
                    "80/tcp": null,
                    "8080/tcp": [
                         {
                              "HostIp": "0.0.0.0",
                              "HostPort": "8080"
                         }
                    ],
                    "8443/tcp": null,
                    "9000/tcp": null
               },
               "SandboxKey": "/run/user/1003/netns/netns-cca41067-029f-4965-7801-ae1db0351ad9",
               "Networks": {
                    "nextcloud-aio": {
                         "EndpointID": "",
                         "Gateway": "10.89.57.1",
                         "IPAddress": "10.89.57.2",
                         "IPPrefixLen": 24,
                         "IPv6Gateway": "fd49:dc34:d0fe:ef6b:cafe::1",
                         "GlobalIPv6Address": "fd49:dc34:d0fe:ef6b:cafe::2",
                         "GlobalIPv6PrefixLen": 80,
                         "MacAddress": "aa:90:38:21:49:19",
                         "NetworkID": "f58b65ccd74094778af2e0f1314b6c698f11a259a59729b2593d33dc3acfe03d",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "0caa3f29d089"
                         ]
                    },
                    "podman": {
                         "EndpointID": "",
                         "Gateway": "10.88.0.1",
                         "IPAddress": "10.88.0.4",
                         "IPPrefixLen": 16,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "56:63:29:c1:2e:a6",
                         "NetworkID": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "nextcloud-aio-mastercontainer",
                              "0caa3f29d089"
                         ]
                    }
               }
          },
          "Namespace": "",
          "IsInfra": false,
          "IsService": false,
          "KubeExitCodePropagation": "invalid",
          "lockNumber": 7,
          "Config": {
               "Hostname": "0caa3f29d089",
               "Domainname": "",
               "User": "root",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": false,
               "OpenStdin": false,
               "StdinOnce": false,
               "Env": [
                    "PHPIZE_DEPS=autoconf \t\tdpkg-dev dpkg \t\tfile \t\tg++ \t\tgcc \t\tlibc-dev \t\tmake \t\tpkgconf \t\tre2c",
                    "PHP_VERSION=8.3.19",
                    "GPG_KEYS=1198C0117593497A5EC5C199286AF1F9897469DC C28D937575603EB4ABB725861C0779DC5C0A9DE4 AFD8691FDAEDF03BDF6E460563F15A9B715376CA",
                    "NEXTCLOUD_MEMORY_LIMIT=1024M",
                    "PHP_URL=https://www.php.net/distributions/php-8.3.19.tar.xz",
                    "APACHE_IP_BINDING=0.0.0.0",
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "container=podman",
                    "PHP_ASC_URL=https://www.php.net/distributions/php-8.3.19.tar.xz.asc",
                    "PHP_SHA256=976e4077dd25bec96b5dfe8938052d243bbd838f95368a204896eff12756545f",
                    "PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64",
                    "PHP_INI_DIR=/usr/local/etc/php",
                    "PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64",
                    "SKIP_DOMAIN_VALIDATION=false",
                    "APACHE_ADDITIONAL_NETWORK=nextcloud_frontend",
                    "NEXTCLOUD_MOUNT=/var/home/nc/scm/all-in-one/aio-on-fcos/backup",
                    "PHP_LDFLAGS=-Wl,-O1 -pie",
                    "WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1003/podman/podman.sock",
                    "APACHE_PORT=11000",
                    "HOME=/root",
                    "HOSTNAME=0caa3f29d089"
               ],
               "Cmd": null,
               "Image": "docker.io/nextcloud/all-in-one:latest",
               "Volumes": null,
               "WorkingDir": "/var/www/docker-aio",
               "Entrypoint": [
                    "/start.sh"
               ],
               "OnBuild": null,
               "Labels": {
                    "PODMAN_SYSTEMD_UNIT": "[email protected]",
                    "com.docker.compose.container-number": "1",
                    "com.docker.compose.project": "aio",
                    "com.docker.compose.project.config_files": "compose.yaml",
                    "com.docker.compose.project.working_dir": "/var/home/nc/scm/all-in-one/aio-on-fcos",
                    "com.docker.compose.service": "nextcloud-aio-mastercontainer",
                    "io.podman.compose.config-hash": "3a910975aa3e57c8cf063ac7a74b9e6a1dc3426ee6dc247c4c77b0d2bbcf422a",
                    "io.podman.compose.project": "aio",
                    "io.podman.compose.version": "1.3.0"
               },
               "Annotations": {
                    "io.container.manager": "libpod",
                    "io.kubernetes.cri-o.SandboxID": "1a29f3a0b922bd18316b02526f761ab63c3f5de78a0739a2a5d6d96edebae7f5",
                    "io.podman.annotations.init": "TRUE",
                    "io.podman.annotations.label": "disable",
                    "org.opencontainers.image.stopSignal": "3",
                    "org.systemd.property.KillSignal": "3",
                    "org.systemd.property.TimeoutStopUSec": "uint64 10000000"
               },
               "StopSignal": "SIGQUIT",
               "Healthcheck": {
                    "Test": [
                         "CMD-SHELL",
                         "/healthcheck.sh"
                    ],
                    "Interval": 30000000000,
                    "Timeout": 30000000000
               },
               "HealthcheckOnFailureAction": "none",
               "HealthLogDestination": "local",
               "HealthcheckMaxLogCount": 5,
               "HealthcheckMaxLogSize": 500,
               "CreateCommand": [
                    "podman",
                    "run",
                    "--name=nextcloud-aio-mastercontainer",
                    "-d",
                    "--pod=pod_aio",
                    "--security-opt",
                    "label:disable",
                    "--label",
                    "io.podman.compose.config-hash=3a910975aa3e57c8cf063ac7a74b9e6a1dc3426ee6dc247c4c77b0d2bbcf422a",
                    "--label",
                    "io.podman.compose.project=aio",
                    "--label",
                    "io.podman.compose.version=1.3.0",
                    "--label",
                    "[email protected]",
                    "--label",
                    "com.docker.compose.project=aio",
                    "--label",
                    "com.docker.compose.project.working_dir=/var/home/nc/scm/all-in-one/aio-on-fcos",
                    "--label",
                    "com.docker.compose.project.config_files=compose.yaml",
                    "--label",
                    "com.docker.compose.container-number=1",
                    "--label",
                    "com.docker.compose.service=nextcloud-aio-mastercontainer",
                    "-e",
                    "APACHE_PORT=11000",
                    "-e",
                    "APACHE_IP_BINDING=0.0.0.0",
                    "-e",
                    "APACHE_ADDITIONAL_NETWORK=nextcloud_frontend",
                    "-e",
                    "NEXTCLOUD_MOUNT=/var/home/nc/scm/all-in-one/aio-on-fcos/backup",
                    "-e",
                    "NEXTCLOUD_MEMORY_LIMIT=1024M",
                    "-e",
                    "SKIP_DOMAIN_VALIDATION=false",
                    "-e",
                    "WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1003/podman/podman.sock",
                    "-v",
                    "/var/home/nc/scm/all-in-one/aio-on-fcos/backup:/mnt/backup:z",
                    "-v",
                    "nextcloud_aio_mastercontainer:/mnt/docker-aio-config:z",
                    "-v",
                    "/run/user/1003/podman/podman.sock:/var/run/docker.sock:z,ro",
                    "--network=bridge:alias=nextcloud-aio-mastercontainer",
                    "--add-host",
                    "nextcloud.breitbandig.de:host-gateway",
                    "-p",
                    "8080:8080",
                    "--restart",
                    "never",
                    "--init",
                    "docker.io/nextcloud/all-in-one:latest"
               ],
               "Umask": "0022",
               "Timeout": 0,
               "StopTimeout": 10,
               "Passwd": true,
               "sdNotifyMode": "container",
               "ExposedPorts": {
                    "80/tcp": {},
                    "8080/tcp": {},
                    "8443/tcp": {},
                    "9000/tcp": {}
               }
          },
          "HostConfig": {
               "Binds": [
                    "nextcloud_aio_mastercontainer:/mnt/docker-aio-config:z,rw,rprivate,nosuid,nodev,rbind",
                    "/var/home/nc/scm/all-in-one/aio-on-fcos/backup:/mnt/backup:rw,rprivate,rbind",
                    "/run/user/1003/podman/podman.sock:/var/run/docker.sock:ro,rprivate,nosuid,nodev,rbind"
               ],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "0B"
               },
               "NetworkMode": "bridge",
               "PortBindings": {
                    "8080/tcp": [
                         {
                              "HostIp": "0.0.0.0",
                              "HostPort": "8080"
                         }
                    ]
               },
               "RestartPolicy": {
                    "Name": "no",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "AutoRemoveImage": false,
               "Annotations": {
                    "io.container.manager": "libpod",
                    "io.kubernetes.cri-o.SandboxID": "1a29f3a0b922bd18316b02526f761ab63c3f5de78a0739a2a5d6d96edebae7f5",
                    "io.podman.annotations.init": "TRUE",
                    "io.podman.annotations.label": "disable",
                    "org.opencontainers.image.stopSignal": "3",
                    "org.systemd.property.KillSignal": "3",
                    "org.systemd.property.TimeoutStopUSec": "uint64 10000000"
               },
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [],
               "CapDrop": [],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [
                    "nextcloud.breitbandig.de:host-gateway"
               ],
               "HostsFile": "",
               "GroupAdd": [],
               "IpcMode": "shareable",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": false,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [
                    "label=disable"
               ],
               "Tmpfs": {},
               "UTSMode": "private",
               "UsernsMode": "",
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 0,
               "NanoCpus": 0,
               "CgroupParent": "user.slice/user-1003.slice/[email protected]/user.slice/user-libpod_pod_1a29f3a0b922bd18316b02526f761ab63c3f5de78a0739a2a5d6d96edebae7f5.slice",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 0,
               "MemorySwappiness": 0,
               "OomKillDisable": false,
               "Init": true,
               "PidsLimit": 2048,
               "Ulimits": [
                    {
                         "Name": "RLIMIT_NOFILE",
                         "Soft": 65000,
                         "Hard": 65000
                    },
                    {
                         "Name": "RLIMIT_NPROC",
                         "Soft": 63175,
                         "Hard": 65000
                    }
               ],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          },
          "UseImageHosts": false,
          "UseImageHostname": false
     }
]

[apparle]

You do have MemorySwappiness: 0 instead of null, so I'm baffled how you're not running into the same issue as me or many other people (all the issues referenced in my previous comments).

Can you double check you're relying on AIO's watchtower to update the mastercontainer and not doing it on the side (say with podman's systemd based auto update) or a manual pull?

If possible, can you track the next time your mastercontainer is updated and share the log file of the container nextcloud-aio-watchtower ? It's done at the same time as AIO's daily backup, but update only happens once a week (roughly Saturday). So you'll need to extract out the log file on a Saturday after a new release has happened.

[aanno]

Dear @apparle ,

let's drill down a bit:

1. It seems that you are talking about nextcloud-aio-watchtower updating the mastercontainer. And you could be right here, I'm not sure if this is happening. Is there anything I could do to trigger this update?
2. However, I have observed that https://<my-server>:8443 aio interface informs me if aio and containers are up-to-date. If the containers are outdated, I could stop and restart containers (from aio interface) to get them up-to-date. But this is another piece of cake, isn't it?
3. I also observed that https://<my-server>:8443 informs my if aio (mastercontainer?!?) 'channel' is out of date. Updating the mastercontainer from the aio interface seems not to work for me. I circumvent this problem by doing a manual pull in this case.

[aanno]

Hm, this is delicate as it might interfere with my Fedora Coreos updates (which happens Saturday nights as well). In addition I'm AFK for a few weeks. I will try this but it might take some time.

[aanno]

Currently, I see the following with podman logs nextcloud-aio-mastercontainer:

...
NOTICE: PHP message: Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
Restarting cron.sh because daily backup time was set, changed or unset.
...

However I could do the following without any problem on the host system:

$ podman pull docker.io/nextcloud/all-in-one:latest
Trying to pull docker.io/nextcloud/all-in-one:latest...
Getting image source signatures
Copying blob 4bcdcf069be9 skipped: already exists  
Copying blob 26966b371bd9 skipped: already exists  
Copying blob 1df8ec9c416f skipped: already exists  
Copying blob f1eeb5b6ce94 skipped: already exists  
Copying blob a6d212e1907c skipped: already exists  
Copying blob 6118175218bb skipped: already exists  
Copying blob 20b4923a81eb skipped: already exists  
Copying blob 8bea583d67ad skipped: already exists  
Copying blob fd92e021672f skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 6e771e15690e skipped: already exists  
Copying blob 4edd9cdb496b skipped: already exists  
Copying blob 770c775ba6d7 skipped: already exists  
Copying blob 7d61f815e2a3 skipped: already exists  
Copying blob 2b0b55cc14be skipped: already exists  
Copying blob ed1f928defcc skipped: already exists  
Copying blob 29b3b25be7e2 skipped: already exists  
Copying blob e58015813eb3 skipped: already exists  
Copying blob 2de8eac2b5ea skipped: already exists  
Copying config 616d69c978 done   | 
Writing manifest to image destination
616d69c978881acfc93de45d32b26159883d2f768022d726cda78054c9d576c6

Inside mastercontainer with podman exec -it nextcloud-aio-mastercontainer bash:

# docker pull docker.io/nextcloud/all-in-one:latest

4bcdcf069be9: Already exists 
f1eeb5b6ce94: Already exists 
1df8ec9c416f: Already exists 
26966b371bd9: Already exists 
6e771e15690e: Already exists 
6118175218bb: Already exists 
20b4923a81eb: Already exists 
4edd9cdb496b: Already exists 
fd92e021672f: Already exists 
2de8eac2b5ea: Already exists 
4f4fb700ef54: Already exists 
7d61f815e2a3: Already exists 
8bea583d67ad: Already exists 
29b3b25be7e2: Already exists 
ed1f928defcc: Already exists 
2b0b55cc14be: Already exists 
770c775ba6d7: Already exists 
e58015813eb3: Already exists 
a6d212e1907c: Already exists 
616d69c97888: Download complete 
docker.io/nextcloud/all-in-one:latest

pulling seems no problem as well. So I wonder what triggers the php error.

[apparle]

Nextcloud AIO switched from docker.io to ghcr.io -- see recent release notes. You probably need to update your images and compose files etc.

[aanno]

Last night my nextcloud-aio-mastercontainer and nextcloud-aio-watchtower died. The are no logs with nextcloud-aio-mastercontainer, but podman logs nextcloud-aio-watchtower says:

time="2025-04-26T05:00:09Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2025-04-26T05:00:10Z" level=debug msg="Making sure everything is sane before starting"
time="2025-04-26T05:00:10Z" level=info msg="Watchtower v0.0.0-unknown"
time="2025-04-26T05:00:10Z" level=info msg="Using no notifications"
time="2025-04-26T05:00:10Z" level=info msg="Only checking containers which name matches \"nextcloud-aio-mastercontainer\""
time="2025-04-26T05:00:10Z" level=info msg="Running a one time update."
time="2025-04-26T05:00:10Z" level=debug msg="Checking containers for updated images"
time="2025-04-26T05:00:10Z" level=debug msg="Retrieving running containers"
time="2025-04-26T05:00:11Z" level=debug msg="Trying to load authentication credentials." container=/nextcloud-aio-mastercontainer image="docker.io/nextcloud/all-in-one:latest"
time="2025-04-26T05:00:11Z" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
time="2025-04-26T05:00:11Z" level=debug msg="Got image name: docker.io/nextcloud/all-in-one:latest"
time="2025-04-26T05:00:11Z" level=debug msg="Checking if pull is needed" container=/nextcloud-aio-mastercontainer image="docker.io/nextcloud/all-in-one:latest"
time="2025-04-26T05:00:11Z" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
time="2025-04-26T05:00:11Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2025-04-26T05:00:11Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2025-04-26T05:00:11Z" level=debug msg="Setting scope for auth token" image=docker.io/nextcloud/all-in-one scope="repository:nextcloud/all-in-one:pull"
time="2025-04-26T05:00:11Z" level=debug msg="No credentials found."
time="2025-04-26T05:00:11Z" level=debug msg="Parsing image ref" host=index.docker.io image=nextcloud/all-in-one normalized=docker.io/nextcloud/all-in-one tag=latest
time="2025-04-26T05:00:11Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/nextcloud/all-in-one/manifests/latest"
time="2025-04-26T05:00:12Z" level=debug msg="Found a remote digest to compare with" remote="sha256:58aaf8da5e8352f7d8db061467c7a9d4dfde0b3a61ef8a64ea0979b35df3c9a0"
time="2025-04-26T05:00:12Z" level=debug msg="Digests did not match, doing a pull."
time="2025-04-26T05:00:12Z" level=debug msg="Pulling image" container=/nextcloud-aio-mastercontainer image="docker.io/nextcloud/all-in-one:latest"
time="2025-04-26T05:00:13Z" level=info msg="Found new docker.io/nextcloud/all-in-one:latest image (616d69c97888)"
time="2025-04-26T05:00:13Z" level=info msg="Stopping /nextcloud-aio-mastercontainer (69c1fd3669ce) with SIGTERM"
time="2025-04-26T05:00:14Z" level=debug msg="Removing container 69c1fd3669ce"
time="2025-04-26T05:00:14Z" level=info msg="Creating /nextcloud-aio-mastercontainer"
time="2025-04-26T05:00:14Z" level=debug msg="Starting container /nextcloud-aio-mastercontainer (afdd1d53209a)"
time="2025-04-26T05:00:15Z" level=error msg="Error response from daemon: crun: cannot set memory swappiness with cgroupv2: OCI runtime error"
time="2025-04-26T05:00:15Z" level=info msg="Session done" Failed=1 Scanned=1 Updated=0 notify=no
time="2025-04-26T05:00:15Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

Nextcloud itself is still up and running. But of course I have to restart AIO now.

[aanno]

The restart was quite interesting. nextcloud-aio-mastercontainer (Nextcloud AIO v10.13.0) first initiated the backup and then has tried to update the containers. However my aio-start.sh includes manual pull for all images.

2025-04-26T13:55:32Z �[0;92mInitial startup of Nextcloud All-in-One complete!
2025-04-26T13:55:32Z You should be able to open the Nextcloud AIO Interface now on port 8080 of this server!
2025-04-26T13:55:32Z E.g. https://internal.ip.of.this.server:8080
2025-04-26T13:55:32Z ⚠️ Important: do always use an ip-address if you access this port and not a domain as HSTS might block access to it later!
2025-04-26T13:55:32Z 
2025-04-26T13:55:32Z If your server has port 80 and 8443 open and you point a domain to your server, you can get a valid certificate automatically by opening the Nextcloud AIO Interface via:
2025-04-26T13:55:32Z https://your-domain-that-points-to-this-server.tld:8443�[0m
2025-04-26T13:55:34Z ++ head -1 /mnt/docker-aio-config/data/daily_backup_time
2025-04-26T13:55:34Z + BACKUP_TIME=05:00
2025-04-26T13:55:34Z + export BACKUP_TIME
2025-04-26T13:55:34Z + export DAILY_BACKUP=1
2025-04-26T13:55:34Z + DAILY_BACKUP=1
2025-04-26T13:55:34Z ++ sed -n 2p /mnt/docker-aio-config/data/daily_backup_time
2025-04-26T13:55:34Z + '[' '' '!=' automaticUpdatesAreNotEnabled ']'
2025-04-26T13:55:34Z + export AUTOMATIC_UPDATES=1
2025-04-26T13:55:34Z + AUTOMATIC_UPDATES=1
2025-04-26T13:55:34Z ++ sed -n 3p /mnt/docker-aio-config/data/daily_backup_time
2025-04-26T13:55:34Z + '[' '' '!=' successNotificationsAreNotEnabled ']'
2025-04-26T13:55:34Z + export SEND_SUCCESS_NOTIFICATIONS=1
2025-04-26T13:55:34Z + SEND_SUCCESS_NOTIFICATIONS=1
2025-04-26T13:55:34Z + set +x
2025-04-26T13:55:34Z Daily backup script has started
2025-04-26T13:55:34Z [26-Apr-2025 13:55:34] NOTICE: fpm is running, pid 169
2025-04-26T13:55:34Z [26-Apr-2025 13:55:34] NOTICE: ready to handle connections
2025-04-26T13:55:34Z [Sat Apr 26 13:55:34.475113 2025] [mpm_event:notice] [pid 158:tid 158] AH00489: Apache/2.4.62 (Unix) OpenSSL/3.3.3 configured -- resuming normal operations
2025-04-26T13:55:34Z [Sat Apr 26 13:55:34.475200 2025] [core:notice] [pid 158:tid 158] AH00094: Command line: 'httpd -D FOREGROUND'
2025-04-26T13:55:34Z {"level":"info","ts":1745675734.5016572,"msg":"using config from file","file":"/Caddyfile"}
2025-04-26T13:55:34Z {"level":"info","ts":1745675734.5052416,"msg":"adapted config to JSON","adapter":"caddyfile"}
2025-04-26T13:55:34Z Error: No such object: nextcloud-aio-apache
2025-04-26T13:55:34Z APACHE_PORT is not set which is not expected...
2025-04-26T13:55:34Z Starting mastercontainer update...
2025-04-26T13:55:34Z (The script might get exited due to that. In order to update all the other containers correctly, you need to run this script with the same settings a second time.)
2025-04-26T13:55:36Z Waiting for watchtower to stop
2025-04-26T13:56:06Z Creating daily backup...
2025-04-26T13:56:09Z Waiting for backup container to stop
2025-04-26T13:56:40Z Waiting for backup container to stop
2025-04-26T13:57:10Z Waiting for backup container to stop
2025-04-26T13:57:40Z Waiting for backup container to stop
2025-04-26T13:58:10Z Waiting for backup container to stop
2025-04-26T13:58:40Z Waiting for backup container to stop
2025-04-26T13:59:11Z Starting and updating containers...
2025-04-26T14:00:18Z NOTICE: PHP message: Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
2025-04-26T14:00:19Z Deleting duplicate sessions
2025-04-26T14:00:29Z NOTICE: PHP message: Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
2025-04-26T14:00:34Z NOTICE: PHP message: 404 Not Found
2025-04-26T14:00:34Z Type: Slim\Exception\HttpNotFoundException
2025-04-26T14:00:34Z Code: 404
2025-04-26T14:00:34Z Message: Not found.
2025-04-26T14:00:34Z File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
2025-04-26T14:00:34Z Line: 76
2025-04-26T14:00:34Z Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(62): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:00:34Z #1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(482): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:00:34Z #2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(177): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
2025-04-26T14:00:34Z #3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(117): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:00:34Z #4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(129): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
2025-04-26T14:00:34Z #5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(36): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:00:34Z #6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(280): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
2025-04-26T14:00:34Z #7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(77): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:00:34Z #8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(129): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
2025-04-26T14:00:34Z #9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(73): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:00:34Z #10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(209): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:00:34Z #11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(193): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:00:34Z #12 /var/www/docker-aio/php/public/index.php(189): Slim\App->run()
2025-04-26T14:00:34Z #13 {main}
2025-04-26T14:00:34Z Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.
2025-04-26T14:00:35Z NOTICE: PHP message: Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
2025-04-26T14:01:08Z Waiting for the Nextcloud container to start
2025-04-26T14:01:39Z Waiting for the Nextcloud container to start
2025-04-26T14:02:09Z Waiting for the Nextcloud container to start
2025-04-26T14:02:18Z NOTICE: PHP message: Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
2025-04-26T14:02:27Z NOTICE: PHP message: Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
2025-04-26T14:02:34Z NOTICE: PHP message: Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
2025-04-26T14:02:39Z Waiting for the Nextcloud container to start
2025-04-26T14:03:09Z Waiting for the Nextcloud container to start
2025-04-26T14:03:40Z Waiting for the Nextcloud container to start
2025-04-26T14:04:10Z Waiting for the Nextcloud container to start
2025-04-26T14:04:32Z NOTICE: PHP message: Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
2025-04-26T14:04:39Z NOTICE: PHP message: Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
2025-04-26T14:04:40Z Waiting for the Nextcloud container to start
2025-04-26T14:05:10Z Waiting for the Nextcloud container to start
2025-04-26T14:05:40Z Connection to nextcloud-aio-nextcloud (10.89.57.25) 9000 port [tcp/*] succeeded!
2025-04-26T14:05:40Z Sending backup notification...
2025-04-26T14:05:45Z NOTICE: PHP message: Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
2025-04-26T14:05:51Z Daily backup script has finished
2025-04-26T14:05:51Z Could not get digest of container docker.io/nextcloud/all-in-one:latest Client error: `HEAD https://registry-1.docker.io/v2/docker.io/nextcloud/all-in-one/manifests/latest` resulted in a `401 Unauthorized` response
2025-04-26T14:06:04Z Deleted Images:
2025-04-26T14:06:04Z deleted: db7652cc4f761ffde611d03cddc71ad0d6e8f109b6732c27bc25c44e8b24732c
2025-04-26T14:06:04Z deleted: b2008bbfaf381d9cffb0195895fc17c368f601ccd4288a99f3fc363c3bd6cdc2
2025-04-26T14:06:04Z deleted: b4afb61a6f8c149420e3669e7840412a71861fa449c0547b0dfb43230df4028b
2025-04-26T14:06:04Z deleted: 7207aafbd8f0e0c9a25a0821ab0a05c263fd05bae9aff2f2fb82ddecc44bd2c2
2025-04-26T14:06:04Z deleted: 61827e7a97bc5339468f8471bc3bcb2fde1d96e53c26496207361dbe297e5ccc
2025-04-26T14:06:04Z deleted: a5659698eff15fcd39f1dcc92cf2742277a8ac445315683d5469f6554486504d
2025-04-26T14:06:04Z deleted: e55079cb40029e11b7f361085badd4fc0ca1456c082ceb5add074d4573111209
2025-04-26T14:06:04Z deleted: 40a677348d86eab164436fc014a5ad9c1a71f50bb27e7e9e69902bf7bb91b1f4
2025-04-26T14:06:04Z deleted: c306f799705603a58d45a8126a8b7c2419b41f55fdf35f547ee976c87ef48e05
2025-04-26T14:06:04Z deleted: fb87954d10a273c52eff053b1c913665f8999b7d05f70a15a5967c017b472511
2025-04-26T14:06:04Z deleted: 0c708e01a2eaef17a5e9b911505fbccc062b5b19c74904ee92f8096fc19c0bbe
2025-04-26T14:06:04Z deleted: 8bd2e9e7ecc334f97d638b1805a9855d605184cca37d3b2213894c215f13ffd6
2025-04-26T14:06:04Z 
2025-04-26T14:06:04Z Total reclaimed space: 5.72GB
2025-04-26T14:06:09Z NOTICE: PHP message: 404 Not Found
2025-04-26T14:06:09Z Type: Slim\Exception\HttpNotFoundException
2025-04-26T14:06:09Z Code: 404
2025-04-26T14:06:09Z Message: Not found.
2025-04-26T14:06:09Z File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
2025-04-26T14:06:09Z Line: 76
2025-04-26T14:06:09Z Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(62): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:06:09Z #1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(482): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:06:09Z #2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(177): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
2025-04-26T14:06:09Z #3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(117): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:06:09Z #4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(129): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
2025-04-26T14:06:09Z #5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(36): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:06:09Z #6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(280): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
2025-04-26T14:06:09Z #7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(77): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:06:09Z #8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(129): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
2025-04-26T14:06:09Z #9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(73): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:06:09Z #10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(209): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:06:09Z #11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(193): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
2025-04-26T14:06:09Z #12 /var/www/docker-aio/php/public/index.php(189): Slim\App->run()
2025-04-26T14:06:09Z #13 {main}
2025-04-26T14:06:09Z Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.
2025-04-26T14:07:04Z ++ head -1 /mnt/docker-aio-config/data/daily_backup_time
2025-04-26T14:07:04Z + BACKUP_TIME=05:00
2025-04-26T14:07:04Z + export BACKUP_TIME
2025-04-26T14:07:04Z + export DAILY_BACKUP=1
2025-04-26T14:07:04Z + DAILY_BACKUP=1
2025-04-26T14:07:04Z ++ sed -n 2p /mnt/docker-aio-config/data/daily_backup_time
2025-04-26T14:07:04Z + '[' '' '!=' automaticUpdatesAreNotEnabled ']'
2025-04-26T14:07:04Z + export AUTOMATIC_UPDATES=1
2025-04-26T14:07:04Z + AUTOMATIC_UPDATES=1
2025-04-26T14:07:04Z ++ sed -n 3p /mnt/docker-aio-config/data/daily_backup_time
2025-04-26T14:07:04Z + '[' '' '!=' successNotificationsAreNotEnabled ']'
2025-04-26T14:07:04Z + export SEND_SUCCESS_NOTIFICATIONS=1
2025-04-26T14:07:04Z + SEND_SUCCESS_NOTIFICATIONS=1
2025-04-26T14:07:04Z + set +x

[aanno]

At present AIO web interface results in 502 (Bad Gateway).

podman logs nextcloud-aio-watchtower attached.

nextcloud-aio-watchtower.log

[xpseudonym]

@apparle

... the only workaround I've found so far is a hacked version of nextcloud-aio-watchtower image...

Is your comment here what you are referring to?

[apparle]

Yes, but that's outdated now.
I wrote a PR to fix all this - #6533 . It was merged very recently, and hasn't been released yet. I'll recommend you just wait until its released (few weeks to a month).

[xpseudonym]

Hmm... It felt like things were on the cusp. Thank you - but, here is me with this w/e set aside for this little project... I see you got two birds with one stone. So, if I wait, I won't have to worry about all this shenanigans...?

[apparle]

Yeah, it should just work once a release goes out with that PR, without any manual steps. That was my goal.

If you want small weekend projects, you can contribute to AIO codebase 🙂

[xpseudonym]

If you want small weekend projects, you can contribute to AIO codebase 🙂

Thank you, you flatter me more than you know - sleuthing, debugging is all good, but my memory fails me with code...

[aanno]

Does #6533 solves this issue?

[apparle]

The watchtower updates issue - Yes.

There's some improvements in #6574 and #6584 which make systemd service start & stop more robust and reliable.

The hairpin NAT for Talk is still an open issue.

[apparle]

Here's my latest setup after all the fixes (v11.3.0 or above) for reference that I use with podman v4.9.3 userspace containers along with traefik. This becomes nextcloud.service and starts/stops automatically on boot/shutdown, and can be manually controlled with systemctl

# ~/.config/containers/systemd/nextcloud.container

[Container]
ContainerName=nextcloud-aio-mastercontainer
Image=ghcr.io/nextcloud-releases/all-in-one:latest
Pull=missing
RunInit=true

Environment=APACHE_PORT=11000 
Environment=APACHE_IP_BINDING=127.0.0.1 
Environment=APACHE_ADDITIONAL_NETWORK=front_net 
Environment=NEXTCLOUD_DATADIR=/media/data 
Environment=NEXTCLOUD_ENABLE_DRI_DEVICE=true 
Environment=WATCHTOWER_DOCKER_SOCKET_PATH=%t/podman/podman.sock 

Volume=nextcloud_aio_mastercontainer.volume:/mnt/docker-aio-config 
Volume=%t/podman/podman.sock:/var/run/docker.sock:ro

Label=traefik.enable=true
Label=traefik.http.routers.aio-nextcloud-router.entrypoints=websecure
Label=traefik.http.routers.aio-nextcloud-router.rule=Host(`aio-nextcloud.${MYDOMAIN}`)
Label=traefik.http.services.aio-nextcloud.loadbalancer.server.port=8080
Label=traefik.http.services.aio-nextcloud.loadbalancer.server.scheme=https
Label=traefik.http.services.aio-nextcloud.loadbalancer.serversTransport="insecure-transport@file" 

Network=front_net.network

Notify=healthy
# Note, above "Notify=healthy" should just work, but there's a bug in quadlet/podman v4.9.3 which is fixed in podman v5.0.0. Below is a workaround that adds the --sdnotify again in the end for v4.9.3
PodmanArgs=--sdnotify=healthy

[Unit]
Description=Nextcloud AIO
After=traefik.service authentik.service
Wants=traefik.service authentik.service
Requires=podman.socket

[Service]
Restart=on-failure
# Below command should only run after mastercontainer is healthy, which is ensured by Notify=healthy above.
ExecStartPost=/usr/bin/podman exec --env START_CONTAINERS=1 nextcloud-aio-mastercontainer /daily-backup.sh

# Below can be split into multi-line with backslashes after quadlet/podman v5.0.0 
ExecStop=/usr/bin/bash -c 'if podman container exists nextcloud-aio-mastercontainer ; then podman exec --env STOP_CONTAINERS=1 nextcloud-aio-mastercontainer /daily-backup.sh; fi'
ExecStop=/usr/bin/bash -c 'while podman ps --format "{{.Names}}" | grep -v nextcloud-aio-mastercontainer | grep nextcloud-aio ; do echo "nextcloud-aio-* containers still running. Attempting to stop them manually."; podman ps --format "{{.Names}}" | grep -v nextcloud-aio-mastercontainer | grep nextcloud-aio | xargs -r -L1 podman stop; sleep 10s; done'

TimeoutStartSec=600
TimeoutStopSec=600

[Install]
WantedBy=default.target

and

# ~/.config/containers/systemd/nextcloud_aio_mastercontainer.volume

[Unit]
Description=Nextcloud AIO Master Container Volume

[Volume]
VolumeName=nextcloud_aio_mastercontainer

Note, traefik's container is setup separately along with it's configuration yaml describing the mapping for nextcloud.${MYDOMAIN}. Above labels only create the mapping for aio's webpage itself.
Also front_net.network is defined separately with traefik's container.

[apparle]

@aanno now that #5568 is merged, you can follow the latest instructions on https://github.com/nextcloud/all-in-one/blob/main/develop.md#how-to-locally-build-and-test-changes-to-mastercontainer to test code change to add Extra Host like I described in #5090 (reply in thread) :

This seems like a small thing to fix in DockerActionManager :: CreateContainer by adding:

if ($container->GetIdentifier() != 'nextcloud-aio-domaincheck') {
        $requestBody['HostConfig']['ExtraHosts'] = [ $this->configurationManager->GetDomain() . ":host-gateway"] ;
}

Maybe it could be guarded under a non-default env variable or even a dynamically detected if container runtime is podman instead of docker, to avoid adding any configuration variables. @szaimen what do you think?

I don't have podman v5.3.0 on my server nor have Nextcloud Talk set up, so I can't test the fix myself. But if above fix does work, I can create a PR to incorporate this feature in AIO as well.