DietPi-Software | Docker: Service fails to start after upgrade (AppArmor)

[samjw-nz]

ADMIN EDIT

Workaround

apt install apparmor
systemctl disable --now apparmor
systemctl restart docker

---

Creating a bug report/issue

Required Information

• DietPi version | 8.13.2
• Distro version | Bullseye
• Kernel version | Linux Sam-Pi 5.15.89-sunxi #22.11.4 SMP Mon Jan 23 21:58:30 UTC 2023 armv7l GNU/Linux
• SBC model | ZeroPi (FriendlyArm/FriendlyElec)
• Power supply used | 5V 2A supply
• SD card used | Samsung Evo Plus microSD

Additional Information (if applicable)

• Software title | Portainer
• Was the software title installed freshly or updated/migrated? | Both. Fresh install after failing to update.
• Can this issue be replicated on a fresh installation of DietPi? | Unsure, have not tried yet. This install has been going for a couple years.

• Bug report ID | 484d3e4a-4e4c-44bd-b2c4-756bdd3c6779

Steps to reproduce

1. Install Docker from dietpi-software (optional)
2. Install Portainer from dietpi-software (with or without first installing Docker, as it's installed if not already present)
3. Allow Portainer image to pull and then fail by itself.

Expected behaviour

• Portainer properly install & accessible from web interface.

Actual behaviour

• Failing to install Portainer witherror message "Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded"

Extra details

• Have had Portainer installed running only one container (UptimeKuma) with no issue.
• Found sometime after apt update/upgrade to Portainer-CE that Portainer web interface is inaccessible and UptimeKuma container has stopped.
• Attempted to both reinstall and fresh install both Docker and Portainer from dietpi-software.
• If I attempt to retry after the failure interrupts isntall, it will fail again with "docker: Error response from daemon: Conflict. The container name "/portainer" is already in use by {imageID here}"
• I am not familiar with AppArmor, and have not knowingly installed it.

[Joulinar]

uname -a (response: command not found)

That worries me a little. This command should display current kernel version. Can you reboot your system and check for kernel error messages afterwards.

dmesg -l err,crit,alert,emerg

As well check Docker logs

journalctl -u docker.service

[samjw-nz]

No kernel errors, or at least no response to the command anyway.

Docker log:

-- Journal begins at Fri 2023-02-03 20:38:57 NZDT, ends at Fri 2023-02-03 20:59:21 NZDT. --
Feb 03 20:39:09 Sam-Pi systemd[1]: Started Docker Application Container Engine.
Feb 03 20:39:12 Sam-Pi dockerd[589]: time="2023-02-03T20:39:12.647336342+13:00" level=error msg="AppArmor enabled on system but the docker-default profile could not be loaded: running `apparmor_parser apparmor_parser --version` failed with output: \nerror: exec: \"apparmor_parser\": executable file not found in $PATH"

Edit: tried formatting it nicely and failed sorry.

[samjw-nz]

Ah, ran uname command again and it worked this time.
Linux Sam-Pi 5.15.89-sunxi #22.11.4 SMP Mon Jan 23 21:58:30 UTC 2023 armv7l GNU/Linux

[Joulinar]

ok I found a known error for Docker v5.23 within Docker docs https://docs.docker.com/engine/release-notes/23.0/#known-issues

can you check the version you are running.

dpkg -l docker-ce

[samjw-nz]

5:23.0.0-1~debian.11~bullseye armhf

Looks like that includes me then. Want me to try their fix and report back?

[Joulinar]

according docker docs

apt install apparmor
systemctl stop apparmor

[samjw-nz]

All working again, no further errors. Cheers for the help, much appreciated.

[Joulinar]

@MichaIng
tried to replicate issue on five different system. All ARM based systems working without issue. It failed on VM x86 only.

• RPi1 ARMv6 ✅
• RPi3B+ ARMv7 ✅
• RPi4B ARMv8 ✅
• NanoPi R5S ARMv8 ✅
• VM x86 AMD64 ❌

Not 100% sure if there is a real trigger or just a random issue. 🤔

[MichaIng]

Good to know that it is no a Bookworm-only issue. Faced it last night on our server. No good instructions in their release notes.

1. Is this expected now or a bug that is being worked on? I guess/hope the second, since AppArmor is/was in fact not installed/enabled on affected systems.
2. Instead of installing all AppArmor utils together with Python 3 and module, do apt install apparmor which is only a very small subset but suffiicient.
3. Disable the service systemctl disable --now apparmor unless you are aware of the implications and know how to configure it to work with your applications.

[MichaIng]

Here is a good bug report, attached to 23.0.1 milestone 🤞: moby/moby#44900

[pedrom34]

@MichaIng tried to replicate issue on five different system. All ARM based systems working without issue. It failed on VM x86 only.

* RPi1 ARMv6 ✅

* RPi3B+ ARMv7 ✅

* RPi4B ARMv8 ✅

* NanoPi R5S ARMv8 ✅

* VM x86 AMD64 ❌

Not 100% sure if there is a real trigger or just a random issue. 🤔

Had the same issue yesterday on my Rock64. Had to apt install apparmor to solve it.

[MichaIng]

Interesting, so not only x86_64.

[Joulinar]

Nope, original issue was reported on a ZeroPi

[cenizo]

So for amd64 there is no solution?

[Joulinar]

Solution is described on Docker docs

apt install apparmor-utils

[cenizo]

This solution doesn't work for me

 apt install apparmor-utils
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  apparmor python3-apparmor python3-libapparmor
Suggested packages:
  apparmor-profiles-extra vim-addon-manager
The following packages will be REMOVED:
  linux-image-5.10.0-21-amd64
The following NEW packages will be installed:
  apparmor apparmor-utils python3-apparmor python3-libapparmor
0 upgraded, 4 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/1023 kB of archives.
After this operation, 314 MB disk space will be freed.
Do you want to continue? [Y/n] y
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 72244 files and directories currently installed.)
Removing linux-image-5.10.0-21-amd64 (5.10.162-1) ...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-5.10.0-21-amd64
/etc/kernel/postrm.d/zz-update-grub:
/usr/sbin/grub-mkconfig: 9: /etc/default/grub: splash: not found
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 127
dpkg: error processing package linux-image-5.10.0-21-amd64 (--remove):
 installed linux-image-5.10.0-21-amd64 package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
 linux-image-5.10.0-21-amd64
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Joulinar]

As already stated on our forum, you have an issue with kernel update blocking your system now.

[cenizo]

I know, don't you know of a way to go back?

[MichaIng]

Please use the solution I added to the OP here:

apt install apparmor
systemctl stop apparmor

Otherwise you pull a lot of additional stuff you do not need here and run the service unnecessarily.

@cenizo
Can you show the output of:

cat /etc/default/grub

[cenizo]

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash acpi_osi=Linux”
GRUB_CMDLINE_LINUX="acpi_enforce_resources=lax"

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

[MichaIng]

Great to see a solution merged 👍.

[madmedix]

Me too last night: Rock64 1G fresh install (cb0d547c-e913-47ca-b648-e40893a0b3e9), nothing else installed - I checked on Docker and Portainer. I give the above a go too.
Cheers,

[Joulinar]

There is no other way around until Docker Devs fixed it. Until a new Docker version is available, the the above workaround is needed.

[MichaIng]

Docker v23.0.1 has been released, issue solved.

[Joulinar]

@MichaIng
not sure if the issue has been fixed fully. There is still an issue for apparmor but with a different error now.

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to apply apparmor profile: apparmor failed to apply profile: write /proc/self/attr/apparmor/exec: no such file or directory: unknown.

[MichaIng]

Your right...

[MichaIng]

Ah, it has been reverted in containerd now, the only real fix: containerd/containerd#8086
But milestone is containerd 1.7 🤔. ... ah no backported to v1.6 as well: containerd/containerd#8087

Let's hope for a soon containerd release then.

[MichaIng]

containerd v1.6.18 with the fix has been released, but while the error message has changed containers still do not start:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to apply apparmor profile: apparmor failed to apply profile: write /proc/self/attr/apparmor/exec: no such file or directory: unknown.

Looks like it will be fully resolved Docker-side with: moby/moby#44982

[MichaIng]

Finally, with the latest docker-ce update from today the issue is solved. I didn't expect this to take so long.

[Joulinar]

related release notes from Docker side https://docs.docker.com/engine/release-notes/23.0/#2302