Hang when containers set to start automatically

[rufoa]

Thanks for this project - it is great. I have noticed a small problem however:

When a container uses this plugin, and its restart policy makes the container start automatically when Docker starts, this plugin appears to hang. This prevents the container from starting, and seems to block the Docker daemon from responding too. If I kill this plugin's process, Docker seems to recover (but the container obviously doesn't come up properly).

If the container is not set to start automatically, and I instead start it manually, everything works fine.

I have narrowed the problem down to this line - it seems the call to NetworkInspect never returns, even after several hours.

I thought the problem might be a race condition, where the network was not fully up before the plugin tries to inspect it. However, inserting a delay before the call does not appear to help.

The logs do not provide any clues.

Because the Docker daemon stops responding, I am unfortunately not able to get a stack trace from it.

Please could you let me know how I might diagnose the problem further? I'm using up to date versions of Docker, Ubuntu and the kernel. The only complicating factor is that it's on an armv7l SBC 🙈

Many thanks

[crolves]

I am seeing this exact same behavior on both a Centos 7 install and Ubuntu 21.04 install both on amd64 hardward.

[Ermodo]

I am also facing this issue on Ubuntu 20.04 armv8 and Ubuntu 18.04 armv7.

[tekgeek1205]

Me also, I'm new to docker and it took me a really long time and 2 reinstalls to realise this was the problem.

Edit:
I was able to get containers with the driver to start on boot by setting the restart policy to On Failure and adding

(sleep 20
docker start bragi) &

to my rc.local

[flywheelnz]

I'm also having this problem. From a quick glance, it looks like the plugin tries to access the Docker socket before it's up and running, and there's no timeout on the call and no way that I can see to defer the starting of the containers until after Docker is more started. 🤷

Logs below give a rough indication of the order of things - the container loads and hangs connecting to Docker - is interrupted (By me), and then starts providing the routes it relies on 🤦

DEBU[2021-12-08T14:55:22.411028372+13:00] Assigning addresses for endpoint portainer's interface on network home

^CINFO[2021-12-08T14:57:44.718633565+13:00] Processing signal 'interrupt'
DEBU[2021-12-08T14:57:44.719427052+13:00] Releasing addresses for endpoint portainer's interface on network home
ERRO[2021-12-08T14:57:44.728083546+13:00] failed to start container                     container=ac48752e9a5c199e3f372109b05701261fc659c46a0d5068b64fdaaae9c1f2ed error="failed to create endpoint portainer on network home: NetworkDriver.CreateEndpoint: failed to get network options: failed to get info from Docker: error during connect: Get \"http://%2Frun%2Fdocker.sock/v1.13.1/networks/7b2bec4e103227f40b09c0e8744c514682b6cfa92a751efa79dbee7a7f406546\": read unix @->/var/run/docker.sock: read: connection reset by peer"
INFO[2021-12-08T14:57:44.728203189+13:00] Loading containers: done.
INFO[2021-12-08T14:57:44.749048232+13:00] Docker daemon                                 commit=847da18 graphdriver(s)=overlay2 version=20.10.11
INFO[2021-12-08T14:57:44.750232282+13:00] Daemon has completed initialization
DEBU[2021-12-08T14:57:44.774791290+13:00] Registering routers

Same problem. I worked around it by starting the container independently, from systemd.

Not very stable though.

[HackerBaloo]

I love that this plugin let my containers get an IP over DHCP.
But I also experienced this problem. When the docker daemon hangs reboot is also hanging on my Ubuntu 20.04, not fun if your are connected remotely.

Any idea on how to solve it?

[sdjnmxd]

Same problem. I tried to research a very tricky and ad hoc way to temporarily circumvent this problem.
First of all, I gave up using docker's restart=always as a boot, because it would cause docker to get stuck. So, I wrote a service file to use systemctl to start the docker container, the file is written like this:

[Unit]
Description=Docker Container
After=network-online.target docker.socket docker.service firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/root/docker-container
ExecStart=docker start ${DOCKER_CONTAINER_LIST}
ExecStop=docker stop ${DOCKER_CONTAINER_LIST}

[Install]
WantedBy=multi-user.target

This file is really tricky. The essence is to execute docker start ${environment variable} to start the container after docker starts normally. Here my environment variable file is placed under /root/docker-container, you can also put it elsewhere. The file for environment variables is written like this:

DOCKER_CONTAINER_LIST=centos nginx phpmyadmin ……

The trouble is, because docker is no longer used to manage boot-started containers, whenever there is a container that needs to be booted up, you need to append the container name or container ID to the environment variable file, so that systemctl can control it to start automatically after booting

---

My system environment is Rocky Linux release 8.6, and the Docker version is 20.10.17. Different distributions may have some different writing on the service file, such as: "After", "Wants", "Requires", etc. , you may need to make adjustments yourself if necessary

[NonaSuomy]

Same problem. I tried to research a very tricky and ad hoc way to temporarily circumvent this problem. First of all, I gave up using docker's restart=always as a boot, because it would cause docker to get stuck. So, I wrote a service file to use systemctl to start the docker container, the file is written like this:

[Unit]
Description=Docker Container
After=network-online.target docker.socket docker.service firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/root/docker-container
ExecStart=docker start ${DOCKER_CONTAINER_LIST}
ExecStop=docker stop ${DOCKER_CONTAINER_LIST}

[Install]
WantedBy=multi-user.target

This file is really tricky. The essence is to execute docker start ${environment variable} to start the container after docker starts normally. Here my environment variable file is placed under /root/docker-container, you can also put it elsewhere. The file for environment variables is written like this:

DOCKER_CONTAINER_LIST=centos nginx phpmyadmin ……

The trouble is, because docker is no longer used to manage boot-started containers, whenever there is a container that needs to be booted up, you need to append the container name or container ID to the environment variable file, so that systemctl can control it to start automatically after booting

My system environment is Rocky Linux release 8.6, and the Docker version is 20.10.17. Different distributions may have some different writing on the service file, such as: "After", "Wants", "Requires", etc. , you may need to make adjustments yourself if necessary

Nov 08 00:55:14 docker001 docker[9228]: Error response from daemon: No such container: mosquitto portainer esphome frigate rtlamr2mqtt lms watchtower homeassistant
Nov 08 00:55:14 docker001 docker[9228]: Error: failed to start containers: mosquitto portainer esphome frigate rtlamr2mqtt lms watchtower homeassistant
Nov 08 00:55:14 docker001 systemd[1]: dockernetdhcp.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 00:55:14 docker001 systemd[1]: dockernetdhcp.service: Failed with result 'exit-code'.
Nov 08 00:55:14 docker001 systemd[1]: Failed to start Docker Net DHCP Container Hang Fix.

Manual test: Seems to work fine.
[root@docker001 system]# docker start mosquitto portainer esphome frigate rtlamr2mqtt lms watchtower homeassistant
mosquitto
portainer
esphome
frigate
rtlamr2mqtt
lms
watchtower
homeassistant

Edit:

Got this working like this:
/etc/systemd/system/dockernetdhcp.service

[Unit]
Description=Docker Net DHCP Container Hang Fix
After=network-online.target docker.socket docker.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/root/docker-container
ExecStart=/usr/bin/dockernetdhcpstart.sh
#ExecStop=/usr/bin/docker stop ${DOCKER_CONTAINER_LIST}
ExecStop=/usr/bin/dockernetdhcpstop.sh

[Install]
WantedBy=multi-user.target

/usr/bin/dockernetdhcpstart.sh

#!/bin/bash
# Start docker containers because of a glitch with netdhcp docker plugin.
docker start mosquitto portainer cloudflared esphome frigate rtlamr2mqtt lms watchtower homeassistant piper whisper openwakeword spoolman

/usr/bin/dockernetdhcpstop.sh

#!/bin/bash
# Stop docker containers because of a glitch with netdhcp docker plugin.
docker stop mosquitto portainer cloudflared esphome frigate rtlamr2mqtt lms watchtower homeassistant piper whisper openwakeword spoolman

[NonaSuomy]

@devplayer0 are you able to comment on the overall OP issue above?

Thank you.

[crkinard]

Getting this issue as well. Time to try some of these solutions.

EDIT:
Well I got nothing. Looks like I'm going back to macvlan. Here is to hoping this gets fixed.

[crkinard]

Same problem. I tried to research a very tricky and ad hoc way to temporarily circumvent this problem. First of all, I gave up using docker's restart=always as a boot, because it would cause docker to get stuck. So, I wrote a service file to use systemctl to start the docker container, the file is written like this:

[Unit]
Description=Docker Container
After=network-online.target docker.socket docker.service firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/root/docker-container
ExecStart=docker start ${DOCKER_CONTAINER_LIST}
ExecStop=docker stop ${DOCKER_CONTAINER_LIST}

[Install]
WantedBy=multi-user.target

This file is really tricky. The essence is to execute docker start ${environment variable} to start the container after docker starts normally. Here my environment variable file is placed under /root/docker-container, you can also put it elsewhere. The file for environment variables is written like this:

DOCKER_CONTAINER_LIST=centos nginx phpmyadmin ……

The trouble is, because docker is no longer used to manage boot-started containers, whenever there is a container that needs to be booted up, you need to append the container name or container ID to the environment variable file, so that systemctl can control it to start automatically after booting

My system environment is Rocky Linux release 8.6, and the Docker version is 20.10.17. Different distributions may have some different writing on the service file, such as: "After", "Wants", "Requires", etc. , you may need to make adjustments yourself if necessary

Ended up just tweaking your script a tad for docker-compose.

[Unit]
Description=Docker-Compose Up
After=network-online.target docker.socket docker.service firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=docker-compose -f /usr/sbin/docker-compose.yml up -d
ExecStop=docker-compose -f /usr/sbin/docker-compose.yml down

[Install]
WantedBy=multi-user.target

[NonaSuomy]

@devplayer0 any chance you can compile the above timeout fix for one last build?

I've been using this plugin for years and this is the only issue I have had. Would be nice to not have to work around it with startup scripts.

[NonaSuomy]

Same issues
#2
#44

docker-net-dhcp/pkg/plugin/plugin.go

Line 78 in 03694af

func NewPlugin(awaitTimeout time.Duration) (*Plugin, error) {

docker-net-dhcp/pkg/plugin/plugin.go

Line 82 in 1bb0ffe

docker.WithTimeout(2*time.Second)) // If local Docker doesn't respond in under 2s, it's probably not ready.