代码之家 › 专栏 › 技术社区 › Greg Bray

dockerd是否支持WatchdogSec sd_通知健康检查?

watchdog hang systemd docker

0

Greg Bray · 技术社区 · 6 年前

Docker守护进程偶尔会在我们的Kubernetes系统上停止响应,但Systemd仍然认为服务正在运行:

systemctl status docker
â docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-04-15 20:40:57 UTC; 3 months 22 days ago
     Docs: https://docs.docker.com
 Main PID: 1281 (dockerd)
    Tasks: 1409
   Memory: 31.0G
      CPU: 5d 17h 3min 4.758s
   CGroup: /system.slice/docker.service
           ââ 1281 /usr/bin/dockerd -H fd://
...

房间里什么都没有 journalctl -u docker 或syslog文件来指示问题所在,但Docker守护进程不再响应请求( docker ps 只是挂了)。我们目前正在使用 17.03.2~ce-0~ubuntu-xenial Ubuntu 16.04的软件包,它有以下服务单元:

cat /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket firewalld.service
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

我注意到尽管这是一个 Type=notify 服务,这里没有 WatchdogSec= 在服务单元中定义。

Docker守护进程是否支持为基于sd_notify的健康检查设置看门狗超时?

0 回复 | 直到 6 年前

1

0

Greg Bray 6 年前

不 ,目前 components/engine/cmd/dockerd/daemon_linux.go 仅文件实现 systemdDaemon.SdNotifyReady 在流程开始时通知Systemd。为了获得看门狗的支持,它必须使用 SdWatchdogEnabled 不断地发送 SdNotifyWatchdog = "WATCHDOG=1" 通知。

如果你试着 WatchdogSec=60s 码头工人。服务文件它将终止并重新启动服务,因为守护进程没有发送所需的通知。

systemctl status docker.service
â docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-08-08 02:09:52 UTC; 50s ago

systemctl status docker.service
â docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: deactivating (stop-sigabrt) (Result: watchdog) since Thu 2019-08-08 02:10:02 UTC; 45ms ago

systemctl status docker.service
â docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: activating (start) since Thu 2019-08-08 02:10:04 UTC; 777ms ago

# Log entries:
Aug 08 02:09:14 kam1 systemd[1]: Starting Docker Application Container Engine...
Aug 08 02:09:15 kam1 systemd[1]: Started Docker Application Container Engine.
Aug 08 02:10:15 kam1 systemd[1]: docker.service: Watchdog timeout (limit 60s)!
Aug 08 02:10:15 kam1 systemd[1]: docker.service: Killing process 12383 (dockerd) with signal SIGABRT.
Aug 08 02:10:16 kam1 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 08 02:10:16 kam1 systemd[1]: docker.service: Failed with result 'watchdog'.
Aug 08 02:10:18 kam1 systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Aug 08 02:10:18 kam1 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Aug 08 02:10:18 kam1 systemd[1]: Stopped Docker Application Container Engine.
Aug 08 02:10:18 kam1 systemd[1]: Starting Docker Application Container Engine...