Healthchecks and Signals

Docker knows whether your process is running. It does not know whether your application works, and it will happily keep a wedged container in the rotation forever unless you tell it how to check.

The other half of the same subject: containers have to stop as well as start, and stopping badly is where dropped requests and corrupted state come from.

Healthchecks

HEALTHCHECK --interval=30s --timeout=3s --start-period=40s --retries=3 \
  CMD curl -fsS http://localhost:3000/healthz || exit 1

Or in Compose, where it does not need to be baked into the image:

services:
  api:
    image: myapp
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:3000/healthz"]
      interval: 30s
      timeout: 3s
      start_period: 40s
      retries: 3

The container gets a status: starting, then healthy or unhealthy.

start_period is the field people leave out and then fight. During it, failures do not count toward retries — it is the grace window for an app that takes thirty seconds to connect to its database and warm a cache. Without it, a slow-starting service is marked unhealthy before it ever had a chance.

Engine 29 also surfaces health in the API's container list, so tooling can read it without inspecting each container individually.

Write the check honestly

A healthcheck that always passes is worse than none, because it makes an outage look like a healthy service.

test: ["CMD-SHELL", "exit 0"]                    # meaningless
test: ["CMD", "curl", "-f", "http://localhost:3000/"]   # is the port open?
test: ["CMD", "curl", "-fsS", "http://localhost:3000/healthz"]  # does the app work?

A useful /healthz checks the things whose failure means this container cannot serve: the database connection, a required cache, a filesystem it must write to. It should not check things it does not control — a downstream API being slow should not take your whole fleet out of rotation.

Note CMD versus CMD-SHELL: the array form execs directly and needs no shell in the image, while CMD-SHELL runs through /bin/sh and lets you use pipes and ||. Distroless images can only use the first, which is why healthchecks for them are usually a small static binary shipped alongside the app.

Stopping: what docker stop actually does

1. SIGTERM to PID 1.
2. Wait — ten seconds by default.
3. SIGKILL if it is still alive.

So a graceful shutdown means catching SIGTERM, refusing new work, finishing in-flight requests, closing connections, and exiting. If your process does not, it gets killed mid-request every single deploy.

Two things routinely break this even in applications that do handle the signal:

Shell form in the Dockerfile. CMD node server.js becomes /bin/sh -c "node server.js". The shell is PID 1, does not forward signals, and your app never hears anything. Use the exec form: CMD ["node", "server.js"].

Nothing reaping children. A PID 1 that spawns processes must wait() on them or they accumulate as zombies. --init inserts a tiny init process that handles both reaping and signal forwarding:

docker run --init myapp

services:
  api:
    init: true

Tuning the grace period

docker run --stop-timeout 30 myapp
docker stop -t 30 api

STOPSIGNAL SIGQUIT

services:
  api:
    stop_grace_period: 30s
    stop_signal: SIGQUIT

STOPSIGNAL matters for software that does not use the convention — nginx treats SIGTERM as a fast shutdown that drops connections and SIGQUIT as the graceful one, which is exactly backwards from what Docker sends by default.

Engine 29.7 added a default-stop-timeout daemon option, so a host can set the fleet-wide default instead of every container carrying its own flag.

docker run --restart unless-stopped myapp

docker inspect -f '{{.RestartCount}}' api
docker events --filter container=api --filter event=restart

Next up: build caching across machines — making CI as fast as your laptop.