Debugging and Cleanup

Two things you will do more than anything else once you actually depend on Docker: work out why a container is not behaving, and stop Docker eating the disk. Neither is difficult; both go faster if you know which command answers which question.

The four commands that answer most questions

docker logs -f --tail 100 web       # what did it print?
docker inspect web                  # how was it configured?
docker stats                        # what is it consuming, live?
docker events                       # what is the daemon doing right now?

inspect returns a wall of JSON, which is why it takes a format string:

That first pair is a complete diagnosis. Exit code 137 is 128 + 9, meaning the process was killed with SIGKILL, and OOMKilled: true says the kernel did it for exceeding a memory limit. No log line, no stack trace — the process was not asked to stop, it was stopped.

Reading exit codes

126 and 127 are almost always an image problem rather than a code problem: a script without the executable bit, a shebang pointing at /bin/bash in an Alpine image that only has /bin/sh, or a binary built for the wrong architecture.

When the container will not start at all

If it exits immediately and the logs are empty, you cannot exec into it — there is nothing running to join. Override the entrypoint and get a shell in the same image instead:

docker run --rm -it --entrypoint sh myapp

Now you are inside the image's filesystem and can check whether the file exists, whether it is executable, and what the environment actually looks like. Nine times out of ten it is a path.

For an image with no shell at all — distroless, scratch — Docker Desktop ships docker debug, which attaches a toolbox of debugging binaries to a running or stopped container without changing the image:

docker debug myapp

Where the disk went

docker system df

Build cache is usually the biggest number and the one people forget exists. Forty gigabytes of it is unremarkable on a machine that builds regularly.

docker container prune          # stopped containers
docker image prune              # untagged images
docker image prune -a           # every image no container is using
docker builder prune            # build cache
docker builder prune --keep-storage 10GB   # keep a working set

docker system prune             # containers + networks + dangling images + cache
docker system prune -a --volumes   # everything. Including your data.

docker system prune -a --volumes is the dangerous one. It deletes every unused volume, and "unused" means "no container currently references it" — which includes the database volume of a stack you stopped for the weekend. Run docker volume ls first and know what you are agreeing to.

Cap the logs before they cap you

The default logging driver writes JSON to disk with no size limit. A chatty container will happily produce hundreds of gigabytes, and because the file is not in /var/lib/docker/volumes it does not show up in docker system df at all.

Set a limit globally in /etc/docker/daemon.json:

{
  "log-driver": "local",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

The local driver is a better default than json-file: it compresses, rotates, and is cheaper to write. Engine 29.5 also added custom attributes to it, so you can tag log entries at the driver level.

Per service in Compose:

services:
  api:
    logging:
      driver: local
      options:
        max-size: "10m"
        max-file: "3"

That is the whole of everyday Docker: run, build, store, connect, compose, debug. Docker in Practice picks up from here with multi-stage builds, BuildKit's cache and secret mounts, multi-platform images, Bake, and the Compose features that make a real development loop.