Observability

docker logs works beautifully for one container on one host. It stops working the moment there are thirty containers across six hosts and the one you need died an hour ago.

Production observability for containers is three separate questions with three separate answers: what did it print, what did the daemon do, and what is it consuming.

Logging drivers

The daemon's logging driver decides where a container's stdout and stderr go.

{
  "log-driver": "local",
  "log-opts": { "max-size": "10m", "max-file": "3", "compress": "true" }
}

local is the right default on a real host. It rotates, compresses, and is cheaper to write than json-file. Engine 29.5 added custom attributes to it, so entries can be tagged at the driver level rather than by the application.

The default json-file driver has no size limit. A chatty container will write until the disk is full, and because the files are not volumes they never appear in docker system df. That second command is the one to run on any host you have inherited.

Forwarding without giving up docker logs

A remote driver like fluentd or gelf breaks docker logs, which is the command everyone reaches for first at 3am. The usual production shape avoids that trade:

Write locally with rotation, and ship the files with a collector. A sidecar or host agent — Fluent Bit, Vector, Promtail, the OpenTelemetry Collector — reads /var/lib/docker/containers/*/*.log, enriches with container metadata, and forwards. docker logs keeps working; the logs also reach your aggregator.

Then, in the application: log JSON to stdout and nothing else. No files, no log rotation inside the container, no syslog. One stream, structured, and the platform decides where it goes. A container writing its own log files is producing data nobody collects and disk nobody bounded.

Events

docker events is the daemon's activity stream, and it answers questions logs cannot:

docker events --since 1h --filter event=oom
docker events --filter container=api --filter event=restart
docker events --filter type=image --filter event=pull
docker events --format '{{.Time}} {{.Type}} {{.Action}} {{.Actor.Attributes.name}}'

Three OOM kills in four minutes, with timestamps. docker ps would have shown that container as Up 12 seconds and looking healthy — because it is up, for the fourth time. Events are how a crash loop stops hiding, and RestartCount is the other half:

docker inspect -f '{{.RestartCount}} {{.State.Status}}' api

Metrics

docker stats
docker stats --no-stream --format '{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}'

Fine interactively, useless as a record — it does not persist. For real metrics, read cgroups directly or run an exporter:

• cAdvisor exposes per-container CPU, memory, network and disk in Prometheus format.
• The daemon itself can expose Prometheus metrics via "metrics-addr" in daemon.json — those are about dockerd, not your containers.
• /sys/fs/cgroup/.../memory.events is the ground truth for OOM kills and reclaim pressure, as covered in the limits lesson.

The container-specific metrics worth alerting on, as opposed to the generic host ones: restart count increasing, OOM kill events, CPU throttling time (a throttled container is slow, not busy, and CPU-percent alerts miss it entirely), and health status transitions. Engine 29 added Health to the container list API, so a collector can read health without inspecting each container.

docker inspect -f '{{.State.Pid}}' api

cat /proc/48213/cgroup
0::/system.slice/docker-7c1f9a3e4d82ab....scope

docker inspect --format '{{.Name}}' 7c1f9a3e4d82

docker top api                                    # its processes
sudo nsenter -t 48213 -n ss -tulpn                # its sockets, host tooling
sudo ls -l /proc/48213/fd | wc -l                 # open file descriptors

Next up: the last lesson — running AI workloads, with Model Runner and Compose's model support.