[{"data":1,"prerenderedAt":590},["ShallowReactive",2],{"mdc--mw6vbf-key":3,"mdc--tesl4r-key":29,"mdc-yh0ss8-key":100,"mdc--5q5u52-key":135,"mdc--8d7q4h-key":147,"mdc-odiqmv-key":224,"mdc--vkjkbq-key":234,"mdc-lz4w59-key":295,"mdc-ld5zcx-key":344,"mdc--r12899-key":356,"mdc-c665j9-key":412},{"data":4,"body":5},{},{"type":6,"children":7},"root",[8,16],{"type":9,"tag":10,"props":11,"children":12},"element","p",{},[13],{"type":14,"value":15},"text","By default a container may use every core and all the memory on the machine. One service with a leak takes down everything else on the host, including whatever you would have used to log in and fix it.",{"type":9,"tag":10,"props":17,"children":18},{},[19,21,27],{"type":14,"value":20},"Limits are cgroups, the mechanism ",{"type":9,"tag":22,"props":23,"children":24},"em",{},[25],{"type":14,"value":26},"Containers From Scratch",{"type":14,"value":28}," sets by hand. Docker's flags write the same files.",{"data":30,"body":31},{},{"type":6,"children":32},[33,40,53,64,86],{"type":9,"tag":34,"props":35,"children":37},"h2",{"id":36},"memory",[38],{"type":14,"value":39},"Memory",{"type":9,"tag":41,"props":42,"children":46},"pre",{"className":43,"code":45,"language":14},[44],"language-text","docker run --memory 512m --memory-reservation 256m myapp\n",[47],{"type":9,"tag":48,"props":49,"children":51},"code",{"__ignoreMap":50},"",[52],{"type":14,"value":45},{"type":9,"tag":41,"props":54,"children":59},{"className":55,"code":57,"language":58,"meta":50},[56],"language-yaml","services:\n  api:\n    mem_limit: 512m\n    mem_reservation: 256m\n","yaml",[60],{"type":9,"tag":48,"props":61,"children":62},{"__ignoreMap":50},[63],{"type":14,"value":57},{"type":9,"tag":10,"props":65,"children":66},{},[67,77,79,84],{"type":9,"tag":68,"props":69,"children":70},"strong",{},[71],{"type":9,"tag":48,"props":72,"children":74},{"className":73},[],[75],{"type":14,"value":76},"--memory",{"type":14,"value":78}," is a hard cap. Cross it and the kernel's OOM killer kills something ",{"type":9,"tag":22,"props":80,"children":81},{},[82],{"type":14,"value":83},"inside that cgroup",{"type":14,"value":85}," — the host is unaffected, which is the entire point.",{"type":9,"tag":10,"props":87,"children":88},{},[89,98],{"type":9,"tag":68,"props":90,"children":91},{},[92],{"type":9,"tag":48,"props":93,"children":95},{"className":94},[],[96],{"type":14,"value":97},"--memory-reservation",{"type":14,"value":99}," is a soft limit: a target the kernel pushes the container back toward under host memory pressure, without killing anything. Setting reservation below limit gives you a container that is squeezed before it is killed.",{"data":101,"body":102},{},{"type":6,"children":103},[104,109],{"type":9,"tag":105,"props":106,"children":108},"terminal-teaser",{":lines":107},"[{\"cmd\":\"docker run -d --name api --memory 512m myapp\",\"out\":\"7c1f9a3e4d82\"},{\"cmd\":\"docker stats --no-stream api\",\"out\":\"NAME   CPU %   MEM USAGE / LIMIT     MEM %   PIDS\\napi    2.14%   487.2MiB / 512MiB     95.16%  23\"},{\"cmd\":\"docker inspect -f '{{.State.OOMKilled}} {{.State.ExitCode}}' api\",\"out\":\"true 137\"}]",[],{"type":9,"tag":10,"props":110,"children":111},{},[112,125,127,133],{"type":9,"tag":68,"props":113,"children":114},{},[115,117,123],{"type":14,"value":116},"Exit code 137 plus ",{"type":9,"tag":48,"props":118,"children":120},{"className":119},[],[121],{"type":14,"value":122},"OOMKilled: true",{"type":14,"value":124}," is a complete diagnosis.",{"type":14,"value":126}," The process was ",{"type":9,"tag":48,"props":128,"children":130},{"className":129},[],[131],{"type":14,"value":132},"SIGKILL",{"type":14,"value":134},"ed by the kernel for exceeding its own limit — which is why the logs simply stop, with no stack trace and no shutdown message. Signal 9 cannot be caught.",{"data":136,"body":137},{},{"type":6,"children":138},[139],{"type":9,"tag":140,"props":141,"children":146},"quiz",{":answer":142,":options":143,"explanation":144,"question":145},"0","[\"The limit is per-cgroup, so free host memory is irrelevant once the container hits its own ceiling\",\"The host is misreporting free memory\",\"Docker reserves the rest of the host memory for itself\"]","That independence is the purpose of the limit. The kernel kills inside the cgroup and the rest of the host never notices. Either the limit is too low for the workload or the workload leaks — `docker stats` over time distinguishes them.","A container is OOMKilled at 512 MB while the host has 60 GB free. What is happening?",[],{"data":148,"body":149},{},{"type":6,"children":150},[151,157,185,205,210,219],{"type":9,"tag":34,"props":152,"children":154},{"id":153},"runtimes-do-not-see-the-limit",[155],{"type":14,"value":156},"Runtimes do not see the limit",{"type":9,"tag":10,"props":158,"children":159},{},[160,162,167,169,175,177,183],{"type":14,"value":161},"The trap that produces most unexplained OOM kills. A limited container still sees the ",{"type":9,"tag":68,"props":163,"children":164},{},[165],{"type":14,"value":166},"host's",{"type":14,"value":168}," total memory through ",{"type":9,"tag":48,"props":170,"children":172},{"className":171},[],[173],{"type":14,"value":174},"/proc/meminfo",{"type":14,"value":176},", because ",{"type":9,"tag":48,"props":178,"children":180},{"className":179},[],[181],{"type":14,"value":182},"/proc",{"type":14,"value":184}," is not namespaced for this. A runtime that sizes its heap from \"available memory\" sizes it against 60 GB and then dies at 512 MB.",{"type":9,"tag":10,"props":186,"children":187},{},[188,190,196,198,203],{"type":14,"value":189},"Modern runtimes are mostly container-aware now — the JVM since 10 with ",{"type":9,"tag":48,"props":191,"children":193},{"className":192},[],[194],{"type":14,"value":195},"UseContainerSupport",{"type":14,"value":197},", .NET Core, recent Node — but \"mostly\" is doing work in that sentence, and anything that reads ",{"type":9,"tag":48,"props":199,"children":201},{"className":200},[],[202],{"type":14,"value":174},{"type":14,"value":204}," directly is not.",{"type":9,"tag":10,"props":206,"children":207},{},[208],{"type":14,"value":209},"Be explicit rather than hopeful:",{"type":9,"tag":41,"props":211,"children":214},{"className":212,"code":213,"language":14},[44],"docker run --memory 512m -e NODE_OPTIONS=\"--max-old-space-size=400\" myapp\ndocker run --memory 2g   -e JAVA_TOOL_OPTIONS=\"-XX:MaxRAMPercentage=75\" myapp\n",[215],{"type":9,"tag":48,"props":216,"children":217},{"__ignoreMap":50},[218],{"type":14,"value":213},{"type":9,"tag":10,"props":220,"children":221},{},[222],{"type":14,"value":223},"Leave headroom. The heap is not the process — thread stacks, native allocations, and the runtime itself all sit outside it, and a heap sized at exactly the container limit is guaranteed to be killed.",{"data":225,"body":226},{},{"type":6,"children":227},[228],{"type":9,"tag":140,"props":229,"children":233},{":answer":142,":options":230,"explanation":231,"question":232},"[\"The heap is only part of the process — thread stacks, metaspace and native allocations count against the cgroup too\",\"The JVM ignores cgroup limits entirely\",\"OOM kills are triggered by CPU, not memory\"]","The cgroup counts every page the process touches. A 480 MB heap in a 512 MB container leaves nothing for metaspace, code cache, or stacks. `MaxRAMPercentage` around 75 is the usual starting point.","A JVM in a 512 MB container is OOMKilled despite a heap that looks small. Why?",[],{"data":235,"body":236},{},{"type":6,"children":237},[238,244,253,278,290],{"type":9,"tag":34,"props":239,"children":241},{"id":240},"cpu",[242],{"type":14,"value":243},"CPU",{"type":9,"tag":41,"props":245,"children":248},{"className":246,"code":247,"language":14},[44],"docker run --cpus 1.5 myapp                # hard: 1.5 cores' worth\ndocker run --cpu-shares 512 myapp          # relative weight under contention\ndocker run --cpuset-cpus 0-3 myapp         # pin to specific cores\n",[249],{"type":9,"tag":48,"props":250,"children":251},{"__ignoreMap":50},[252],{"type":14,"value":247},{"type":9,"tag":10,"props":254,"children":255},{},[256,265,267,276],{"type":9,"tag":68,"props":257,"children":258},{},[259],{"type":9,"tag":48,"props":260,"children":262},{"className":261},[],[263],{"type":14,"value":264},"--cpus",{"type":14,"value":266}," is a quota — a ceiling enforced whether or not the machine is busy. ",{"type":9,"tag":68,"props":268,"children":269},{},[270],{"type":9,"tag":48,"props":271,"children":273},{"className":272},[],[274],{"type":14,"value":275},"--cpu-shares",{"type":14,"value":277}," is a weight that only matters when cores are contended; an unloaded machine lets the container use everything.",{"type":9,"tag":10,"props":279,"children":280},{},[281,283,288],{"type":14,"value":282},"The distinction is worth caring about, because a hard CPU limit is more aggressive than it looks. A throttled process is ",{"type":9,"tag":68,"props":284,"children":285},{},[286],{"type":14,"value":287},"stopped mid-period and resumed at the start of the next",{"type":14,"value":289},", which shows up as latency spikes at the period boundary rather than uniform slowness. A great many \"mysterious p99\" investigations end at a CPU limit somebody set for safety on a service that was never going to starve anything.",{"type":9,"tag":10,"props":291,"children":292},{},[293],{"type":14,"value":294},"Weights first. Quotas when you genuinely need a ceiling — a noisy batch job, or a tenant you are billing.",{"data":296,"body":297},{},{"type":6,"children":298},[299,305,314,333],{"type":9,"tag":34,"props":300,"children":302},{"id":301},"processes-and-file-descriptors",[303],{"type":14,"value":304},"Processes and file descriptors",{"type":9,"tag":41,"props":306,"children":309},{"className":307,"code":308,"language":14},[44],"docker run --pids-limit 200 myapp\ndocker run --ulimit nofile=65536:65536 myapp\n",[310],{"type":9,"tag":48,"props":311,"children":312},{"__ignoreMap":50},[313],{"type":14,"value":308},{"type":9,"tag":10,"props":315,"children":316},{},[317,323,325,331],{"type":9,"tag":48,"props":318,"children":320},{"className":319},[],[321],{"type":14,"value":322},"--pids-limit",{"type":14,"value":324}," caps processes in the cgroup and is the answer to a fork bomb, accidental or otherwise. Without it, a runaway ",{"type":9,"tag":48,"props":326,"children":328},{"className":327},[],[329],{"type":14,"value":330},"fork()",{"type":14,"value":332}," exhausts the host's PID space and nothing on the machine can start a process — including your shell.",{"type":9,"tag":10,"props":334,"children":335},{},[336,342],{"type":9,"tag":48,"props":337,"children":339},{"className":338},[],[340],{"type":14,"value":341},"nofile",{"type":14,"value":343}," matters for anything holding many connections; Docker's default is often lower than a busy server needs, and the failure looks like random connection errors under load.",{"data":345,"body":346},{},{"type":6,"children":347},[348],{"type":9,"tag":349,"props":350,"children":355},"fill-blank",{":answer":351,"hint":352,"placeholder":353,"prompt":354},"[\"docker run --memory 512m --cpus 1.5 myapp\",\"docker run --cpus 1.5 --memory 512m myapp\",\"docker run -m 512m --cpus 1.5 myapp\",\"docker run --memory=512m --cpus=1.5 myapp\"]","Two flags before the image name.","docker run ...","Run `myapp` limited to 512 MB of memory and 1.5 CPUs.",[],{"data":357,"body":358},{},{"type":6,"children":359},[360,366,371,380,399],{"type":9,"tag":34,"props":361,"children":363},{"id":362},"setting-the-numbers",[364],{"type":14,"value":365},"Setting the numbers",{"type":9,"tag":10,"props":367,"children":368},{},[369],{"type":14,"value":370},"Do not guess. Run the workload under realistic load and watch:",{"type":9,"tag":41,"props":372,"children":375},{"className":373,"code":374,"language":14},[44],"docker stats\ndocker stats --no-stream --format '{{.Name}}\\t{{.MemUsage}}\\t{{.CPUPerc}}'\n",[376],{"type":9,"tag":48,"props":377,"children":378},{"__ignoreMap":50},[379],{"type":14,"value":374},{"type":9,"tag":10,"props":381,"children":382},{},[383,385,390,392,397],{"type":14,"value":384},"Then set the memory limit ",{"type":9,"tag":68,"props":386,"children":387},{},[388],{"type":14,"value":389},"above observed peak with real headroom",{"type":14,"value":391}," — for a garbage-collected runtime, peak is not steady state, and a limit at peak means the first unusual request kills you. Set CPU as a ",{"type":9,"tag":68,"props":393,"children":394},{},[395],{"type":14,"value":396},"weight",{"type":14,"value":398}," unless you specifically need a ceiling.",{"type":9,"tag":10,"props":400,"children":401},{},[402,404,410],{"type":14,"value":403},"And instrument the outcome, because a limit that is too low produces exactly the symptom of a limit that is absent: an application that stops working. ",{"type":9,"tag":48,"props":405,"children":407},{"className":406},[],[408],{"type":14,"value":409},"docker events --filter event=oom",{"type":14,"value":411}," tells you which it was.",{"data":413,"body":414},{},{"type":6,"children":415},[416,585],{"type":9,"tag":417,"props":418,"children":420},"deep-dive",{"title":419},"cgroup v2, `memory.high`, and reading the truth from the kernel",[421,426,443,452,461,489,543],{"type":9,"tag":10,"props":422,"children":423},{},[424],{"type":14,"value":425},"Docker's flags are a thin layer over cgroup files, and reading them directly is often faster than reasoning about the flags.",{"type":9,"tag":10,"props":427,"children":428},{},[429,434,436,441],{"type":9,"tag":68,"props":430,"children":431},{},[432],{"type":14,"value":433},"cgroup v2",{"type":14,"value":435}," is the unified hierarchy every current distribution ships. ",{"type":9,"tag":68,"props":437,"children":438},{},[439],{"type":14,"value":440},"cgroup v1 is deprecated",{"type":14,"value":442},", with support continuing until May 2029 — long enough not to panic, short enough that a new host should be v2.",{"type":9,"tag":41,"props":444,"children":447},{"className":445,"code":446,"language":14},[44],"stat -fc %T /sys/fs/cgroup     # cgroup2fs = v2, tmpfs = v1\ncat /sys/fs/cgroup/system.slice/docker-\u003Cid>.scope/memory.events\n",[448],{"type":9,"tag":48,"props":449,"children":450},{"__ignoreMap":50},[451],{"type":14,"value":446},{"type":9,"tag":41,"props":453,"children":456},{"className":454,"code":455,"language":14},[44],"low 0\nhigh 0\nmax 3428\noom 2\noom_kill 2\n",[457],{"type":9,"tag":48,"props":458,"children":459},{"__ignoreMap":50},[460],{"type":14,"value":455},{"type":9,"tag":10,"props":462,"children":463},{},[464,466,472,474,480,482,487],{"type":14,"value":465},"That file is the one to read when a container dies without explanation. ",{"type":9,"tag":48,"props":467,"children":469},{"className":468},[],[470],{"type":14,"value":471},"oom_kill",{"type":14,"value":473}," counts actual kills. A large ",{"type":9,"tag":48,"props":475,"children":477},{"className":476},[],[478],{"type":14,"value":479},"max",{"type":14,"value":481}," count with ",{"type":9,"tag":68,"props":483,"children":484},{},[485],{"type":14,"value":486},"zero",{"type":14,"value":488}," kills is the more interesting case: the container is repeatedly hitting its ceiling and reclaiming, which means it is thrashing page cache and running slowly rather than dying. No log line anywhere says so.",{"type":9,"tag":10,"props":490,"children":491},{},[492,503,505,511,513,518,520,526,528,533,535,541],{"type":9,"tag":68,"props":493,"children":494},{},[495,501],{"type":9,"tag":48,"props":496,"children":498},{"className":497},[],[499],{"type":14,"value":500},"memory.high",{"type":14,"value":502}," is the control Docker does not expose.",{"type":14,"value":504}," Where ",{"type":9,"tag":48,"props":506,"children":508},{"className":507},[],[509],{"type":14,"value":510},"memory.max",{"type":14,"value":512}," is a cliff, ",{"type":9,"tag":48,"props":514,"children":516},{"className":515},[],[517],{"type":14,"value":500},{"type":14,"value":519}," is a throttle — cross it and the process is put under heavy reclaim pressure and slowed, but not killed. Setting ",{"type":9,"tag":48,"props":521,"children":523},{"className":522},[],[524],{"type":14,"value":525},"high",{"type":14,"value":527}," somewhat below ",{"type":9,"tag":48,"props":529,"children":531},{"className":530},[],[532],{"type":14,"value":479},{"type":14,"value":534}," gives a warning zone that shows up in ",{"type":9,"tag":48,"props":536,"children":538},{"className":537},[],[539],{"type":14,"value":540},"memory.events",{"type":14,"value":542}," long before anything dies. Kubernetes does not expose it either. For a service where a slow response beats a restart, it is worth setting on the cgroup directly.",{"type":9,"tag":10,"props":544,"children":545},{},[546,548,557,559,564,566,575,577,583],{"type":14,"value":547},"Two more worth knowing by name: ",{"type":9,"tag":68,"props":549,"children":550},{},[551],{"type":9,"tag":48,"props":552,"children":554},{"className":553},[],[555],{"type":14,"value":556},"pids.max",{"type":14,"value":558}," is what ",{"type":9,"tag":48,"props":560,"children":562},{"className":561},[],[563],{"type":14,"value":322},{"type":14,"value":565}," writes, and ",{"type":9,"tag":68,"props":567,"children":568},{},[569],{"type":9,"tag":48,"props":570,"children":572},{"className":571},[],[573],{"type":14,"value":574},"io.max",{"type":14,"value":576}," rate-limits block device throughput per cgroup — the answer to one container saturating a disk everyone shares, and something Docker exposes only partially through ",{"type":9,"tag":48,"props":578,"children":580},{"className":579},[],[581],{"type":14,"value":582},"--device-read-bps",{"type":14,"value":584}," and friends.",{"type":9,"tag":10,"props":586,"children":587},{},[588],{"type":14,"value":589},"Next up: the supply chain — SBOMs, provenance, and proving where an image came from.",1787908868428]