[{"data":1,"prerenderedAt":602},["ShallowReactive",2],{"mdc--usvfjz-key":3,"mdc--ucxj5g-key":41,"mdc-r4r52t-key":94,"mdc--nl8p5z-key":123,"mdc--kq2mwb-key":171,"mdc-kddbm4-key":183,"mdc-di935e-key":270,"mdc--2egvy-key":365,"mdc-fi06cw-key":395,"mdc--d7j47a-key":450,"mdc--il33ip-key":460,"mdc--p89eu8-key":585},{"data":4,"body":5},{},{"type":6,"children":7},"root",[8,31],{"type":9,"tag":10,"props":11,"children":12},"element","p",{},[13,16,22,24,29],{"type":14,"value":15},"text","Namespaces control what a process can ",{"type":9,"tag":17,"props":18,"children":19},"strong",{},[20],{"type":14,"value":21},"see",{"type":14,"value":23},". They say nothing about what it can ",{"type":9,"tag":17,"props":25,"children":26},{},[27],{"type":14,"value":28},"consume",{"type":14,"value":30}," — a process in every namespace we have built can still allocate all the memory on the machine and spin every core.",{"type":9,"tag":10,"props":32,"children":33},{},[34,39],{"type":9,"tag":17,"props":35,"children":36},{},[37],{"type":14,"value":38},"Control groups",{"type":14,"value":40}," are the other half. A cgroup is a set of processes with limits attached, and the kernel enforces the limits on the whole group.",{"data":42,"body":43},{},{"type":6,"children":44},[45,52,57,70,89],{"type":9,"tag":46,"props":47,"children":49},"h2",{"id":48},"which-version-are-you-on",[50],{"type":14,"value":51},"Which version are you on",{"type":9,"tag":10,"props":53,"children":54},{},[55],{"type":14,"value":56},"There are two generations, and the commands differ:",{"type":9,"tag":58,"props":59,"children":63},"pre",{"className":60,"code":62,"language":14},[61],"language-text","stat -fc %T /sys/fs/cgroup\n",[64],{"type":9,"tag":65,"props":66,"children":68},"code",{"__ignoreMap":67},"",[69],{"type":14,"value":62},{"type":9,"tag":10,"props":71,"children":72},{},[73,79,81,87],{"type":9,"tag":65,"props":74,"children":76},{"className":75},[],[77],{"type":14,"value":78},"cgroup2fs",{"type":14,"value":80}," means v2 — the unified hierarchy, and what every current distribution ships. ",{"type":9,"tag":65,"props":82,"children":84},{"className":83},[],[85],{"type":14,"value":86},"tmpfs",{"type":14,"value":88}," means v1, with a separate tree per controller. Docker on a modern host uses v2; a 2019-era CentOS box uses v1.",{"type":9,"tag":10,"props":90,"children":91},{},[92],{"type":14,"value":93},"The concepts are identical. The filenames are not, which is the whole reason a container tool has version-detection code in it.",{"data":95,"body":96},{},{"type":6,"children":97},[98,104,109,118],{"type":9,"tag":46,"props":99,"children":101},{"id":100},"the-filesystem-is-the-api",[102],{"type":14,"value":103},"The filesystem is the API",{"type":9,"tag":10,"props":105,"children":106},{},[107],{"type":14,"value":108},"There is no cgroup syscall. You create a group by making a directory, and set a limit by writing to a file:",{"type":9,"tag":58,"props":110,"children":113},{"className":111,"code":112,"language":14},[61],"sudo mkdir /sys/fs/cgroup/my-container\n",[114],{"type":9,"tag":65,"props":115,"children":116},{"__ignoreMap":67},[117],{"type":14,"value":112},{"type":9,"tag":10,"props":119,"children":120},{},[121],{"type":14,"value":122},"The kernel populates it immediately with the controls:",{"data":124,"body":125},{},{"type":6,"children":126},[127,132,143,152,157,166],{"type":9,"tag":128,"props":129,"children":131},"terminal-teaser",{":lines":130},"[{\"cmd\":\"sudo mkdir /sys/fs/cgroup/my-container\",\"out\":\"\"},{\"cmd\":\"ls /sys/fs/cgroup/my-container\",\"out\":\"cgroup.procs  cpu.max  cpu.weight  memory.max  memory.high\\nmemory.current  memory.events  pids.max  io.max\"},{\"cmd\":\"cat /sys/fs/cgroup/my-container/memory.max\",\"out\":\"max\"}]",[],{"type":9,"tag":10,"props":133,"children":134},{},[135,141],{"type":9,"tag":65,"props":136,"children":138},{"className":137},[],[139],{"type":14,"value":140},"max",{"type":14,"value":142}," means no limit. Change it by writing a number:",{"type":9,"tag":58,"props":144,"children":147},{"className":145,"code":146,"language":14},[61],"# 512 MB hard cap\necho 536870912 | sudo tee /sys/fs/cgroup/my-container/memory.max\n\n# 50% of one CPU: 50000 microseconds of runtime per 100000 microsecond period\necho \"50000 100000\" | sudo tee /sys/fs/cgroup/my-container/cpu.max\n",[148],{"type":9,"tag":65,"props":149,"children":150},{"__ignoreMap":67},[151],{"type":14,"value":146},{"type":9,"tag":10,"props":153,"children":154},{},[155],{"type":14,"value":156},"And you put a process into the group by writing its PID:",{"type":9,"tag":58,"props":158,"children":161},{"className":159,"code":160,"language":14},[61],"echo $$ | sudo tee /sys/fs/cgroup/my-container/cgroup.procs\n",[162],{"type":9,"tag":65,"props":163,"children":164},{"__ignoreMap":67},[165],{"type":14,"value":160},{"type":9,"tag":10,"props":167,"children":168},{},[169],{"type":14,"value":170},"That is the entire interface. Children inherit membership, so putting a shell in the group puts everything it launches in the group too.",{"data":172,"body":173},{},{"type":6,"children":174},[175],{"type":9,"tag":176,"props":177,"children":182},"quiz",{":answer":178,":options":179,"explanation":180,"question":181},"0","[\"The group may use 50000 microseconds of CPU time in every 100000 microsecond period — half of one core\",\"The group is capped at 50000 processes\",\"The group gets 50000 shares out of 100000 when the CPU is contended\"]","It is quota and period, in microseconds — a hard ceiling enforced whether or not the machine is busy. `cpu.weight` is the other kind: a *relative* share that only matters under contention, and lets a group use the whole machine when nothing else wants it.","What does `cpu.max` set to `50000 100000` mean?",[],{"data":184,"body":185},{},{"type":6,"children":186},[187,193,198,258],{"type":9,"tag":46,"props":188,"children":190},{"id":189},"quota-versus-weight",[191],{"type":14,"value":192},"Quota versus weight",{"type":9,"tag":10,"props":194,"children":195},{},[196],{"type":14,"value":197},"The distinction matters more in practice than the syntax does.",{"type":9,"tag":199,"props":200,"children":201},"ul",{},[202,232],{"type":9,"tag":203,"props":204,"children":205},"li",{},[206,215,217,222,224,230],{"type":9,"tag":17,"props":207,"children":208},{},[209],{"type":9,"tag":65,"props":210,"children":212},{"className":211},[],[213],{"type":14,"value":214},"cpu.max",{"type":14,"value":216}," is a ",{"type":9,"tag":17,"props":218,"children":219},{},[220],{"type":14,"value":221},"hard ceiling",{"type":14,"value":223},". The group is throttled at its quota even on a completely idle machine. This is Kubernetes' CPU ",{"type":9,"tag":225,"props":226,"children":227},"em",{},[228],{"type":14,"value":229},"limit",{"type":14,"value":231},".",{"type":9,"tag":203,"props":233,"children":234},{},[235,244,245,250,252,257],{"type":9,"tag":17,"props":236,"children":237},{},[238],{"type":9,"tag":65,"props":239,"children":241},{"className":240},[],[242],{"type":14,"value":243},"cpu.weight",{"type":14,"value":216},{"type":9,"tag":17,"props":246,"children":247},{},[248],{"type":14,"value":249},"relative share",{"type":14,"value":251},", used only when the CPU is contended. A group with weight 50 competing against one with weight 100 gets a third of the time — but on an idle machine it can use everything. This is Kubernetes' CPU ",{"type":9,"tag":225,"props":253,"children":254},{},[255],{"type":14,"value":256},"request",{"type":14,"value":231},{"type":9,"tag":10,"props":259,"children":260},{},[261,263,268],{"type":14,"value":262},"Setting a hard CPU limit is a much more aggressive thing to do than it looks. A process throttled by ",{"type":9,"tag":65,"props":264,"children":266},{"className":265},[],[267],{"type":14,"value":214},{"type":14,"value":269}," is stopped mid-period and resumed at the start of the next one, which shows up as latency spikes at the period boundary rather than as uniform slowness. Plenty of production CPU-throttling incidents are a limit set \"for safety\" on a service that was never going to starve anything.",{"data":271,"body":272},{},{"type":6,"children":273},[274,286,306,315,328,337],{"type":9,"tag":46,"props":275,"children":277},{"id":276},"cgroup-tools-the-convenience-layer",[278,284],{"type":9,"tag":65,"props":279,"children":281},{"className":280},[],[282],{"type":14,"value":283},"cgroup-tools",{"type":14,"value":285},", the convenience layer",{"type":9,"tag":10,"props":287,"children":288},{},[289,291,297,299,304],{"type":14,"value":290},"Writing to ",{"type":9,"tag":65,"props":292,"children":294},{"className":293},[],[295],{"type":14,"value":296},"/sys/fs/cgroup",{"type":14,"value":298}," by hand is fine, but the walkthrough this course follows uses ",{"type":9,"tag":65,"props":300,"children":302},{"className":301},[],[303],{"type":14,"value":283},{"type":14,"value":305},", which wraps it:",{"type":9,"tag":58,"props":307,"children":310},{"className":308,"code":309,"language":14},[61],"sudo apt update && sudo apt install -y cgroup-tools\n\nsudo cgcreate -g cpu,memory:/$CONTAINER_ID\nsudo cgset -r cpu.weight=50 $CONTAINER_ID          # relative CPU share\nsudo cgset -r memory.max=536870912 $CONTAINER_ID   # hard cap at 512 MB\n",[311],{"type":9,"tag":65,"props":312,"children":313},{"__ignoreMap":67},[314],{"type":14,"value":309},{"type":9,"tag":10,"props":316,"children":317},{},[318,320,326],{"type":14,"value":319},"Then ",{"type":9,"tag":65,"props":321,"children":323},{"className":322},[],[324],{"type":14,"value":325},"cgexec",{"type":14,"value":327}," launches a process directly into the group:",{"type":9,"tag":58,"props":329,"children":332},{"className":330,"code":331,"language":14},[61],"sudo cgexec -g cpu,memory:$CONTAINER_ID /bin/sh\n",[333],{"type":9,"tag":65,"props":334,"children":335},{"__ignoreMap":67},[336],{"type":14,"value":331},{"type":9,"tag":10,"props":338,"children":339},{},[340,342,348,350,356,358,364],{"type":14,"value":341},"Same mechanism, one command instead of a ",{"type":9,"tag":65,"props":343,"children":345},{"className":344},[],[346],{"type":14,"value":347},"mkdir",{"type":14,"value":349},", two ",{"type":9,"tag":65,"props":351,"children":353},{"className":352},[],[354],{"type":14,"value":355},"echo",{"type":14,"value":357},"s, and a ",{"type":9,"tag":65,"props":359,"children":361},{"className":360},[],[362],{"type":14,"value":363},"tee",{"type":14,"value":231},{"data":366,"body":367},{},{"type":6,"children":368},[369,375],{"type":9,"tag":46,"props":370,"children":372},{"id":371},"what-happens-at-the-memory-limit",[373],{"type":14,"value":374},"What happens at the memory limit",{"type":9,"tag":10,"props":376,"children":377},{},[378,380,386,388,393],{"type":14,"value":379},"Nothing gentle. A process that allocates past ",{"type":9,"tag":65,"props":381,"children":383},{"className":382},[],[384],{"type":14,"value":385},"memory.max",{"type":14,"value":387}," gets an allocation failure, and the kernel's OOM killer chooses a victim ",{"type":9,"tag":17,"props":389,"children":390},{},[391],{"type":14,"value":392},"within the cgroup",{"type":14,"value":394}," — not on the whole machine, which is exactly the point.",{"data":396,"body":397},{},{"type":6,"children":398},[399,403,429],{"type":9,"tag":128,"props":400,"children":402},{":lines":401},"[{\"cmd\":\"sudo cgexec -g memory:my-container sh -c 'head -c 900M /dev/zero | tail -c 1'\",\"out\":\"Killed\"},{\"cmd\":\"cat /sys/fs/cgroup/my-container/memory.events\",\"out\":\"low 0\\nhigh 0\\nmax 214\\noom 1\\noom_kill 1\"}]",[],{"type":9,"tag":10,"props":404,"children":405},{},[406,412,414,420,422,427],{"type":9,"tag":65,"props":407,"children":409},{"className":408},[],[410],{"type":14,"value":411},"memory.events",{"type":14,"value":413}," is the file to read when a container dies without explanation. ",{"type":9,"tag":65,"props":415,"children":417},{"className":416},[],[418],{"type":14,"value":419},"oom_kill 1",{"type":14,"value":421}," says the kernel killed something for exceeding the limit; a high ",{"type":9,"tag":65,"props":423,"children":425},{"className":424},[],[426],{"type":14,"value":140},{"type":14,"value":428}," count with no kills means the group is repeatedly hitting the ceiling and reclaiming, which is a container that needs more memory and is currently paying for it in page-cache thrashing.",{"type":9,"tag":10,"props":430,"children":431},{},[432,434,440,442,448],{"type":14,"value":433},"This is ",{"type":9,"tag":65,"props":435,"children":437},{"className":436},[],[438],{"type":14,"value":439},"OOMKilled",{"type":14,"value":441}," in ",{"type":9,"tag":65,"props":443,"children":445},{"className":444},[],[446],{"type":14,"value":447},"kubectl describe pod",{"type":14,"value":449},", and exit code 137 — 128 plus signal 9.",{"data":451,"body":452},{},{"type":6,"children":453},[454],{"type":9,"tag":176,"props":455,"children":459},{":answer":178,":options":456,"explanation":457,"question":458},"[\"The limit is per-cgroup — free host memory is irrelevant to a group that has hit its own ceiling\",\"The host is misreporting available memory\",\"The container is leaking memory faster than the host can allocate\"]","The whole purpose of the limit is to make the group's ceiling independent of the machine's. The kernel kills inside the cgroup and the rest of the host never notices. Either the limit is too low or the workload genuinely leaks — and `memory.events` plus `memory.current` over time will tell you which.","A container is repeatedly OOMKilled at 512 MB while the host has 60 GB free. What is going on?",[],{"data":461,"body":462},{},{"type":6,"children":463},[464],{"type":9,"tag":465,"props":466,"children":468},"deep-dive",{"title":467},"`memory.high` — the throttle before the cliff",[469,474,511,545,550],{"type":9,"tag":10,"props":470,"children":471},{},[472],{"type":14,"value":473},"cgroup v2 adds a second memory control that has no v1 equivalent and is under-used.",{"type":9,"tag":199,"props":475,"children":476},{},[477,490],{"type":9,"tag":203,"props":478,"children":479},{},[480,488],{"type":9,"tag":17,"props":481,"children":482},{},[483],{"type":9,"tag":65,"props":484,"children":486},{"className":485},[],[487],{"type":14,"value":385},{"type":14,"value":489}," is the hard limit. Cross it and something gets killed.",{"type":9,"tag":203,"props":491,"children":492},{},[493,502,504,509],{"type":9,"tag":17,"props":494,"children":495},{},[496],{"type":9,"tag":65,"props":497,"children":499},{"className":498},[],[500],{"type":14,"value":501},"memory.high",{"type":14,"value":503}," is a throttle. Cross it and the process is put under heavy reclaim pressure and ",{"type":9,"tag":225,"props":505,"children":506},{},[507],{"type":14,"value":508},"slowed down",{"type":14,"value":510},", but not killed.",{"type":9,"tag":10,"props":512,"children":513},{},[514,516,522,524,529,531,536,538,543],{"type":14,"value":515},"Setting ",{"type":9,"tag":65,"props":517,"children":519},{"className":518},[],[520],{"type":14,"value":521},"high",{"type":14,"value":523}," somewhat below ",{"type":9,"tag":65,"props":525,"children":527},{"className":526},[],[528],{"type":14,"value":140},{"type":14,"value":530}," gives you a warning zone: a workload drifting upward gets progressively slower and shows up in ",{"type":9,"tag":65,"props":532,"children":534},{"className":533},[],[535],{"type":14,"value":411},{"type":14,"value":537}," as a rising ",{"type":9,"tag":65,"props":539,"children":541},{"className":540},[],[542],{"type":14,"value":521},{"type":14,"value":544}," count, long before anything is killed. For a service where a slow response beats a restart, that is a much better failure mode than a cliff.",{"type":9,"tag":10,"props":546,"children":547},{},[548],{"type":14,"value":549},"Kubernetes exposes only the hard limit, which is one reason a memory limit there is such a blunt instrument.",{"type":9,"tag":10,"props":551,"children":552},{},[553,555,564,566,572,574,583],{"type":14,"value":554},"Two other controllers are worth knowing by name. ",{"type":9,"tag":17,"props":556,"children":557},{},[558],{"type":9,"tag":65,"props":559,"children":561},{"className":560},[],[562],{"type":14,"value":563},"pids.max",{"type":14,"value":565}," caps the number of processes in the group and is the answer to a fork bomb — Docker's ",{"type":9,"tag":65,"props":567,"children":569},{"className":568},[],[570],{"type":14,"value":571},"--pids-limit",{"type":14,"value":573},". ",{"type":9,"tag":17,"props":575,"children":576},{},[577],{"type":9,"tag":65,"props":578,"children":580},{"className":579},[],[581],{"type":14,"value":582},"io.max",{"type":14,"value":584}," rate-limits block device throughput per cgroup, which is what stops one noisy container from saturating a disk everyone shares.",{"data":586,"body":587},{},{"type":6,"children":588},[589,597],{"type":9,"tag":590,"props":591,"children":596},"fill-blank",{":answer":592,"hint":593,"placeholder":594,"prompt":595},"[\"echo 536870912 | sudo tee /sys/fs/cgroup/my-container/memory.max\",\"echo 536870912 > /sys/fs/cgroup/my-container/memory.max\",\"sudo sh -c \\\"echo 536870912 > /sys/fs/cgroup/my-container/memory.max\\\"\"]","Write the byte count into the group's `memory.max` file.","echo 536870912 ...","Cap the cgroup `my-container` at 512 MB of memory by writing directly to the cgroup v2 filesystem. (512 MB is 536870912 bytes.)",[],{"type":9,"tag":10,"props":598,"children":599},{},[600],{"type":14,"value":601},"Next up: assembling everything — one command, four cages, and a shell that believes it owns the machine.",1787908868385]