[{"data":1,"prerenderedAt":585},["ShallowReactive",2],{"mdc--zbur9x-key":3,"mdc--mutjj0-key":21,"mdc--fw0yjq-key":63,"mdc--koqlde-key":159,"mdc--kw5812-key":171,"mdc--flziju-key":310,"mdc--b3sfb7-key":320,"mdc-mddll1-key":408,"mdc--aoxmh5-key":473,"mdc-immjpo-key":485},{"data":4,"body":5},{},{"type":6,"children":7},"root",[8,16],{"type":9,"tag":10,"props":11,"children":12},"element","p",{},[13],{"type":14,"value":15},"text","Running a model is not like running a web service. The artifact is gigabytes of weights rather than megabytes of code, the process wants a GPU it must be granted explicitly, and the thing your application talks to is an inference server with its own lifecycle.",{"type":9,"tag":10,"props":17,"children":18},{},[19],{"type":14,"value":20},"Docker's answer to all three arrived recently enough that most of it postdates the tutorials you will find, so this lesson is deliberately specific about what exists today.",{"data":22,"body":23},{},{"type":6,"children":24},[25,32,50],{"type":9,"tag":26,"props":27,"children":29},"h2",{"id":28},"docker-model-runner",[30],{"type":14,"value":31},"Docker Model Runner",{"type":9,"tag":10,"props":33,"children":34},{},[35,41,43,48],{"type":9,"tag":36,"props":37,"children":38},"strong",{},[39],{"type":14,"value":40},"Model Runner",{"type":14,"value":42}," runs models locally and serves them over an OpenAI-compatible API. Models are pulled from Docker Hub or Hugging Face as ",{"type":9,"tag":36,"props":44,"children":45},{},[46],{"type":14,"value":47},"OCI artifacts",{"type":14,"value":49}," — the same distribution mechanism as images, so the same registries, the same auth, the same caching.",{"type":9,"tag":51,"props":52,"children":56},"pre",{"className":53,"code":55,"language":14},[54],"language-text","docker model status\ndocker model pull ai/qwen2.5-coder\ndocker model list\ndocker model run ai/qwen2.5-coder \"explain this stack trace\"\ndocker model ps\n",[57],{"type":9,"tag":58,"props":59,"children":61},"code",{"__ignoreMap":60},"",[62],{"type":14,"value":55},{"data":64,"body":65},{},{"type":6,"children":66},[67,72,147],{"type":9,"tag":68,"props":69,"children":71},"terminal-teaser",{":lines":70},"[{\"cmd\":\"docker model pull ai/smollm2\",\"out\":\"Downloaded 1.79 GB\\nModel pulled successfully\"},{\"cmd\":\"docker model list\",\"out\":\"MODEL           PARAMETERS  QUANTIZATION  SIZE\\nai/smollm2      361.82 M    IQ2_XXS       1.79 GB\"},{\"cmd\":\"docker model ps\",\"out\":\"MODEL           BACKEND     MODE\\nai/smollm2      llama.cpp   completion\"}]",[],{"type":9,"tag":10,"props":73,"children":74},{},[75,77,83,85,91,93,99,101,107,109,115,116,122,123,129,131,137,139,145],{"type":14,"value":76},"The rest of the surface is what you would expect of something meant for production rather than demos: ",{"type":9,"tag":58,"props":78,"children":80},{"className":79},[],[81],{"type":14,"value":82},"docker model configure",{"type":14,"value":84}," sets context size and runtime flags, ",{"type":9,"tag":58,"props":86,"children":88},{"className":87},[],[89],{"type":14,"value":90},"docker model package",{"type":14,"value":92}," turns local GGUF or Safetensors files into an OCI artifact you can ",{"type":9,"tag":58,"props":94,"children":96},{"className":95},[],[97],{"type":14,"value":98},"push",{"type":14,"value":100},", ",{"type":9,"tag":58,"props":102,"children":104},{"className":103},[],[105],{"type":14,"value":106},"docker model bench",{"type":14,"value":108}," measures performance at different concurrency levels, and ",{"type":9,"tag":58,"props":110,"children":112},{"className":111},[],[113],{"type":14,"value":114},"docker model logs",{"type":14,"value":100},{"type":9,"tag":58,"props":117,"children":119},{"className":118},[],[120],{"type":14,"value":121},"df",{"type":14,"value":100},{"type":9,"tag":58,"props":124,"children":126},{"className":125},[],[127],{"type":14,"value":128},"unload",{"type":14,"value":130}," and ",{"type":9,"tag":58,"props":132,"children":134},{"className":133},[],[135],{"type":14,"value":136},"purge",{"type":14,"value":138}," handle the operational side. On Docker Engine without Desktop, ",{"type":9,"tag":58,"props":140,"children":142},{"className":141},[],[143],{"type":14,"value":144},"docker model install-runner",{"type":14,"value":146}," sets it up.",{"type":9,"tag":10,"props":148,"children":149},{},[150,152,157],{"type":14,"value":151},"Note that the runner is ",{"type":9,"tag":36,"props":153,"children":154},{},[155],{"type":14,"value":156},"disabled by default",{"type":14,"value":158}," in current Docker Desktop and has to be enabled explicitly.",{"data":160,"body":161},{},{"type":6,"children":162},[163],{"type":9,"tag":164,"props":165,"children":170},"quiz",{":answer":166,":options":167,"explanation":168,"question":169},"0","[\"Models reuse the registry, authentication, caching and mirroring you already run for images\",\"It makes the models smaller\",\"It lets the model run without a runtime\"]","No new infrastructure. A private registry, a pull-through cache, credentials, retention policy and air-gapped mirroring all work unchanged — which is the difference between a model being deployable and being a special case someone maintains by hand.","Why does distributing models as OCI artifacts matter operationally?",[],{"data":172,"body":173},{},{"type":6,"children":174},[175,181,186,197,224,289],{"type":9,"tag":26,"props":176,"children":178},{"id":177},"models-in-compose",[179],{"type":14,"value":180},"Models in Compose",{"type":9,"tag":10,"props":182,"children":183},{},[184],{"type":14,"value":185},"Compose can declare models as first-class dependencies, alongside services:",{"type":9,"tag":51,"props":187,"children":192},{"className":188,"code":190,"language":191,"meta":60},[189],"language-yaml","services:\n  chat:\n    build: .\n    models:\n      llm:\n        endpoint_var: AI_MODEL_URL\n        model_var: AI_MODEL_NAME\n    ports:\n      - \"127.0.0.1:8080:8080\"\n\nmodels:\n  llm:\n    model: ai/smollm2\n    context_size: 4096\n    runtime_flags:\n      - \"--temp=0.7\"\n","yaml",[193],{"type":9,"tag":58,"props":194,"children":195},{"__ignoreMap":60},[196],{"type":14,"value":190},{"type":9,"tag":10,"props":198,"children":199},{},[200,202,208,210,215,217,222],{"type":14,"value":201},"The top-level ",{"type":9,"tag":58,"props":203,"children":205},{"className":204},[],[206],{"type":14,"value":207},"models",{"type":14,"value":209}," block declares what the application needs. The service-level ",{"type":9,"tag":58,"props":211,"children":213},{"className":212},[],[214],{"type":14,"value":207},{"type":14,"value":216}," key binds one in and ",{"type":9,"tag":36,"props":218,"children":219},{},[220],{"type":14,"value":221},"injects environment variables",{"type":14,"value":223}," pointing at it.",{"type":9,"tag":10,"props":225,"children":226},{},[227,229,234,236,242,244,250,251,257,259,264,266,272,273,279,281,287],{"type":14,"value":228},"Two forms. The ",{"type":9,"tag":36,"props":230,"children":231},{},[232],{"type":14,"value":233},"short",{"type":14,"value":235}," form is a plain list of names, and Compose derives the variable names — ",{"type":9,"tag":58,"props":237,"children":239},{"className":238},[],[240],{"type":14,"value":241},"llm",{"type":14,"value":243}," becomes ",{"type":9,"tag":58,"props":245,"children":247},{"className":246},[],[248],{"type":14,"value":249},"LLM_URL",{"type":14,"value":130},{"type":9,"tag":58,"props":252,"children":254},{"className":253},[],[255],{"type":14,"value":256},"LLM_MODEL",{"type":14,"value":258},". The ",{"type":9,"tag":36,"props":260,"children":261},{},[262],{"type":14,"value":263},"long",{"type":14,"value":265}," form above names them explicitly with ",{"type":9,"tag":58,"props":267,"children":269},{"className":268},[],[270],{"type":14,"value":271},"endpoint_var",{"type":14,"value":130},{"type":9,"tag":58,"props":274,"children":276},{"className":275},[],[277],{"type":14,"value":278},"model_var",{"type":14,"value":280},", which is what you want when the application already expects ",{"type":9,"tag":58,"props":282,"children":284},{"className":283},[],[285],{"type":14,"value":286},"OPENAI_BASE_URL",{"type":14,"value":288}," or similar.",{"type":9,"tag":10,"props":290,"children":291},{},[292,294,300,302,308],{"type":14,"value":293},"Your application then talks to an OpenAI-compatible endpoint at whatever ",{"type":9,"tag":58,"props":295,"children":297},{"className":296},[],[298],{"type":14,"value":299},"AI_MODEL_URL",{"type":14,"value":301}," says, with no knowledge that a model runner exists. Swap the model in ",{"type":9,"tag":58,"props":303,"children":305},{"className":304},[],[306],{"type":14,"value":307},"compose.yaml",{"type":14,"value":309},", and nothing in the code changes.",{"data":311,"body":312},{},{"type":6,"children":313},[314],{"type":9,"tag":164,"props":315,"children":319},{":answer":166,":options":316,"explanation":317,"question":318},"[\"`LLM_URL` and `LLM_MODEL` — the endpoint to call and the model name to send\",\"The model weights, mounted as a volume\",\"Nothing; the service must discover the runner itself\"]","Compose provisions the model through the runner and passes the connection details as environment variables derived from the key name, uppercased. The long form lets you choose the variable names to match what the application already reads.","With `models: [llm]` in short form, what does Compose inject into the service?",[],{"data":321,"body":322},{},{"type":6,"children":323},[324,330,335,344,364,373],{"type":9,"tag":26,"props":325,"children":327},{"id":326},"gpus",[328],{"type":14,"value":329},"GPUs",{"type":9,"tag":10,"props":331,"children":332},{},[333],{"type":14,"value":334},"A container gets no GPU unless you give it one. On Linux this needs the NVIDIA Container Toolkit installed on the host.",{"type":9,"tag":51,"props":336,"children":339},{"className":337,"code":338,"language":14},[54],"docker run --gpus all nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi\ndocker run --gpus '\"device=0,1\"' myapp\ndocker run --gpus 1 myapp\n",[340],{"type":9,"tag":58,"props":341,"children":342},{"__ignoreMap":60},[343],{"type":14,"value":338},{"type":9,"tag":10,"props":345,"children":346},{},[347,349,354,356,362],{"type":14,"value":348},"In Compose, the modern ",{"type":9,"tag":58,"props":350,"children":352},{"className":351},[],[353],{"type":14,"value":326},{"type":14,"value":355}," key rather than the older ",{"type":9,"tag":58,"props":357,"children":359},{"className":358},[],[360],{"type":14,"value":361},"deploy.resources.reservations.devices",{"type":14,"value":363}," block:",{"type":9,"tag":51,"props":365,"children":368},{"className":366,"code":367,"language":191,"meta":60},[189],"services:\n  inference:\n    image: myorg/inference:1.0\n    gpus: all\n",[369],{"type":9,"tag":58,"props":370,"children":371},{"__ignoreMap":60},[372],{"type":14,"value":367},{"type":9,"tag":10,"props":374,"children":375},{},[376,378,383,385,390,392,398,400,406],{"type":14,"value":377},"Two things that consistently go wrong. ",{"type":9,"tag":36,"props":379,"children":380},{},[381],{"type":14,"value":382},"The CUDA version in the image must be compatible with the host driver",{"type":14,"value":384}," — the driver is not in the container, and a mismatch fails at runtime with an unhelpful message. And ",{"type":9,"tag":36,"props":386,"children":387},{},[388],{"type":14,"value":389},"GPU memory is not managed by cgroups",{"type":14,"value":391},": ",{"type":9,"tag":58,"props":393,"children":395},{"className":394},[],[396],{"type":14,"value":397},"--memory",{"type":14,"value":399}," does not cap VRAM, two containers on one GPU will happily exhaust it, and the failure is an out-of-memory error from CUDA rather than an OOM kill you can find in ",{"type":9,"tag":58,"props":401,"children":403},{"className":402},[],[404],{"type":14,"value":405},"docker events",{"type":14,"value":407},".",{"data":409,"body":410},{},{"type":6,"children":411},[412,418,437],{"type":9,"tag":26,"props":413,"children":415},{"id":414},"the-mcp-toolkit",[416],{"type":14,"value":417},"The MCP Toolkit",{"type":9,"tag":10,"props":419,"children":420},{},[421,423,428,430,435],{"type":14,"value":422},"Docker's ",{"type":9,"tag":36,"props":424,"children":425},{},[426],{"type":14,"value":427},"MCP Toolkit",{"type":14,"value":429}," packages Model Context Protocol servers — the tools an AI agent can call — as containers, with a catalog, an OAuth flow for the ones needing credentials, and ",{"type":9,"tag":36,"props":431,"children":432},{},[433],{"type":14,"value":434},"profiles",{"type":14,"value":436}," for grouping servers into named collections.",{"type":9,"tag":10,"props":438,"children":439},{},[440,442,448,449,455,457,463,465,471],{"type":14,"value":441},"The reason it is in a production course rather than a novelty section: an MCP server is arbitrary code that an agent invokes with your credentials attached. Everything in this course applies to it with more force than usual — non-root, ",{"type":9,"tag":58,"props":443,"children":445},{"className":444},[],[446],{"type":14,"value":447},"--cap-drop ALL",{"type":14,"value":100},{"type":9,"tag":58,"props":450,"children":452},{"className":451},[],[453],{"type":14,"value":454},"--read-only",{"type":14,"value":456},", a ",{"type":9,"tag":58,"props":458,"children":460},{"className":459},[],[461],{"type":14,"value":462},"--pids-limit",{"type":14,"value":464},", a memory cap, and no Docker socket. An MCP server with ",{"type":9,"tag":58,"props":466,"children":468},{"className":467},[],[469],{"type":14,"value":470},"/var/run/docker.sock",{"type":14,"value":472}," mounted has root on the host, and it is being driven by a model.",{"data":474,"body":475},{},{"type":6,"children":476},[477],{"type":9,"tag":478,"props":479,"children":484},"fill-blank",{":answer":480,"hint":481,"placeholder":482,"prompt":483},"[\"docker model pull ai/smollm2\",\"docker model pull ai/smollm2:latest\"]","Same verb as images, different noun.","docker model ...","Pull the model `ai/smollm2` with Docker Model Runner.",[],{"data":486,"body":487},{},{"type":6,"children":488},[489,559],{"type":9,"tag":490,"props":491,"children":493},"deep-dive",{"title":492},"What is different about operating a model",[494,499,509,527,537,554],{"type":9,"tag":10,"props":495,"children":496},{},[497],{"type":14,"value":498},"Every earlier lesson still applies, but four assumptions change enough to be worth naming.",{"type":9,"tag":10,"props":500,"children":501},{},[502,507],{"type":9,"tag":36,"props":503,"children":504},{},[505],{"type":14,"value":506},"Image size stops being the lever.",{"type":14,"value":508}," A 15 MB application image serving a 4 GB model has a 4 GB deploy. Keep the weights out of the image and pull them as artifacts, cached on the node — otherwise every application patch redistributes the model.",{"type":9,"tag":10,"props":510,"children":511},{},[512,517,519,525],{"type":9,"tag":36,"props":513,"children":514},{},[515],{"type":14,"value":516},"Startup is slow and the healthcheck must know it.",{"type":14,"value":518}," Loading weights into memory or VRAM takes tens of seconds. ",{"type":9,"tag":58,"props":520,"children":522},{"className":521},[],[523],{"type":14,"value":524},"start_period",{"type":14,"value":526}," needs to reflect that or the container is marked unhealthy while it is legitimately loading — the lesson from the healthcheck section, with a much bigger constant.",{"type":9,"tag":10,"props":528,"children":529},{},[530,535],{"type":9,"tag":36,"props":531,"children":532},{},[533],{"type":14,"value":534},"Memory limits interact badly with inference.",{"type":14,"value":536}," A model's resident set is largely the weights, so it is close to constant and close to the limit, all the time. There is no comfortable headroom, and a limit set at observed peak will be hit. Size against the model's known requirement, not against a measurement.",{"type":9,"tag":10,"props":538,"children":539},{},[540,545,547,552],{"type":9,"tag":36,"props":541,"children":542},{},[543],{"type":14,"value":544},"Concurrency is not free.",{"type":14,"value":546}," A web service handles a hundred concurrent requests on one core. An inference server serialises, and queueing shows up as latency rather than errors — which is why ",{"type":9,"tag":58,"props":548,"children":550},{"className":549},[],[551],{"type":14,"value":106},{"type":14,"value":553}," measures at different concurrency levels, and why the metric to alert on is queue depth rather than CPU.",{"type":9,"tag":10,"props":555,"children":556},{},[557],{"type":14,"value":558},"The general point is the one this course keeps returning to: the primitives do not change. It is still a process in namespaces with a cgroup, still an artifact in a registry, still an image you should sign and scan. What changes is which numbers are large.",{"type":9,"tag":10,"props":560,"children":561},{},[562,564,570,572,577,578,583],{"type":14,"value":563},"That is ",{"type":9,"tag":565,"props":566,"children":567},"em",{},[568],{"type":14,"value":569},"Docker in Production",{"type":14,"value":571},": the engine underneath, security, limits, supply chain, Scout, hardened bases, signing, observability, and AI workloads. Together with ",{"type":9,"tag":565,"props":573,"children":574},{},[575],{"type":14,"value":576},"Docker Fundamentals",{"type":14,"value":130},{"type":9,"tag":565,"props":579,"children":580},{},[581],{"type":14,"value":582},"Docker in Practice",{"type":14,"value":584},", that is the whole of Docker as it stands at the versions on this course's page — and when those numbers move, the notes on this site should move with them.",1787908868468]