[{"data":1,"prerenderedAt":400},["ShallowReactive",2],{"mdc-8afrv8-key":3},{"data":4,"body":5},{},{"type":6,"children":7},"root",[8,17,24,30,35,42,47,58,68,86,96,102,107,183,188,193,199,204,209,214,219,244,249,255,260,272,277,283,288,293,336,350,356,361,373,377],{"type":9,"tag":10,"props":11,"children":13},"element","h3",{"id":12},"agents-in-production",[14],{"type":15,"value":16},"text","Agents in production",{"type":9,"tag":18,"props":19,"children":21},"h1",{"id":20},"long-running-agents-belong-in-a-queue-not-a-request",[22],{"type":15,"value":23},"Long-running agents belong in a queue, not a request",{"type":9,"tag":25,"props":26,"children":27},"p",{},[28],{"type":15,"value":29},"The moment your feature needs more than one model call, it has outgrown the HTTP request. Put the work on a queue, stream progress back over a separate channel, and make every step individually retryable. This is not premature engineering — it is the cheapest point at which to make the change, and everything after it gets easier.",{"type":9,"tag":25,"props":31,"children":32},{},[33],{"type":15,"value":34},"The reason teams resist is that the request-response version works in development. One user, one call, eight seconds, fine. It keeps working right up until two things happen at once: real concurrency, and a chain long enough that some proxy in front of you gives up first.",{"type":9,"tag":36,"props":37,"children":39},"h2",{"id":38},"why-cant-the-work-just-live-in-the-request",[40],{"type":15,"value":41},"Why can't the work just live in the request?",{"type":9,"tag":25,"props":43,"children":44},{},[45],{"type":15,"value":46},"Four separate reasons, any one of which is sufficient.",{"type":9,"tag":25,"props":48,"children":49},{},[50,56],{"type":9,"tag":51,"props":52,"children":53},"strong",{},[54],{"type":15,"value":55},"Timeouts are not yours to set.",{"type":15,"value":57}," Your framework's timeout is one of several. There is a load balancer, possibly a CDN, possibly a corporate proxy, and a browser. A six-step generation that takes ninety seconds will be killed by whichever of those is least patient, and you will not always be told which.",{"type":9,"tag":25,"props":59,"children":60},{},[61,66],{"type":9,"tag":51,"props":62,"children":63},{},[64],{"type":15,"value":65},"A blocked worker is a wasted worker.",{"type":15,"value":67}," A synchronous LLM call holds a web worker for its entire duration, doing nothing but waiting on a socket. Ten concurrent generations on a ten-worker pool means the eleventh visitor cannot load your homepage. You have coupled the availability of your entire application to the latency of a third-party API.",{"type":9,"tag":25,"props":69,"children":70},{},[71,76,78,84],{"type":9,"tag":51,"props":72,"children":73},{},[74],{"type":15,"value":75},"Retries become all-or-nothing.",{"type":15,"value":77}," If step five of six fails inside a request, the honest options are to fail the whole thing or to redo steps one through four. Both are bad, and the second is bad ",{"type":9,"tag":79,"props":80,"children":81},"em",{},[82],{"type":15,"value":83},"and",{"type":15,"value":85}," expensive, because you pay for those tokens again.",{"type":9,"tag":25,"props":87,"children":88},{},[89,94],{"type":9,"tag":51,"props":90,"children":91},{},[92],{"type":15,"value":93},"Concurrency control has nowhere to live.",{"type":15,"value":95}," Your model provider has rate limits and a practical throughput ceiling. In a request-based design the only thing regulating how many calls you make at once is how many users happen to click at once, which is not a control system.",{"type":9,"tag":36,"props":97,"children":99},{"id":98},"what-does-the-queue-based-shape-look-like",[100],{"type":15,"value":101},"What does the queue-based shape look like?",{"type":9,"tag":25,"props":103,"children":104},{},[105],{"type":15,"value":106},"Three components, and the separation between them is the whole design:",{"type":9,"tag":108,"props":109,"children":110},"table",{},[111,130],{"type":9,"tag":112,"props":113,"children":114},"thead",{},[115],{"type":9,"tag":116,"props":117,"children":118},"tr",{},[119,125],{"type":9,"tag":120,"props":121,"children":122},"th",{},[123],{"type":15,"value":124},"Component",{"type":9,"tag":120,"props":126,"children":127},{},[128],{"type":15,"value":129},"Job",{"type":9,"tag":131,"props":132,"children":133},"tbody",{},[134,151,167],{"type":9,"tag":116,"props":135,"children":136},{},[137,146],{"type":9,"tag":138,"props":139,"children":140},"td",{},[141],{"type":9,"tag":51,"props":142,"children":143},{},[144],{"type":15,"value":145},"API",{"type":9,"tag":138,"props":147,"children":148},{},[149],{"type":15,"value":150},"Accepts the request, creates a session record, enqueues a task, returns immediately with a session id",{"type":9,"tag":116,"props":152,"children":153},{},[154,162],{"type":9,"tag":138,"props":155,"children":156},{},[157],{"type":9,"tag":51,"props":158,"children":159},{},[160],{"type":15,"value":161},"Worker",{"type":9,"tag":138,"props":163,"children":164},{},[165],{"type":15,"value":166},"Runs the actual graph — model calls, validation, retries — and publishes progress events",{"type":9,"tag":116,"props":168,"children":169},{},[170,178],{"type":9,"tag":138,"props":171,"children":172},{},[173],{"type":9,"tag":51,"props":174,"children":175},{},[176],{"type":15,"value":177},"Channel",{"type":9,"tag":138,"props":179,"children":180},{},[181],{"type":15,"value":182},"A WebSocket (or SSE) the client subscribes to with that session id, receiving partial output as it lands",{"type":9,"tag":25,"props":184,"children":185},{},[186],{"type":15,"value":187},"The client's flow is: ask, get an id, subscribe, watch. Nothing is blocked, nothing times out, and the user sees the work happening.",{"type":9,"tag":25,"props":189,"children":190},{},[191],{"type":15,"value":192},"This is the architecture I run. A studio client opens a socket, sends a prompt with a session id, and the server dispatches to a worker; progress streams back through a channel-layer group keyed on that session, so the browser shows units appearing one at a time rather than a spinner for ninety seconds. The HTTP API handles everything that is genuinely a request — CRUD, uploads, publishing. Generation never touches it.",{"type":9,"tag":36,"props":194,"children":196},{"id":195},"how-do-you-stop-it-from-melting-the-model-provider",[197],{"type":15,"value":198},"How do you stop it from melting the model provider?",{"type":9,"tag":25,"props":200,"children":201},{},[202],{"type":15,"value":203},"By sizing worker concurrency against the provider, not against the machine.",{"type":9,"tag":25,"props":205,"children":206},{},[207],{"type":15,"value":208},"This is the detail that most queue setups get wrong, because the default is wrong for this workload. Most task runners default their worker count to the number of CPU cores, which is the right heuristic for CPU-bound work and exactly the wrong one here. Your workers are not computing anything. They are waiting on a network call. The right number is a function of what your model endpoint will actually serve concurrently before it starts rate-limiting or degrading — and that number has nothing to do with your hardware.",{"type":9,"tag":25,"props":210,"children":211},{},[212],{"type":15,"value":213},"Make it a configuration value, not a derived one, so you can tune it when you change provider or tier without redeploying a different machine. In my setup the worker concurrency defaults to core count but is explicitly overridable by an environment variable for exactly this reason: the sizing question is \"how much fan-out will the LLM endpoint tolerate,\" and that answer changes independently of everything else.",{"type":9,"tag":25,"props":215,"children":216},{},[217],{"type":15,"value":218},"Two more settings matter, and both are non-obvious:",{"type":9,"tag":220,"props":221,"children":222},"ul",{},[223,234],{"type":9,"tag":224,"props":225,"children":226},"li",{},[227,232],{"type":9,"tag":51,"props":228,"children":229},{},[230],{"type":15,"value":231},"Prefetch of one.",{"type":15,"value":233}," By default many workers grab a batch of tasks to reduce broker round-trips. With tasks that run for two minutes, a worker that has grabbed four of them has created a two-minute queue behind itself while another worker sits idle. Set the prefetch multiplier to 1 so tasks are pulled one at a time and distribute evenly.",{"type":9,"tag":224,"props":235,"children":236},{},[237,242],{"type":9,"tag":51,"props":238,"children":239},{},[240],{"type":15,"value":241},"Acknowledge late.",{"type":15,"value":243}," Acknowledge a task after it completes, not when it is received. If a worker dies mid-generation — deploy, OOM, spot instance reclaimed — late acknowledgement means the task is redelivered instead of vanishing. Early acknowledgement means a user's generation silently never finishes and nothing anywhere records that it should have.",{"type":9,"tag":25,"props":245,"children":246},{},[247],{"type":15,"value":248},"These two together are the difference between a queue that degrades gracefully and one that loses work under exactly the conditions where losing work is most expensive.",{"type":9,"tag":36,"props":250,"children":252},{"id":251},"what-should-you-actually-stream-back",[253],{"type":15,"value":254},"What should you actually stream back?",{"type":9,"tag":25,"props":256,"children":257},{},[258],{"type":15,"value":259},"Partial results, not percentages. A progress bar that moves is a lie you have to maintain; a unit appearing in a list is the real thing arriving.",{"type":9,"tag":25,"props":261,"children":262},{},[263,265,270],{"type":15,"value":264},"This is a product decision disguised as an infrastructure one. Streaming genuine partial output changes what the wait feels like — the user is reading while the system is still working, so the perceived latency is the time to ",{"type":9,"tag":79,"props":266,"children":267},{},[268],{"type":15,"value":269},"first",{"type":15,"value":271}," useful output rather than the time to last. It also gives them a reason to intervene early when the direction is wrong, which is worth more than the waiting time you saved.",{"type":9,"tag":25,"props":273,"children":274},{},[275],{"type":15,"value":276},"It has a second effect that is easy to miss: streaming partial output forces your pipeline to have meaningful intermediate states. A design where nothing is showable until the end is usually a design where nothing is checkable until the end either. If you can stream it, you can validate it, and you can retry just that piece.",{"type":9,"tag":36,"props":278,"children":280},{"id":279},"what-does-this-buy-you-beyond-not-timing-out",[281],{"type":15,"value":282},"What does this buy you beyond not timing out?",{"type":9,"tag":25,"props":284,"children":285},{},[286],{"type":15,"value":287},"The ability to be careful, which is the real payoff.",{"type":9,"tag":25,"props":289,"children":290},{},[291],{"type":15,"value":292},"Once you are off the request path, latency stops being the constraint on how much work a generation is allowed to do. That changes what you can build:",{"type":9,"tag":220,"props":294,"children":295},{},[296,306,316,326],{"type":9,"tag":224,"props":297,"children":298},{},[299,304],{"type":9,"tag":51,"props":300,"children":301},{},[302],{"type":15,"value":303},"Validate and retry.",{"type":15,"value":305}," Output fails its schema? Retry that step with the validation error fed back in. In a request budget you cannot afford a second attempt. In a queue it costs you seconds you were never counting.",{"type":9,"tag":224,"props":307,"children":308},{},[309,314],{"type":9,"tag":51,"props":310,"children":311},{},[312],{"type":15,"value":313},"Check the work.",{"type":15,"value":315}," A second pass that reviews the first is affordable when nobody is holding a connection open.",{"type":9,"tag":224,"props":317,"children":318},{},[319,324],{"type":9,"tag":51,"props":320,"children":321},{},[322],{"type":15,"value":323},"Fan out.",{"type":15,"value":325}," Ten units of a course generate in parallel across workers rather than in sequence inside one request.",{"type":9,"tag":224,"props":327,"children":328},{},[329,334],{"type":9,"tag":51,"props":330,"children":331},{},[332],{"type":15,"value":333},"Resume.",{"type":15,"value":335}," A session that fails at step five restarts at step five, because the first four results were persisted as they landed.",{"type":9,"tag":25,"props":337,"children":338},{},[339,341,348],{"type":15,"value":340},"Every one of these is a reliability improvement that is simply unavailable in a synchronous design. That is the argument: not that queues are faster, but that they are the precondition for building something careful enough to trust. It is the same trade as ",{"type":9,"tag":342,"props":343,"children":345},"a",{"href":344},"/blog/narrow-agents-beat-smart-ones",[346],{"type":15,"value":347},"narrowing the task",{"type":15,"value":349}," — you give up an appealing simplicity and you get back a system whose failures are bounded.",{"type":9,"tag":36,"props":351,"children":353},{"id":352},"what-this-costs-you",[354],{"type":15,"value":355},"What this costs you",{"type":9,"tag":25,"props":357,"children":358},{},[359],{"type":15,"value":360},"Honesty about the trade: it is more moving parts. You now operate a broker, workers, and a channel layer. There is a session model to persist, a subscription to authenticate, and a set of failure states — worker died, task orphaned, client disconnected mid-stream — that did not exist before and all need handling.",{"type":9,"tag":25,"props":362,"children":363},{},[364,366,371],{"type":15,"value":365},"That is a real cost, and for a single model call behind a button it is not worth paying. The threshold is roughly: ",{"type":9,"tag":51,"props":367,"children":368},{},[369],{"type":15,"value":370},"more than one model call, or any single call that can exceed ten seconds.",{"type":15,"value":372}," Below that, keep it in the request. Above it, the queue is not overengineering, it is the version that survives having users.",{"type":9,"tag":374,"props":375,"children":376},"hr",{},[],{"type":9,"tag":25,"props":378,"children":379},{},[380],{"type":9,"tag":79,"props":381,"children":382},{},[383,385,390,392,398],{"type":15,"value":384},"Part 3 of ",{"type":9,"tag":342,"props":386,"children":388},{"href":387},"/blog?series=agents-in-production",[389],{"type":15,"value":16},{"type":15,"value":391},". Next: ",{"type":9,"tag":342,"props":393,"children":395},{"href":394},"/blog/an-agent-is-a-state-machine",[396],{"type":15,"value":397},"an agent is a state machine with an LLM picking the transitions",{"type":15,"value":399},".",1787908866326]