Agents in production
Everything between a working demo and a system you can leave running.
Software
EngineerWriting
Notes from building generative-AI systems that have to survive real users — agents, evaluation, and the architecture underneath.
14 posts · 3 series · RSS
Latest
Context is the hard part, and nobody sets out to build it
Why the same AI feature works at one company and fails at another: canonical definitions, current signals, and quality enforcement — the layers nobody budgets for.
7 min read
Read by series
Making LLM output trustworthy
Evaluation, constraints and observability for systems that are allowed to be wrong.
The architecture and cost of LLM features
Where the model belongs in your system, and what it costs when it is there.
Everything, newest first
14 posts
The architecture and cost of LLM features · Part 4
Context is the hard part, and nobody sets out to build it
Why the same AI feature works at one company and fails at another: canonical definitions, current signals, and quality enforcement — the layers nobody budgets for.
7 min read
The architecture and cost of LLM features · Part 3
The most expensive call is the one that fails after trying hardest
In a step-billed system your costliest interactions are the ones the model works hardest on and still can't finish. The effort paradox, and the four controls that bound it.
6 min read
The architecture and cost of LLM features · Part 2
What an LLM feature actually costs to run
Cost is steps times tokens times retries, the failures dominate, and the pilot will not show you. The four buckets to model and the levers that actually move the bill.
6 min read
The architecture and cost of LLM features · Part 1
Where AI belongs in your architecture — and where it doesn't
Models at the edges, deterministic code in the core. The patterns that don't need a model, the ones where it earns its keep, and the question to ask before adding it.
6 min read
Making LLM output trustworthy · Part 4
You can't debug what you can't see: observability for LLM features
What to record on every model call, why the rendered prompt is the field that matters most, and why instrumentation has to exist before the first incident.
6 min read
Making LLM output trustworthy · Part 3
Test one prompt a hundred times, not a hundred prompts once
Single-run prompt testing hides variance, tail latency, token spread and intermittent misbehaviour. How to build the batch harness that surfaces them.
6 min read
Making LLM output trustworthy · Part 2
Constrain the model, don't review its output
Inconsistent generated output is a constraint problem, not a capability problem. Schemas and closed vocabularies beat review queues, and here is why.
6 min read
Making LLM output trustworthy · Part 1
How to evaluate an LLM feature before you ship it
Why normal testing doesn't work on a probability distribution, what goes in an adversarial case set, and how to score it without fooling yourself.
7 min read
Agents in production · Part 4
An agent is a state machine with an LLM picking the transitions
Orchestration is not reasoning. Which decisions to encode in code, which to leave to the model, and what an explicit generation graph buys you.
6 min read
Agents in production · Part 3
Long-running agents belong in a queue, not a request
Why multi-step generation cannot live in an HTTP request, what the queue-plus-stream architecture looks like, and the worker settings that decide whether it degrades gracefully.
7 min read
Agents in production · Part 2
Your agent doesn't need to be smart. It needs to be narrow.
A boring agent that produces a useful artifact every time beats a clever one that mostly does. How to scope a task narrowly enough to be validated, tested and trusted.
7 min read
Agents in production · Part 1
What it actually takes to put an LLM agent in production
The checklist that separates a working demo from a system you can leave running: scope, control flow, queues, evaluation, constraints, cost and visibility.
9 min read
Driving HPC Forward
Modernizing OnPrem Infrastructure
Bringing cloud-native tools into your on-prem infrastructure for automation and resiliency.
5 min read
Driving HPC Forward
Automating HPC Infrastructure
Using Apache CloudStack to manage cloud resources in your on-prem infrastructure.
8 min read