Seven phases for shipping production AI agents.
Most production AI agent failures share a root cause. The team built the agent first and the eval framework second. By the time something breaks in production, “the agent did it” is the only answer leadership gets. The eval framework was an afterthought, not a contract.
The corrected sequence is grounded theory applied to AI systems. Ship a v0. Read 100 traces. Code the failures in plain language. Build the eval framework from observed failure modes. Binary per axis. Theoretical saturation as the stopping criterion. Domain expert calibration on the judgment standard.
We run this methodology across the agent fleet at Magnetiz, wrapped in a seven-phase lifecycle that takes an idea from triage to autonomous production. The lifecycle is how the methodology ships repeatably across deployments.
Here is the lifecycle.
What Is Phase 0 in the Agent Build Lifecycle?
Phase 0. Triage. The go/no-go before any build.
Not every problem needs an agent. Phase 0 asks three questions. Is the work encodable. Is the business outcome measurable. Is the failure mode contained.
Most teams skip triage and end up building agents for problems that should have stayed human. Phase 0 introduces explicit kill criteria. If the problem fails any of the three tests, the project does not start. Vanity automation gets cut here.
Watch for agents proposed against problems that are judgment-heavy with messy inputs and no clear outcome. Those problems get sharper operators, not agents.
What Does Phase 1 Look Like in an AI Agent Build?
Phase 1. Discovery and scoping.
The business problem gets defined before the technical solution. What metric moves. Who owns the outcome. What the existing manual workflow looks like. What success looks like at the 90-day mark.
The scoping output is a short brief that names the metric, the owner, the workflow, and the failure modes the team can already anticipate. The brief is not the eval framework. Anticipated failure modes are speculation until they show up in real traces.
Why Ship a v0 Before Writing the Eval Framework?
Phase 2. Instrumented v0 ship.
This is the phase most teams sequence wrong. The eval framework cannot be written before the agent exists because there is nothing to evaluate. Failure modes are speculation until traces exist.
Phase 2 ships a v0 that is instrumented from the first run. Telemetry on every output. Cost tracking. Full input and output logging. The v0 does not need to be polished. It needs to be observable. The traces are the product of this phase, not the agent itself.
Watch for the temptation to write a beautiful eval rubric upfront. It produces evals that grade what the team thinks matters, not what actually breaks.
How Do You Build AI Evals from Real Traces Instead of Assumptions?
Phase 3. Open coding into a taxonomy.
Most teams building AI evals have never read 100 of their own traces. That gap is the entire reason Phase 3 exists.
Phase 3 is reading every output the v0 produced and labeling the failures in plain language. No predefined categories. No structured rubric. Just open coding, a methodology term from grounded theory that means capturing what is actually there before imposing structure.
Once 100 traces are coded, the second pass clusters the labels into a taxonomy. Axial coding. Categories emerge from the data rather than being imposed before the data existed. The taxonomy is what the eval framework will target.
What Is the Right Scoring Framework for AI Agent Evals?
Phase 4. Binary scoring per axis.
Each category in the taxonomy becomes an axis. Each axis is binary. No 1-to-10 scales. Scalar scoring introduces noise across raters and turns evals into debates about the scale rather than the agent.
Word count meets the constraint or it does not. Schema is valid or it is not. Factual grounding holds or it breaks. Refusal behavior fires or it does not. When the rubric is binary, the conversation moves from “how much is this worth” to “what failed, specifically.”
Rule checks handle the deterministic axes. LLM-as-judge handles the subjective ones. Both binary. Both auditable.
When Does an AI Agent Reach Theoretical Saturation?
Phase 5. Iterate to saturation.
Saturation is the stopping criterion borrowed from grounded theory. The agent is iterated until new traces stop surfacing new failure modes. The eval taxonomy stops growing because the data stops producing new categories.
Most teams under-invest here. They ship after one round of fixes and call it production-ready. Saturation requires patience and a willingness to iterate on the same agent across dozens or hundreds of runs. The payoff is that the agent does not surprise the team in production.
How Do Domain Experts Calibrate an LLM-as-Judge?
Phase 6. Domain expert calibration.
The LLM-as-judge has its own failure modes. It will pass outputs a human would catch and fail outputs a human would approve. The fix is the benevolent dictator model. One domain expert reads twenty to thirty outputs and labels them. The judge scores the same outputs. Where they disagree, the human is right and the rubric gets tuned.
The output is a calibrated judge that approximates the domain expert’s standard. After calibration, the agent can scale without the expert reading every output.
When Does an AI Agent Earn Autonomous Status in Production?
Phase 7. Autonomous status with on-call.
The agent runs autonomously after five consecutive runs pass every axis of the binary eval framework. Telemetry continues. Aggregate failure patterns surface before they become production incidents.
Autonomy is not unconditional. Every agent has an on-call protocol. When something breaks, the failure is specific because the rubric is binary. Debug starts at the line that broke.
Watch for teams that ship to autonomous status and stop watching. The eval framework gates ongoing autonomy, not just initial deployment.
The Pattern Across All Seven Phases
Each phase has a different kind of work. Phase 0 separates problems worth building agents for from problems that should stay human. Phase 1 defines the business outcome. Phase 2 produces traces. Phase 3 codes the failures. Phase 4 sets binary contracts. Phase 5 iterates to saturation. Phase 6 calibrates against a domain expert. Phase 7 ships with continuous gating.
Most teams that fail in production failed at the sequencing. They wrote the eval before the v0 and graded outputs against what they imagined would break.
Production agents are not graded against assumptions. They are graded against observed failures.
FAQ
What is the right order to build an AI agent in production?
Ship an instrumented v0 first, then read 100 traces, then build the eval framework from the observed failure modes. The eval framework cannot be written before the agent exists because there is nothing to evaluate. The corrected order is Phase 0 (triage), Phase 1 (scoping), Phase 2 (v0 ship), Phase 3 (open coding 100 traces), Phase 4 (binary evals from the taxonomy), Phase 5 (iterate to saturation), Phase 6 (domain expert calibration), Phase 7 (autonomous status with on-call).
Why is binary scoring better than Likert scales for AI evals?
Scalar scoring introduces noise across raters. A 7-out-of-10 means different things to different reviewers. The eval becomes a debate about the scale rather than a debate about what failed. Binary scoring per axis collapses the disagreement to a single decision. Did this output meet the constraint or not. Inter-rater reliability is higher on binary than on Likert across subjective qualities.
What is open coding in the context of AI agent evaluation?
Open coding is a methodology term from grounded theory. It means reading every output and labeling the failures in plain language, without predefined categories. The labels are then clustered into a taxonomy in a second pass called axial coding. The taxonomy is what the eval framework targets. The approach grounds the eval in observed failures rather than anticipated ones.
What is theoretical saturation for AI agent evals?
Theoretical saturation is the point at which new traces stop surfacing new failure modes. The eval taxonomy stops growing because the data stops producing new categories. Saturation is the stopping criterion for the iterate-and-fix loop in Phase 5. It signals that the eval framework is comprehensive enough to support autonomous production.
Why is domain expert calibration necessary for an LLM-as-judge?
LLM-as-judge has its own failure modes. It will pass outputs a human would catch and fail outputs a human would approve. Calibration brings the judge into alignment with a designated domain expert by having the expert label twenty to thirty outputs and tuning the judge’s rubric on the disagreements. Without calibration, the judge scales the wrong standard.
When can an AI agent run autonomously in production?
After five consecutive runs pass every axis of the binary eval framework. Autonomy is gated on the rubric, not on time elapsed or human approval. Telemetry continues after autonomous status to surface aggregate failure patterns. The on-call protocol activates when patterns appear, allowing intervention before production incidents.