The human gate wasn't a safety feature. It was the eval set.

The first version of our incident agent could not do anything. It read alerts, retrieved context, wrote a root-cause analysis, and stopped. Every action it proposed went to an engineer, who clicked approve or reject.

We built it that way for the boring reason. No SRE team is going to hand write access to a language model on the strength of a demo, and we were not going to ask them to. The gate was the price of getting deployed at all.

It took about three weeks to notice that the gate was the most valuable component in the system.

The measurement problem nobody solves in time

Ask a team how good their agent is and you get one of three answers: a vibe, a benchmark that does not resemble their traffic, or a promise that they are "setting up evals." The third answer is the honest one, and it usually stays true for months.

The reason is not laziness. Building an eval set for an open-ended task is genuinely expensive. Someone has to define what a correct root-cause analysis looks like, gather real incidents, write reference answers, and keep all of it current as the system it describes changes underneath them. That is a quarter of work before you learn anything.

Meanwhile the agent is running. It is producing outputs. And every one of those outputs is being read by an expert who forms an instant, confident judgement about whether it is right — and then throws that judgement away.

The gate is a labelling interface wearing a disguise

An on-call engineer looking at a proposed remediation at 3am is doing free, expert, in-distribution annotation. They are not guessing at a hypothetical. They are looking at a real incident, on their own service, with the consequences in front of them. That is a better label than anything you would get from a contractor with a rubric.

So we stored it. Not just the boolean — the whole context:

• the alert and the deduplicated incident
• every retrieved artefact and its relevance
• the ranked hypotheses, including the ones the agent discarded
• the verification tool calls and their results
• the drafted RCA
• the decision, and where possible a one-line reason

That record is an evaluation example. It cost the engineer nothing beyond the click they were already making.

Within a month we had a precision number that meant something, and it came with a breakdown: precision by service, by alert class, by whether the top hypothesis survived verification. That last one turned out to be the most useful cut we had. Hypotheses that survived a verification step were approved at a dramatically higher rate than hypotheses that had merely been ranked highly — which told us where to spend engineering effort.

Rejections are worth more than approvals

An approval says the agent did something reasonable. A rejection says it did something that looked reasonable and was not, which is the failure mode that actually costs you.

The rejections clustered. A large fraction were the same shape: the agent had matched a strong prior from a similar postmortem, verification had come back ambiguous rather than confirmatory, and it had proposed the action anyway with a confidence that did not reflect how thin the evidence was. Once you can see that pattern in labelled data, you can fix it — in our case by refusing to surface any candidate whose supporting evidence had not survived an explicit verification step, and by surfacing the confidence drop rather than burying it.

You cannot see that pattern if the only thing you record is that MTTR went down.

What I would tell someone starting this

Build the gate first. Not second, not "once we've proven it out."

The instinct is to make the agent good and then add the guardrails, because guardrails feel like a constraint on a system that is not working yet. It is the wrong order. The gate is what makes the system deployable, and it is what generates the data that makes it good. Retrofitting it after someone asks "how do you know it works?" means starting the measurement clock from zero at exactly the moment you need an answer.

The gate also does something quieter that is hard to put in a metric. It makes the agent a colleague rather than an authority. Engineers argue with it, correct it, and — because they can see the trace and the citations — occasionally learn something from it. An agent that cannot be overruled does not get that relationship, and it does not get the data either.