Skip to content
JS
All case studies

Autonomous incident investigation on LangGraph

An SRE that reads the runbook before it pages you

Apple · Oct 2025 — Present

A multi-agent platform that takes a production alert, retrieves grounded evidence, proposes a root cause with citations, and asks a human before it touches anything.

LangGraphPythonRAGKnowledge GraphsAWS
reduction in MTTR

30%

reduction in MTTR

alerts triaged per day

10k+

alerts triaged per day

unattended production writes

0

unattended production writes

The problem

On-call engineers were spending the expensive part of an incident on the cheap part of the work: opening six dashboards, grepping for the last time this fired, and finding the runbook. The reasoning was fast; the retrieval was slow.

Alert volume made this unfixable by hiring. At 10k+ alerts a day, any approach that scales linearly with human attention is already lost.

But an LLM that confidently invents a root cause is worse than no automation at all — it costs the engineer a rebuttal on top of the incident. Every generated claim had to be traceable to a real artifact.

Constraints that shaped it

  • No unattended writes. Anything that changes production state passes a human gate.
  • Every root-cause sentence cites a retrievable artifact — a runbook section, a prior postmortem, or a telemetry window.
  • Investigations are resumable. A node that fails mid-graph restarts from its checkpoint, not from the alert.
  • The system has to be measurable, which means the human's decision is data, not just a button press.

Architecture

refutedapproverejectAlertwebhook10k+/dayTriageclassify + dedupeKnowledge Graphservices, deps, ownersRetrieveGraphRAGLong-term memoryprior incidentsHypothesizeranked causesVerifytool callsDraft RCAwith citationsHuman gateapprove / reject100% of writesRemediateguarded actionEscalatewith the trace

Select any box for the design decision behind it.

The investigation graph. Solid edges are the forward path; the dashed arc is the retry loop back into hypothesis when verification fails.

What I owned

  • 01The graph itself — node contracts, checkpointing, and retry semantics on LangGraph.
  • 02The GraphRAG retrieval layer over the service knowledge graph, runbooks, postmortems, and telemetry.
  • 03Long-term memory across incidents, so a recurring failure is not investigated from scratch every time.
  • 04The human-in-the-loop gate and the guardrails around every consequential action.
  • 05The evaluation harness that turns accept/reject decisions into a precision number the team can argue with.
  • 06Backend execution engine in Python on AWS, wired into the live telemetry pipelines.

What I would do differently

The instinct is to measure an agent on whether its answer sounds right. The thing that actually moved MTTR was measuring whether the engineer accepted it — and building the gate first, so that number existed from day one rather than being retrofitted after someone asked how well it worked.

This system is employer-internal, so there is no repository to link. Everything above describes system shape, trade-offs, and results already stated on my résumé — no proprietary implementation detail.