Green matrix-style code representing AI systems

Stop Your AI Agent from Hallucinating Root Causes: The 6-Layer Architecture That Works

by yaniskes

Stop Your AI Agent from Hallucinating Root Causes: The 6-Layer Architecture That Works

A PM’s solution discovery on why naive AI fails at root cause analysis in large systems — and a follow-able blueprint for fixing it.


TL;DR

  • The problem. Naive AI on a large enterprise stack hallucinates root causes 15–52% of the time. That destroys engineering trust faster than it saves time.
  • What works. Six layers: scoped MCP tools, documentation RAG, agent skills, a topology graph, persistent memory, and a citation contract with evals. Meta’s published version of this pattern hits 42% RCA accuracy on first pass.
  • The operator interface. A /triage command with explicit parameters (--env, --depth, --time-window) that produces a structured report — symptom, hypotheses passed and failed, root cause, flow diagram, recommended action.
  • The PM lesson. This is an integration program, not a feature program. ~60% access plumbing, 20% evals, 15% skills/topology, 5% prompts. Anyone telling you otherwise is selling something.

Contents

  1. Why this post exists
  2. The problem is real, and it’s measurable
  3. Why naive AI fails on a large enterprise stack
  4. What actually works: the public benchmark
  5. The architecture: six layers
  6. Concrete commands and behaviors to encode
  7. The /triage command and report
  8. Rollout checklist
  9. Common failure modes
  10. Lessons for product leaders

Architecture at a glance

👤 Engineer
/triage INC-4521 --env prod --depth deep

Operator Interface
/triage command
parses args · enforces scope
⇄
📄 Triage Report
markdown · JSON · Mermaid

Agent Loop
🧠 triage-playbook skill
5-stage workflow

Six Architectural Layers
LAYER 1
🔌 Grounding
Scoped MCP tools
Splunk · Git · DB · K8s · Kafka
LAYER 2
📚 Documentation RAG
Confluence · SharePoint
chunked · embedded · re-ranked
LAYER 3
🧠 Expertise
Agent Skills
procedural knowledge
LAYER 4
🗺️ Topology
Knowledge graph
services · deps · owners
LAYER 5
💾 Memory
Past incidents
+ corrections
LAYER 6
🛡️ Guardrails
Citation contract
+ eval harness

Why this post exists

If you’ve tried to point a capable LLM at your enterprise stack to speed up defect triage, you’ve probably watched it confidently invent table names, fabricate commit references, and send engineers down dead-end investigations. That failure mode is the central problem of enterprise AI in 2026, and it’s solvable.

This is a working blueprint: the metrics that prove the problem is real, the architectural pattern that fixes it, and the specific tools, libraries, and checklists you can use to reproduce the result.

Reading paths. If you have ten minutes: TL;DR → architecture diagram → /triage command. If you’re going to build this: skip to the checklist and use the rest as reference.


Server datacenter racks
Photo by Taylor Vick on Unsplash

The problem is real, and it’s measurable

The short version: Investigation eats most of MTTR. Outages are expensive. AI-grounded systems measurably help. Adoption isn’t the bottleneck — trust is.

A few numbers worth carrying into any conversation with leadership:

  • Investigation eats most of MTTR. For most teams, the investigation and root cause analysis phase consumes the largest share of mean time to resolve — not the fix itself. (Cyberhaven, 2026)
  • Most outages cost at least $100,000; severe ones regularly exceed $1 million. (Uptime Institute, 2025, via New Relic)
  • AI-grounded incident response shortens breach lifecycles by 80 days and saves $1.9M per incident for organizations that use it extensively. (IBM Cost of a Data Breach 2025)
  • Mature, AI-augmented detection cuts MTTR 30–40% vs. signature-based approaches. (Palo Alto Networks)
  • 57% of organizations now have AI agents in production, and 32% cite quality as the top barrier to deployment. (LangChain State of AI Agents 2026)

That last one is the crux. Adoption is no longer the bottleneck — trust is. And trust is what naive AI burns.

Why naive AI fails on a large enterprise stack

The short version: Models hallucinate 15–52% of the time without grounding, under 2% with it. The architecture is the entire game.

The hallucination rates are not subtle:

  • Frontier models still hallucinate 15–52% of the time on structured analysis tasks, depending on benchmark and conditions. (SQ Magazine, 2026)
  • AI-generated code contains security vulnerabilities 29–45% of the time, and roughly 20% of package recommendations point to libraries that don’t exist. (diffray, 2026)
  • Citation fabrication rates reach 94% in adversarial testing. (SQ Magazine, 2026)
  • In retrieval-grounded tasks, OpenAI’s own evals show hallucination rates drop below 2%. (SQ Magazine, 2026)

Ungrounded models hallucinate at rates that destroy operational trust. Grounded models don’t.

The mechanics are well understood. Frontier context windows top out around a million tokens; a real enterprise stack — hundreds of repos, hundreds of tables, hundreds of Kafka topics, runbooks across Confluence and SharePoint — is several million tokens before you’ve loaded the relevant logs. Even within the context window, attention degrades in the middle of long inputs (the “lost in the middle” problem), so models miss evidence you fed them. Without a tool layer to fetch authoritative data on demand, the model defaults to plausible-sounding generation.

The first time engineers chase a hallucinated RCA, they lose two hours. The second time, they lose trust. The third time, they stop using the system. That is how enterprise AI initiatives die.

Circuit board close-up representing AI system internals
Photo by Alexandre Debiève on Unsplash

What actually works: the public benchmark

The short version: Meta published the canonical pattern — heuristic retrieval feeding an LLM ranker, confidence-gated, with explicit suppression of low-confidence answers. 42% first-pass RCA accuracy.

The pattern that works is not theoretical. Meta published it (Engineering at Meta, 2024), and the numbers are the right anchor for any program:

  • 42% accuracy in identifying root causes at investigation creation time for incidents in their web monorepo.
  • Achieved by combining heuristic-based retrieval (using code ownership, directory structure, and runtime code graphs to reduce thousands of changes to a few hundred) with an LLM-based ranker (a fine-tuned Llama 2 7B reducing hundreds to the top five).
  • Results presented with explicit confidence measurements, with low-confidence recommendations suppressed rather than shown.

Meta’s stack is unique, but the architectural primitives are not: retrieval first, ranking second, confidence-gated output, and a hard rule that low confidence escalates rather than guesses. Vendor systems built on the same pattern (NeuBird Hawkeye, Rootly) report MTTR reductions of 40–90% in production. The 90% number is vendor marketing; treat it as a ceiling, not a floor. The 30–40% range from independent analysis is the realistic operating zone.


The architecture: six layers

The short version: Most teams stop after MCP + evals. That’s the demo, not the production system. The full stack adds documentation RAG, agent skills, a topology graph, and persistent memory.

Most teams stop after they wire up MCP servers and an eval harness. That’s where the impressive demos live and where production systems quietly fail. MCP gives the agent transport — the ability to fetch evidence. It does not give the agent expertise. Without explicit procedural knowledge, indexed documentation, infrastructure mapping, and persistent memory, the agent rediscovers your stack from scratch on every invocation, wastes turns on dead-end queries, and re-makes the same mistakes a human engineer would have learned from once.

The full stack has six layers, ordered roughly by what to build first:

1
Grounding
Scoped MCP tools to live systems
⚠ The agent can’t see anything
2
Documentation RAG
Semantic search over Confluence/SharePoint
⚠ The agent re-reads stale pages and misses the right one
3
Expertise
Agent Skills with procedural knowledge
⚠ The agent does the wrong things in the wrong order
4
Topology
Knowledge graph of services and ownership
⚠ The agent re-discovers your stack every time
5
Memory
Persistent record of past incidents
⚠ The agent makes the same mistakes forever
6
Guardrails
Citation contract + eval harness
⚠ You can’t trust any of the above

Skip layers 2–5 and you have a fast retriever with no judgment. Build all six and you have a system engineers actually use in month six.

Layer 1 — Grounding through scoped MCP servers

Build one MCP server per authoritative source, each exposing 3–6 narrowly scoped tools. Generic “query everything” tools are the dominant agent failure mode at scale.

▸Tools, tradeoffs, and why scope matters → expand

Build one Model Context Protocol (MCP) server per authoritative source. Each exposes a small, namespaced toolset, not a single generic “query everything” endpoint:

Source Tools
Splunk splunk_search_logs, splunk_get_trace
GitLab/GitHub git_get_commit_diff, git_search_repo, git_blame
Oracle/MySQL db_describe_table, db_recent_schema_changes, db_query_readonly
Kafka/RabbitMQ kafka_topic_metadata, kafka_consumer_lag, mq_queue_depth
Kubernetes k8s_pod_logs, k8s_recent_deployments, k8s_describe_resource

Documentation sources (Confluence, SharePoint, runbooks) belong in Layer 2 — they need RAG, not direct tool access. More on that below.

Why scoped tools matter: Anthropic’s research on tool design shows that agents lose accuracy fast when tools are too generic or too numerous. Anthropic’s own production guidance is that Claude models hallucinate less when tools are well-described and narrowly scoped (ClickHouse, 2026). Namespacing reduces wrong-tool selection — the dominant agent failure mode at scale.

MCP is now standard infrastructure, not a vendor bet. It was donated to the Linux Foundation’s Agentic AI Foundation in December 2025, has 97M+ monthly SDK downloads, 10,000+ active public servers, and adoption from ChatGPT, Cursor, Gemini, Microsoft Copilot, and VS Code (Anthropic).

Tools and libraries:

  • Official MCP SDKs (Python, TypeScript, Java, Kotlin, C#, Rust, Swift): github.com/modelcontextprotocol
  • FastMCP (Python) — the productivity framework most teams ship on
  • MCP-Framework (TypeScript) — mcp create app scaffolding
  • MCP Inspector — official protocol-level test harness
  • Reference servers (Filesystem, Git, Fetch, Memory) at the same repo
  • Claude Agent SDK (Python/TypeScript) — easiest agent loop to drop on top
  • Streamable HTTP for remote/cloud deployment; stdio for local. SSE is deprecated.

Layer 2 — Documentation RAG over Confluence, SharePoint, and runbooks

Native enterprise search returns dozens of weakly-relevant pages. RAG with semantic chunking and re-ranking lets the agent find the right page in one call — with citations. Permission-aware sync is non-negotiable.

▸Pipeline, retrieval contract, and tools → expand

A direct Confluence search tool is fine for fetching a known page. It’s badly insufficient for finding the right page when the agent doesn’t yet know where the answer lives. Native enterprise search returns dozens of weakly-relevant pages; the agent burns turns reading through them, often misses the one that matters, and sometimes confidently summarizes a stale page from 2019.

This is what RAG (Retrieval-Augmented Generation) is actually for. The agent semantically searches an indexed corpus and gets back the few chunks most likely to contain the answer, with citations.

The signal that this layer matters: the average enterprise employee spends two to three hours per week simply looking for information that already exists somewhere (Towards Data Science, 2026). Senior engineers absorb this cost as accidental support work. A well-built documentation RAG gives that time back.

What goes in the index:

  • Confluence (architecture pages, runbooks, post-mortems, design docs)
  • SharePoint (specs, contracts, ops manuals)
  • Markdown READMEs and /docs folders from each repo
  • Past incident reports and root cause writeups
  • Onboarding and architecture decision records (ADRs)

The pipeline (offline ingestion):

  1. Connect to the source via API. LlamaIndex ships native connectors for Confluence, SharePoint, Notion, Google Drive, S3, and ~100 others — and tracks document hashes so only changed files are re-indexed.
  2. Chunk the content. This is the single biggest quality lever — chunking strategy alone can move retrieval relevance by 30–40% (Velaris, 2026). Avoid naive fixed-size windows; prefer recursive or semantic chunking that respects document structure (headers, sections, code blocks). Recent published benchmarks show recursive chunking with TF-IDF-weighted embeddings achieving 82.5% precision vs. content-only baselines (ArXiv, Dec 2025).
  3. Embed with a strong model. text-embedding-3-large (OpenAI), voyage-3-large (Voyage AI), BGE-M3 (open source, multilingual), or Cohere Embed v3 are reasonable defaults. Pick one and stay with it — switching embedding models requires re-indexing the entire corpus.
  4. Store in a vector database. pgvector (PostgreSQL extension) handles up to ~2M vectors with sub-100ms latency on modest hardware and reuses your existing DBA expertise. For larger or more demanding workloads: Qdrant, Weaviate, or Milvus. For most enterprise documentation corpora (500K–1.5M chunks), pgvector is the right call.
  5. Sync incrementally. Subscribe to Confluence/SharePoint webhooks or poll their delta APIs. The edge case that catches most teams: document moves and renames show up as delete-then-add pairs — handle this explicitly or you’ll get duplicate and orphaned results.

The retrieval contract (runtime):

Expose two MCP tools, not one:

  • docs_search(query, top_k, source_filter) — returns chunks with page IDs, section anchors, and source URLs
  • docs_get(page_id) — fetches the full canonical document when the agent needs more than the chunk

Always add a re-ranker between retrieval and the LLM. Cohere Rerank, Voyage Rerank, or a cross-encoder like bge-reranker will measurably lift relevance — re-ranking the top 50 chunks down to top 5 is a common pattern. And always include freshness metadata (last-modified date, author, source) so the agent can flag stale or contested content rather than treating all hits as equally authoritative.

Tools and libraries:

  • LlamaIndex — best-in-class document connectors and incremental indexing
  • LangChain — broader ecosystem, comparable connector coverage
  • pgvector — PostgreSQL extension, the pragmatic enterprise default
  • Qdrant, Weaviate, Milvus — purpose-built vector databases for larger corpora
  • Unstructured — handles the messy parts of enterprise document parsing (PDFs, Office docs, HTML)
  • Haystack — end-to-end RAG framework with strong evaluation tooling
  • Managed alternatives with permission-aware sync: Glean, Vectara, Katonic AI

⚠️ Permissions are not optional. Confluence and SharePoint have rich, layered ACLs. Your RAG layer must respect them — either by indexing per-user (expensive) or by storing permissions as metadata and filtering at retrieval time. An agent that surfaces content the requesting user shouldn’t see is a security incident, not a quality issue. This is why managed vendors (Glean, Vectara, Katonic) lead with “permission-aware sync” — it’s the hardest part to get right.

Layer 3 — Expertise via Agent Skills

This is the layer most teams miss. Agent Skills are folders of procedural knowledge (“when triaging consumer-lag, first check brokers, not DBs”). Loaded dynamically; small context footprint; the right place for everything senior engineers carry in their heads.

▸Skills to write, examples, and resources → expand

Agent Skills are folders of instructions, scripts, and resources that the agent loads dynamically when a task matches their description. Anthropic introduced them in October 2025 and published them as an open standard in December 2025 (see agentskills.io). They are the right place to encode all the procedural knowledge that your senior engineers carry in their heads:

  • “When triaging a consumer-lag incident, first check broker health, then partition assignment, then downstream consumer thread counts. Do not query the database before ruling out broker issues.”
  • “When a service times out intermittently, check K8s recent deployments before checking application code.”
  • “Logs older than 30 days live in cold storage. Use splunk_archive_search, not splunk_search_logs.”
  • “The customer_events table is partitioned by event_date. Always include a date predicate. Queries without one will be killed.”
  • “Schema changes to the billing_* tables require runbook BILL-17. Do not propose fixes that touch them without flagging it.”

A SKILL.md file is dead simple — YAML frontmatter (name and description) plus markdown instructions. The agent reads the description on every turn and pulls the full skill into context only when it’s relevant. This is progressive disclosure: many specialized skills, small context cost.

Skills you should write for triage:

Skill Purpose
triage-playbook The decision tree senior engineers use — check X before Y, escalate when Z
log-search-patterns Where each service logs, retention windows, indexed fields, sampling rules
database-conventions Which columns are indexed, partition keys, query cost limits, read replicas
kafka-topology Topic ownership, retention policies, common consumer-lag causes
deploy-and-test What is allowed to be tested in which environment, how to verify a hypothesis without touching prod
escalation-rules Severity definitions, on-call routing, who owns which service
runbook-index A pointer to the canonical runbook for each known failure pattern

Resources:

Layer 4 — Topology via a knowledge graph

The agent needs to know what depends on what, who owns what, what changed where. A knowledge graph answers those questions in milliseconds. GraphRAG patterns lift retrieval accuracy ~15–30% over vector-only.

▸What to put in the graph, tools, and benchmarks → expand

The agent needs to know how your infrastructure fits together — which service owns which database, which Kafka topic feeds which consumer, which deployments touch which K8s namespace. Without this, the agent has to re-discover topology on every incident.

A knowledge graph is the right structure for this because the questions you actually ask during triage are graph questions: “What depends on this service? What did this commit change downstream? Who owns this table?”

The data points worth carrying:

  • Microsoft’s GraphRAG: 29.6% faster resolution times in customer support compared to vector retrieval alone (Atlan, 2026).
  • Agent-as-a-Graph (PwC, ArXiv 2025): 14.9% Recall@5 improvement and 14.6% nDCG@5 improvement on tool selection benchmarks when MCP tool retrieval is graph-aware vs. vector-only (arxiv.org/pdf/2511.18194).

What to put in the graph:

  • Services, daemons, jobs — and which repos own them
  • Databases and tables — with ownership, criticality, and indexed columns
  • Kafka topics, RMQ queues — with producers and consumers
  • API contracts and proxy routes
  • K8s namespaces, deployments, and pods
  • Documentation pages and runbooks linked to the systems they cover
  • People and teams — who owns what

Tools and libraries:

  • Graphiti — open-source, real-time knowledge graph specifically for AI agents; supports Neo4j, FalkorDB, Kuzu, and Amazon Neptune; combines vector, BM25, and graph traversal for retrieval
  • Neo4j — the canonical property graph; mature ecosystem, Cypher query language
  • GraphRAG (Microsoft) — open-source pattern combining graph retrieval with LLM reasoning
  • Backstage software catalog as a topology source — most enterprises already have one or should
  • For schema-level mapping, expose graph queries to the agent as a small set of MCP tools: topology_what_depends_on, topology_owners_of, topology_path_between

The graph does not have to be perfect on day one. Start with services, databases, and topics. Add depth as the agent’s failure modes reveal what’s missing.

Layer 5 — Memory: don’t let the agent re-learn every incident

Without persistent memory, every incident starts from zero. 23% better outcomes after three months of accumulated learning in published deployments. Memory is also an attack surface — provenance tracking is mandatory.

▸Within-session vs. cross-session, what to store, security caveats → expand

Without persistent memory, agents cannot reuse past failure patterns, resolution paths, or root cause analyses across sessions (Augment Code, 2026). Every incident starts from zero. That’s a structural waste.

There are two memory layers worth building:

Within-session memory — keeps a long investigation coherent. Without it, the context window fills up and the model loses track of what it has already checked. Solutions: a working scratchpad file the agent writes to, or a short-term store like Redis. AWS Bedrock AgentCore Memory and Mem0 are managed alternatives.

Cross-session memory — the agent learns from incidents over time. Patterns that worked, dead ends that didn’t, corrections engineers made. One published reference point: 23% better outcomes after three months of accumulated memory vs. a baseline without learning, in a financial AI deployment (Hindsight, 2026).

What to store:

  • Every resolved incident: symptoms, hypotheses checked, root cause, fix
  • Engineer corrections: “the agent suggested X, the actual cause was Y because Z”
  • Confirmed false leads: “if you see symptom A, do not check B — that’s a known dead end”
  • Stack-specific gotchas the agent has learned (and that should eventually become Skills)

Tools and libraries:

⚠️ Memory is an attack surface. The 2025 research paper MemoryGraft (arxiv.org/pdf/2512.16962) showed that poisoned memory entries can persistently compromise agent behavior — a more covert attack than prompt injection because the corruption survives across sessions. Memory should have provenance tracking and an audit trail. Treat the memory store as production data with the same access controls as a database.

Layer 6 — Guardrails: citations and evals

Two simple rules: no citation, no claim, and every change ships through the eval set. These are the single most effective hallucination controls — they make lying require effort and regression-test it on every change.

▸The citation contract, the eval harness, and tools → expand

The model is asked to investigate, not to generate. Every assertion in the RCA must include a source reference:

  • A specific log timestamp range
  • A commit SHA
  • A table or schema version
  • A Confluence/SharePoint page ID
  • A runbook section

Output is structured (JSON), not prose, so citations are machine-checkable. No citation, no claim. This is the single most effective hallucination control I’ve found — it makes lying require effort.

The eval harness is the unglamorous piece doing the heaviest lifting. Build a held-out set of historical incidents with known root causes. Run the agent against it on every change to prompts, tools, skills, or models. A regression blocks the change. Confidence below threshold triggers escalation, not a guess.

This mirrors what production systems do: NeuBird Hawkeye uses adversarial dual-model agreement scoring with LLMs as judges; Meta uses fine-tuned ranker confidence with explicit suppression of low-confidence recommendations.

Eval frameworks:

  • Ragas — open-source, RAG-specific, reference-free metrics
  • Open RAG Eval (Vectara + University of Waterloo) — academically rigorous
  • DeepEval / Confident AI — battle-tested metric library
  • Arize Phoenix — OpenTelemetry-based observability + eval
  • LangSmith or Langfuse for production tracing
  • LLM-as-judge for cases without ground truth — pair with adversarial dual-model scoring

Concrete commands and behaviors to encode

Explicit instructions in skills beat implicit hopes the model will get it right. Below: querying, log search, test execution, and escalation rules — examples to start from.

▸Specific behaviors per category → expand

Things that should be explicit instructions in your skills, not implicit hopes the model will do them right:

Querying behavior

  • Always include partition predicates on partitioned tables; agent must check the partition key before issuing a query
  • Use indexed columns in WHERE clauses; if the schema map says no index exists, narrow by another column first
  • Read replicas only for analytical queries; write-leader only when explicitly required
  • Set query timeouts; never run open-ended scans on large tables

Log search behavior

  • Hot logs vs. cold logs — different tools, different latencies
  • Sampling rules — high-traffic services log a fraction; the agent must compensate
  • Time bounds always required; default to a 15-minute window around the incident timestamp

Test execution behavior

  • Allowed environments: dev, staging, perf, prod-readonly
  • Tests that may run in each: unit, integration (staging only), smoke (prod-readonly only)
  • Never run tests that mutate state in any environment without explicit human approval
  • Synthetic test data only; never use real customer records as test inputs

Escalation behavior

  • Confidence < 0.6 → return “insufficient evidence” with a list of next checks
  • Touches a billing-, payments-, or auth-related path → escalate before answering
  • Symptom matches a known false-lead pattern in memory → flag the dead end, don’t repeat it

These rules belong in skills, not in the system prompt. The system prompt should be short and stable; skills are where the operational knowledge lives.


Code on monitors in a developer environment
Photo by Kevin Ku on Unsplash

The /triage command: turning the agent into an operator interface

A chat box works for a demo. In production, ship a structured /triage command with explicit parameters (--env, --depth, --time-window) that runs a five-stage workflow and produces a persistent report. The command is the surface that determines adoption.

Everything above describes capability. This is how engineers actually invoke it.

A chat box where engineers type “investigate INC-4521” works for a demo. In production, you want a structured command — one that takes the incident as a parameter, runs the full triage workflow with validated arguments and explicit options, and produces a deterministic, auditable artifact. This is the surface that determines adoption.

Implementation paths. A triage command can ship as:

  • A Claude Code slash command (a markdown file in .claude/commands/triage.md) for engineers working in the IDE
  • A Slack slash command (/triage INC-4521) for on-call rotation
  • A CLI (triage INC-4521 --env prod) for automation and scripting
  • A CI/CD step that runs on every Sev-2+ alert and posts findings to the incident channel

All four wrap the same underlying skill. Build the skill once; expose it through the surfaces your team actually uses.

What the command does

/triage <incident-id-or-description>
        [--env dev|staging|perf|prod-readonly]
        [--depth quick|standard|deep]
        [--time-window 15m|1h|6h|24h]
        [--scope service-name|repo-name]
        [--skip-validation]
        [--dry-run]

Parameters explained:

Parameter Purpose Default
incident-id-or-description Either a ticket ID (INC-4521, JIRA-1234) the agent looks up, or a free-text description required
--env Which environment to investigate. Constrains tool scopes (a prod investigation should never read dev logs) inferred from incident, else prompt
--depth quick (5 min, single hypothesis), standard (15 min, top 3 hypotheses), deep (45 min, full RCA with timeline reconstruction) standard
--time-window How far back to search logs and deployments 1h
--scope Restrict investigation to a service or repo when the engineer already has a hypothesis none
--skip-validation Bypass the validation step (for cases where the engineer has already confirmed the symptoms) false
--dry-run Show the plan without executing tools — useful for reviewing what the agent is about to do false

The triage workflow the command runs

Every invocation follows the same five-stage workflow, encoded in the triage-playbook skill:

1. Intake and validation

  • Parse the incident ID or description; if an ID, fetch it from Jira/ServiceNow/PagerDuty
  • Extract structured facts: affected service(s), symptom, first-seen timestamp, environment, severity
  • Validate that the symptom is reproducible or observable in the named environment (e.g., does the error actually appear in logs in the time window?)
  • If validation fails, return early: “I cannot reproduce this symptom in prod logs between 14:00–15:00 UTC. Please confirm the time window or environment.”
  • Cross-check against the memory store for known false-lead patterns: “Symptom matches dead-end #47 — last 3 incidents with this signature were caused by X, not Y. Skipping that hypothesis.”

2. Hypothesis generation

  • Query the topology graph for blast radius: what services depend on the affected one?
  • Query memory for similar past incidents and their resolutions
  • Query documentation RAG for relevant runbooks
  • Produce a ranked list of hypotheses with prior probabilities (think Meta’s heuristic-based retrieval feeding the LLM ranker)

3. Evidence gathering

  • For each top-N hypothesis (N = 3 for standard, 5 for deep), execute a small batch of scoped tool calls in parallel
  • Logs, recent deployments, schema changes, consumer-lag, runbook lookups — all bounded by the --env and --time-window parameters
  • Every fact retrieved is tagged with its source for the citation contract

4. RCA synthesis (with optional one-level-deeper analysis)

  • For each hypothesis, summarize: supporting evidence, contradicting evidence, confidence
  • If --depth=deep is set, drill one level further on the highest-confidence hypothesis: reconstruct the timeline, identify the introducing change, propose a fix and a verification step
  • If two hypotheses are within 10% confidence of each other, run an adversarial comparison: “Hypothesis A predicts log pattern X; hypothesis B predicts log pattern Y. Check which is present.”

5. Output and escalation

  • Structured report: the ranked hypotheses, their evidence with citations, a recommended next action, and a confidence score
  • If top-confidence < 0.6, escalate explicitly with “what I checked, what I couldn’t determine, what to look at next”
  • If touching a sensitive path (billing, payments, auth), flag for human review before any further action
  • Write the investigation to memory for future incidents to learn from

Sample output

$ /triage INC-4521 --env prod-readonly --depth standard --time-window 1h

📋 INC-4521: "Checkout API returning 503s for ~5% of requests since 14:32 UTC"
   Affected: checkout-service (prod)
   Severity: Sev-2

✅ Validation passed
   - 503 errors confirmed in prod logs (623 events, 14:32–15:32 UTC)
   - Error rate: 4.8% (within reported range)
   - No matching dead-end pattern in memory

🔍 Top hypotheses (3 of 8 considered)

  1. Database connection pool exhaustion (confidence: 0.78)
     ├─ Evidence: 47 "connection timeout" errors at db-checkout-1 starting 14:31 UTC
     │   [splunk: 2026-05-09T14:31:00Z to 14:35:00Z]
     ├─ Evidence: connection pool size=20, active=20, queue depth=140
     │   [k8s: checkout-service-7d8f, metrics endpoint]
     ├─ Topology: db-checkout-1 also serves cart-service (no errors there)
     └─ Suggests: pool sizing, not DB capacity

  2. Recent deployment regression (confidence: 0.41)
     ├─ Deploy at 14:28 UTC: checkout-service v2.14.3 → v2.14.4
     │   [gitlab: commit a3f9c12, "refactor: connection lifecycle"]
     └─ ⚠ Touches connection handling — relevant to hypothesis #1

  3. Upstream payment provider degradation (confidence: 0.18)
     └─ No supporting evidence; eliminated by /payments_health check

📌 Recommended next action
   Investigate hypothesis #1 ∩ #2: deploy a3f9c12 likely changed connection
   lifecycle, exposing pool exhaustion under load. Roll back or increase pool.

   Verification: deploy v2.14.3 to one canary pod; observe error rate.

📚 Citations: 7 evidence references, 2 runbooks (RUN-checkout-23, RUN-db-pool-7)
🧠 Written to incident memory for future similar patterns.
📄 Full report: ./reports/triage/INC-4521/2026-05-09T15-32Z.md

The triage report: persistent artifact, not just CLI output

The condensed CLI summary above is what the engineer reads in 30 seconds. The report is what they share with the team, attach to the incident ticket, hand off at shift change, and reference six months later when a similar incident recurs.

Every /triage invocation writes a structured markdown report to a known location:

./reports/triage/<incident-id>/<timestamp>.md       # primary artifact
./reports/triage/<incident-id>/<timestamp>.json     # machine-readable copy
./reports/triage/<incident-id>/<timestamp>.flow.mmd # Mermaid flow diagram

The directory structure matters: one folder per incident, multiple timestamped reports per folder. A long incident generates several reports as new evidence comes in; the chronology stays intact and replayable.

For Slack-invoked triage, the report is also attached to the thread as a markdown file. For CI-invoked triage, it’s posted to the incident channel and stored in object storage with a link. Same report, different surfaces.

What goes in the report

The report follows a fixed template — every incident, every time. Engineers learn it once and can scan any report in 60 seconds.

The nine sections, in order:

  1. Symptom — what’s happening, where, magnitude
  2. Incident description — customer-facing impact, severity
  3. Investigation timeline — table of events with timestamps
  4. Hypotheses considered — passed (✅) and failed (❌), each with status, supporting evidence, and explicit reasoning. The most underrated part.
  5. Root cause analysis — primary cause, contributing factors, why it wasn’t caught earlier
  6. Behavior and scenario flow — Mermaid sequence diagram
  7. Recommended next action — sequenced (immediate → short-term → medium-term → process)
  8. References and citations — every claim traceable
  9. Confidence and limitations — what the agent doesn’t know; what humans must verify
▸Full report template (markdown) → expand
# Triage Report: INC-4521

**Generated:** 2026-05-09 15:32 UTC by /triage v1.4.2
**Invocation:** `/triage INC-4521 --env prod-readonly --depth standard --time-window 1h`
**Confidence:** 0.78 (above escalation threshold; no human escalation required)
**Status:** ✅ Investigation complete | ⚠ Recommended action requires deploy

---

## 1. Symptom

Checkout API returning HTTP 503 errors for approximately 4.8% of requests
since 14:32 UTC. Affects `checkout-service` in `prod`. No regional skew.
First-seen by automated alert; reported by 3 engineers in #incident-checkout.

## 2. Incident description

Customer-facing: a portion of checkout requests fail outright, breaking
the purchase flow. Sev-2 severity per the on-call rubric (revenue impact,
< 10% affected, no data loss). 47 customer-support tickets opened in the
window.

## 3. Investigation timeline

| Time (UTC) | Event |
|---|---|
| 14:28 | Deploy: checkout-service v2.14.3 → v2.14.4 (commit a3f9c12) |
| 14:32 | First 503 in prod logs |
| 14:34 | Error rate exceeds 1% threshold; alert fires |
| 14:41 | Manual investigation begins |
| 15:32 | /triage invoked; this report generated |

## 4. Hypotheses considered

### ✅ Hypothesis 1: Database connection pool exhaustion (confidence: 0.78)

**Status:** Confirmed as primary contributing factor.

**Supporting evidence:**
- 47 "connection timeout" errors at db-checkout-1 starting 14:31 UTC
  [splunk: 2026-05-09T14:31:00Z to 14:35:00Z]
- Connection pool: size=20, active=20, queue depth=140
  [k8s: checkout-service-7d8f, metrics endpoint]
- Wait-time histogram shows p99 spike from 8ms → 4.2s at 14:32
  [splunk: pool_wait_time metric]

**Why this hypothesis stuck:** all three independent signals (timeouts,
pool saturation, wait-time degradation) point to the same root. Topology
check ruled out shared-DB capacity issue (cart-service uses the same
db-checkout-1 and is healthy).

### ✅ Hypothesis 2: Recent deployment regression (confidence: 0.41 → 0.74 after combining with #1)

**Status:** Confirmed as triggering change.

**Supporting evidence:**
- Deploy at 14:28 UTC, 4 minutes before symptom onset
  [gitlab: commit a3f9c12, "refactor: connection lifecycle"]
- Diff inspection: connection close moved from `finally` block to an
  async cleanup path, increasing time-to-release under load
  [gitlab: a3f9c12 +18 −7]

**Why this hypothesis became central:** standalone confidence was
moderate, but the diff directly explains hypothesis #1. The two are
not independent — #2 is the cause; #1 is the manifestation.

### ❌ Hypothesis 3: Upstream payment provider degradation (confidence: 0.18 → eliminated)

**Status:** Rejected.

**Why eliminated:** `/payments_health` check returned green for the
window. Payment provider's status page shows no incidents. 503s in
checkout precede any payment call (errors fire at API gateway, not at
provider boundary).

### ❌ Hypothesis 4: Memory pressure / OOM (confidence: 0.09 → eliminated)

**Status:** Rejected.

**Why eliminated:** k8s pod memory metrics flat across the window;
no OOMKilled events; no GC pauses in JVM logs.

## 5. Root cause analysis

**Primary cause:** Commit a3f9c12 ("refactor: connection lifecycle")
moved database connection cleanup from a synchronous `finally` block
to an asynchronous post-response handler. Under normal load this is
indistinguishable; under sustained traffic, connections are held ~80ms
longer per request. With a fixed pool size of 20 and ~250 RPS, the pool
saturates and incoming requests queue, then time out at the gateway
(2s timeout), surfacing as 503s.

**Why it wasn't caught pre-prod:**
- Staging load tests run at 50 RPS — well below the saturation point
- Unit tests passed because they don't exercise pool dynamics
- Code review focused on correctness, not concurrency timing

**Contributing factors:**
- Connection pool sizing (20) was set in 2023 for a smaller traffic
  baseline; current peak is 5x higher
- No automated check for pool-saturation regressions in CI

## 6. Behavior and scenario flow

The flow diagram below shows the failure mode under load. (Mermaid
source at `./reports/triage/INC-4521/2026-05-09T15-32Z.flow.mmd`.)

```mermaid
sequenceDiagram
    participant Client
    participant Gateway
    participant Checkout as checkout-service v2.14.4
    participant Pool as Connection Pool (size=20)
    participant DB as db-checkout-1

    Note over Checkout: Pre-deploy: connection released in finally{} (sync)
    Note over Checkout: Post-deploy a3f9c12: released in async cleanup (~80ms later)

    Client->>Gateway: POST /checkout
    Gateway->>Checkout: forward request
    Checkout->>Pool: acquire connection
    Pool-->>Checkout: connection (1 of 20)
    Checkout->>DB: query order
    DB-->>Checkout: result
    Checkout-->>Gateway: 200 OK
    Note over Pool: Connection NOT released yet — async cleanup pending

    loop Sustained load (~250 RPS)
        Client->>Gateway: POST /checkout
        Gateway->>Checkout: forward
        Checkout->>Pool: acquire connection
        Note over Pool: All 20 in use, queueing
        Pool--xCheckout: wait 4.2s (p99)
        Gateway--xClient: 503 (gateway 2s timeout exceeded)
    end

7. Recommended next action

  1. Immediate (mitigation): Roll back checkout-service to v2.14.3
    on a canary pod; verify error rate drops within 5 minutes; then
    roll back fleet-wide.
  2. Short-term (verification): Once mitigated, deploy v2.14.4 to a
    single canary pod with pool size raised to 60; confirm 503s do not
    recur. This confirms RCA.
  3. Medium-term (prevention): Add a pool-saturation regression
    test to CI at 250 RPS staging load; raise pool size in production
    config to 60.
  4. Process: Add “concurrency timing review” to the code review
    checklist for connection-lifecycle changes.

8. References and citations

  • 7 evidence sources (Splunk, GitLab, K8s metrics) — full list in JSON
  • 2 runbooks consulted: RUN-checkout-23, RUN-db-pool-7
  • 3 similar past incidents in memory: INC-3819 (similar mechanism,
    different service), INC-4102 (same service, different cause)

9. Confidence and limitations

Overall confidence: 0.78. Primary hypothesis is well-supported
by three independent signals; deploy correlation is strong.

What I could not verify:

  • Exact connection-hold time under load — would require a load test
    to confirm the 80ms estimate
  • Whether other services using db-checkout-1 are at risk — depends on
    their pool sizing, which I did not inspect

What a human should double-check before acting:

  • The recommended canary deploy (step 1 above) is reversible and safe
  • The pool-size change in step 3 should be reviewed by the DBA team

Report written to incident memory. Future invocations of /triage on
similar symptoms will surface this resolution.

Why the report has this shape

Each section earns its place:

  • Symptom and description — what an engineer needs to brief a manager in 30 seconds
  • Investigation timeline — the audit trail; satisfies post-incident review and compliance needs
  • Hypotheses (passed and failed) with reasoning — the most important section. Showing failed hypotheses with explicit reasons builds trust faster than any other element. It also lets the next engineer skip the dead ends.
  • Root cause analysis — the primary cause, contributing factors, and why it wasn’t caught earlier (the question every post-mortem asks)
  • Flow diagram — engineers process diagrams faster than prose for sequence-of-events failures. Mermaid is the right format because it’s text-based, version-controllable, and renders in GitHub, GitLab, Confluence, and most modern markdown viewers
  • Recommended next action — sequenced by urgency, with verification steps
  • References and citations — every claim is traceable
  • Confidence and limitations — what the agent does not know, and what a human must verify before acting

Implementation notes

  • Generate the report from a Jinja2 template — the agent fills sections; the template enforces structure. This keeps reports comparable across incidents.
  • Store reports in version control or object storage, not just on the engineer’s laptop. A shared reports/triage/ directory in a dedicated git repo works well; so does S3 with a known prefix.
  • The JSON copy is for tooling, not humans — it feeds the eval harness, the memory store, and any downstream dashboards.
  • The Mermaid diagram is generated by the agent as part of the workflow; it’s part of the skill’s instructions, not a manual step. Anthropic’s pre-built mermaid skill or a small custom skill can handle this.
  • Reports are write-once. If new evidence arrives, generate a new timestamped report — don’t edit the old one. This preserves the chronology and prevents accidental rewrites of history.

Why this matters for the program, not just the engineer

The report is also the single most valuable input to your eval harness and your memory store. Every triage report is a labeled training example: symptom, hypotheses, evidence, outcome. Two months in, you have hundreds of these. They become:

  • The eval set that gates every prompt and skill change
  • The memory the agent retrieves on similar future incidents
  • The corpus a fine-tuned ranker can train on (the Meta pattern)
  • The audit trail for regulators and post-incident reviewers

Build the report well on day one and the rest of the program compounds for free.

Why this shape of command works

  • The engineer stays in control. The agent investigates and recommends; the engineer decides whether to roll back, scale, or escalate. The --dry-run and confidence-gated escalation make it safe to invoke without ceremony.
  • The output is auditable. Every claim has a citation. Every confidence score is calibrated against the eval set. The whole investigation can be replayed from the memory store.
  • The parameters constrain hallucination space. An agent told to investigate prod-readonly in a 1h window cannot wander off into dev logs from last week. Scope is enforced by the command, not hoped-for from the prompt.
  • The workflow is the same every time. Engineers don’t have to remember which skills to invoke or which tools to call. The command is the skill.

Variants worth building

Once the base /triage works, the same pattern produces useful siblings:

  • /postmortem <incident-id> — runs after resolution, drafts the post-mortem document with timeline reconstruction, contributing factors, and action items. Writes back to Confluence with a citation trail.
  • /blast-radius <service-or-commit> — uses the topology graph to enumerate downstream impact before a deploy or change.
  • /runbook-find <symptom> — pure documentation RAG; returns the top runbooks for a symptom without doing the full investigation.
  • /regression-check <incident-id> — runs against the eval set: “have we seen this before? Did the previous fix introduce this?”
  • /triage --learn-from <incident-id> — explicit memory write: capture this incident’s resolution as a pattern for future agents.

Where to put the command file

For Claude Code, the simplest path is a markdown file at .claude/commands/triage.md:

---
name: triage
description: Run a full triage workflow on an incident
---

# /triage <incident-id-or-description> [options]

Invoke the `triage-playbook` skill with the following arguments:
- incident: $1
- environment: ${env:-inferred}
- depth: ${depth:-standard}
- time_window: ${time-window:-1h}
- scope: ${scope:-none}
- skip_validation: ${skip-validation:-false}
- dry_run: ${dry-run:-false}

Follow the five-stage workflow defined in the skill. Output the structured
report in the format specified in the skill examples.

The command is intentionally thin — it’s a parameter parser. The actual logic lives in the skill, which is the right place because skills are versioned, evaluable, and reusable across the Slack bot, the CLI, and the CI pipeline.


Rollout checklist

Order matters. Don’t skip steps to ship faster — the trust budget is the binding constraint. Six phases. Each builds on the last.

Phase Focus
1 Define and measure
2 Grounding layer (MCP)
3 Docs RAG, skills, topology
4 Eval harness + citation contract
5 Soft launch
6 Measure and decide

Define and measure

  • [ ] Pick one defect class to start (recommend: internal tooling, batch jobs, or dev-environment issues — not customer-impacting)
  • [ ] Document current median triage time for that class — this is your baseline
  • [ ] Define your metric stack: North Star, guardrails, safety counter-metric, escalation metric
  • [ ] Get explicit success criteria signed off by engineering leadership before coding
  • [ ] Identify the 3–5 most-used data sources for that defect class — start there, not with all of them

Build the grounding layer

  • [ ] Stand up one MCP server per source using FastMCP (Python) or MCP-Framework (TypeScript)
  • [ ] Each server exposes 3–6 narrowly scoped, namespaced tools — not one generic query tool
  • [ ] All access is read-only; auth via short-lived tokens, scoped to the agent’s identity
  • [ ] Test each server in isolation with MCP Inspector before integration
  • [ ] Wire servers into the Claude Agent SDK (or your client of choice)

Build documentation RAG, expertise, and topology

  • [ ] Index Confluence and SharePoint via LlamaIndex connectors; chunk recursively, embed with a strong model, store in pgvector
  • [ ] Set up incremental sync (webhooks or delta API polling); verify document moves are handled
  • [ ] Verify permission filtering — the agent must never surface content the requesting user can’t see
  • [ ] Add a re-ranker between retrieval and the LLM (Cohere Rerank, Voyage Rerank, or bge-reranker)
  • [ ] Write the first 3–5 skills (start with triage-playbook, database-conventions, log-search-patterns)
  • [ ] Build a minimal knowledge graph: services, databases, topics, ownership — even if it’s a YAML file initially
  • [ ] Expose graph queries as MCP tools: topology_what_depends_on, topology_owners_of
  • [ ] Seed the memory store with the last 20–30 resolved incidents — symptoms, root causes, fixes

Build the eval harness and citation contract

  • [ ] Collect 50–200 historical incidents with confirmed root causes
  • [ ] Pick an eval framework: Ragas (lightweight), Open RAG Eval (rigorous), or DeepEval (battle-tested)
  • [ ] Define metrics: faithfulness, context relevance, citation precision, confidence calibration
  • [ ] Force structured JSON output with required citation fields; reject responses without valid citations at the orchestration layer
  • [ ] Set a confidence threshold; below it, the system returns “insufficient evidence — here’s what I checked and what to look at next”
  • [ ] Wire the eval into CI so prompt, tool, or skill changes can’t ship without passing
  • [ ] Run adversarial tests: feed it incidents the data sources can’t actually explain; verify it escalates instead of fabricating

Soft launch

  • [ ] Ship the /triage command (Claude Code slash command, Slack, or CLI — pick the surface engineers actually use)
  • [ ] Report generation: structured markdown + JSON + Mermaid flow diagram per invocation, stored in reports/triage/<incident-id>/<timestamp>
  • [ ] Pilot with 3–5 friendly engineers on the chosen defect class
  • [ ] Track every output: was the hypothesis right? Were citations real? Did escalation happen when it should have?
  • [ ] Weekly review — fix the top three failure patterns, no more
  • [ ] When engineers correct the agent, capture the correction in memory and (if structural) into a skill update
  • [ ] Resist the urge to expand scope. Other teams will ask. Say “soon.”

Measure and decide

  • [ ] Compare median triage time to baseline. If you don’t have a 30%+ improvement, the issue is almost certainly retrieval quality, skills coverage, or topology gaps — not the model
  • [ ] Review the eval scores. Hallucination rate trending toward zero? Confidence well-calibrated?
  • [ ] Audit the memory store — anything stale, wrong, or potentially poisoned?
  • [ ] If foundation is solid: graduate to the next defect class. If not: do not expand scope; fix the foundation
  • [ ] Write the post-implementation review. Share the failure modes openly — they teach more than the wins

Common failure modes and how to avoid them

Failure Cause Fix
Confident wrong answers No citation contract Reject any output without source references
“Looks right but engineers don’t use it” No baseline metric Measure median triage time before launch
Slow agent loops Too many or too generic tools Namespace, scope, and limit to 5–8 tools per source
Agent re-discovers stack every incident No skills, no topology Encode procedural knowledge in skills; build a topology graph
Agent can’t find documentation it knows exists Native search exposed instead of RAG Index docs into a vector store; expose semantic search + re-ranking
RAG returns stale or contradicting pages No freshness metadata, no re-indexing Subscribe to webhooks; surface last-modified dates; flag conflicts
Agent leaks restricted documents Permissions ignored at retrieval Filter results by requesting user’s ACLs at query time
Same mistakes repeating No cross-session memory Add a memory layer; store resolved incidents and corrections
Eval scores plateau Bad retrieval, not bad model Improve tool definitions, skills, and topology coverage before tuning prompts
Security blocks rollout Asked for too much access at once Start read-only, scoped, audited; expand later
Trust collapses in month 4 Under-investment in escalation Track time-to-correct-escalation as a first-class metric
Eval set drifts from production Evals not refreshed Add new incidents to the eval set monthly; retire stale ones
Memory poisoned or corrupted No provenance tracking Audit memory entries; require provenance and access controls (see MemoryGraft)
Engineers don’t invoke the agent No structured command, just chat Ship /triage with parameters (--env, --depth, --time-window); meet engineers where they work
Agent investigates the wrong environment No env scoping in the command Make --env required or inferred-and-confirmed before tool execution
Investigation isn’t reusable or shareable CLI output only, no persistent report Generate timestamped markdown + JSON + Mermaid diagram per invocation; store in a known location
Eval set doesn’t grow Reports not feeding back into evals Treat each report as a labeled example; pipe into Ragas/DeepEval and the memory store

Lessons for product leaders

The four lessons that hold across every team I’ve seen build this:

  1. It’s an integration program, not a feature program. ~60% access plumbing, 20% evals, 15% skills/topology, 5% prompts.
  2. Evals before prompts. A held-out test set with a confidence floor is the biggest unlock, not a cleverer prompt.
  3. Skills, topology, and memory are where engineering judgment becomes durable. They turn an LLM into something that learns from your team — not just an expensive intern that forgets everything.
  4. Protect the credibility budget. Every confident wrong answer spends from a finite pool. Conservatism on day one buys ambition in year two.

Where the field is going

The convergence is visible. Cloud providers and infrastructure vendors are all shipping variants of the same six-layer pattern: grounded reasoning over enterprise sources via MCP, RAG over documentation, agent skills for procedural expertise, knowledge graphs for topology, persistent memory for learning, and eval harnesses as the safety net. The protocol matured through 2025, the open-source ecosystem around it is now substantial, and enterprise governance primitives — auth, audit, scoping — are arriving.

The teams that get the most value won’t be the ones with the best models. They’ll be the ones that treat grounding, documentation retrieval, expertise, topology, memory, and evaluation as first-class product disciplines — the same way a previous generation treated APIs, observability, and feature flags.

Days to hours isn’t the ceiling. It’s where the trust curve starts.


Reference reading

▸Sources organized by topic → expand

Foundations

Documentation RAG

Skills

Knowledge graphs

Memory

Evaluation

—–

*If you’re building or hiring for AI-grounded operations in regulated, high-asset-surface environments, I’d enjoy the conversation.*

Related Posts

Leave a Comment