Stop Your AI Agent from Hallucinating Root Causes: The 6-Layer Architecture That Works
A PM’s solution discovery on why naive AI fails at root cause analysis in large systems — and a follow-able blueprint for fixing it.
TL;DR
- The problem. Naive AI on a large enterprise stack hallucinates root causes 15–52% of the time. That destroys engineering trust faster than it saves time.
- What works. Six layers: scoped MCP tools, documentation RAG, agent skills, a topology graph, persistent memory, and a citation contract with evals. Meta’s published version of this pattern hits 42% RCA accuracy on first pass.
- The operator interface. A
/triagecommand with explicit parameters (--env,--depth,--time-window) that produces a structured report — symptom, hypotheses passed and failed, root cause, flow diagram, recommended action.- The PM lesson. This is an integration program, not a feature program. ~60% access plumbing, 20% evals, 15% skills/topology, 5% prompts. Anyone telling you otherwise is selling something.
Contents
- Why this post exists
- The problem is real, and it’s measurable
- Why naive AI fails on a large enterprise stack
- What actually works: the public benchmark
- The architecture: six layers
- Concrete commands and behaviors to encode
- The
/triagecommand and report - Rollout checklist
- Common failure modes
- Lessons for product leaders
Architecture at a glance
/triage INC-4521 --env prod --depth deepparses args · enforces scope
markdown · JSON · Mermaid
5-stage workflow
Splunk · Git · DB · K8s · Kafka
chunked · embedded · re-ranked
procedural knowledge
services · deps · owners
+ corrections
+ eval harness
Why this post exists
If you’ve tried to point a capable LLM at your enterprise stack to speed up defect triage, you’ve probably watched it confidently invent table names, fabricate commit references, and send engineers down dead-end investigations. That failure mode is the central problem of enterprise AI in 2026, and it’s solvable.
This is a working blueprint: the metrics that prove the problem is real, the architectural pattern that fixes it, and the specific tools, libraries, and checklists you can use to reproduce the result.
Reading paths. If you have ten minutes: TL;DR → architecture diagram → /triage command. If you’re going to build this: skip to the checklist and use the rest as reference.
The problem is real, and it’s measurable
The short version: Investigation eats most of MTTR. Outages are expensive. AI-grounded systems measurably help. Adoption isn’t the bottleneck — trust is.
A few numbers worth carrying into any conversation with leadership:
- Investigation eats most of MTTR. For most teams, the investigation and root cause analysis phase consumes the largest share of mean time to resolve — not the fix itself. (Cyberhaven, 2026)
- Most outages cost at least $100,000; severe ones regularly exceed $1 million. (Uptime Institute, 2025, via New Relic)
- AI-grounded incident response shortens breach lifecycles by 80 days and saves $1.9M per incident for organizations that use it extensively. (IBM Cost of a Data Breach 2025)
- Mature, AI-augmented detection cuts MTTR 30–40% vs. signature-based approaches. (Palo Alto Networks)
- 57% of organizations now have AI agents in production, and 32% cite quality as the top barrier to deployment. (LangChain State of AI Agents 2026)
That last one is the crux. Adoption is no longer the bottleneck — trust is. And trust is what naive AI burns.
Why naive AI fails on a large enterprise stack
The short version: Models hallucinate 15–52% of the time without grounding, under 2% with it. The architecture is the entire game.
The hallucination rates are not subtle:
- Frontier models still hallucinate 15–52% of the time on structured analysis tasks, depending on benchmark and conditions. (SQ Magazine, 2026)
- AI-generated code contains security vulnerabilities 29–45% of the time, and roughly 20% of package recommendations point to libraries that don’t exist. (diffray, 2026)
- Citation fabrication rates reach 94% in adversarial testing. (SQ Magazine, 2026)
- In retrieval-grounded tasks, OpenAI’s own evals show hallucination rates drop below 2%. (SQ Magazine, 2026)
Ungrounded models hallucinate at rates that destroy operational trust. Grounded models don’t.
The mechanics are well understood. Frontier context windows top out around a million tokens; a real enterprise stack — hundreds of repos, hundreds of tables, hundreds of Kafka topics, runbooks across Confluence and SharePoint — is several million tokens before you’ve loaded the relevant logs. Even within the context window, attention degrades in the middle of long inputs (the “lost in the middle” problem), so models miss evidence you fed them. Without a tool layer to fetch authoritative data on demand, the model defaults to plausible-sounding generation.
The first time engineers chase a hallucinated RCA, they lose two hours. The second time, they lose trust. The third time, they stop using the system. That is how enterprise AI initiatives die.
What actually works: the public benchmark
The short version: Meta published the canonical pattern — heuristic retrieval feeding an LLM ranker, confidence-gated, with explicit suppression of low-confidence answers. 42% first-pass RCA accuracy.
The pattern that works is not theoretical. Meta published it (Engineering at Meta, 2024), and the numbers are the right anchor for any program:
- 42% accuracy in identifying root causes at investigation creation time for incidents in their web monorepo.
- Achieved by combining heuristic-based retrieval (using code ownership, directory structure, and runtime code graphs to reduce thousands of changes to a few hundred) with an LLM-based ranker (a fine-tuned Llama 2 7B reducing hundreds to the top five).
- Results presented with explicit confidence measurements, with low-confidence recommendations suppressed rather than shown.
Meta’s stack is unique, but the architectural primitives are not: retrieval first, ranking second, confidence-gated output, and a hard rule that low confidence escalates rather than guesses. Vendor systems built on the same pattern (NeuBird Hawkeye, Rootly) report MTTR reductions of 40–90% in production. The 90% number is vendor marketing; treat it as a ceiling, not a floor. The 30–40% range from independent analysis is the realistic operating zone.
The architecture: six layers
The short version: Most teams stop after MCP + evals. That’s the demo, not the production system. The full stack adds documentation RAG, agent skills, a topology graph, and persistent memory.
Most teams stop after they wire up MCP servers and an eval harness. That’s where the impressive demos live and where production systems quietly fail. MCP gives the agent transport — the ability to fetch evidence. It does not give the agent expertise. Without explicit procedural knowledge, indexed documentation, infrastructure mapping, and persistent memory, the agent rediscovers your stack from scratch on every invocation, wastes turns on dead-end queries, and re-makes the same mistakes a human engineer would have learned from once.
The full stack has six layers, ordered roughly by what to build first:
Skip layers 2–5 and you have a fast retriever with no judgment. Build all six and you have a system engineers actually use in month six.
Layer 1 — Grounding through scoped MCP servers
Build one MCP server per authoritative source, each exposing 3–6 narrowly scoped tools. Generic “query everything” tools are the dominant agent failure mode at scale.
Layer 2 — Documentation RAG over Confluence, SharePoint, and runbooks
Native enterprise search returns dozens of weakly-relevant pages. RAG with semantic chunking and re-ranking lets the agent find the right page in one call — with citations. Permission-aware sync is non-negotiable.
Layer 3 — Expertise via Agent Skills
This is the layer most teams miss. Agent Skills are folders of procedural knowledge (“when triaging consumer-lag, first check brokers, not DBs”). Loaded dynamically; small context footprint; the right place for everything senior engineers carry in their heads.
Layer 4 — Topology via a knowledge graph
The agent needs to know what depends on what, who owns what, what changed where. A knowledge graph answers those questions in milliseconds. GraphRAG patterns lift retrieval accuracy ~15–30% over vector-only.
Layer 5 — Memory: don’t let the agent re-learn every incident
Without persistent memory, every incident starts from zero. 23% better outcomes after three months of accumulated learning in published deployments. Memory is also an attack surface — provenance tracking is mandatory.
Layer 6 — Guardrails: citations and evals
Two simple rules: no citation, no claim, and every change ships through the eval set. These are the single most effective hallucination controls — they make lying require effort and regression-test it on every change.
Concrete commands and behaviors to encode
Explicit instructions in skills beat implicit hopes the model will get it right. Below: querying, log search, test execution, and escalation rules — examples to start from.
The /triage command: turning the agent into an operator interface
A chat box works for a demo. In production, ship a structured
/triagecommand with explicit parameters (--env,--depth,--time-window) that runs a five-stage workflow and produces a persistent report. The command is the surface that determines adoption.
Everything above describes capability. This is how engineers actually invoke it.
A chat box where engineers type “investigate INC-4521” works for a demo. In production, you want a structured command — one that takes the incident as a parameter, runs the full triage workflow with validated arguments and explicit options, and produces a deterministic, auditable artifact. This is the surface that determines adoption.
Implementation paths. A triage command can ship as:
- A Claude Code slash command (a markdown file in
.claude/commands/triage.md) for engineers working in the IDE - A Slack slash command (
/triage INC-4521) for on-call rotation - A CLI (
triage INC-4521 --env prod) for automation and scripting - A CI/CD step that runs on every Sev-2+ alert and posts findings to the incident channel
All four wrap the same underlying skill. Build the skill once; expose it through the surfaces your team actually uses.
What the command does
/triage <incident-id-or-description>
[--env dev|staging|perf|prod-readonly]
[--depth quick|standard|deep]
[--time-window 15m|1h|6h|24h]
[--scope service-name|repo-name]
[--skip-validation]
[--dry-run]
Parameters explained:
| Parameter | Purpose | Default |
|---|---|---|
incident-id-or-description |
Either a ticket ID (INC-4521, JIRA-1234) the agent looks up, or a free-text description | required |
--env |
Which environment to investigate. Constrains tool scopes (a prod investigation should never read dev logs) | inferred from incident, else prompt |
--depth |
quick (5 min, single hypothesis), standard (15 min, top 3 hypotheses), deep (45 min, full RCA with timeline reconstruction) |
standard |
--time-window |
How far back to search logs and deployments | 1h |
--scope |
Restrict investigation to a service or repo when the engineer already has a hypothesis | none |
--skip-validation |
Bypass the validation step (for cases where the engineer has already confirmed the symptoms) | false |
--dry-run |
Show the plan without executing tools — useful for reviewing what the agent is about to do | false |
The triage workflow the command runs
Every invocation follows the same five-stage workflow, encoded in the triage-playbook skill:
1. Intake and validation
- Parse the incident ID or description; if an ID, fetch it from Jira/ServiceNow/PagerDuty
- Extract structured facts: affected service(s), symptom, first-seen timestamp, environment, severity
- Validate that the symptom is reproducible or observable in the named environment (e.g., does the error actually appear in logs in the time window?)
- If validation fails, return early: “I cannot reproduce this symptom in
prodlogs between 14:00–15:00 UTC. Please confirm the time window or environment.” - Cross-check against the memory store for known false-lead patterns: “Symptom matches dead-end #47 — last 3 incidents with this signature were caused by X, not Y. Skipping that hypothesis.”
2. Hypothesis generation
- Query the topology graph for blast radius: what services depend on the affected one?
- Query memory for similar past incidents and their resolutions
- Query documentation RAG for relevant runbooks
- Produce a ranked list of hypotheses with prior probabilities (think Meta’s heuristic-based retrieval feeding the LLM ranker)
3. Evidence gathering
- For each top-N hypothesis (N = 3 for
standard, 5 fordeep), execute a small batch of scoped tool calls in parallel - Logs, recent deployments, schema changes, consumer-lag, runbook lookups — all bounded by the
--envand--time-windowparameters - Every fact retrieved is tagged with its source for the citation contract
4. RCA synthesis (with optional one-level-deeper analysis)
- For each hypothesis, summarize: supporting evidence, contradicting evidence, confidence
- If
--depth=deepis set, drill one level further on the highest-confidence hypothesis: reconstruct the timeline, identify the introducing change, propose a fix and a verification step - If two hypotheses are within 10% confidence of each other, run an adversarial comparison: “Hypothesis A predicts log pattern X; hypothesis B predicts log pattern Y. Check which is present.”
5. Output and escalation
- Structured report: the ranked hypotheses, their evidence with citations, a recommended next action, and a confidence score
- If top-confidence < 0.6, escalate explicitly with “what I checked, what I couldn’t determine, what to look at next”
- If touching a sensitive path (billing, payments, auth), flag for human review before any further action
- Write the investigation to memory for future incidents to learn from
Sample output
$ /triage INC-4521 --env prod-readonly --depth standard --time-window 1h
📋 INC-4521: "Checkout API returning 503s for ~5% of requests since 14:32 UTC"
Affected: checkout-service (prod)
Severity: Sev-2
✅ Validation passed
- 503 errors confirmed in prod logs (623 events, 14:32–15:32 UTC)
- Error rate: 4.8% (within reported range)
- No matching dead-end pattern in memory
🔍 Top hypotheses (3 of 8 considered)
1. Database connection pool exhaustion (confidence: 0.78)
├─ Evidence: 47 "connection timeout" errors at db-checkout-1 starting 14:31 UTC
│ [splunk: 2026-05-09T14:31:00Z to 14:35:00Z]
├─ Evidence: connection pool size=20, active=20, queue depth=140
│ [k8s: checkout-service-7d8f, metrics endpoint]
├─ Topology: db-checkout-1 also serves cart-service (no errors there)
└─ Suggests: pool sizing, not DB capacity
2. Recent deployment regression (confidence: 0.41)
├─ Deploy at 14:28 UTC: checkout-service v2.14.3 → v2.14.4
│ [gitlab: commit a3f9c12, "refactor: connection lifecycle"]
└─ ⚠ Touches connection handling — relevant to hypothesis #1
3. Upstream payment provider degradation (confidence: 0.18)
└─ No supporting evidence; eliminated by /payments_health check
📌 Recommended next action
Investigate hypothesis #1 ∩ #2: deploy a3f9c12 likely changed connection
lifecycle, exposing pool exhaustion under load. Roll back or increase pool.
Verification: deploy v2.14.3 to one canary pod; observe error rate.
📚 Citations: 7 evidence references, 2 runbooks (RUN-checkout-23, RUN-db-pool-7)
🧠 Written to incident memory for future similar patterns.
📄 Full report: ./reports/triage/INC-4521/2026-05-09T15-32Z.md
The triage report: persistent artifact, not just CLI output
The condensed CLI summary above is what the engineer reads in 30 seconds. The report is what they share with the team, attach to the incident ticket, hand off at shift change, and reference six months later when a similar incident recurs.
Every /triage invocation writes a structured markdown report to a known location:
./reports/triage/<incident-id>/<timestamp>.md # primary artifact
./reports/triage/<incident-id>/<timestamp>.json # machine-readable copy
./reports/triage/<incident-id>/<timestamp>.flow.mmd # Mermaid flow diagram
The directory structure matters: one folder per incident, multiple timestamped reports per folder. A long incident generates several reports as new evidence comes in; the chronology stays intact and replayable.
For Slack-invoked triage, the report is also attached to the thread as a markdown file. For CI-invoked triage, it’s posted to the incident channel and stored in object storage with a link. Same report, different surfaces.
What goes in the report
The report follows a fixed template — every incident, every time. Engineers learn it once and can scan any report in 60 seconds.
The nine sections, in order:
- Symptom — what’s happening, where, magnitude
- Incident description — customer-facing impact, severity
- Investigation timeline — table of events with timestamps
- Hypotheses considered — passed (✅) and failed (❌), each with status, supporting evidence, and explicit reasoning. The most underrated part.
- Root cause analysis — primary cause, contributing factors, why it wasn’t caught earlier
- Behavior and scenario flow — Mermaid sequence diagram
- Recommended next action — sequenced (immediate → short-term → medium-term → process)
- References and citations — every claim traceable
- Confidence and limitations — what the agent doesn’t know; what humans must verify
Why the report has this shape
Each section earns its place:
- Symptom and description — what an engineer needs to brief a manager in 30 seconds
- Investigation timeline — the audit trail; satisfies post-incident review and compliance needs
- Hypotheses (passed and failed) with reasoning — the most important section. Showing failed hypotheses with explicit reasons builds trust faster than any other element. It also lets the next engineer skip the dead ends.
- Root cause analysis — the primary cause, contributing factors, and why it wasn’t caught earlier (the question every post-mortem asks)
- Flow diagram — engineers process diagrams faster than prose for sequence-of-events failures. Mermaid is the right format because it’s text-based, version-controllable, and renders in GitHub, GitLab, Confluence, and most modern markdown viewers
- Recommended next action — sequenced by urgency, with verification steps
- References and citations — every claim is traceable
- Confidence and limitations — what the agent does not know, and what a human must verify before acting
Implementation notes
- Generate the report from a Jinja2 template — the agent fills sections; the template enforces structure. This keeps reports comparable across incidents.
- Store reports in version control or object storage, not just on the engineer’s laptop. A shared
reports/triage/directory in a dedicated git repo works well; so does S3 with a known prefix. - The JSON copy is for tooling, not humans — it feeds the eval harness, the memory store, and any downstream dashboards.
- The Mermaid diagram is generated by the agent as part of the workflow; it’s part of the skill’s instructions, not a manual step. Anthropic’s pre-built
mermaidskill or a small custom skill can handle this. - Reports are write-once. If new evidence arrives, generate a new timestamped report — don’t edit the old one. This preserves the chronology and prevents accidental rewrites of history.
Why this matters for the program, not just the engineer
The report is also the single most valuable input to your eval harness and your memory store. Every triage report is a labeled training example: symptom, hypotheses, evidence, outcome. Two months in, you have hundreds of these. They become:
- The eval set that gates every prompt and skill change
- The memory the agent retrieves on similar future incidents
- The corpus a fine-tuned ranker can train on (the Meta pattern)
- The audit trail for regulators and post-incident reviewers
Build the report well on day one and the rest of the program compounds for free.
Why this shape of command works
- The engineer stays in control. The agent investigates and recommends; the engineer decides whether to roll back, scale, or escalate. The
--dry-runand confidence-gated escalation make it safe to invoke without ceremony. - The output is auditable. Every claim has a citation. Every confidence score is calibrated against the eval set. The whole investigation can be replayed from the memory store.
- The parameters constrain hallucination space. An agent told to investigate
prod-readonlyin a1hwindow cannot wander off into dev logs from last week. Scope is enforced by the command, not hoped-for from the prompt. - The workflow is the same every time. Engineers don’t have to remember which skills to invoke or which tools to call. The command is the skill.
Variants worth building
Once the base /triage works, the same pattern produces useful siblings:
/postmortem <incident-id>— runs after resolution, drafts the post-mortem document with timeline reconstruction, contributing factors, and action items. Writes back to Confluence with a citation trail./blast-radius <service-or-commit>— uses the topology graph to enumerate downstream impact before a deploy or change./runbook-find <symptom>— pure documentation RAG; returns the top runbooks for a symptom without doing the full investigation./regression-check <incident-id>— runs against the eval set: “have we seen this before? Did the previous fix introduce this?”/triage --learn-from <incident-id>— explicit memory write: capture this incident’s resolution as a pattern for future agents.
Where to put the command file
For Claude Code, the simplest path is a markdown file at .claude/commands/triage.md:
---
name: triage
description: Run a full triage workflow on an incident
---
# /triage <incident-id-or-description> [options]
Invoke the `triage-playbook` skill with the following arguments:
- incident: $1
- environment: ${env:-inferred}
- depth: ${depth:-standard}
- time_window: ${time-window:-1h}
- scope: ${scope:-none}
- skip_validation: ${skip-validation:-false}
- dry_run: ${dry-run:-false}
Follow the five-stage workflow defined in the skill. Output the structured
report in the format specified in the skill examples.
The command is intentionally thin — it’s a parameter parser. The actual logic lives in the skill, which is the right place because skills are versioned, evaluable, and reusable across the Slack bot, the CLI, and the CI pipeline.
Rollout checklist
Order matters. Don’t skip steps to ship faster — the trust budget is the binding constraint. Six phases. Each builds on the last.
| Phase | Focus |
|---|---|
| 1 | Define and measure |
| 2 | Grounding layer (MCP) |
| 3 | Docs RAG, skills, topology |
| 4 | Eval harness + citation contract |
| 5 | Soft launch |
| 6 | Measure and decide |
Define and measure
- [ ] Pick one defect class to start (recommend: internal tooling, batch jobs, or dev-environment issues — not customer-impacting)
- [ ] Document current median triage time for that class — this is your baseline
- [ ] Define your metric stack: North Star, guardrails, safety counter-metric, escalation metric
- [ ] Get explicit success criteria signed off by engineering leadership before coding
- [ ] Identify the 3–5 most-used data sources for that defect class — start there, not with all of them
Build the grounding layer
- [ ] Stand up one MCP server per source using FastMCP (Python) or MCP-Framework (TypeScript)
- [ ] Each server exposes 3–6 narrowly scoped, namespaced tools — not one generic query tool
- [ ] All access is read-only; auth via short-lived tokens, scoped to the agent’s identity
- [ ] Test each server in isolation with MCP Inspector before integration
- [ ] Wire servers into the Claude Agent SDK (or your client of choice)
Build documentation RAG, expertise, and topology
- [ ] Index Confluence and SharePoint via LlamaIndex connectors; chunk recursively, embed with a strong model, store in pgvector
- [ ] Set up incremental sync (webhooks or delta API polling); verify document moves are handled
- [ ] Verify permission filtering — the agent must never surface content the requesting user can’t see
- [ ] Add a re-ranker between retrieval and the LLM (Cohere Rerank, Voyage Rerank, or
bge-reranker) - [ ] Write the first 3–5 skills (start with
triage-playbook,database-conventions,log-search-patterns) - [ ] Build a minimal knowledge graph: services, databases, topics, ownership — even if it’s a YAML file initially
- [ ] Expose graph queries as MCP tools:
topology_what_depends_on,topology_owners_of - [ ] Seed the memory store with the last 20–30 resolved incidents — symptoms, root causes, fixes
Build the eval harness and citation contract
- [ ] Collect 50–200 historical incidents with confirmed root causes
- [ ] Pick an eval framework: Ragas (lightweight), Open RAG Eval (rigorous), or DeepEval (battle-tested)
- [ ] Define metrics: faithfulness, context relevance, citation precision, confidence calibration
- [ ] Force structured JSON output with required citation fields; reject responses without valid citations at the orchestration layer
- [ ] Set a confidence threshold; below it, the system returns “insufficient evidence — here’s what I checked and what to look at next”
- [ ] Wire the eval into CI so prompt, tool, or skill changes can’t ship without passing
- [ ] Run adversarial tests: feed it incidents the data sources can’t actually explain; verify it escalates instead of fabricating
Soft launch
- [ ] Ship the
/triagecommand (Claude Code slash command, Slack, or CLI — pick the surface engineers actually use) - [ ] Report generation: structured markdown + JSON + Mermaid flow diagram per invocation, stored in
reports/triage/<incident-id>/<timestamp> - [ ] Pilot with 3–5 friendly engineers on the chosen defect class
- [ ] Track every output: was the hypothesis right? Were citations real? Did escalation happen when it should have?
- [ ] Weekly review — fix the top three failure patterns, no more
- [ ] When engineers correct the agent, capture the correction in memory and (if structural) into a skill update
- [ ] Resist the urge to expand scope. Other teams will ask. Say “soon.”
Measure and decide
- [ ] Compare median triage time to baseline. If you don’t have a 30%+ improvement, the issue is almost certainly retrieval quality, skills coverage, or topology gaps — not the model
- [ ] Review the eval scores. Hallucination rate trending toward zero? Confidence well-calibrated?
- [ ] Audit the memory store — anything stale, wrong, or potentially poisoned?
- [ ] If foundation is solid: graduate to the next defect class. If not: do not expand scope; fix the foundation
- [ ] Write the post-implementation review. Share the failure modes openly — they teach more than the wins
Common failure modes and how to avoid them
| Failure | Cause | Fix |
|---|---|---|
| Confident wrong answers | No citation contract | Reject any output without source references |
| “Looks right but engineers don’t use it” | No baseline metric | Measure median triage time before launch |
| Slow agent loops | Too many or too generic tools | Namespace, scope, and limit to 5–8 tools per source |
| Agent re-discovers stack every incident | No skills, no topology | Encode procedural knowledge in skills; build a topology graph |
| Agent can’t find documentation it knows exists | Native search exposed instead of RAG | Index docs into a vector store; expose semantic search + re-ranking |
| RAG returns stale or contradicting pages | No freshness metadata, no re-indexing | Subscribe to webhooks; surface last-modified dates; flag conflicts |
| Agent leaks restricted documents | Permissions ignored at retrieval | Filter results by requesting user’s ACLs at query time |
| Same mistakes repeating | No cross-session memory | Add a memory layer; store resolved incidents and corrections |
| Eval scores plateau | Bad retrieval, not bad model | Improve tool definitions, skills, and topology coverage before tuning prompts |
| Security blocks rollout | Asked for too much access at once | Start read-only, scoped, audited; expand later |
| Trust collapses in month 4 | Under-investment in escalation | Track time-to-correct-escalation as a first-class metric |
| Eval set drifts from production | Evals not refreshed | Add new incidents to the eval set monthly; retire stale ones |
| Memory poisoned or corrupted | No provenance tracking | Audit memory entries; require provenance and access controls (see MemoryGraft) |
| Engineers don’t invoke the agent | No structured command, just chat | Ship /triage with parameters (--env, --depth, --time-window); meet engineers where they work |
| Agent investigates the wrong environment | No env scoping in the command | Make --env required or inferred-and-confirmed before tool execution |
| Investigation isn’t reusable or shareable | CLI output only, no persistent report | Generate timestamped markdown + JSON + Mermaid diagram per invocation; store in a known location |
| Eval set doesn’t grow | Reports not feeding back into evals | Treat each report as a labeled example; pipe into Ragas/DeepEval and the memory store |
Lessons for product leaders
The four lessons that hold across every team I’ve seen build this:
- It’s an integration program, not a feature program. ~60% access plumbing, 20% evals, 15% skills/topology, 5% prompts.
- Evals before prompts. A held-out test set with a confidence floor is the biggest unlock, not a cleverer prompt.
- Skills, topology, and memory are where engineering judgment becomes durable. They turn an LLM into something that learns from your team — not just an expensive intern that forgets everything.
- Protect the credibility budget. Every confident wrong answer spends from a finite pool. Conservatism on day one buys ambition in year two.
Where the field is going
The convergence is visible. Cloud providers and infrastructure vendors are all shipping variants of the same six-layer pattern: grounded reasoning over enterprise sources via MCP, RAG over documentation, agent skills for procedural expertise, knowledge graphs for topology, persistent memory for learning, and eval harnesses as the safety net. The protocol matured through 2025, the open-source ecosystem around it is now substantial, and enterprise governance primitives — auth, audit, scoping — are arriving.
The teams that get the most value won’t be the ones with the best models. They’ll be the ones that treat grounding, documentation retrieval, expertise, topology, memory, and evaluation as first-class product disciplines — the same way a previous generation treated APIs, observability, and feature flags.
Days to hours isn’t the ceiling. It’s where the trust curve starts.
Reference reading
—–
*If you’re building or hiring for AI-grounded operations in regulated, high-asset-surface environments, I’d enjoy the conversation.*
