An open map · v0.3 · July 2026
The tools commoditize. The judgment doesn't.
Every AI engineering roadmap teaches the same tool sequence. But teams don't fail on tools anymore. They fail on judgment: evals nobody trusts, context nobody budgets, agents nobody can stop, costs nobody owns.
This is a map of that judgment, broken into microskills: the smallest teachable units of AI-era engineering. Each one is a short page with a 20-minute kata, a picture of what good looks like, and how it's tested. Free, no accounts, no course to buy. The katas are meant to be done, not read.
Branch 01 · live
Evals
The discipline of proving an AI system works. The single most screened-for skill in 2026 hiring, and the least taught.
- 01The golden setBuild a 30-case eval set from real failureslive
- 02LLM-as-judgeCalibrate a judge before you trust itlive
- 03Failure taxonomiesName the ways it breaks before counting themlive
- 04Regression gatesEvals in CI, and when to block a shiplive
- 05Offline vs onlineWhat to measure before shipping vs afterlive
- 06Eval-driven iterationChange one thing, re-run, read the difflive
- 07Cost-aware evalsEvaluate without burning the budgetlive
- 08When metrics lieGoodhart traps in LLM evalslive
Branch 02 · live
Harness engineering
A harness is the operating system you build around a probabilistic CPU: a loop, memory, context, tools, and evals. Two of those organs are big enough to be branches of their own (01 and 03). This branch covers the rest: the system around the model.
- 01Harness anatomyLoop, memory, context, tools, evals: the OS for a probabilistic CPUlive
- 02The loopBudgets, stopping conditions, and the exit nobody designslive
- 03Tool schema designThe API your model actually seessoon
- 04Memory designWhat to remember, what to forget, who corrects itsoon
- 05Human-in-the-loop placementWhere approval gates earn their latencysoon
- 06State & persistenceAgents that survive a restartsoon
- 07The org chart is the promptMulti-agent structure is context designsoon
Branches 03–08 · planned
The rest of the map
Named now so the shape of the discipline is visible. Built one branch at a time, in the open.
03 · Context engineering
Context budgeting · retrieval quality vs quantity · caching economics · structured context · context rot
04 · Guardrails & trust
Tool authorization · spend caps · sandboxing · audit trails · OWASP LLM risks · failure containment
05 · AI-assisted development
Reviewing AI-written code · codegen prompt patterns · when to hand-write · test discipline · cognitive debt
06 · Cost & performance
Model selection · routing & fallbacks · caching strategy · latency budgets · per-task unit economics
07 · Judgment calls
Build vs buy vs wait · when not to use AI · the escalation ladder · sunset criteria for AI features
08 · The agent-design interview
For hirers: probing eval literacy, context judgment, and guardrail thinking. Rubrics that beat leetcode-for-prompts.
Why this exists
AI can write the code. It can't own the outcome.
I'm Vishal Kannankara. I've spent two decades in production engineering: distributed systems, payments at $1B+/month, and the teams that run them. In the last few years I shipped production RAG and agent systems and led 200+ engineers through AI-assisted development, before most teams had a vocabulary for any of it. This map is the vocabulary I wish existed. I write the longer arguments on Vishal's Archive.