The AI Agents Stack Summary (2026)
The zoom-out note: the six infrastructure layers between “a model in a while loop” and an agent you can put in production. The other notes in this sequence each go deep on one slice; this one is the map that shows where each slice sits and — just as important — which layers a given project can skip.
Read LLM Tool Calling and Building-an-AI-Agent-from-Scratch first: the think-act-observe loop they teach is the atomic unit, and everything in this note is infrastructure wrapped around that loop to make it reliable at scale. The layer model here follows Paolo Perrone’s 2026 stack article (all statistics below are from it and its cited sources), which redraws Letta’s widely-shared 2024 diagram.
Related: Model-Context-Protocol-MCP, A2A-Protocol, LangGraph vs CrewAI Research, Frameworks, runtimes, and harnesses, Deep Agents, NAT-Lesson-0-Getting-Started.
1. Why think in layers at all
The same reason network engineers think in OSI layers: when something breaks, you debug by layer. “The agent gave a wrong answer” is as useless a bug report as “the internet is down” — is it the model (layer 1), a poisoned tool description (layer 2), the right fact evicted from context (layer 3), a retry loop that gave up (layer 4)? A layer map turns one vague failure into a small set of testable hypotheses, exactly like ping → traceroute → curl walks you up the network stack.
The map also protects you from the opposite failure: over-building. Perrone opens with a team three weeks into a 14-node LangGraph graph, custom Redis checkpointing, and retry logic — for a refund-question chatbot that calls one API and could have shipped as a 50-line script with two MCP servers. Nobody had asked which layers the problem actually needed. The operating rule for the whole stack: add a layer when something specific breaks, not before.
Three questions score every layer, and they’re worth memorizing because they apply to any infrastructure choice:
- How much state does this layer make you manage? Stateless layers are easy; the stack’s pain concentrates where state lives.
- How much vendor lock-in does this choice create? Open standards port; orchestration code and proprietary memory stores do not.
- How big is the demo-to-production gap? Some layers work in production exactly as they demo; others (eval, guardrails) barely exist in demos at all.
2. The six layers
| # | Layer | In one sentence | Deep-dive note |
|---|---|---|---|
| 1 | Models & inference | How the model runs: closed API, managed open-weight, or self-hosted | — |
| 2 | Protocols & tools | How the agent reaches tools and other agents: MCP, A2A | Model-Context-Protocol-MCP, A2A-Protocol |
| 3 | Memory & knowledge | What the agent knows: in-context state, retrieval, persistent memory | Deep Agents |
| 4 | Frameworks & SDKs | How the loop, tools, and control flow are wired together | Frameworks, runtimes, and harnesses, LangGraph vs CrewAI Research |
| 5 | Eval & observability | How you know the agent works, and catch it when it stops | — |
| 6 | Guardrails & safety | How you stop it doing what it shouldn’t | — |
Layer 1 — Models & inference
The engine choice, and the layer that’s commoditizing fastest. What changed since 2024: reasoning models (o-series, DeepSeek R1, Claude extended thinking) let a single call solve what used to need a hand-built multi-step chain, and open-weight models (Llama 3.3, DeepSeek V3, Qwen 2.5) closed enough of the quality gap that “always use the biggest closed model” stopped being default advice. The emerging pattern is prototype on closed-source, deploy on open-weight.
Think of this layer like buying compute: once parts are instruction-set compatible, you stop arguing about which is “smartest” and start comparing price and latency. That’s where models are heading — the real decision is cost/latency, and the OpenAI-compatible API shape (the ABI from Building-an-AI-Agent-from-Scratch §4) is what makes the vendors swappable. Scores: no state to manage, high lock-in for closed APIs (prompts and evals are tuned per model), the smallest demo-to-prod gap in the stack. Self-host only when call volume makes API pricing untenable or you need latency an API round-trip can’t deliver.
Layer 2 — Protocols & tools
In 2024 this layer didn’t exist as a category — every framework had its own tool-definition schema. Then MCP standardized the wire format, and the layer went the way of the network protocol wars: once TCP/IP won over IPX and SNA, the interesting work stopped being “which protocol” and became “how do we secure it.” Perrone’s markers of the win: 97M monthly SDK downloads, adoption by OpenAI, Google, and Microsoft, donation to the Linux Foundation (December 2025). Model-Context-Protocol-MCP covers the mechanics.
Security is the open problem, and the numbers are ugly: Invariant Labs’ MCPTox benchmark got an 84.2% tool-poisoning success rate against agents with auto-approval enabled, and Endor Labs’ scan of 2,614 MCP servers found 82% prone to path traversal and 67% to code injection. This is the “tool results are untrusted input” warning from LLM Tool Calling §7 measured at ecosystem scale.
Agent-to-agent communication lives here too — Google’s A2A and IBM’s ACP — but neither has reached critical mass, so multi-agent coordination today is mostly built by hand at layer 4. A2A-Protocol explains what these protocols add over plain tool calls.
Layer 3 — Memory & knowledge
In 2024, “memory” meant “pick a vector database and do RAG.” In 2026 it’s a first-class design problem with three tiers, and the tiers are exactly the memory hierarchy you already know: in-context state is RAM (the context window — fast, small, wiped between processes), retrieval is demand paging (vector or graph search pulls the needed page off disk into RAM at call time), and persistent memory is the filesystem (state that survives across sessions). Bigger context windows (1M+ tokens) didn’t kill the hierarchy any more than bigger RAM killed filesystems — they just moved the swap threshold.
What changed: “context engineering” replaced “prompt engineering” as the discipline — architecting what the model sees on every call rather than wordsmithing one prompt. Memory blocks (named in-context fields the agent itself reads and overwrites) made state agent-managed. pgvector became the default retrieval store — it’s just Postgres with an extension — and Graph RAG (Neo4j) added relationship-following as an alternative to embedding similarity.
This is the highest-state, highest-complexity layer, and the honest take is that most teams overcomplicate it: start with conversation history in Postgres plus a structured system prompt; add vector search when history outgrows the context window; add agentic memory management only when the agent must learn across sessions. The harness-side view of the same problem — compression, filesystem-as-swap, subagents — is Deep Agents.
Layer 4 — Frameworks & SDKs
Three camps now, where 2024 had only LangChain: provider SDKs (OpenAI Agents SDK, Google ADK — fastest to start, locked to one vendor’s models and orchestration), graph-based frameworks (LangGraph — portable across models, heavy setup, explicit state machines), and no framework — the thin-wrapper-plus-MCP camp, which is Building-an-AI-Agent-from-Scratch grown up. One recurring confusion worth flagging: LangChain is the integration layer (connectors, prompts, tool calling); LangGraph is the orchestration engine where agent logic lives — most production teams use both.
Lock-in is the highest in the stack: orchestration code doesn’t port, and a LangGraph agent rewritten for CrewAI is a new codebase. MCP is the one investment that transfers across all three camps. The tier vocabulary for this layer is Frameworks, runtimes, and harnesses; the paradigm comparison is LangGraph vs CrewAI Research. The honest take matches both: most teams pick too much framework for what they’re building.
Layer 5 — Eval & observability
The layer that barely existed in 2024 and is now where production quality dies. LangChain’s State of Agent Engineering survey: 89% of teams with production agents have observability, only 52% have evals. That 37-point gap is running production with syslog but no test suite — you can watch the crash in beautiful traces, but nothing stops the regression from shipping, and users find the failures for you.
The converging shape is three-tier: fast checks on every PR (did the agent call the right tools?), nightly regression suites with an LLM judging output quality, and continuous production monitoring for drift. And evals must score steps, not just final output: if the agent picked the wrong tool at step 3 of 12, steps 4–12 were doomed — checking only the final answer is checking only the last exit code of a pipeline with no pipefail. This layer has the biggest demo-to-prod gap of all, for the simple reason that most prototypes have zero eval.
Layer 6 — Guardrails & safety
Agent guardrails split from LLM guardrails as a discipline: filtering text in and out was enough when models only talked, but an agent calls tools, spends money, and acts. The pattern teams learned the hard way is guardrails before action — authorization enforced at the tool-execution layer, not the output layer. It’s the difference between seccomp at the syscall boundary and grepping the audit log afterward: by the time you filter the response, the agent already sent the email. OWASP’s MCP Top 10 (December 2025) is the first real security checklist for tool-connected agents.
This is the least mature layer — no dominant framework, no established patterns, mostly policy code you write yourself (NVIDIA’s NeMo Guardrails is the closest thing to a framework, and you’ll still write most rules). The demo-to-prod gap is effectively infinite: your demo has no guardrails because nobody is trying to break it. Production will be tried. Guardrail propagation — policies following a task as agents delegate to other agents — is unsolved; if you’re doing multi-agent delegation (A2A-Protocol), you’re building that yourself.
3. What are you building? (the question that picks your layers)
The agent type determines which layers you invest in — a tool-calling chatbot and a multi-agent research system share almost no infrastructure:
| Agent type | Example | Stack | Watch out for |
|---|---|---|---|
| Stateless tool caller | look up an order, answer from a knowledge base | Provider SDK + MCP + Postgres | Nothing — this is a weekend project; don’t add a framework |
| Multi-step workflow | process a refund end-to-end, triage tickets | Framework (e.g. LangGraph) + MCP + eval | Silent breakage — build evals before deploying |
| Agent that learns | remembers preferences, improves on your codebase over weeks | Memory-first architecture + retrieval + eval | Deciding what to remember/forget; old context polluting new answers |
| Multi-agent system | specialists delegating and running in parallel | The full stack | Two agents passing context is already hard to debug; five is impossible without per-handoff traces |
Coding agents (Claude Code, Cursor, Codex) are the proof that all six layers can work together in one product: routed inference (1), MCP into editors/terminals/git (2), codebase-aware retrieval (3), purpose-built orchestration rather than any framework (4), eval running continuously in production — Cursor retrains its suggestion-acceptance model every 90 minutes on live accept/reject data (5), and sandboxed execution containing what the agent can touch (6).
4. The prediction to hold loosely
Perrone’s closing claim: the stack will collapse. Provider SDKs are already absorbing memory, tool calling, and basic eval into single APIs, and by 2027 most teams will take an opinionated stack from their model provider — fine for ~80% of use cases, with custom builds surviving at scale where defaults break. Even if that lands, the layer map keeps its value for the same reason OSI outlived the protocols it described: when the opinionated stack fails in production, you still have to know which layer failed.
5. A second taxonomy, and what it adds
A second treatment cuts the same territory a different way: Neo Kim and Paul Hoekstra, Agentic Engineering, The System Design Newsletter, 2026-06-17 — 30 concepts in six groupings. Worth holding alongside the layer map, because where two independent taxonomies disagree is where the boundary is genuinely arguable rather than settled.
Read the newsletter, not the copy. A widely shared Medium retelling, 30 Core Agentic Engineering Concepts Every Developer Should Know, was published four days later and is openly derivative by its own editor’s note. It drops every hard number, loses at least one concept, and is partly AI-assisted. Everything quantitative below comes from the newsletter.
| That primer’s grouping | Maps onto | Note |
|---|---|---|
| Core building blocks — agent, execution model, state, patterns | Below layer 1 | The think-act-observe loop itself, which the six-layer map treats as the given and never names. |
| Configuration — config files, workflow files, workflow frameworks, prompt caching, context rot | Splits layer 3 | Its sharpest divergence: it treats instructions as first-class infrastructure. The six-layer map files this under memory, which undersells it. |
| Capability — MCP, live docs, web search, visual output, memory, knowledge search | Layers 2 and 3 | Same content, merged. |
| Orchestration — subagents, agent loops, orchestration tools, managed agents | Layer 4 | Broader than “frameworks & SDKs”: it counts the multi-agent supervisor as its own tier. |
| Guardrails — sandboxing, permissions, hooks, prompt-injection defense, structural linting, pre-commit gates | Layer 6 | Adds the code-quality gates, which Perrone’s guardrails layer does not cover at all. |
| Observability — tracing, logging, metrics | Layer 5 | Same, minus eval. |
The two real disagreements are worth stating plainly. The primer promotes configuration to its own layer and pulls code-quality gates into guardrails; Perrone does neither. Both moves come from writing for coding agents specifically, where a CLAUDE.md and a pre-commit hook do more day-to-day work than a memory store does.
What the primer has that this note did not
- A cheap model with good instructions beat an expensive one without them. SkillsBench — 86 tasks across 11 domains — found Claude Haiku scoring 27.7% with human-curated skills against Opus’s 22.0% without them. Note the absolute numbers: both are low, so this is a hard benchmark and the finding is about the gap, not about Haiku being adequate. The gain vanished when the model wrote its own skills, which is the part that matters: generic self-authored instructions add tokens without adding guidance, and land as noise. Direct evidence for the discipline argued in AGENTS md vs SKILLS md.
- Config files are code, not documentation. Keep them short — the primer says under ~100 lines — review changes, and delete rules that stopped earning their place. “Write clean code” and “use best practices” cost context and teach nothing the model does not already have; “this project uses
uv, notpip” prevents a specific recurring failure. The newsletter puts this more strongly than the retelling did: generic AI-generated boilerplate in a config file “produced worse results than having no file at all.” A bad config is not a neutral cost — it is negative. The split it draws is the useful part: config = always-true rules, workflow file = repeatable procedure, live prompt = this task. - Prompt caching changes the economics of long instructions, not their quality. The stable prefix — system prompt, config, loaded workflows, tool schemas — is cached after the first turn, so a good config file costs far less than its token count suggests. The catch is TTL: caches expire on idle, so a long enough break forces the cache to be written again. It makes good context cheaper and does nothing for bad context, which still has to be read past on every turn.
- Deferred tool loading is what makes a large MCP stack affordable, and the numbers are stark. A GitHub + Exa + Context7 + DeepWiki stack costs ~607 tokens before any tool is called with deferred loading, against ~5,500 fully loaded — roughly a 9x difference in standing context. Plain skills shelling out to CLIs come in near ~300. That is the concrete form of the layer-2 tradeoff: MCP buys standardization, auth and shared access, and you pay for it in permanent context. See Model-Context-Protocol-MCP.
- Structural linting catches what formatters and type checks miss. AI-written code often passes lint, types and tests while carrying a bad pattern underneath — the canonical case being a mutable default argument in Python, which a model reproduces because it is common in training data, not because it is correct. AST-grep rules match against the syntax tree rather than the characters. The operating rule is the valuable bit: when an agent makes the same mistake twice, turn it into a rule instead of correcting it again.
- Pre-commit gates work better on agents than on humans, for an unflattering reason. A gate that rejects a commit is an annoyance to a person and a correction loop to an agent — it reads the error, fixes the code, retries. That makes the gate a teacher rather than a speed bump. It still needs CI behind it, because local hooks can be misconfigured, skipped with
--no-verify, or behave differently on another machine. The newsletter’s baseline stack is four layers in order: standard hooks (trailing whitespace, file size, valid YAML/TOML) → Ruff (lint and format, with--fix) → Bandit (hardcoded credentials,eval()) → AST-grep (structural anti-patterns). It also suggests a CI concurrency block that cancels stale runs, since an agent pushing small commits quickly will otherwise burn CI minutes on already-superseded code. - Pre-tool hooks are the last interception point. A
PreToolUsehook fires after the model has decided on a call and before the system runs it — the only moment a dangerous command can still be stopped. Bash is the case that matters, and validators like Tirith exist to screen for pipe-to-shell, homoglyph hostnames and ANSI injection. Hooks and sandboxing are complements, not alternatives: the hook tries to prevent the bad action, the sandbox limits it when prevention fails. - Agent loops are subagents applied every iteration. A subagent pushes state out of the parent’s context once, for one delegated task; an agent loop restarts the same agent with a fresh context each round, keeping progress in files and Git rather than in the prompt. It suits bounded repetitive work — migrating a codebase file by file, draining a queue — and terminates on a checked completion condition rather than on the model declaring itself finished.
- Metrics divide into proxy and outcome, and only one of them is evidence. Latency, tokens, cost, tool-call counts and failure counts all fall out of the logs, and they catch waste and stuck loops. An agent reporting “task complete” is a claim, not a result — the outcome signals are external: tests passed in CI, the PR merged, the deploy held. Harder to wire up, and the only ones that answer whether the work succeeded.
- Capability-layer tooling the retelling glossed over: Context7 for live library documentation and DeepWiki for AI-generated GitHub docs (the two that answer “what does this API actually look like today”), Exa as the AI-native search engine, Figma MCP for design input, draw.io, frontend-slides and Remotion for visual and video output, and episodic-memory plus QMD for searching past sessions and a local knowledge base respectively. QMD is Tobi Lütke’s, and it is the closest published analogue to a shared Markdown knowledge base that a fleet of agents reads and writes.
- Named orchestration tooling, absent from the sections above: Conductor and JetBrains Air for parallel isolated sessions, Vibe Kanban and Cline Kanban for board-style task assignment, and Paperclip at the ambitious end. Workflow frameworks it names: Superpowers, Get Shit Done, and Compound Engineering, whose plan-work-review-compound cycle exists to feed each task’s lessons into the next.
One line worth stealing outright, on why permissions are not optional: “Agents are not malicious. They are reward-hacky problem solvers.” That is the whole argument for deny-lists in eight words — the failure mode is an agent removing a failing assertion to make the task look done, not an agent deciding to do harm.
On sourcing, and what is still unverified. The figures above are the newsletter’s own and are reported without citations — it names SkillsBench but not a paper, and gives the context-rot and boilerplate results as “recent research” with no reference. Verified across two independent reads of the newsletter; not verified against the underlying studies. Chase SkillsBench and the context-rot benchmark to source before quoting either to a customer.
The Medium retelling is demonstrably lossy. Read directly, it runs 28 concepts under 30 numbers, with no 26 and no 29. The newsletter does have a 29 — “replay/debugging failed runs” — so that one was dropped in the retelling, and replay is a real gap in this note too. Whether a 26 exists upstream is unresolved: one read of the newsletter reported it as “CI/CD checks,” a second reported the article jumping 25 → 27. Recorded as unsettled rather than guessed.
6. Where this goes next
- The loop all of this wraps: Building-an-AI-Agent-from-Scratch.
- Layer 2 in depth: Model-Context-Protocol-MCP, then A2A-Protocol for agent-to-agent.
- Layer 3/harness side: Deep Agents.
- Layer 4 vocabulary and the paradigm choice: Frameworks, runtimes, and harnesses → LangGraph vs CrewAI Research.
- All the layers exercised in a small real project: NAT-Lesson-0-Getting-Started.
References
- Neo Kim and Paul Hoekstra, Agentic Engineering, The System Design Newsletter, 2026-06-17: newsletter.systemdesign.one/p/agentic-engineering — the source of section 5, and of every figure in it.
- Paolo Perrone, The AI Agents Stack (2026 Edition): medium.com/data-science-collective; the layer model, statistics, and honest-takes above are his.
- Letta, The AI Agents Stack (Nov 2024): letta.com/blog/the-ai-agents-stack — the original diagram this updates.
- Invariant Labs, MCPTox tool-poisoning benchmark: invariantlabs.ai
- OWASP, Top 10 for MCP Servers: owasp.org/www-project-mcp-top-10
- LangChain, State of Agent Engineering (Dec 2025): langchain.com/state-of-agent-engineering