The AI Agents Stack (2026)
The zoom-out note: the six infrastructure layers between “a model in a while loop” and an agent you can put in production. The other notes in this sequence each go deep on one slice; this one is the map that shows where each slice sits and — just as important — which layers a given project can skip.
Read LLM Tool Calling and Building-an-AI-Agent-from-Scratch first: the think-act-observe loop they teach is the atomic unit, and everything in this note is infrastructure wrapped around that loop to make it reliable at scale. The layer model here follows Paolo Perrone’s 2026 stack article (archived in the vault as The AI Agents Stack (2026 Edition) — all statistics below are from it and its cited sources), which redraws Letta’s widely-shared 2024 diagram.
Related: Model-Context-Protocol-MCP, A2A-Protocol, LangGraph vs CrewAI Research, Frameworks, runtimes, and harnesses, Deep-Agents, NAT-Lesson-0-Getting-Started.
1. Why think in layers at all
The same reason network engineers think in OSI layers: when something breaks, you debug by layer. “The agent gave a wrong answer” is as useless a bug report as “the internet is down” — is it the model (layer 1), a poisoned tool description (layer 2), the right fact evicted from context (layer 3), a retry loop that gave up (layer 4)? A layer map turns one vague failure into a small set of testable hypotheses, exactly like ping → traceroute → curl walks you up the network stack.
The map also protects you from the opposite failure: over-building. Perrone opens with a team three weeks into a 14-node LangGraph graph, custom Redis checkpointing, and retry logic — for a refund-question chatbot that calls one API and could have shipped as a 50-line script with two MCP servers. Nobody had asked which layers the problem actually needed. The operating rule for the whole stack: add a layer when something specific breaks, not before.
Three questions score every layer, and they’re worth memorizing because they apply to any infrastructure choice:
- How much state does this layer make you manage? Stateless layers are easy; the stack’s pain concentrates where state lives.
- How much vendor lock-in does this choice create? Open standards port; orchestration code and proprietary memory stores do not.
- How big is the demo-to-production gap? Some layers work in production exactly as they demo; others (eval, guardrails) barely exist in demos at all.
2. The six layers
| # | Layer | In one sentence | Deep-dive note |
|---|---|---|---|
| 1 | Models & inference | How the model runs: closed API, managed open-weight, or self-hosted | — |
| 2 | Protocols & tools | How the agent reaches tools and other agents: MCP, A2A | Model-Context-Protocol-MCP, A2A-Protocol |
| 3 | Memory & knowledge | What the agent knows: in-context state, retrieval, persistent memory | Deep-Agents |
| 4 | Frameworks & SDKs | How the loop, tools, and control flow are wired together | Frameworks, runtimes, and harnesses, LangGraph vs CrewAI Research |
| 5 | Eval & observability | How you know the agent works, and catch it when it stops | — |
| 6 | Guardrails & safety | How you stop it doing what it shouldn’t | — |
Layer 1 — Models & inference
The engine choice, and the layer that’s commoditizing fastest. What changed since 2024: reasoning models (o-series, DeepSeek R1, Claude extended thinking) let a single call solve what used to need a hand-built multi-step chain, and open-weight models (Llama 3.3, DeepSeek V3, Qwen 2.5) closed enough of the quality gap that “always use the biggest closed model” stopped being default advice. The emerging pattern is prototype on closed-source, deploy on open-weight.
Think of this layer like buying compute: once parts are instruction-set compatible, you stop arguing about which is “smartest” and start comparing price and latency. That’s where models are heading — the real decision is cost/latency, and the OpenAI-compatible API shape (the ABI from Building-an-AI-Agent-from-Scratch §4) is what makes the vendors swappable. Scores: no state to manage, high lock-in for closed APIs (prompts and evals are tuned per model), the smallest demo-to-prod gap in the stack. Self-host only when call volume makes API pricing untenable or you need latency an API round-trip can’t deliver.
Layer 2 — Protocols & tools
In 2024 this layer didn’t exist as a category — every framework had its own tool-definition schema. Then MCP standardized the wire format, and the layer went the way of the network protocol wars: once TCP/IP won over IPX and SNA, the interesting work stopped being “which protocol” and became “how do we secure it.” Perrone’s markers of the win: 97M monthly SDK downloads, adoption by OpenAI, Google, and Microsoft, donation to the Linux Foundation (December 2025). Model-Context-Protocol-MCP covers the mechanics.
Security is the open problem, and the numbers are ugly: Invariant Labs’ MCPTox benchmark got an 84.2% tool-poisoning success rate against agents with auto-approval enabled, and Endor Labs’ scan of 2,614 MCP servers found 82% prone to path traversal and 67% to code injection. This is the “tool results are untrusted input” warning from LLM Tool Calling §7 measured at ecosystem scale.
Agent-to-agent communication lives here too — Google’s A2A and IBM’s ACP — but neither has reached critical mass, so multi-agent coordination today is mostly built by hand at layer 4. A2A-Protocol explains what these protocols add over plain tool calls.
Layer 3 — Memory & knowledge
In 2024, “memory” meant “pick a vector database and do RAG.” In 2026 it’s a first-class design problem with three tiers, and the tiers are exactly the memory hierarchy you already know: in-context state is RAM (the context window — fast, small, wiped between processes), retrieval is demand paging (vector or graph search pulls the needed page off disk into RAM at call time), and persistent memory is the filesystem (state that survives across sessions). Bigger context windows (1M+ tokens) didn’t kill the hierarchy any more than bigger RAM killed filesystems — they just moved the swap threshold.
What changed: “context engineering” replaced “prompt engineering” as the discipline — architecting what the model sees on every call rather than wordsmithing one prompt. Memory blocks (named in-context fields the agent itself reads and overwrites) made state agent-managed. pgvector became the default retrieval store — it’s just Postgres with an extension — and Graph RAG (Neo4j) added relationship-following as an alternative to embedding similarity.
This is the highest-state, highest-complexity layer, and the honest take is that most teams overcomplicate it: start with conversation history in Postgres plus a structured system prompt; add vector search when history outgrows the context window; add agentic memory management only when the agent must learn across sessions. The harness-side view of the same problem — compression, filesystem-as-swap, subagents — is Deep-Agents.
Layer 4 — Frameworks & SDKs
Three camps now, where 2024 had only LangChain: provider SDKs (OpenAI Agents SDK, Google ADK — fastest to start, locked to one vendor’s models and orchestration), graph-based frameworks (LangGraph — portable across models, heavy setup, explicit state machines), and no framework — the thin-wrapper-plus-MCP camp, which is Building-an-AI-Agent-from-Scratch grown up. One recurring confusion worth flagging: LangChain is the integration layer (connectors, prompts, tool calling); LangGraph is the orchestration engine where agent logic lives — most production teams use both.
Lock-in is the highest in the stack: orchestration code doesn’t port, and a LangGraph agent rewritten for CrewAI is a new codebase. MCP is the one investment that transfers across all three camps. The tier vocabulary for this layer is Frameworks, runtimes, and harnesses; the paradigm comparison is LangGraph vs CrewAI Research. The honest take matches both: most teams pick too much framework for what they’re building.
Layer 5 — Eval & observability
The layer that barely existed in 2024 and is now where production quality dies. LangChain’s State of Agent Engineering survey: 89% of teams with production agents have observability, only 52% have evals. That 37-point gap is running production with syslog but no test suite — you can watch the crash in beautiful traces, but nothing stops the regression from shipping, and users find the failures for you.
The converging shape is three-tier: fast checks on every PR (did the agent call the right tools?), nightly regression suites with an LLM judging output quality, and continuous production monitoring for drift. And evals must score steps, not just final output: if the agent picked the wrong tool at step 3 of 12, steps 4–12 were doomed — checking only the final answer is checking only the last exit code of a pipeline with no pipefail. This layer has the biggest demo-to-prod gap of all, for the simple reason that most prototypes have zero eval.
Layer 6 — Guardrails & safety
Agent guardrails split from LLM guardrails as a discipline: filtering text in and out was enough when models only talked, but an agent calls tools, spends money, and acts. The pattern teams learned the hard way is guardrails before action — authorization enforced at the tool-execution layer, not the output layer. It’s the difference between seccomp at the syscall boundary and grepping the audit log afterward: by the time you filter the response, the agent already sent the email. OWASP’s MCP Top 10 (December 2025) is the first real security checklist for tool-connected agents.
This is the least mature layer — no dominant framework, no established patterns, mostly policy code you write yourself (NVIDIA’s NeMo Guardrails is the closest thing to a framework, and you’ll still write most rules). The demo-to-prod gap is effectively infinite: your demo has no guardrails because nobody is trying to break it. Production will be tried. Guardrail propagation — policies following a task as agents delegate to other agents — is unsolved; if you’re doing multi-agent delegation (A2A-Protocol), you’re building that yourself.
3. What are you building? (the question that picks your layers)
The agent type determines which layers you invest in — a tool-calling chatbot and a multi-agent research system share almost no infrastructure:
| Agent type | Example | Stack | Watch out for |
|---|---|---|---|
| Stateless tool caller | look up an order, answer from a knowledge base | Provider SDK + MCP + Postgres | Nothing — this is a weekend project; don’t add a framework |
| Multi-step workflow | process a refund end-to-end, triage tickets | Framework (e.g. LangGraph) + MCP + eval | Silent breakage — build evals before deploying |
| Agent that learns | remembers preferences, improves on your codebase over weeks | Memory-first architecture + retrieval + eval | Deciding what to remember/forget; old context polluting new answers |
| Multi-agent system | specialists delegating and running in parallel | The full stack | Two agents passing context is already hard to debug; five is impossible without per-handoff traces |
Coding agents (Claude Code, Cursor, Codex) are the proof that all six layers can work together in one product: routed inference (1), MCP into editors/terminals/git (2), codebase-aware retrieval (3), purpose-built orchestration rather than any framework (4), eval running continuously in production — Cursor retrains its suggestion-acceptance model every 90 minutes on live accept/reject data (5), and sandboxed execution containing what the agent can touch (6).
4. The prediction to hold loosely
Perrone’s closing claim: the stack will collapse. Provider SDKs are already absorbing memory, tool calling, and basic eval into single APIs, and by 2027 most teams will take an opinionated stack from their model provider — fine for ~80% of use cases, with custom builds surviving at scale where defaults break. Even if that lands, the layer map keeps its value for the same reason OSI outlived the protocols it described: when the opinionated stack fails in production, you still have to know which layer failed.
5. Where this goes next
- The loop all of this wraps: Building-an-AI-Agent-from-Scratch.
- Layer 2 in depth: Model-Context-Protocol-MCP, then A2A-Protocol for agent-to-agent.
- Layer 3/harness side: Deep-Agents.
- Layer 4 vocabulary and the paradigm choice: Frameworks, runtimes, and harnesses → LangGraph vs CrewAI Research.
- All the layers exercised in a small real project: NAT-Lesson-0-Getting-Started.
References
- Paolo Perrone, The AI Agents Stack (2026 Edition) — archived as The AI Agents Stack (2026 Edition); the layer model, statistics, and honest-takes above are his.
- Letta, The AI Agents Stack (Nov 2024): letta.com/blog/the-ai-agents-stack — the original diagram this updates.
- Invariant Labs, MCPTox tool-poisoning benchmark: invariantlabs.ai
- OWASP, Top 10 for MCP Servers: owasp.org/www-project-mcp-top-10
- LangChain, State of Agent Engineering (Dec 2025): langchain.com/state-of-agent-engineering