AGENTS.md vs Skills

Where do you put knowledge a coding agent needs — in a context file it always sees, or in a skill it can choose to load? This note explains the two mechanisms from zero, then works through the Vercel eval that made the trade-off concrete (archived in the vault as AGENTS.md outperforms SKILLS.md in our AI Agent Evaluations, research by Jude Gao), and finishes with how to read that eval critically — because the answer is a division of labor, not a winner.

Read LLM Tool Calling first for the context-window basics; Deep-Agents is the natural companion, since both mechanisms are tools of the context engineering it describes.

Related: Deep-Agents, The-AI-Agents-Stack-Summary (layer 3 — this is context engineering in miniature), Frameworks, runtimes, and harnesses.

1. The two mechanisms

AGENTS.md (Claude Code’s CLAUDE.md is the same idea) is a markdown file in your project root that the harness injects into the agent’s context on every turn. It is passive context: the agent never decides to read it, the same way a login shell never decides to source .bashrc — it just happens, and the settings are in effect for the whole session. Whatever lives there — project conventions, build commands, “never touch this directory” — is simply always present.

Skills are an open standard (agentskills.io) for packaging domain knowledge — prompts, docs, sometimes tools — that an agent loads on demand. They are active retrieval: nothing happens until the agent recognizes “this task needs the Next.js skill” and invokes it. The systems analogy is dlopen() versus static linking — the capability exists on disk, but somebody has to decide to load it, and that decision is a new failure point that statically-linked context doesn’t have.

2. What Vercel measured

The eval design is worth understanding before the numbers, because it’s a model of how to test agent-knowledge mechanisms honestly. Next.js 16 shipped APIs ('use cache', connection(), forbidden()…) that postdate model training cutoffs, so no model can answer from memory — any pass above baseline must come from the docs-access mechanism. That isolates the variable. Vercel also hardened the suite first (removed test leakage, switched to behavior-based assertions) after finding their initial evals measured the wrong things — a layer-5 lesson straight out of The-AI-Agents-Stack-Summary.

Four configurations against the same suite:

Two findings besides the headline:

• The skill sat unused. In 56% of cases the agent never invoked it — docs available, agent chose not to look. Default-behavior skills produced zero improvement, and on some sub-metrics scored worse than no docs at all, suggesting an un-invoked skill is context noise. Models not reliably using available tools is a known limitation, not something unique to this setup.
• Trigger wording was fragile. “You MUST invoke the skill” made the agent read docs first and anchor on doc patterns, missing project context (in one test it wrote a correct page.tsx but skipped the required next.config.ts change). “Explore the project first, then invoke the skill” got both. Same skill, same docs — the outcome swung on phrasing, which is a brittle thing to build production behavior on.

3. Why passive beat active

Vercel’s three-factor theory, which generalizes past this one eval:

1. No decision point. The information is already there; there is no moment where the agent can fail to look it up. This is the checklist printed on the wall versus the runbook the on-call engineer has to remember exists — most incident postmortems feature the second kind.
2. Consistent availability. Passive context is in every call. A skill’s contents exist only in the turns after invocation.
3. No ordering problem. Skills force a sequencing choice (read docs first? explore code first?) that the wording-fragility finding shows agents handle badly. Passive context has no sequence to get wrong.

4. The pattern worth stealing: a compressed index, not the docs

The naive version of “put the docs in AGENTS.md” bloats the context window — Vercel’s full injection was ~40KB. The fix is the note’s most reusable idea: put an 8KB compressed index in context and leave the full docs as files on disk the agent reads on demand:

[Next.js Docs Index]|root: ./.next-docs
|IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning
|01-app/01-getting-started:{01-installation.mdx,02-project-structure.mdx,...}

In the memory-hierarchy terms from The-AI-Agents-Stack-Summary layer 3: the index is the page table, resident in RAM; the doc files are pages on disk, faulted in only when touched. The agent always knows what exists and where (passive), and pays the token cost of full content only when a task needs it (active). The 100% pass rate held after the 80% compression — knowing where to look was worth as much as having it all in view.

The embedded IMPORTANT: line names the deeper principle: retrieval-led reasoning over pre-training-led reasoning. An agent’s training data is a snapshot that ages; project-local, version-matched docs are ground truth. The instruction tells the model which to trust when they disagree — the same reason you check man on the box you’re on instead of trusting what some other distro’s manpage said years ago. For Next.js projects the whole setup is one command: npx @next/codemod@canary agents-md.

5. How to read this critically

The numbers are real but the frame needs three qualifiers before you generalize:

• It’s one vendor’s eval of its own framework, ending in a pitch for its own codemod. That doesn’t invalidate it — the design (post-cutoff APIs, hardened suite, retries for variance) is sound — but 100%-on-Next.js is not “100% on your stack.” Treat the direction as credible and the magnitude as local.
• The gap is partly a model-capability snapshot. Skills lose largely because current models under-trigger them; Vercel says plainly the gap may close as tool-use improves. A conclusion that rests on “models are currently bad at deciding X” has a shelf life.
• It compares default-triggered skills, not user-triggered ones. A skill the user explicitly invokes (a slash command in Claude Code) skips the unreliable trigger decision entirely — the 56%-never-invoked failure mode doesn’t apply. The eval measured agent-initiated triggering only.

Vercel’s own conclusion is the balanced one: passive context for horizontal knowledge (framework conventions relevant to every task), skills for vertical workflows explicitly triggered (“migrate to the App Router”, “upgrade my version”). The two mechanisms complement; the mistake is putting always-relevant knowledge behind a trigger decision, or bloating every call’s context with a workflow used once a month.

6. Where this goes next

• The harness-tier context machinery both mechanisms plug into: Deep-Agents.
• The stack layer this lives in, and “context engineering > prompt engineering”: The-AI-Agents-Stack-Summary layer 3.
• Why an unused tool in context still costs you — the “more tools ≠ better” point: LLM Tool Calling §7.

References

• Jude Gao / Vercel, AGENTS.md outperforms skills in our agent evals — archived as AGENTS.md outperforms SKILLS.md in our AI Agent Evaluations; all numbers above are from it.
• Agent Skills standard: agentskills.io; skills directory: skills.sh
• AGENTS.md convention: agents.md