Introducing CAFE(S): A framework for evaluating AI coding agent effectiveness

September 24, 2026 — Today, researchers from DX, Capital One, GitHub, the University of Victoria, and Google announced CAFE(S), a diagnostic framework designed to evaluate the context provided to AI agents. Published in ACM Queue, the research establishes an industry standard for diagnosing, designing, and maintaining the information environments that power AI coding agents across the software development lifecycle.

As software engineering organizations scale investments in AI coding agents, task failures are often attributed to issues with model capabilities or harness orchestration. However, research demonstrates that even frontier models degrade when provided with ambiguous, incomplete, or stale context.

The CAFE(S) framework introduces a shared diagnostic vocabulary for platform teams, developer productivity leaders, and software engineers to evaluate assembled context across five dimensions:

1. Clarity: Can the agent interpret the request the way the person intended? Ambiguity is often invisible to the person writing the request. CAFE(S) helps teams ensure the agent understands the work the same way a teammate would.

2. Actionability: Does the agent have clear goals, useful boundaries, and a way to know when it is done? A request can be unambiguous but still leave out the goal, say too little about the constraints, or give no way to tell that the work is finished.

3. Fidelity: Is the context true at the moment the agent reads it? Accuracy decays over time. Stale documentation, old architectural decisions, and conflicting sources lead agents down the wrong path.

4. Efficiency: Is the context scoped to the task, without unnecessary token load? More context is not always better. Parsing an entire repository when an agent needs a single function inflates costs and degrades performance.

5. Security: Should the agent have this context at all? The “S” is kept at a remove because the first four dimensions determine whether context helps the task, whereas security asks if the context is fundamentally appropriate, compliant, and safe to access in the first place.

   “AI does not create a new need for good knowledge management; it dramatically increases the cost of poor knowledge management.”

   Brian Houck, Distinguished Scientist, DX

The research emphasizes that capable models alone are insufficient to ensure reliable task completion and stronger delivery outcomes. Teams that systematically strengthen the information environments feeding their agents will see better results. CAFE(S) becomes a quality scorecard for context, designed to sit on top of the existing stack.

When context goes wrong

One useful way to understand the CAFE(S) dimensions is to look at what happens when they break down across three primary areas:

• Cost: Context is where token costs accumulate. When agents lack scoped inputs or boundaries for what “done” looks like, teams risk runaway executions, such as autonomous tasks running over a weekend and burning thousands of dollars in unnecessary token spend.
• Liability: Organizations are increasingly held legally and operationally liable for their agents’ actions and hallucinations (e.g., cases where companies were required to honor hallucinated discounts or agreements). Compromised or unsafe context also introduces risks like prompt injection and unintended data leaks.
• Developer experience: AI is designed to accelerate engineering, but poor context forces developers to spend valuable time diagnosing, debugging, and fixing avoidable mistakes. When specifications are ambiguous, agents are forced to guess early in the task loop, compounding errors into downstream code. Furthermore, dumping excessive information leads to “Lost in the Middle” syndrome, where agents get distracted and miss critical context buried in noisy repositories.

What’s next?

This research defines a framework, not a measurement system. Future work will be needed to develop reliable ways of assessing these properties at scale, and to understand how improvements in context quality influence collaboration, software delivery, and organizational outcomes. But meaningful measurement requires a shared definition of what is being measured. Before teams can improve context quality, they need a vocabulary for discussing it.

Download the full research paper: https://getdx.com/whitepaper/the-cafes-framework/

About the authors

Brian Houck is a Distinguished Scientist at DX. His work combines large-scale telemetry analysis, field experiments, surveys, and qualitative research to uncover the technical, cultural, environmental, and organizational factors that shape developer productivity and well-being. He is best known for his work on the SPACE framework for developer productivity and for measuring and improving developer experience in large engineering organizations.

Max Kanat-Alexander is an Executive Distinguished Engineer for Developer Experience at Capital One, where he leads technical strategy for AI-assisted developer experience for more than 14,000 technologists. He is the author of Code Simplicity and Understanding Software, and has previously led code health initiatives at Google and the Bugzilla Project.

Eirini Kalliamvakou is a research advisor at GitHub and leads research that deciphers developers’ motivations, needs, behavior, and how tools support them. Her insights shape product thinking, strategic storytelling, and leadership decisions across the company. She holds a Ph.D. in computer science from the University of Victoria and speaks about developer productivity, happiness, and the impact of AI on individuals, teams, and organizations.

Margaret-Anne Storey is a professor of computer science at the University of Victoria and a Canada research chair in human and social aspects of software engineering. She is co-author of the SPACE framework and a leading researcher in Developer Experience (DevEx). Her research focuses on how developers and teams understand complex software systems and how tools, AI, and collaborative practices shape that understanding. Her recent work examines how generative AI is transforming software engineering by changing how understanding is created, shared, and maintained. She collaborates with industry partners including Microsoft and DX. She holds an honorary doctorate from Lund University.

Nicole Forsgren is the Sr. Director of Developer Intelligence at Google. She is an expert in DevOps and the author of the Shingo Publication Award-winning book Accelerate: The Science of Lean Software and DevOps. Her work on technical practices and development has been published in industry and academic journals and is used to guide organizational transformations around the world.