Brian Houck: Thank you everyone for joining today. Absolutely thrilled to be talking about a new paper, CAFE(S), that just was published this morning. And so it is perfect timing. The stars aligned. I am thrilled to be joined by my CAFE(S) co-authors today who I’ll ask to introduce themselves in just a second. But just a little bit of run of show for you.
Today we’re going to be discussing the paper. I’m going to be asking my co-authors some of their thoughts around it and around the broader space of context quality, which is what the paper is based about.
But to kick us off, I’m Brian Houck. I am your host today. I am one of the resident mad scientists over here at DX studying all things developer productivity. And I will hand it over to Max.
Max Kanat-Alexa…: Hi everybody. I am Max Kanat-Alexander and I am a principal software engineer at Capital One where I help lead Capital One’s efforts around AI assisted software development.
Brian Houck: Awesome. Thank you. Eirini.
Eirini Kalliamv…: Hi, I’m Eirini. I recently joined DX as a researcher as of three days ago. But when I worked on this paper, I was at GitHub as a research advisor. So yeah, that’s what we’ll talk about today.
Brian Houck: And last but certainly not least, Peggy.
Margaret-Anne S…: Great. So I’m Margaret-Anne. Everybody calls me Peggy. Story from the University of Victoria. I’ve been studying developer experience and how developers use tools for a long time. And I’ve worked a lot with Brian and Eirini on these very topics and chatted a lot with Max. So this is the first time we’ve actually worked together. So thrilled to be working with Max and all of you as well on this framework.
Brian Houck: I also would like to point out that we had the great privilege of having Dr. Nicole Forsgren be a co-author on the paper. She unfortunately couldn’t join us today for the panel, but was a key contributor to the paper itself. And so Nicole, if you are watching this, we are thinking of you for sure.
And so with that, I’m going to give a little bit of background on what CAFE(S) even is, what’s the problem space we are looking at before I kick off a round of questions with my fabulous co-authors. For those of you who might have watched my session at this week’s Atlassian State of AI SDLC event, the next five minutes are largely going to be a repeat of what you saw. So this is your chance to go grab a cup of coffee. But once I get through bringing us all up to speed, then it will be all new material for you. And so with that, let me go ahead and share.
CAFE(S) is a framework ultimately for measuring the health and quality of the context that we are feeding into our AI agents. And I firmly believe that if we want to improve and maximize AI agent effectiveness, we have to have ever increasingly better context. And as I sort of think about what do I mean by context and how does it fit in human agent collaboration? And as I sort of trace this from left to right, ultimately intent is what it is a human wants to accomplish. And that intent has to be translated into context, which is the task that you want to complete, all of the information, the constraints, the data associated with it, your code, that’s the context. All of the information, the knowledge that we are passing to our agents, which gets orchestrated through our harnesses, which deal with retrieval and memory and execution logic before ultimately passing it to the models which supply the underlying intelligence.
Now, if we want to maximize our human agent collaboration, get the most out of our agents, everything in this flow is important. Everything has to get better. But we already spend enormous effort improving our harnesses through harness engineering, improving our models through frontier development. Intent, we improve through improving our product planning processes to better synthesize and think about customer data, excuse me, things of that nature. But what I found is that almost nobody is holistically evaluating the quality of context itself.
And this is so important right now, especially because of a variety of compounding risks that low quality context in our agent workflows puts onto us. One example of that is just liability. Organizations are being held accountable for what their agents say and do. If your chat agent hallucinates a sale that doesn’t really exist, courts have held that you are responsible for honoring that.
Cost. We all see cost skyrocketing and is growing exponentially and every token has a cost associated with it. And context is really where those costs accumulate. It is everything that you are asking your agents to read and what you are having them produce is all a function of the context.
And also developer experience. We are seeing, even though we are accomplishing more, we are building more and more and more than ever, we are seeing some downward pressure on developer experience. Annie Vella has called this the productivity experience paradox. And part of that is because poor context turns AI into not something that’s helping us, it’s turning AI into another source of developer toil.
And there are lots of ways that our context fails us. And so what does it mean to have bad context? Well, let’s talk through just some examples of named real world failures that result from having bad context beyond just hallucination, which is what oftentimes we talk about.
The first example is what we call weekend runaway, which is you may set your agents about on a task Friday afternoon and you don’t clearly define what done means. And so those agents just keep going and going and going and going, expanding their scope. And you don’t realize that until you come in on Monday morning and you see that you’ve blown through your entire month’s token allocation.
Ambiguity, agents don’t stop to negotiate requirements. They will pick an interpretation often silently and proceed often confidently. And the problem is if that first assumption is wrong early on, it just snowballs and compounds.
The next failure mode is what’s called lost in the middle. Agents get distracted. And so you may have incredibly valuable, important, relevant context buried in a much larger context window and the agents may overlook the part that really matters because of placement. They truly get distracted. So oftentimes I hear people talk about how can we get larger and larger context windows? And I think that is the wrong way to look at it. If we had infinite size context windows, we may have substantially worse outcomes as a result because of lost in the middle.
And just the last example that I’ll talk through real briefly is prompt injection. Much like we have SQL injection, there are real world failure cases where instructions were embedded in the context that the agent then acted on, resulting in sensitive information getting leaked, perhaps even customer data getting linked. And while yes, our harnesses and models may be able to help protect us against this, it doesn’t change the fact that the underlying context itself was compromised.
And so as we look at some of these named failure modes, the ways that context might fail, we though about, well, what are all of the durable properties of high quality context? And the first is clarity. Is can the agent interpret the request the way that you intended it to be interpreted? Ambiguity is invisible to the person defining the task. We know what we meant, but thankfully the agents don’t live in our heads. And so clarity is a dimension that has to be specified.
And I see we have a raised hand. Unfortunately, we don’t have time for questions today, but Adam, please leave your question in chat and we’ll be happy to get to it asynchronously.
The next dimension is actionability. Have you specified a clear goal that the agent can work towards? Does the agent know when it is done or will it just keep on going forever? And clear is not the same thing as sufficient. You may have a perfectly clear task that is impossible to act on because it is not actionable.
The third dimension of CAFE(S) is fidelity, truthfulness. Can the agent trust that the context is true relative to the task that it is trying to accomplish? And accuracy, truthfulness is actually a pretty hard property to wrap your mind around because we all have different definitions sometimes. And accuracy is not a property you set once. It decays over time. And we’ve seen this with existing documentation where an organization, maybe your organizations have tons of documentation, but if they were written two years ago, are they still relevant? Can you trust them? Is that functionally the same thing as not having any documentation at all? And so the accuracy of that documentation is part of the very definition of the context quality.
The fourth dimension is efficiency. Is the context lean enough to be usable and affordable? And I want to note that this doesn’t necessarily mean the smallest possible context. It means the highest signal to noise. And this is a place where within your organizations you see individual outliers and token usage, oftentimes it’s a failure and efficiency. Where it’s like, oh, I wanted help understanding this single method and so I passed in my entire repo. Or it’s like, hey, I’m trying to debug this one failure, so I pass in all of my built logs. And every time you do that, the agents have to read it and that just uses tokens.
And then the last property that we think defines what it means to be high quality context is security. Is that context safe and compliant? And so the first four dimensions really talk about is the context itself high quality? The last dimension talks about should we have been using that context at all?
Now you may be asking yourselves, well, why these five dimensions? Why not something else? And ultimately, the principle that we had is a property sort of gets its own pillar if it is not derivable from any of the others, if it can be acted on independently, and it names a real world failure mode that the others miss. And ultimately we think these properties, if they are true, ultimately result in better context, which results in more effective agents, which ultimately results in what we care about, better outcomes.
Again, that was a seven-minute overview of a very long paper. I would love it if you all went and read it, let us know what you think, but that gives you a high level overview of what is the CAFE(S) framework? How do we think about its place in the world? And with that, I now want to dive into some questions for my co-authors.
And my first question is we will often blame failures on our actual models, on the LLMs themselves. And I’m curious, and maybe Peggy, we can start with you, what are some failures that you’ve seen blamed on these models that was actually a context failure?
Margaret-Anne S…: Yeah. Let me talk about the experience that I had in a course. I mentioned that I’m an academic, a professor at the university, and I teach a course called Startup Studio where I encourage the students to build a startup in about 12 weeks and to use AI, of course, to go quickly and learn quickly about what their customers might need and learn about the problem that they’re trying to address. And I’ll chat about the triple debt model a little bit later on perhaps, Brian. But one of the things that I saw last year is that the students were blaming the model for hallucinating features that they were adding to their MVPs, so to the early versions of their startup apps. And I would say to them, “Well, where did that come from?” And they were like, “Oh, I don’t know. I guess the model hallucinated them.” And I’m like, “But where are you writing down to the model what it is that you want it to do because you’re trying to build this minimal kind of set of features? Where have you told the model that this is what you need to do?”
And of course, at that point they had not. So that was kind of the first clue that, “Oh, we need to really take our time in describing to the agent and to the other people on our team what it is that we want to do and what it is that we don’t want to do.”
Brian Houck: Absolutely. Absolutely. Kicking it over to you, Max, what are some failures you’ve seen assigned to the models that might’ve actually been context failures?
Max Kanat-Alexa…: Yeah. Every so often I’ll run into a developer who will tell me, “It takes me longer to write code with coding agents than it does to write it by hand.” And that’s mystifying because that should never be true. And whenever I dig into it’s one of two things. One of them is, which is totally fine and acceptable, sometimes people are new to writing with coding agents and that’s a learning process, and at first it’s going to slow them down and that’s normal and we expect that as part of the learning process. But usually when I dig into it and I ask them about it, I’m like, “What was the problem?” And they’re like, “Well, it doesn’t know any of the things that are in my head and in the heads of all of my teammates.” And I’m like, “Well, that’s the problem. The problem is that those things are in your head and in the heads of all of your teammates and they aren’t written down anywhere.”
Brian Houck: Yeah. Eirini, closing us out, what are some examples that you might’ve seen of context failures that people are like, “Oh no, it’s just these dumb models.”
Eirini Kalliamv…: So I’m going to call out an example that I saw published at some point in a tech article, and I’ll dig up the link and give it so that we can share with everybody as well. But this happened, I think, back in April. It was a small company that was called PocketOS, I think, and it runs software for car rental businesses. So I had a coding agent was working on something pretty routine. It hit a permissions problem, and instead of stopping there, it went looking and somewhere it found an API token completely unrelated to the task and an unrelated file. And long story short, the whole production database was deleted, and it took a few seconds. Then the backups, of course, we lived in the same place, so those were gone too. And then the last good copy was from several months ago. When this was written up, it was written up as the agent goes rogue and it’s something about the model and alignment and so on.
And yes, the agent of course ignored an instruction, but if we think about it, it had access to a credential that had nothing to do with a task. It was within reach and it shouldn’t have been. And that’s the failure that I think has to do with context. It had context that allowed the model to make a bad call or the agent to make a bad call.
Brian Houck: Yeah. And you can clearly see the potential security ramifications of failures like that.
Eirini Kalliamv…: Absolutely.
Brian Houck: Which is why it’s a dimensions of CAFE(S). So one of the things that is the thesis, a key thesis of the paper is that for agents to do their best work, it will be limited by the knowledge environment around them. And knowledge management problems have existed in software organizations forever. And we’ve known about them, we’ve seen them survey and survey again, that documentation is one of the key sources of frustration for developers, and we largely just learned to live through them. And so Max, I’m curious, what’s different now that the consumers of this knowledge are agents versus humans?
Max Kanat-Alexa…: Yeah. So one of the great secrets of agent experience is that it’s all the same things that we knew about developer experience already. And really knowledge management in many ways is no exception. If you gave a bunch of humans a bunch of confusing documentation, they would also do the wrong things. You just couldn’t see it and it wasn’t a direct artifact that cost you money. So there was always a problem. If a person couldn’t find the information they needed, they would spend a bunch of very expensive time either trying to discover that information or building the wrong thing. If the instructions that you had were wrong in your documentation, the exact same sort of thing would happen. This is a perennial complaint of developers. All these things always needed to be written down. It’s just become dramatically more apparent in present time.
Brian Houck: You have said on my own podcast, AI is an amplifier and it’s like, well, we could sort of absorb a little bit of being crappy about this and it’s just as you shine a bigger and bigger spotlight on it. So one thought that I’ve had, and I’ve actually been asked this already, is like, well, what’s the most important dimension of CAFE(S)? And a question to all of you is if you were to tell an organization, if you could only improve one dimension of CAFE(S), what dimension would you have them to start with and why? Eirini, you want to kick us off?
Eirini Kalliamv…: Yes. Most important… So I toggle between two because when we were working on the paper, they were the two that it was harder for me to disambiguate. And I’m talking about clarity and actionability. And both of those in my mind are very important because the result, it looks the same from the outside. It’s an agent that can’t really do the work and you can’t really tell why that is, but they do have different fixes, and I think those are equally important. If you really press me to pick one, I would say probably clarity. And it’s so easy to, as you said, the ambiguity is very easy to propagate because I know what I want and it’s in my head, but that doesn’t necessarily mean that I said it in a way that is unambiguous. So both clarity and actionability I think are key. They look the same from the outside when they don’t work, but they have different fixes.
So if I was going to fix something by wording it more precisely, that means that’s clarity. I’m taking the ambiguity out of it. If I was going to fix it by adding something, adding a goal, adding a constraint and so on, that means it’s kind of actionability. So in my mind, those are absolutely critical, and I would say the clarity is number one.
Brian Houck: Awesome. Max, I got to interrupt you from geeking out with people about your amazing novel collection back there. Where would you recommend that organization start? What dimension?
Max Kanat-Alexa…: I think the most important thing is probably clarity and actionability. I mean, so in a combination, the first thing is just you, the human being, typing things into the agent has to be providing clear instructions. You need to know because you are the final fallback. If there is no other information gathering mechanism, the information gathering mechanism is you, and so you have to be the thing that knows how to type out the thing that is actually your intent in a way that will actually get executed.
Brian Houck: Peggy?
Margaret-Anne S…: I’m going to be boring and say clarity as well. I mean, not surprising, we all worked on the paper together, so we kind of think the same. But the reason that I think clarity is the most important is that it’s the least visible of the problems. I mean, I think a lot of organizations know that they need to… Well, maybe not all, but many organizations know that they need to pay attention to security and efficiency is often top of mind because of token costs, so money hurts when you spend too much. Maybe fidelity and clarity are kind of the two that I would say if you’re going to make a change, focus on those. I think Eirini described it well when she said that we’re pretty good at telling the agent through our instructions what it is that we wanted to do, but it’s the stuff that… And this is when you see robots fail, they fail because of the things that we just take for granted about the physicality of the world and what our senses tell us and implicit knowledge, and that’s the stuff that we don’t tell the agents. And then it’s like, duh, why didn’t you know that we drive on the right-hand side of the road?
I saw a funny video the other day of somebody driving down AI generated and the car, actually, I think it was a Pelican, was on the wrong side of the road and I’m like, “It’s going to get hurt.” But things like that that we just take for granted and we don’t tell the agent that we need to make explicit, I think is super important with clarity. And then fidelity is like, “Is this really true? Is this changing since you last gave instructions to the agent?”
Brian Houck: It is interesting. I sort of like to believe all of the pillars are equally important, but you’re right, there are some that will result in failures substantially quicker than others.
And so Peggy, staying with you for a moment, so you in your triple debt paper, which for those in the audience who have not read Peggy’s triple debt paper, please, please go and do it. It is, I think, sort of foundational to so much of the way I think about AI these days. You described intent debt as the erosion of rationale, goals, and constraints a system needs to evolve safely. An organization can carry these invisibly for years. Does delegating work to agents make that debt more visible? Does it make it more costly? Perhaps both? How do you think about that?
Margaret-Anne S…: Yeah, I mean, maybe I’ll just say one quick word about what triple debt is because some folks may not know what that means. So triple debt was this paper that I wrote quite some months ago about system health. So it ties together technical debt, cognitive debt, and intent debt. And technical debt is the thing that we all already know about. I don’t need to say much about that. There was a lot of discussion in the industry about AI is making technical debt worse while other people were saying maybe it makes it better, improves it. But cognitive debt referred to what we have in our minds. So Max kind of touched on this a little bit, expecting the agents to read our minds, but if we don’t actually know what it is that we’re trying to build and we don’t have a good decision across our team, a shared understanding of what it is we’re trying to do, we can build cognitive debt over time.
And intent debt is when we don’t explicitly document the things that we need the agents or the models to take into consideration or other people on our team. So that’s things like the constraints, the things that we want to achieve, as well as the decisions we make about what we are not going to do.
So thinking about intent debt and how agents in particular or thinking about if we have good context, how that can make it both better and worse, I think the answer is both. On the one hand, if you have really good context and everything seems to be working really well, you may over time become complacent and not write things down explicitly. And on the other hand, it can be better. You could imagine you have an agent that is asking you questions about, yes, you want to implement this feature, but what are the underlying requirements behind that?
What are the disagreements about this? What are the things that you haven’t told me that I might need to know? What are the parts of this task that I should do first? I think that having better context, having quieter or higher quality context can help. But on the other hand, I think of the triple debts as being consequences, right?
Brian Houck: Mm-hmm.
Margaret-Anne S…: So the dimensions of CAFE(S) are kind of the properties of the context, but if we don’t have those properties well-designed, then we might lead to these consequences of cognitive debt, technical debt, and intent debt.
Brian Houck: Eirini, question for you. When an agent doesn’t have enough context or certainly not enough high quality context, humans often have to compensate for that. We have to clarify, we have to redirect, we have to add information. Sometimes we just abandon the task entirely because we’re like, “Oh, the models can’t do this.” What can those behaviors from humans as part of the human agent interaction, what can those human behaviors tell us about the quality of the context we’re providing?
Eirini Kalliamv…: I think they probably tell us almost everything because I think that if you look at each of these reactions that you mentioned, when do we do it? When do we clarify? We clarify when a request is ambiguous. It could be read more than one way. We add information when the agent has very little to go on or has nothing to go on, like Peggy mentioned, no goals, constraints. You mentioned earlier what done looks like. We tend to redirect when the agent is using something, is acting on something that used to be true, but it no longer is. And you mentioned sometimes we abandoned the task, and I think that’s the person deciding that the context cannot be fixed in the moment. And so this compensation behavior that you’re describing, I think it means the context is being reviewed live by the only person that knows what was the actual intent.
And right now, that review of that is kind of invisible. We don’t know which one of these failures it is. Nobody records that. It just shows up as the agent didn’t do the thing that I was thinking it was going to do where it was slower than I expected. If we could see it, which part broke, basically, which compensation reaction we are using and where it came from, you would know what’s the property that is failing. I think that would be actually really cool and really helpful if we could instrument it somehow. [inaudible 00:29:46].
Brian Houck: It’s a like [inaudible 00:29:46] indicator of the quality of context. I saw we had a question in chat, and I’ll tease a little bit how I’m thinking about measurement systems towards the end today, but this is also a measurement system, is what were the human behaviors? A bit lagging and you hope to not catch it at that point, but absolutely.
So Eirini, staying with you, a previous paper that you and I worked on together was The AI-Native Developer. And in it, you described four phases of AI fluency and described how people could mature through those phases. And I’m curious, do you think as we get better at providing high quality context or at least better at recognizing and recovering from context failures, is that part of that maturity journey to become more AI fluent? Is it a prerequisite to begin with? How do you think about that?
Eirini Kalliamv…: Oh, can I say both?
Brian Houck: Of course.
Eirini Kalliamv…: I think that both providing good context and being able to recognize and recover when the context is bad is both things that happened through the stages, and they probably happen in that order too. I think nobody starts by knowing how to write good context. And that was true in the interviews that we did as well. People start by getting burned and they learn from that. The agent does something that sounds kind of confident, but also is wrong. And we start working backwards from what we realized it didn’t know. That is definitely the recovery behavior. That’s where the learning seems to happen. And so the people that were in the further stages of AI fluency, they had now built a habit to do things the other way. They would front load the things that they have by being burned by those in the past, they’ve realized the agent absolutely needs to know. And that meant goals and what is off limits and signaling when something is done, like what’s the success criteria?
So all that to say, I think providing good context is kind of what recovery looks like when you’ve done it enough times. And so in the stages, I would say the stages are about how the behavior changes relative to the work. And a lot of that change is what you hand to the agent and how you structure that. So I think thinking well about context is part of what the stages are. And of course you exhibit that as you go through the stages.
Brian Houck: That sort of makes me think that today, part of how we evaluate a great engineer is how good are they at writing code? And things change particularly as you get much more senior, but it’s like how we write code and the quality of the code we write is part of how we’re evaluated. I’m curious for anyone, if anyone has a thought, in the future, will more of the evaluation criteria for what makes a good builder or maker or software engineer or whatever we call roles in the future be the quality of the context that we are creating? I don’t know, does anyone have any thoughts on that by chance?
Max Kanat-Alexa…: So even now, when you are a senior engineer in an organization, a lot of the nature of your job has been like I describe in English clearly what I need and then the quality of the clarity of my description determines the quality of the result that occurs. So I think that actually already was a thing. And it’s not a thing that most, I think, developers experience unless you have become a staff engineer basically or hire at a company, especially at a larger company. But once you do, you see that that experience has already been very real and you already also see the consequences of failure and how expensive they can be. And you see that… I think that that discipline will spread more and more through everybody who works and creates software.
Margaret-Anne S…: And I think companies like CodeScene is leaning heavily into how to evaluate and improve quality of the code. And this could also be an important part of your context or maybe a constitution where you put in, here are my expectations for the quality of the code. And that could also be a direction I think that you could improve your context to lead towards that quality.
Brian Houck: Yeah, absolutely.
So Eirini, I earlier said that a property of context only earns its own pillar, its own dimension if it isn’t derivable from the others, can be acted on independently and names a failure mode, a real world failure mode that the others miss. And I’m curious, which of our five dimensions is weakest under that test? You sort of already hinted at it when talking about what your favorite dimensions were, but also can you remember any dimensions that we thought about that then failed that test and so didn’t make it back in?
Eirini Kalliamv…: Yes. So yes to my earlier point, the clarity actionability, I think that we distinguish them based on how they fail and what would fix that failure. In terms of what we left out, we left out accessibility. And we did that on purpose. And I think we described that in the paper too, whether the agent can reach the information at all, that is something that comes before any of the five dimensions. And so it does sit outside.
Brian Houck: A prerequisite.
Eirini Kalliamv…: Exactly. So it does sit outside the framework. I think that there were a few other candidates that we talked about how current the context is, where it came from. I think we folded that into fidelity. But yeah, the accessibility one was a big one.
Brian Houck: Yeah. Ryan, I see your comment in there about security and that was another one we debated on quite a bit and also debated should it be included and should it be somewhat visually separated with the parenthetical? And we do talk in the paper on why we made those decisions.
And so last question, Max, I’ll kick this off to you and then by all means, Eirini and Peggy, if you have additional thoughts, please ladder them in. So in the paper, we argue for a context review discipline, sort of like we have a code review discipline today with effort proportional to how often a piece of context gets reviewed. So obviously if you have context that you’re feeding it to your agents every time, like a shared skills file and agent MD file, your effort on reviewing that will be much higher.
And code review has taken the industry decades to normalize if we ever have. Documentation work is often one of the first things that developers cut when they feel time pressure. Everyone likes to have documentation, no one likes to write it. And so my question to you is, as a real world practitioner on the panel, is context review actually realistic?
Max Kanat-Alexa…: That’s a great question. So the goal that you should have is to have great context. Regardless of how you get there, that’s what you’re trying to accomplish. And there are times when putting a human review gate in front of that process actually causes you to have lower quality context. We saw this for years with documentation where if you allow people to do very easy low review or no review documentation updates, you get a lot higher quality documentation because people all over the company, they’ll just show up and they’ll fix your documentation. On the other hand, there’s also always been this same problem of there’s a few types of documents or a few things that are so critical and so important and hard to get right and require expertise that somebody has to control what goes in there. And it has to be the person who has the expertise or the skill required in order to control the quality of that thing.
And so basically your goal is to do whatever gets you that quality, but sometimes there are things that are so critical and in particular usually can only be correctly done by a small group of people or even sometimes a single individual because they’re the person who has all of the business knowledge in order to make sure that that thing is saying the right thing that you really would benefit from having a fixed and specific review process that is controlled and enforced on those things.
Brian Houck: Yeah, absolutely.
Margaret-Anne S…: Can I add something to that too?
Brian Houck: Please.
Margaret-Anne S…: So thinking about code review, I mean, code review was done to check the quality of the code, also to make sure that it could be maintainable and easy to change in the future. But when we asked developers what was one of the main benefits that they got from doing code review, it was shared understanding about what the code is doing. And I think that if we have a context review, say in a retrospective, that one of the things that we’ll get from it is this shared understanding and this collective, I guess, evaluation. Is this enough context? Is this the right context? But also kind of shared across people. They know what it is that we’re trying to do. So I just wanted to mention that as well.
I’m so excited to see this. I love this idea, by the way, Max, of having context review. Please tell me when you try it and I know how to go about doing it myself.
Brian Houck: And I will say in the paper we go into much more depth on what are real world recommendations for validating and improving the quality of your context. Because I think we talked about code review and code quality. So much about the quality of our resulting software has historically been dictated by the quality of the code that makes it up. And low quality code can have compounding failures. And we think that the same will be true with context where so much of the end result will be dictated by the quality of the context that we hand to our agents, particularly as they work more autonomously. And it’s going to create this flywheel effect where high quality context results in good outcomes, which becomes tomorrow’s high quality context. And on the flip side, crappy context will result in bad outcomes that will feed future bad agent interactions. And so there is this compounding value to improving your context now.
Eirini Kalliamv…: Yeah. I was going to make the same point about leverage because you asked the question, is this realistic? And I think keeping in mind that it’s realistic if it’s proportionate is what makes sense here. The discipline that we’re talking about is there’s a small set of things that every agent on the team reads, whether it’s instructions or architecture docs and so on, and that’s where the leverage is. If you get that right, then every agent becomes more capable with each right. We get that wrong, then the same mistake runs in every session. So I think the effort is supposed to travel where the impact is, which is something that we also apply to code today. You don’t give the same review and attention to a throwaway script versus something that is a shared library. So I do think that keeping that proportionality and where’s the leverage I think makes sense as a criteria. Yeah.
Brian Houck: And with that, we are just about at time. I would like to thank all of my panelists and co-authors for joining me today. I’d like to thank everyone for attending. Again, we have pasted the link in chat we’ll be sending out afterwards. Please go and read the paper. Let us know what you think. We’d like to know does it actually match your real world lived experiences? Do you find it relevant for what you’re thinking about right now? I saw some questions pop up about, “Great, well, how do you measure this?” CAFE(S) is explicitly not a measurement framework. It is a definition framework. It is shared language. What I am working on right now is great. Well, how do you actually go and measure that in the real world? And I think we’re going to borrow things from how we measure code quality. I think we’ll test and analyze context, but we’ll also evaluate agentic interactions and transcripts against the context that was given to them. We have a feature called agent experience that is this very thing, and so I will be sharing more about that in the future.
And the last thing I’ll say, if you have any questions, feel free to forward them to Brian, B-R-I-A-N, @team-gitdx.com. And like I said, we’ll try to encapsulate all of these questions and answer them in a dedicated newsletter. And so thank you all so very, very much.
Eirini Kalliamv…: Thank you.
