Brian Houck:
Hello, welcome to the Engineering Enablement Podcast. I’m your host, Brian Houck. Very excited for this week’s guest. We are joined by Andrew Swerdlow, who is VP of engineering at Roblox, where he leads AI acceleration and core platforms. Andrew’s working on some really interesting problems on what it takes to make software engineering genuinely AI-native.
Before Roblox, Andrew led trust and safety engineering and AI fairness at Instagram. He spent nearly 16 years at Google working across developer tooling, privacy, and products like YouTube and Android. Andrew is also the author of Tech Leadership: The Blueprint for Evolving From Individual Contributor to Tech Leader. And on a more personal note, Andrew and I are of the same sort of branch of the developer productivity tree he studied under the amazing Dr. Peggy Storey. And so it’s always fun to have someone that sort of came from the same lineage of developer productivity as I did.
Andrew, welcome to the show.
Andrew Swerdlow:
Thank you so much. I am so happy to be here. I’m so grateful that you gave me the invite and happy to always talk with you. So yeah, I’m looking forward to today’s conversation.
Brian Houck:
Great. Let’s jump in. So you’ve said that we’ve solved typing, but we haven’t solved trust.
Andrew Swerdlow:
Yes.
Brian Houck:
You’ve said that we’ve solved typing, but we haven’t solved trust. What does it actually take to close that gap and get from AI that can generate code to AI that you’re comfortable letting act autonomously?
Andrew Swerdlow:
Yeah, I think that is really the hot topic right now because what we’ve noticed is that it’s pretty easy to generate code and that’s what we’re seeing, a lot of PRs are being generated, but the overall lead time into production hasn’t really moved all that much. It still takes about the same amount of time to ship into production. And so we start looking at, “Well, we have all this code and where’s it getting bottlenecked at?” And the first stop is actually, it’s code review. And hopefully we’re going to talk a little bit about what we’ve been doing to solve code review, but there’s also deployments and testing and debugging production and all these other areas where I don’t think AI has been as effective yet. And so what we really need is we need AI to be infused across the entire software development life cycle to actually get the full promise of it.
But to do that, to actually give AI the keys to be able to do a deployment, you have to trust it that it’s not going to create a sev. And so that’s where the gap is right now, is being able to know that an AI agent is not going to cause a regression in production or create a sev or ultimately ship some sort of slop or mistakes into your product. And so one of the ways that we’ve been thinking about this at Roblox is how do we put in place those guardrails? What are the sets of controls and system design that would be needed to prevent bad things from happening? And as it turns out, those sets of guardrails, those sets of trust systems are actually the same ones that we’ve been saying for almost a decade are important for us to invest in. So it’s things like great unit test coverage, it’s integration testing, it’s canaries into production, it’s the ability to detect and roll back issues.
And so actually the companies who I see moving the fastest are ones that had been investing in continuous deployment and good testing infrastructure all along because now they have those guardrails that are in place that help them establish trust. So that’s a piece of it.
Brian Houck:
Yeah. So I mean, I’ve said on this podcast before that AI acts as an amplifier. It shines a spotlight on a lot of the properties of your engineering system that were always true. As many organizations think about trust, a big question is where do we actually need humans in the loop? And I know you’ve talked about humans in the loop can often act as a tax. And so when is human oversight actually good safety engineering and when is it a crutch because we haven’t built the right systems in order to operate without having a human there?
Andrew Swerdlow:
Yeah. And I think to be clear, humans need to be accountable for the products and code that we build. And so I’m a strong believer in that. I do think where we’ve decided right now to put humans in the loop are not necessarily the right spots. And so really been thinking about how do we move from things like humans reviewing every single line of code that AI generates to humans teaching systems about what kind of code could be good in production and what kind of code is bad in production. And so that way you’re actually teaching a system versus reviewing all the output of AI. But to do that, we found that there are some systemic issues with the current technology suite that is available to us. So if you look at most of the Frontier models or open weight models, these things are trained off of the internet data, publicly available data or smaller sets of data. We’re talking petabytes of data, but companies store exabytes of data. That’s orders of magnitude different in terms of the amount of data.
And so it’s no wonder that a Frontier model or an external model doesn’t understand your organization like internal employees do because it doesn’t have access to the same level of knowledge base and experience as businesses. And so one of the things that we thought about early at Roblox is how do we take Roblox best engineers and capture their knowledge and infuse it into agentic interactions? And that’s been a journey that we’ve been on for the last year and a half now and some really exciting results with that.
Brian Houck:
So there’s actually a few really interesting places we can go with that. I think one of the things I’d like to dig into a little bit more is this notion that Frontier model can know an enormous amount about software engineering, but doesn’t know anything about your actual organization. I think certainly trying to encode some of those best practices and the guidance from your senior engineers is important. But what else does an agent need to understand about your engineering system that isn’t already in the model itself?
Andrew Swerdlow:
Yeah. So if you’re working at a large company with a large code base, you might have picked up some patterns that might be different than the industry norm. So for example, maybe you’re using an older version of Python internally, and so maybe you’re still on 2.x or whatever, and meanwhile, the entire industry is multiple versions further. A good model would probably tell you, “Hey, there’s some wrong pattern here. You need to fix that.” But actually there might be a conscious decision internally inside of a company to say, “We don’t want to migrate yet. That’s actually a large set of work and we’re not ready for that. And so we actually want to use an old outdated language version or library.” And so how would a model know that? You have to kind of instruct it on what behavior is expected.
And so the way that we think about this is that all this knowledge of how you should build inside of your company is already there. We have this very rich historical information of what it took to build the existing product that we have in production now. And so being able to mine that data and be able to extract all of the interesting insights from that evolution of your product and turn it into some type of judgment that can be used by agents either for code review or even generating new code, that was our goal.
And so we actually built a system and it sort of has a couple different components to it. The first is this idea of extracting out of the 20 years of historical information of Roblox, can we actually pull meaningful insights out of the code base? So looking at all of the different commit histories, looking at code review comments, looking at sev data, Slack information, all of those sorts of things and turn that into interesting signals that could be used to guide agentic interactions. And so we did a whole bunch of research around this and extracted a ton of data and did some analysis. And what we found is that the highest fidelity signal is actually code review comments because in code review comments, this is like your engineer is telling people how they should write code inside of your organization, but it’s a noisy set of data. So people are kind of nitpicking on things or making relevant comments or using different language patterns.
And so a tough part of the challenge is how do you extract a huge amount of data, normalize it, cluster it and make it useful for agents? And so we came up with a method to do that. And one of the things that we found is that there is sort of a temporal element to it that actually most of those comments from 20 years of history are not relevant. It’s actually probably only about the last year and a half to two years that are relevant.
Brian Houck:
Sort of like decays over time.
Andrew Swerdlow:
Exactly, decays over time because patterns change, the software changes. And so we found the sweet spot for this data mining activity is looking at just the past 18 months to two years worth of data. And so we have this process where we extract all the comments, we use an LLM to normalize those comments and put them into sort of a rule format. And then we discard a bunch of things that don’t look like a rule that are hard for us to parse. And then we actually, we do a clustering of them and we essentially have a simple process of just inserting those comments into a vector database and looking at clusters of similar comments. And once we have those, we extract those into something that’s like a rule. So we have another LLM process that actually takes that and puts it into a rule format. And then we created this nice UI that takes this rule, which we call an exemplar, and puts it in front of different repo owners.
And so you can actually go to a nice UI and see a set of 50 rules for your repo that were automatically extracted through this data analysis. And the cool thing about it is that these have citations. So you can see this particular rule was cited 50 times. And if you click on it, you can see who are the people that were citing this rule. And you can see some reputation level signals around those people. “Oh, that’s the senior engineer. That’s the person who’s most knowledgeable. This must be an authoritative rule.” And so we say that a repo owner can go through that list and then easily click to add each of those rules to the repo. Essentially it gets stored as a YAML file in that repo, and then becomes a rule that gets followed for things like code review or even new code generation. And so that’s the exemplar piece of pulling out what we consider historical information.
Brian Houck:
So I feel that good code reviewers, people who leave high quality comments, they have always been worth their weight in gold. And I think that we historically have dramatically undervalued engineers who are phenomenal at leaving code review comments. And I’m curious, more than just frequency, do you have some sort of weighting for senior devs versus junior devs? And even amongst senior devs, those who have a strong reputation, is there some sort of weighting in how you treat these exemplars?
Andrew Swerdlow:
So right now it’s actually more about frequency seems to be the strongest signal. We did actually build an expertise classifier that looked at a bunch of different interesting signals like tenure, level, and then also experience within a repo itself because you might have maybe a relatively new engineer that is just most familiar with that code base. And so we actually did build a classifier to help us with ranking of these exemplars, but we found that that was less effective than actually just the repeated citation. And so we kind of combined the two a little bit so that you can see a ranked ordering list of exemplars for your repo, and then you can actually look and spot check who the people are that are citing them and implicitly know whether or not that’s an authoritative voice. But we don’t do any formal ranking based off of that right now. Maybe something to experiment again with in the future.
Brian Houck:
So as you mine out all of this wealth of engineering knowledge from the code review comments, in order to make your own AI code reviewers more effective, does that run the risk of you are going to dry up that future source of engineering knowledge for the future as you no longer have to have your engineers leave their own comments?
Andrew Swerdlow:
Yeah. So let’s talk through that. What would that lead to? So suppose that we’ve built a system so good that our code review agent is used to write all code review feedback for everyone and that people are happy with it. They accept it, they don’t reject it. I’m not sure that would be a bad thing. I feel like that’s actually sort of the end state, but we’re not there right now. We’re at a point where I think about 60%, or actually this week’s data is almost 70% of all PR feedback from our code review agent is actually accepted by humans. That still leaves on the table the 30% that wasn’t accepted for some reason, as well as things that might have been missing. And so I think there’s still a fairly big gap that needs to be closed for code review to be perfect. But if it was perfect, if it was only being used and that was the main source of truth and all comments were generated by AI, I think that means that we’ve achieved a high level of abstraction where we can start to trust that system fully.
Brian Houck:
Got it. So it’s like we aren’t going to need that source anymore. So you had said 70% of AI generated code review comments are accepted. How does that compare to human generated code review?
Andrew Swerdlow:
Oh, that’s a great question. So what we’ve observed and corresponds with the literature is approximately 50 to 55% of code review comments are accepted by humans that were written by humans. And so we’ve been able to do this before and after as well as we have some level of holdout groups, not every repo has opted into code review agents at this point. And so we have some strong conviction that our code review agent actually produces more acceptable comments than even humans at this point.
Brian Houck:
Oh, that’s really interesting. I would also suspect-
Andrew Swerdlow:
But it was a journey, I would say. It was a journey.
Brian Houck:
Yeah. It didn’t quite get there overnight.
Andrew Swerdlow:
Our first attempt was just kind of a naive, like, “Let’s run an LLM on a PR and get it to provide some feedback for us.” And with the noisy feedback that it was providing, only about 20% of the feedback was actually accepted by humans. And so it’s sort of been this long journey over the last year and a half to get to 70%.
Brian Houck:
So we got to this part of the conversation because we were talking about what is all of the organization specific knowledge that you have to be able to feed into the models that they weren’t sort of natively trained on. And you have this wealth of engineering knowledge that comes from the code review comments, but that is all about engineering artifacts that have already been created, like from the decisions we have made, what can we comment about them?
Andrew Swerdlow:
Yes.
Brian Houck:
And it’s so much of why we are making the decisions to begin with, why we are building these things are often never written down. And I’m curious, was that a problem that you faced in Roblox where you have all this information that just lived in engineers’ heads, lived in hallway conversations, or have you always had processes to make sure that those things get written down?
Andrew Swerdlow:
I think like most companies, documentation is very hard at Roblox and every company I’ve ever worked at. And so what I’ve learned is the code, the systems around the code, those are the sources of truth. Those are the most accurate sources of references that we can look at. And so yes, trying to extract this information directly out of docs was not the highest fidelity signal. And Slack was actually pretty good. We were able to see some interesting opportunities in Slack. But for us, code review comments was the most solid and attributable and turned into the most concrete forms of knowledge encapsulation. There are some other experiments that we’re doing, looking at things like incident reports and these sorts of things.
We’ve also provided a way for people to author their own rules that don’t need to necessarily be implicitly extracted. We have a playground where you can author rules and you can test it against sets of PRs and see whether or not it actually finds the right issues. So we do have some rules that are hand authored. A lot of those come out of sevs. So we say we have an incident, we want to make sure that we have a exemplar that would prevent that sev from happening again. We go to the authoring tool, we can write up in natural language with some one shot examples about what we want to be able to detect and prevent, and then we can test it on the PR that caused a sev or even build some PRs that try to tickle that particular rule.
And so that’s another way for us to merge both explicit and implicit exemplar rules. And I think that has bridged the gap for us. I think we’re always looking for more signals, but we also want to do it in a way that doesn’t become noisy and that is high value. We don’t want things that are nits. We want things that are really important and would potentially prevent defects from escaping into production.
The interesting thing about this is that after we started this process, we had a whole new surface area where we could start to add things like postmortem follow-up action items too. Because in the past, whenever you had an incident, usually you’d be like, “Hey, we need to add some tests. We need to add an alert. We need to do these things.” And maybe you update a runbook or engineers just know not to do a certain thing and you create a doc for it, but those things decay over time, people leave the company, tribal knowledge leaves. Now there’s a way to programmatically encapsulate those learnings into something that gets injected into every code review. And that’s something that’s an entirely new capability for us.
Brian Houck:
So I am a huge proponent of code reviews, not just as a way of finding defects. In fact, I think they’re probably not a particularly great way of finding defects. I think they are a phenomenal way of transferring knowledge, of raising transparency around what our peers are working on. It’s a way of creating sort of shared understanding.
And as we automate this away, we sort of lose some of that. Now, some of that we might not need anymore as you mentioned, but we still need some… I still feel like there’s something lost if we don’t understand what it is our peers are doing, what it is that is being built by the engineering system. I’m curious how you think about that and what you’ve seen so far.
Andrew Swerdlow:
Yeah. I think my intuition here is that the things that matter the most right now is still intent. It’s the intent of what this change is supposed to be doing. It’s the architecture designs about how you’re achieving those goals. And it’s about maybe the spec is about as detailed as maybe we’ll end up going in the future because I think the underlying syntax of how a function is labeled and what sort of naming convention we’re using, all that kind of stuff, I feel like will be below the agentic abstraction layer in very short period of time. We’re already seeing it now where we’ve seen this shift from everyone wanting to operate in an IDE, looking at every line of code, to being functioning mostly in a terminal at this point. And it’s not to say that people aren’t still looking at the code, but it’s less and less.
And so if you project that out into the future, six months, one year, two years, I think most people will be living at a higher level of abstraction for these systems. And those three things that I just mentioned feel like they matter a lot to me. It’s the intent, it’s the architecture, and it’s the spec of the system that you’re building.
Brian Houck:
So I love that. I agree that as we move away from code as sort of the first order artifact and we really move to intent as the atomic unit of work, then it’s like how do we track that? And that’s the thing that we’re trying to understand. But I believe I’ve heard you say that design docs are dead as we move to a world where it’s just so easy to rapidly create prototypes. And help me resolve that in my head.
Andrew Swerdlow:
Yeah. First of all, I want to give credit. That was Malte from Vercel, the CTO of Vercel. He was giving a talk at Roblox. I think you were there for that one. But he used to be one of the main design doc folks from Google back in the Google days and evangelizing the value of design docs. And his thoughts on this were that you don’t need a design doc when you can make a prototype faster. And I agree with that. I think that if your goal is to get feedback on the contours of a system and iterate on some of the details, then a prototype feels like a good way to do that. And so I think the time that you would spend doing those sorts of reviews should actually just be going into prototype evaluation and iteration.
So I think yes, design docs might be dead, long-lived specs. And specs, I suppose you could say is a form of a design doc, but it’s more, I think, tied to the direct implementation of the thing that you’re trying to build and going through the details of the exact implementation and architecture.
Brian Houck:
So as we move more towards then spec-driven development, spec being the sort of engineering artifact that expresses our intent, what makes a specification good enough that you can just sort of hand it to an agent and it can work autonomously off of it? What are the attributes of a good spec?
Andrew Swerdlow:
Yeah. So I feel like this is still an area of active discussion. I don’t feel like we’ve really consolidated on a canonical spec format for all of Roblox. It’s sort of picked up and used in different areas at different various degrees. But the things that I like about spec-driven development is that it’s a lot cheaper and it’s a lot easier to iterate and align on a higher level design and specification than it is to actually go and spend the tokens on building something. And so when we’re moving into budget conscious faster iterations, doing the cheap thing first, which is iterating on some words on a paper is a lot easier to change than even the fast prototype that you can build with AI.
And so what does a good spec encapsulate? It allows you to see what the overall system is trying to achieve, what are the approaches for building the sub-components, separation of concerns, all of those sorts of things. And what we found more recently is that even code reviews themselves are actually just being replaced by spec review. And so you can actually just send out the spec for people and have them provide feedback on that and that actually gets incorporated. It’s almost like the code review process itself.
Brian Houck:
I mean, I don’t feel like we have it figured out yet. I think everyone is trying to scramble on what is the right level of detail and the right level of consistency. I think one of the things we’ve been wildly inconsistent about as an industry is specs and how we define specs. And one of the things I’m thinking about as you talk through this, Andrew, is historically the way that we might evaluate a great engineer is, do they provide helpful code review comments? Are they great at mentoring the engineers around them? And do they write high quality code? Do our principal devs, are they great coders? And do you envision that we’re going to start shifting how we evaluate what it means to be a great engineer to the quality and the consistency of the specifications we write? How does it change what makes a great software engineer?
Andrew Swerdlow:
Yeah. So in my experience, what I’m seeing so far is that very experienced seasoned engineers, they’re still able to leverage these systems to a higher degree of impact than more junior folks. The pattern that I see is a junior person who hasn’t built the scar tissue of building large scale systems or having to maintain those systems, sometimes they’re a little too trusting of AI, and AI in its sort of sycophantic way will kind of build a system that you ask and maybe not push back enough. And therefore you’ll end up with something that might be suboptimal. It might work initially, but it might not be something that will scale or be maintainable or those sorts of things. Whereas an engineer who’s built a bunch of these systems, who knows the operating problems, they’re going to be a little more skeptical of maybe the first response from an agent and they’re going to push back on it and they’re going to be asking questions about boundary conditions and these sorts of things where maybe the junior person wouldn’t have.
And so I do think it’s this level of skepticism, this level of pushback, this level of iteration to get to really the ideal outcome is something that we’re going to want to see in more junior engineers. They have to develop that. And I think it’s not going to come naturally to them until they actually see that, “Hey, if I just go with a basic spec and take whatever the AI has given me as the output, if that’s going to have to be rebuilt five times, so the half life of the code in production is pretty short because we didn’t do a good job at coming up with a good spec or a good prompt to design it,”
And so yeah, I think there’s going to be a lot of new dimensions for how we think about evaluating people. Certainly most companies were not looking at code output as the single metric anyway. And so all the other things that we were evaluating on teamwork and cross-functional behaviors and leadership and all that kind of stuff isn’t going away. I think it’s just more across the execution dimension where output is easy, longevity and quality and product fit is stuff that matters more.
Brian Houck:
You have to have those scars.
Andrew Swerdlow:
Yeah, exactly. And scar tissue.
Brian Houck:
As you say, yes, I think on one hand, these are things we always should have valued more, I think. And it’s just now it’ll be a bit of a forcing function, which I think is a good thing. I also, as you are talking through this, wonder how will we change the way that we interview for junior engineers? And in the future, will junior dev interviews look a lot more like senior dev interviews where it’s a lot more sort of system design and system architecture versus I think we’ll move away from coding whiteboard problems.
Andrew Swerdlow:
Yeah. I’ve seen a few different approaches that companies are taking. Some of them actually want to see how well you can use AI. And so instead of doing a coding interview, it’s a coding interview with AI. Can you actually build a system zero to one? What’s your process? Is it efficient? I think efficiency is a key dimension that we’re going to dig into and prize a lot more than we did in the past. Because if you think about it this way, you could have two engineers. One engineer is sort of prompting their way to sort of vibe coding to get to a solution and maybe burning a lot of tokens. And actually the cost is a lot higher for that engineer opposed to another engineer that’s maybe kind of working in plan mode, iterating on a spec, and then ultimately coming up with a very nice and elegant solution and then single sort of pass to get it implemented, leveraging a lot less tokens to do that.
And so my intuition is that we need ways to evaluate that, to look at how people are building better, looking at their tool set up, their process. And that becomes the thing that we evaluate, not just whether or not they were able to converge to a solution because I think the convergence to a solution will become a commodity, but how you get there is a thing that we need to evaluate.
Brian Houck:
So we’ve touched on a lot of different places where software engineering may be changing, changing the way we do workflows around things like code review or even sort of incident management. And a term I hear thrown around a lot is AI-native development. And so when you think about what makes an engineering organization AI-native beyond just like, “Oh, we’re using a lot of AI tools,” what does that even mean? What are those attributes to truly be AI-native?
Andrew Swerdlow:
Yeah, I feel like this is a term that will probably have a shelf life by the way, because I don’t think it’ll be long before every company can claim to be AI-native.
Brian Houck:
You’re just an engineering organization.
Andrew Swerdlow:
Yeah, you’re just an engineering organization, right? But I do think there’s probably a spectrum of where people are thinking about in terms of AI-native. And the V0 here is that if you use AI deeply for all of your engineering work, that’s probably the base table stakes there. And I think most companies are getting to that point. And in fact, even at Roblox, we’ve been trying to measure how AI-native we are. And one of the dimensions that we’ve been looking at, which many other companies do, is how much of our code is actually written by AI? I think that was probably the first question that we all started asking ourselves about a year and a half, two years ago.
So for Roblox, most of our code is written by AI. And we can say that concretely and scientifically because we’ve built a pretty sophisticated fingerprinting system that looks at every single generated line of code across any different tool, whether it’s Cursor, Copilot, Codex, Claude Code, and we store that fingerprint. And then every day we look in our GitHub repo and we try to do that fingerprint matching about what actually made it to a merged line of code in our production code base. And so we’re able to see if all the code is generated and see what goes into production and see what’s not generated by AI. And so that’s actually been the interesting metric, is how much is actually still brain coded or hand authored.
Brian Houck:
So real quick question on that, Andrew, because even a given line of code might not be all AI versus all human. And so how do you think about that? How much of the code has to be AI for it to be an AI authored line?
Andrew Swerdlow:
It has to be all. So we actually take a hash of the line of code and we do a direct hash mapping. And so for us, if someone does go in and tweaks a case, we would consider that hand authored. And so we’re kind of giving humans the benefit of the doubt there a little bit. But if someone had to go and specifically look at that piece of line of code, it’s talking about attention and it’s talking about their understanding.
And then the interesting question is like, well, that metric no longer has value for us. But actually the system that we built has great amount of value for us because now what we can see is we can see all the lines of code, which sort of harness, model, version were used to produce that code. And then you can start to map that to things like, “Well, how good is that code? What is the half-life of that code in production? Does it get rolled back because there was a bug or a sev? So which models produce bugs and sevs and those sorts of things?” And then we even try to store things like session information, prompt information associated to what did it take to generate that line of code, how much did it cost, how many turns of an engineer did it take to actually produce that production line. So then we can start to ask some very interesting things about the cost of a line of production code or whether or not-
Brian Houck:
Cost as in actual, not just human time and effort, but also token cost, I assume?
Andrew Swerdlow:
Well, token cost. Not just to do the last generation, but maybe even all of the successive turns in a single session or maybe even doing session mapping for similar sessions used to prototype before actually implementing that line. So there’s some very interesting metrics that we’re starting to just scratch the surface with, but also capturing that prompt stream has been super useful for us because it helps us understand overall efficiency and waste. And some very fascinating metrics that come out of that around things like 60% of all of our token uses, it’s actually not to generate code, but it’s to understand code or to ask questions and that sort of thing. And so that’s a huge amount of spend for essentially knowledge management and understanding and only 40% is actually used to generate production code. Or not even production code, just to generate code and only some subset actually survives to make it into main. And then we look at things like waste. So we saw a huge amount of waste around context bloats and tool failure silently, cache misses, all that kind of stuff can be looked at once you have great observability into your Agentic coding tools. And so we’ve actually modified some version of OTEL tracing to be able to leverage our existing observability stack based off of Grafana and OpenTelemetry and using that to instrument the agents as much as possible to be able to do that analysis and find opportunities for efficiency improvements, not just at the system level, but also using it to coach engineers on how they might do a better job with their Agentic software development, like, “Hey, you have this 30 turn session with a totally full context window that’s going through compaction all the time. Did you know that that actually might not produce the best results for you and is awfully expensive?” And so, “Hey, this can be great information to teach people how to use these tools even better.”
Andrew Swerdlow:
One other thing that we’re experimenting with that we’ve noticed is compaction, which is this process that harness will do try to reduce your context window size to try to make it more efficient. It’s sort of a little opaque and non-deterministic. And so we actually have been trying to build something around checkpointing sessions so that if you create actual forced checkpoints of longer running sessions and then prompt people to actually start a new session with the previous checkpoint, we’re seeing some interesting value in that, one, because you can actually pick up a session that maybe drops maybe unexpectedly quit or works out in some way, or you can actually be more efficient with your tokens by starting from a previous checkpoint in a new context window. So I think there’s lots of interesting technologies that can be built to help engineers. And the more automatic you make them for people, I think that will be the win.
Brian Houck:
That’s really interesting because not only is it more efficient, but there’s sort of this agentic failure mode where agents get distracted. And so if you have some hyper relevant context buried in a much larger context window, agents will get distracted. I’m curious as you start new sessions over, if that allows the agent to refocus. So not only is it more efficient, I wonder if you get better outcomes actually out of it as well.
Andrew Swerdlow:
Yeah, that’s a great point. So for code review, we have a very specific example of that. So one of the things that we do for code review is we say for every exemplar for a particular PR, each exemplar has to run in its own agent, its own agent. So that way it has its own context window. And what we found is we found better results and more determinism by lowering the amount of things that you’re asking an agent to do at any specific time. And so the more that you can separate concerns, the better because LLMs by their nature can be non-deterministic. And the more that you put in the context window, the less likely it’s going to be that you’ll actually achieve all the outcomes of the things that you’re trying to put in there.
So one of our first attempts of doing our code review agent was actually just taking all of the exemplars, shoving them into a single rule file and having one agent just go and run them all. And what we found is that that decayed very quickly. It was sort of proportional to the size of the context window and the number of things that you put in there. And so separating things into their own agentic runs certainly in our experience is producing much better outcomes.
Brian Houck:
So if you have this incredible map of tasks that’s trying to be completed to what is the model, the harness that you’re using and can trace it through some outcomes, then I’m curious how much that feeds into things like model selection and how fine-tuned that is or you still leave that to the discretion of individual developers?
Andrew Swerdlow:
Yeah. So model routing is a huge topic right now. It’s all the hotness. We’ve had an intuition like many other companies that certain models perform better on certain tasks and are more efficient. And so we have experimented a lot with building some of our own model routing technologies. But what I would say is that most of the Frontier companies are actually trying to provide some of these services as well.
So there’s a very famous IDE that was recently acquired by SpaceX that has model routing capabilities that are pretty good and they are tunable so that you can balance things like cost or intelligence and those sorts of things. My expectation is that all of the Frontier Labs will provide their own routers. And so I wouldn’t spend too much time on building a brand new router from scratch itself. I would leverage some existing thing out there.
This is one of the mistakes that we made early in our sort of agentic transformation of the company was we were trying to build things that we shouldn’t have probably built because the industry is moving fast, faster than we can move. And so trying to focus on the things that we have the most leverage on that are unique to us has been where we want to put our energy. And so that’s why the whole idea of extracting all of the judgment and information from our own knowledge sources and inject that into agentic interactions was an important effort for us because that’s something that most companies are not going to do at the scale that we wanted it to. But building a router for a harness that already has thousands of engineers trying to improve the product is probably not a winning bet to invest your time on.
Brian Houck:
Although my cynical side does need to point out that leaving that to the Frontier Labs or organizations that are associated with Frontier Labs as xAI is sort of in that family over there, then they have different economic incentives and also are constrained by a set number of families of models. And I’m curious, who will end up filling that gap, I wonder?
Andrew Swerdlow:
Yeah. So when I talk to other folks doing similar jobs to me, other companies, one of the pieces of philosophy I try to leave them with is that this is still a very dynamic, fast moving domain right now. And I would say almost on a weekly or biweekly basis, there are emerging new technologies that really change the landscape. And so one of the things that you should do as an engineering leader building an agentic organization is try to create agility and flexibility.
And so at Roblox, we are very multi-harness, multi-model, trying not to close the door. We want to use whatever is the best tool, most economically efficient. And so we allow engineers to use a broad set of Frontier level tools as well as open weight open source models. And so we believe that trying to create vendor neutral abstractions or not go through one way doors for specific product features kind of allows us to very easily migrate between whether it’s Codex or Claude Code or Cursor or Gemini. These things should be up to the discretion of engineer and they should vote with their feet on which tool might be the best.
And so if someone’s providing a model router capability and it’s not effective and it’s not doing a good job, then engineers are going to vote with their feet. And we’ve seen that. We’ve seen that already before where someone will launch a feature, they don’t like it, and then within days people have migrated already to a new model and harness almost seamlessly. There is some work to do, but I feel like that’s the job of the dev infra for teams is to make that seamless, make sure that skills are transferable and MCPs and no dependencies on special metadata files that can’t easily just be rewritten or sim-linked, that sort of thing.
Brian Houck:
As you describe this new future of everyone is an AI-native engineering organization and we are really focused on driving specs through the system and our understanding of sort of line level code might decrease over time, a lot of the metrics that we have historically used to track developer productivity, certainly, I don’t know if they become meaningless, but their meaning changes dramatically. And so how are you thinking about measuring if you’re truly getting to that AI-native engineering organization essentially?
Andrew Swerdlow:
Yeah. I think one of the key things still goes back to one of the other topics we were talking about is autonomy that truly agentic-native companies that are leveraging autonomy really, really well are going to see the most benefits. But autonomy has to be earned through trust and through safety. And so really thinking about those early guardrails that we talked about at the beginning around test coverage, integration testing, canaries, those are the things that are going to get you to autonomy. But how do you actually know whether or not you’re running autonomously, I think is a question.
And so what we’ve put together is a rudimentary metric around this. And essentially it’s long-term session that converges. And so we have a metric that we look at that looks at the P50, the P75, the P99, P99.9 and looks for how long are sessions getting because that means that an agent was able to do something that took more than 30 seconds. Because if it’s only 30 seconds, a person’s going to have to be on their keyboard interacting every 30 seconds to give a new prompt. And what we want to see is more long-running tasks that actually achieve tough projects. And so we haven’t quite-
Brian Houck:
How do you tie that together? Because I’m like, “Okay. I could make a task run for a real long time that doesn’t do a whole lot.”
Andrew Swerdlow:
Yes, yes, yes, yes. So we actually use AI to evaluate the sessions because as I said before, we have this very rich session information and we can see all the behaviors of the agents, whether or not code actually ended up merged into production. We’ve built this sort of end-to-end pipeline of data. And then when we start analyzing long-running sessions, autonomous sessions, we actually ask LLMs, did this survive into production? Does this look like a good use of time? How complex was this task? What kind of task was it? We developed a taxonomy that we had PRs with. We use an LLM to determine whether or not a PR is a new feature, a bug fix, all those sorts of things, mostly because we wanted another metric besides autonomy, which was actually feature velocity.
And so this is something that I think most of us have wanted for a really long time. And I have had several failed attempts in my career to build a feature velocity metric by annotating code, that sort of thing. But we’ve kind of reached a point now where LLMs do a very good job at analyzing a PR and being able to determine whether or not something is a feature, a bug request or bug fix, that sort of thing. And so every PR that we have right now is tagged with one of those dimensions and we can actually look at a normalized feature velocity rate per engineer and we can see whether or not the introduction of new tools actually increases feature velocity. Because in my opinion, feature velocity is the closest proxy to innovation that a company can have because if you’re shipping new and interesting features, you’re most likely an innovative company. And if you’re innovating, you’re probably going to find some great new ways to delight customers. And so that would be an interesting metric that I like to use.
Besides that, I do think there are still some other really important metrics that you and me both share as a thing that we like in terms of how are engineers feeling. Do they feel like they’re productive? Do they feel like their output is more and are they happy with the tools and all those sorts of things? And so we can give everyone access to all these tools, but if they feel like they’re still being slowed down by something, code review overload or whatever it is, issues with debugging production, then we haven’t really solved the productivity problem with these tools. And so still looking at the qualitative feedback coming in from our engineers, a set of quantitative metrics that are shifting away from just the raw PR metrics into something newer, maybe more like a feature velocity metric, those are the things that I think will matter most in terms of measurement in the future.
Brian Houck:
I love that because I’m a huge proponent of measuring across lots of dimensions and having checks and balances. And I think if you are analyzing the sessions themselves, I can tell you a lot about efficiency. Obviously developer sentiment can tell you about what is the human cost of doing it. And then feature velocity, I am a firm believer as we move to intent-driven development, then our innovation velocity, our idea to customers is the way I talk about it, is the most important speed metric and that is effectively feature velocity. How quickly can we deliver new capabilities to our customers? And so I love that.
I’m curious, as you were looking at evaluating the sessions to see how long-running are they, that in and of itself can get pretty token-intensive. And so do you sample? Are you evaluating everything? In practice, how do you recommend organizations look at that?
Andrew Swerdlow:
Yeah. Efficiency is still really top of mind for us. So we’ve successfully driven the adoption of all these tools. Everyone has all the tools. They’re using them all the time. And now we’re kind of looking at the ROI and the budget. And so that’s always top of mind for us. And we have an entire AI efficiency team that is looking at how do we sort of optimize all of the different types of workflows. So for example, our code review agent, it actually does an ensemble based approach for code review where it uses multiple Frontier models to evaluate the same exemplars and it’s sort of like alignment engine react loop. And that could be very expensive when you’re talking about many thousands of PRs per day, fanning that out for every exemplar, could be 50, 100 exemplars per PR and then multiple models.
And so we focus a lot on trying to make these systems optimal. And so one of the engineers, a great engineer on the team is the tech lead. His name is Kushan. Go Kushan. He came up with a really nice caching strategy for all of this where we noticed that when you select a set of exemplars, you’re actually operating in a very specific area of the DIF. So maybe it’s like exemplars around testing, for example. So actually the type of context that you need to evaluate those exemplars for that PR is going to be all the tests. And so what we do is we pre-warm the cache with a bunch of this sort of metadata around the PR, around those tests, and then we leverage that cache between agents to be able to run those exemplar evaluations across the same context data. And it reduces the overall cost to run each of those exemplars by 90% because most of the cost is actually the context data.
And so that’s just an example. But for the session analysis, so for session analysis, actually there’s a lot of deterministic, just normal code that’s being used to capture that data and storing it in an efficient way for us. And so we actually use, like I said, OpenTelemetry as our tracing data plus some other pieces that we ended up building ourselves and probably might open source in the near future and storing it in a really efficient database. So we actually use a StarRocks database that is quite efficient. And then we do run deep analysis on most of those prompts, but we try to do it in a more cost-effective way because it’s actually, it’s not real-time data. This is more like async data that can be run at nighttime on cheaper models and more efficiently even open weight can be used to extract insights from these things. This is not the stuff that you need the highest level of reasoning, like extra high, top Frontier model. You can actually offload that to more efficient systems. So it’s about choosing the right thing for the right system.
Brian Houck:
Yeah. The right tool for the right task.
Andrew Swerdlow:
I do think the hardest part of AI right now is actually system design and infrastructure. And so the hardest problems to solve for us right now are how do we build trust into our systems? How do we make these things cost-effective? And how do we extract the ROI that we all want?
Brian Houck:
So my final question for you today, as you describe the future of what engineering organizations might be looking like and operating at these higher levels of abstraction, we’ve talked a lot about how the role of a software engineer may change. I’m curious, in the future, are we going to call each other software engineers or is there another term for it, do you think?
Andrew Swerdlow:
Yeah. We already have a different name inside of Roblox. We call ourselves builders. And everyone is a builder at Roblox. And what we’ve seen is the lines blur between engineers, designers, product managers to the point where these tools have been like an equalizer. And we have product managers that are fixing bugs and shipping new features into production by themselves. We had less of that before. There was always a few really hands-on PMs, but now we’re seeing that as almost part of the job is that you can lean in and actually build as much as a software engineer. But I do think software engineering as a discipline and a domain and a skillset is still here to stay. I do believe that some of our most senior engineers that have experience building high scale systems, they know where the pitfalls are. They know how to come up with the right spec and the right system architecture so that we build the right thing once and not have to rebuild it over and over again. And so I don’t see that changing. I just see the details of the work shifting.
But I think this is a transformation that we’ve seen in our industry multiple times. When my parents were in university and studying, they were using punch cards and writing assembly and all these sorts of things. And then higher abstractions languages came in. And for a while I think people were probably still looking underneath the hood and checking the assembly and maybe correcting things. And then eventually the abstractions became strong enough that you don’t need to go and poke under the hood and that people just trusted those language abstractions. And if there was a problem, you would solve the problem using the abstraction as well. And I think we’re just in that transition period with AI right now where we’re still trying to poke under the hood and make sure everything is optimal and correct. But as we build trust-
Brian Houck:
Yeah, we’re still building trust. Yep.
Andrew Swerdlow:
Exactly. Exactly.
Brian Houck:
As you think about then automating code review, one of the concerns is you have the model vouching for itself. And so I’m curious then, how do you know how much you can trust the AI to say, “Oh yeah, this AI generated code is good”? How much trust do you have there?
Andrew Swerdlow:
Yeah. It’s interesting. It’s like AI writes the code, AI reviews the code, AI deploys the code. So where is actually the human judgment in the loop there? And so I talked a little bit before where I’m a strong believer that people need to be accountable for their code. And so I think that has just shifted. So right now, the way we think about it is that all exemplars have to be explicitly reviewed and approved by a human. And that essentially creates this boundary condition to say, “This is how our code review agents should behave.” And so that way it can approve classes of changes and not just a specific code review change.
And we believe that that’s the right spot for humans to be in the loop. And we found that this has improved accuracy and acceptability of the feedback to the point where we’ve surpassed even the human level of feedback acceptance. And so much so that we’ve actually decided that we want to allow some PRs to flow into production autonomously just with AI code review only. And so the approach to that has been we have a programmatic check that looks at did the code review agent find anything that would be an issue? No. Okay. Passes that check. Is this a repo that people have opted in for autonomous deployment check? The set of these other things that are verified explicitly by code.
If it passes all those checks, we’ll do one more final pass where we’ll have a LLM review the overall PR and try to do a risk-based assessment of it to be able to say, “Hey, even though it’s a small PR, this looks like something risky.” And so yes or no to whether or not we should do deployment for that. And what this means is this means that if it passes those two checks, we can actually do a full auto merge into production. And right now we’re at about 5% of our PRs at the company are going through this automated pipeline. And so we feel like AI has earned the right to do this because of all the safety checks that we put in place. And that’s the cost of autonomy, is really putting in place all of those safety checks.
Brian Houck:
It reminds me a lot of Meta’s system. They call it DIF risk score, which is can you assess the risk of allowing this change through and if the ramifications of letting it through are low or it’s easy to call back, then you can do it fully autonomously. It sounds like you’ve built something very similar.
Andrew Swerdlow:
Yeah. And right now we’re taking a very conservative approach. So the agent looks at things like, "Hey, is this updating documentation? Is it updating tests? Is it doing these sorts of things where we have pretty high confidence that a blast radius would be quite low if something didn’t go right, if there was an issue there. And our job right now is to tune this system to the point where we kind of adjust the risk spectrum to a point where we’re able to incorporate more automated deployments, but also not cause sevs. So that’s actually the work that’s ahead of us.
Brian Houck:
How has that been received by developers themselves?
Andrew Swerdlow:
So actually, an interesting part of this journey has been the people side. So the technology has been interesting and fun to build and lots of experimentation, but there’s also culture change that has to happen as well. We have entire engineering practices and culture that are just predicated on this idea that every production, PR, needs to go through a human code review. And so convincing people that these systems are safe enough, that they do as good or a better job than humans has been a journey, not just for engineers, but also for our compliance team and for leadership. They all want to see the data as well. And so we spent a great deal of time doing evals, doing measurements, even injecting faults into the system so that we can see whether or not we catch them, just to build confidence and trust. And so that’s actually been an important part of the transformation as well.
I think one of the things that I really wanted to tell folks is stop counting tokens. That metric doesn’t matter. Start counting what code gets accepted, what got rolled back, and what shipped with no human intervention. I think those are the things that are going to matter most in the future for agentic native development companies.
Brian Houck:
Totally agree. I think counting tokens is a good way of measuring cost and not a ton else.
Brian Houck:
Andrew, this was phenomenal. Thank you so much for joining us.
Andrew Swerdlow:
Thank you. Thank you. Yeah, it’s been a really pleasure and you’re really excited to have these conversations. It’s always very thrilling to talk to you, Brian. Thank you.
Brian Houck:
Have a great day.
Andrew Swerdlow:
Thank you.