Laura: Nathen, welcome back. We’ve had so many great conversations about DORA metrics, what DORA metrics even are, and now, with this new world of AI, we see the world changing around us. I want to set the scene for a conversation I’ve had many times in the last couple months, which is me sharing a little bit about measuring AI impact, measuring return on investment, and then I have someone sitting opposite of me that says, “Oh, yeah. We’re totally covered, we’re measuring the four key DORA metrics, and that’s the basis of our AI measurement strategy.” What do you say to leaders who have that approach? Is it sufficient?
Nathen: Yeah. Interesting, interesting. First, I might start off with a snarky comment, something along the lines of which four DORA metrics do you have? Because the reality is there are so many metrics within DORA, and I’ll grant that individual… Everyone knows DORA for the four key measures, but just to be very clear, those are the software delivery performance metrics, and even what might surprise that particular leader, maybe even some of our listeners, is that, for the past two years, DORA has researched five software delivery performance metrics. Because we’ve done it for two years there now, we now kind of consider them… They’re real. There’s five metrics, not just four, and they look at software delivery performance, which is really important to look at. If that’s all you have to help measure AI, okay, it’s-
Laura: Better than nothing, right?
Nathen: It is way better than nothing, and the nice thing is that those software delivery performance really look at what happens at the end of a software delivery process, so in order to improve those software delivery performance metrics, you’ve got to have the right system in place. The right organizational systems, the right maybe incentives, certainly the right technologies, like your pipeline has to be built well and provide fast feedback along that process, but just looking at those four metrics, the challenge with them is when they start to go off. Maybe you’ve seen those four metrics or five metrics improving and, all of a sudden, they’re not.
Laura: Then what?
Nathen: Now, what? I often equate those metrics as like having a compass so we can see directionally where are we headed, and that’s great. That’s really, really great. But if you just look at the compass and you don’t have the map to associate compass to, yes, you’ll know that you’re headed north, northwest, but without a map, you don’t know if that’s the right way to be heading or what’s coming up ahead of you. Is there a giant mountain that you’re going to have to cross? Are you about to walk off of a cliff? Who knows? You have to look beyond those four metrics. We always say those metrics are leading indicators for what’s happening organizationally. The better you are at software delivery performance, you’re probably doing better with your business. You probably have better wellbeing for the people on your team.
That’s all good, but they’re lagging indicators for your engineering practices, and if you want to improve your engineering practices, you have to understand how you’re doing with things like, I don’t know, continuous integration, with documentation quality, with your change approval process. All of these things contribute to those software delivery performance metrics, with AI adoption as a good example. You have to be able to measure those things as well because you’re going to want to make changes to those and see how they impact the overall system, so while those four metrics are certainly better than nothing, they are probably almost necessary but not sufficient. The only reason I hesitate there is because are they actually necessary? Maybe, but they’re definitely not sufficient in and of themselves.
Laura: I absolutely couldn’t agree more, and I think some listeners out there might be very surprised to understand that this is what I’ll dub as the official position of DORA. I think you all were extremely clear about this in this year’s report, that software delivery capabilities are important. As you said, it’s a leading indicator toward organizational performance, but a lagging indicator of all of the other parts of the SDLC. They give you just a glimpse into performance as a whole and we can’t bet on those four metrics at all, especially for investments the size of AI strategy.
Nathen: Totally. I feel like to be complete here in our podcast, because we’ve talked about four, or is it five metrics, I just want to list-
Laura: Or is it five? Yeah.
Nathen: I just want to list them off really quickly.
Laura: Please.
Nathen: DORA thinks of these five metrics in sort of two different categories. The first is throughput and the second is instability. For throughput, the three things that we measure are lead time for changes, deployment frequency, and your failed deployment recovery time. Instability, we measure your change fail rate and your deployment rework rate. Those are the five metrics altogether. But in the 2025 report, as an example, we said very explicitly, looking at those five metrics is important but not enough to help you understand how your team is doing, and so we actually did a cluster analysis across eight different categories or eight different factors in the report. Team performance, product performance, individual effectiveness, time spent doing valuable work, the amount of friction that you encounter in your daily work, burnout, and then software delivery throughput and instability. By looking across all of those factors, it gives you a much more comprehensive view of how a team is doing, and it might even give you some directional signal about where you might want to lean into help this team improve.
Laura: Yeah. I appreciate this very thorough and comprehensive approach, because I think where a lot of leaders get AI and AI strategy and AI measurement wrong is they think about AI as just the technology or just about the tool, when in fact how I describe it, which is maybe a bit unconventional, is I describe that AI is not a tooling problem or a technology problem. It’s a change management problem with cool, shiny technology on top of it. At its heart, AI is not going to do anything for your organization. It’s applied to a system, and then that system works in a different way. If we think about it that way, then we can understand there is so much more visibility that we need on a systems level.
Where there’s so much more than those five key DORA metrics for measuring software delivery capabilities. There’s just so much more to look at outside of them, where things like code authoring, looking at adoption, looking at impact, looking at teamwork, looking at individual team organizational outcomes, it’s just not sufficient.
Nathen: Yeah. I would say, if I took the 2025 DORA report and summarized it in a sentence, is that AI is an amplifier. It’s an amplifier of the things that you already have in your organization. We have this DORA community, one of our DORA community guides, Steve Fenton, said it best, I think. He’s like, “It’s truly an amplifier.” Think about a band, you plug that band into the amplification system and you turn it up. Well, you might be listening to a high school band full of amateurs and you just made it a lot louder, maybe you didn’t want to, but then on the other hand, you might be listening to a band of professional musicians and you turned it up, and that’s way better.
Laura: Might be Yo-Yo Ma on stage, and then-
Nathen: Exactly.
Laura: … game on.
Nathen: Yeah, let’s go. We see that in our research and in the data.
Laura: Yeah. When you say it this way, you make me think of often advice that I give about DORA metrics and the book Accelerate. As you said, this is like having a compass without a map, and you could be accelerating yourself right off of a cliff. You can just be getting nowhere faster. Some of the advice I give to executives very frequently about AI is, as you said, it is an amplifier. AI will amplify the great parts of your organization, but if there is dysfunction, it will make you have more dysfunction. It’s just going to shine a light on all of the places that are already broken. What great organizations do is that they know about the places that are broken and they use AI to address those gaps. Unfortunately, that’s not a common pattern that I see at every single organization, but I think that is what the promise of AI, to unlock theoretically unlimited potential in productivity in our systems, is applying AI to those bottlenecks, but we need to know where they are first. Right?
Nathen: Yeah. Given AI’s role as an amplifier, maybe it’s going to just help you feel those pains more acutely, which, in an optimistic view, that’s actually better, because now it becomes much more apparent where is there pain that we need to go and address.
Laura: Yeah, yeah.
Nathen: Like the classic example that we’ve seen, honestly, we’ve seen this for three or four years in our research, code review process is often a bottleneck for teams. If we go back, I think it was three years ago in DORA’s research, we found that teams with faster code reviews have about a 50% better software delivery performance, so clearly, this is a place where there’s a problem. AI comes along, and as an industry, we are so focused, as you said, on authoring more code. Yes, I can author a whole lot more code a whole lot faster as a team when I’m using AI, but just authoring doesn’t improve my capability to review code. In fact, if I have to review 10X, 100X more code, that code review process is going to become even more of a constraint on throughput in that system, and so you’re going to feel that pain much more acutely.
Laura: Yeah, absolutely. We see that pop up in DX data, where one interesting thing that came out in the 2025 Q4 AI impact report, which we just released a few weeks ago and we’ll link to that in the show notes here, is looking at the proportion of heavy users, moderate users, light users, and seniority or job title, I think, unsurprisingly, we find that junior engineers are adopting AI or using AI with much more intensity than their more senior counterparts. Interesting thing, we see that staff engineers actually save more time than junior engineers, and this tells us… There’s a couple of interesting patterns. It could be that AI is just more suited to the very specific, discrete scoped tasks of a junior engineer. That’s definitely one thing, right?
Nathen: Yup.
Laura: We have also junior engineers are typically younger, and younger people are generally more experimental or using more current tools. That’s another thing. When we look at how this pattern plays out though, we’re you’re having junior engineers generate a lot more code, but who’s reviewing it? It’s not other junior engineers. We’re putting load on other parts of the system, and so that’s a pain that some organizations are dealing with right now. They started with a broken code review process or not a great build-and-test strategy, and now that’s being overloaded and those problems are exacerbated by AI putting additional pressure on the system. That’s definitely one place.
Nathen: Yeah, absolutely.
Laura: On that note about amplifier dysfunction, as well like AI is an accelerant, some of the other things, and I wanted to get your take on this, because I think DORA research and DX research, we are investing in a lot of the same research questions. We have different methodologies and different sample sets, but we do find quite a lot in common, which is really validating. One of the things that we found was that some organizations are doubling their customer-facing incidents with AI. Their change confidence of their quality metrics and their code maintainability metrics are going down the tubes. We have other organizations that are doing the exact opposite. They actually have fewer customer-facing incidents now that they use AI. Their batch size is going down, their PR review times are going down, they’re kind of moving… They’re accelerating in really positive ways, like up into the right, totally. Why do you think that is?
Nathen: Well, it’s a super fascinating question, because as I talked about those eight different characteristics that we looked at teams across, what we found was kind of seven different prototypes or team profiles that emerged from that analysis, and yet we do see nearly universal adoption of AI. This led us to this question of not… It’s becoming clearer and clearer that it’s not that you’re using AI that’s leading to great outcomes, but maybe there’s something in how you’re using AI, and with DORA, DORA has been around for over a decade, we’ve seen patterns like this before.
If we go back to the early days of cloud adoption, it wasn’t that you were using the cloud that led to great outcomes. It was how you were using the cloud, and so we sort approached this year’s research with that same idea in mind. What are the key characteristics, the conditions or the capabilities that a team needs to get the most out of AI? We started off with a hypothesis that included a bunch of different characteristics or conditions, but through our data and our analysis, including interviews with folks, we whittled that down to seven key characteristics. We call those characteristics or capabilities the DORA AI Capabilities Model.
Laura: I want to hear them. What are they?
Nathen: All right. The first and probably… I mean, let’s be honest, Laura, I love all of our capabilities equally, but-
Laura: You can’t pick a favorite.
Nathen: I can’t pick a favorite.
Laura: But if you had to?
Nathen: I can’t do it, I can’t do it. But I will tell you the one that we always list first, so take that for whatever it’s worth, is a clear and communicated AI stance. We find that having this within your organization, it eliminates a bunch of friction and it opens up so much for everyone in your organization. The other thing I think that’s fascinating about this though, this clear and communicated stance, this space, unquestionably, is moving super fast, so if your clear and communicated stance locks you into specific tools and locks you in for a long time to those specific tools, I worry about how that’s going to impact your organization, because things are changing so frequently. We have to have to be nimble, we have to be able to take in new capabilities.
Laura: Agile. Were you going to say agile?
Nathen: Maybe even agile, so I think it is important to have a clear and communicated AI stance. It’s also, I think, very important to revisit that stance over time and not… You can rush to consolidation and to very strict rules too quickly. I don’t know that any of us have the right way to balance that, it’s just something to be aware of. All right. That’s one of seven, so let me get to the other. The next two are kind of related.
Having a healthy data ecosystem, where you don’t have a bunch of data siloed and the right people have the right access to the right data when they need it. Speaking of the right people having access, that takes us to our third one, that internal data needs to be accessible to AI as well. I think about things like, I don’t know, compliance or security policies. You’ve probably got those written down. Are they accessible to engineers and are they accessible to the AI that’s generating your code or the AI that’s reviewing your code and so forth, then we have some other practices, things like having strong version control practices. In a lot of our interviews, we’re hearing from folks that when they’re using AI to generate code, they find themselves checking in their code diversion control much more frequently.
Let’s get lots of little checkpoints so that if it starts to go off the rails, we can roll back. That safety net, that ability to roll back really provides a lot of value. Next up is one that you already mentioned, working in small batches. It turns out that if you give the AI a smaller task to accomplish, it’s going to do better at that task than it is to a large task, and I think as engineers, our job has always been to take large insurmountable problems and break them down into small things that we can deliver on, so working in small batches is something that DORA has been looking at for many, many years. In fact, it’s probably one of the best ways to improve those five software delivery performance metrics, ship smaller changes, right?
Laura: Yeah, smaller changes more frequently. I think there’s so much emphasis on the acceleration and the more frequently part but not about smaller change sets, reducing risk. Smaller means more frequently just by nature. We’re not trying to do more work in the same amount of time, we just want to chop it up differently. It’s almost like there’s nothing new under the sun in a lot of ways, because like you said, batch size, strong version control practices, these are things that have been the fundamentals of high-performing software engineering organizations for literally decades, and we’re still finding evidence in the data validated that these things are still important, even arguably more so important when we are going through a transformation as monumental as AI right now.
Nathen: Absolutely, absolutely, and this AI Capabilities Model kind of complements the DORA core model, which is based on the research that we have for over a decade. At a high level, that core model looks at things like having a climate for learning, having fast feedback, having fast flow within your organization. All of these things still matter, but there are two additional capabilities on our DORA AI Capabilities Model. I just want to cover them really quickly. The first is a user-centric focus. Again, talk about something that’s been around for a long time. We should be using technology for the sake of technology. We have to remember that every piece of software that we build is for a human. It’s for some human endeavor. We want to make sure that we’re focused on the needs of those users that are at the end of this. Finally, a quality internal platforms. Quality internal platforms help you scale so many things, so that instead of having just localized pockets of improvement, you can actually scale those things across the organization.
Laura: Yeah. It’s a systems problem and not just a technology problem. This really comes across really clear.
Nathen: Yeah, it is definitely not just a technology problem. Like you said, there’s a lot of complimentary research here across DX and DORA. I particularly like some of the research that you’ve done around how do rollouts happen and the value of it’s not just that we bought and handed everyone a license. It’s the enablement and those enablement practices that we’ve put in place, and there’s the ability to share and learn collectively as an organization. We talk about that a lot in DORA through creating communities of practice around a particular idea, maybe around a particular way of working, or even a particular technology, and so I think these ideas of creating a community of practice around how to get the most out of AI in your software development lifecycle, that’s huge.
Laura: Absolutely. I want to go back to that first one, not your favorite.
Nathen: Yeah, yeah. No, no, no. No favorites.
Laura: Not your favorite but it’s first, clear and communicated AI stance. Thinking back to not this most recent state of AI-assisted engineering report, but the one that came out in… Was it April?
Nathen: Yes.
Laura: Earlier in spring. There was some figure in there that companies that have a very clearly communicated acceptable use policy have… It’s like 451% better outcomes. They have higher adoption, they have more satisfaction with the tooling. Is that still true? Is that still true today? The research, and data, and habits age pretty fast, but this seems to be something that’s kind of an evergreen truth.
Nathen: Yeah, I would say that we report on it because we analyze it slightly differently now than we did back then, but what we are seeing is that a clear and communicated stance. When you have a really good clear and communicated stance, it has a large increase on AI’s impact on individual performance, which maybe should help me clarify here a little bit about how this AI Capabilities Model works. We know that everyone is adopting AI, so our question was not how does AI impact these outcomes, but rather how does AI, when combined with these capabilities and conditions, how does it impact the outcomes that we care about? AI adoption times or with a clear and communicated stance has a large increase on individual effectiveness, has a medium increase on individual performance, has a medium decrease on friction that you have in your organization, and of course, you want a decrease in friction.
Laura: We want a decrease there, yeah.
Nathen: Yeah. It has a medium increase on software delivery throughput, so the individuals on your team experience less friction, they’re able to ship more software, throughput is going up, organizational performance is improving, and, as an individual, I’m more effective in my role all because there’s a clear and communicated stance.
Laura: Yeah. One of the interesting things that we found in the most recent AI impact report that we did at DX was that we’re looking at system-level data from the individual tools. We have data connectors from… We’re looking at their cloud, their copilot, et cetera, et cetera, all those friends, and there were some engineers who had no evidence of usage for the tool that they had an enterprise license agreement. However, when we analyzed the time savings questions, questions about how much code they shipped that was AI-authored, it was obvious that they’re using AI to the point of a weekly or daily user. My takeaway from this, when I looked at all the data, was not that they truly were using no AI tools, it was that they’re using AI tools that are perhaps not approved or not part of their enterprise license.
Nathen: Maybe not sanctioned.
Laura: To wrap up our conversation about what should leaders do with this information, clear and communicated AI stance is critical for those situations specifically, because we need to balance security and compliance with experimentation. As you said, we don’t want to be locked into a single vendor, single model, single usage pattern. We want to have flexibility for experimentation, and I see it so clearly in the data. There are lots of engineers doing a lot of experimentation, but in order to do that safely and compliant, we have to have a clear and communicated AI stance about governance, compliance, what’s acceptable, PII, all of these things. It’s so critical for organizations during this Cambrian explosion that we’re going through right now of so many different tools, so many capabilities. It’s just a lot of change.
Nathen: Yeah, absolutely. Absolutely. I think that as we do that, we also have to, as leaders, have to set some boundaries. Yes, I want experimentation all the time, but given the pace of how things are moving right now, as an engineer, I could spend 80% of my time this week experimenting with a new tool, figuring out how to get it worked into my workflow, and then I spend 20% delivering value. But next week, there’s a new tool, so do I-
Laura: Am I going to do that 80% again?
Nathen: Do I spend 80% again trying out this new tool? We have to have some rational boundaries, and I think that that falls as much to the individual as it does to the overall organizational stance. Frankly, if we lean back to one of those other capabilities, a user-centric focus, let’s make sure we’re focused on delivering value to the user of our system. that becomes super important.
Laura: Yeah. The physics of great software that it should be solving user problems delivered quickly and of high quality, easy to maintain, those haven’t changed just because AI is here. Nathen, to just wrap up our conversation, I want to go back to that leader that we spoke about at the very beginning. They’re trying to measure ROI and AI impact, and they were taking the approach of just looking at the four-key DORA metrics. We’ve talked about the seven capabilities, the AI capability model from DORA as being a good addition to enrich that perspective. What else would you recommend that leader do in order to truly understand if their AI strategy was working or not?
Nathen: This is maybe going to sound obvious and it’s definitely going to sound not scalable, go talk to your teams. Go talk to your teams and ask them, “Where are you experiencing friction, what’s difficult to do, and how is AI helping you either overcome that friction or introducing that friction?” By doing so, you’re going to get a much better perspective of how work is actually happening, and that, as a leader, is always going to give you better insights and more ability to drive that team in the right direction.
Laura: Yeah. It all comes back to developer experience, doesn’t it?
Nathen: It really does.
Laura: And reducing friction. That’s a great thought to end this conversation on. I’ll make sure to drop a link to the AI Capabilities Model. Can you talk about the report that’s about to come out?
Nathen: Yeah. In September, we released the State of AI-assisted Software Development, and in December of 2025, we should be releasing the DORA AI Capabilities Model report. I won’t give you a specific date in December because I know how those go, but-
Laura: How things work.
Nathen: Yeah, exactly. But December of 2025, watch for that. You could stay up to date on everything at dora.dev.
Laura: Wonderful. Thank you so much for the conversation and the camaraderie, Nathen. It’s always a pleasure.
Nathen: Absolutely. Thank you, Laura.