Brian Houck:
We are incredibly excited for this conversation. Justin and the entire research team have been hard at work to pull together a set of really interesting and I think novel insights on how is AI changing engineering across the entire industry. And so with that, I’d like to hand it over to Justin to just talk a little bit about what the AI impact report that we just released even is, what were the motivations behind it, and a little bit about how the discussion today is going to go.
Justin Reock:
Thanks, Brian. Appreciate it. Always great to do an episode with you. Yeah, we are really excited about this one. So this is the third AI impact report that we’ve put out, but because of how quickly things are changing in the industry. Because of all the changes that we’ve seen, this sort of now non-binary relationship with AI where it’s like, okay, are you using it or you’re not using it? It’s like, no, it’s no longer a binary condition. Everyone is using it. We don’t have a control group really of folks who aren’t using it anymore to do these comparisons against.
It really forced us to rethink the perspective that we wanted to have on the way that we were looking at this. And so it made us get really creative in different ways that we cut the data. So one of the things that we did was that instead of having maybe this kitchen sink approach that we had in the previous two reports where we were just finding some interesting trends and putting them there, we instead sort of reframed the report around our Core 4 and our DX AI Measurement Framework. So we took that same framework for measurement and then we looked at our industry averages based on that. So we’re looking at speed, we’re looking at effectiveness, we’re looking at developer experience, we’re looking at individual developer experience drivers. And that really helped us find more interesting insights that go beyond looking at what’s happening with velocity and even quality to a degree, to now really trying to understand how this is impacting the more granular aspects of developer workflows.
We’re just going to go through some of these insights. And Brian, I just want to hear your reaction. I of course have my thoughts too about what these data points actually mean. How can we action them to help leaders make decisions around this data that we’re seeing? So start with adoption, and this is the underlying reason why we had to change the format of this report so much. It’s like, yeah, we’re at 95%. And we know to say, and this is coming from telemetry, this is coming from API metrics. And what we can also see is that in this 5% cohort that’s not showing up in the API telemetry, they are merging AI-generated code into production. So that AI-generated code is coming from somewhere. It may not be the enterprise license that they’re using to generate that code, but we are now in this position where we’re pretty convinced that this is just closer to 100%.
And so this has then forced us to think beyond comparing a cohort of users to non-users to now looking at these other firmographics and demographics.
Brian Houck:
I think the question no longer is interesting on, are people using AI? It’s now pretty settled that yes, virtually everyone is using AI. It’s sort of interesting what’s that remaining 5% and are those special niche cases? And I think now the question is not just are we using it, but how are we using it? How sophisticated are we in our usage? And that’s some research that I recently did called the AI native developers tries to answer that question on what I think is that next question rather on how mature are we in our usage? Are we using it in more complicated, complex, and hopefully higher impact ways? And I think that’s now where we have to move the conversation.
Justin Reock:
I totally agree. And also, I think this calls into question over the last year, adoption was the main metric. Everyone’s like, “Well, we got to get 100% of everybody in the organization using AI.” And now it’s like, “Okay, well that happened. So what did that do?”
And that’s kind of the rest of this report. Well, here’s a bright spot for sure. We see this nice satisfying linear progression of time savings per week. This is of course a qualitative metric, but gathered at a very high fidelity signal, millions of data points from millions of engineers at this point that we have in our data set. And we’re just seeing this increase pretty steadily at the rate of about an hour a quarter per week that we’re gaining on this one. Now, time is an abstract resource. We can do what we want with this time. So I think just saving the time alone doesn’t create new value. So Mohawk, this is a qualitative signal. This is gathered through survey data, but looking at millions of data points across millions of developers. So very high fidelity qualitative signal.
Brian Houck:
Yeah. And asked very simply, how much time is AI saving you? We’re not trying to infer this from a collection of other signals.
Justin Reock:
Directionally, it’s interesting because we see it go up. Yeah.
Brian Houck:
Yep. Yeah, I think the directionality’s interesting. I think it begs the question, where are these time savings going? And that’s actually some work that I’m exploring right now and hopefully will be the topic of a future newsletter. Where are we reinvesting these time savings?
Justin Reock:
And it speaks to some of the data points that we’re going to look at later in this report as well, but I think that’s exactly the right question. Okay, adoption, 95%, time savings continuing to just move up steadily. What does that mean in terms of outcome? But it also, again, it’s like what did these metrics really mean? To your point, with time savings, what does that tell us in terms of shipping new value? Nothing really. Neither does adoption. What are we doing in those sessions? What are the use cases?
So yeah, this is part of why we had to change our perspective on the research was like, well, maybe these signals aren’t actually telling us enough about outcomes, enough about ROI, enough about shipping new value and that sort of thing. And here’s another one of those metrics too. 50% of merged code is now AI authored, and this has increased more than linearly. We’re starting to see an actual curve here. So this has now crossed the 50% threshold. Again, what is this doing in terms of increasing our capacity?
Brian Houck:
Yeah, I think this is really interesting, but again, you have to look at this also through the lens and we’re about to, and well, what are the knock-on effects to things like quality? This could be a really good thing, but it also could be a really bad thing depending on what the other signals we see look like.
Justin Reock:
What does that code look like? How complex is it? So let’s keep digging through. Another steady increase. Some people might be familiar with the longitudinal study that we released from November 2024 to February of this year. We looked at 16 months worth of data and found what we would’ve used to have celebrate, a 10% improvement in the proxy metric of PR throughput. But of course, we compare that 10% to what we hear from the industry. Oh, 2X, 5X, 10X. We didn’t see that in the report. We’re still not seeing 2X, 5X, 10X, but we are seeing improvement. I mean, the last report showed a mean 13%, a median P50, 8%. We’re seeing this increase in PR throughput now moving up to more like 25%. So this is steadily increasing as well. But again, we don’t know what’s in those PRs. We don’t know how many of these metrics have been incentivized and then may have a bit of a bias because of that incentivization. But still directionally interesting to see what we would consider a decent proxy velocity metric continuing to improve.
Brian Houck:
And one thing I’ll just add on this, when I look at charts like this, oftentimes we think correlation versus causation. And I think that is also a largely settled question. There was some great research recently published by Alex Heilman at Microsoft that actually looked causally at, is this increase we are seeing in throughput actually causally from the usage of AI? And it very clearly is. And so we know that while there is confoundment, AI is explaining this increase.
Justin Reock:
Yeah, I think that there’s always going to be other factors. And I saw a question about if 90% or 95% adoption, but only 50% code showing up in production, that is specifically merged code. So that is code that was AI generated purely not human generated that made it into production. So just to clear that up. The DORA metric that’s associated with velocity deployment frequency, we also see increasing, which is I think just validating what we see in our PR throughput increase. I think then to your point, yeah, we can certainly prescribe some causality here between this increased velocity with these metrics, which are not full end-to-end metrics, but at least tell us about how much velocity are we shipping activity through. So I just see this. I mean, this is just like, let’s take the DORA metric and use it to validate the Core 4 metric. And I think we see that validation here. So I don’t know that there’s a lot to add on this one really.
Brian Houck:
One thing I will note that I think is really interesting when I see this is there has been research recently published from the National Bureau of Economics Research. Writing Code Vs. Shipping Code was a great study that actually found something a little different. And it was finding that very little of the code that AI writes actually survives all the way to deployment. And so it sort of attenuates through the system. And the fact that we’re still seeing this upward trend, even if we know that a large portion of the AI code might not actually be surviving all the way, shows that there might be multiple factors at play. And I think that’s something worth diving into as well.
Justin Reock:
That is really interesting, especially we start looking at that 50% merged code, but what is that in terms of features? So which would be like-
Brian Houck:
As a scientist, it’s great when we see some conflicting signals because it’s just more evidence of places we have to dive in further.
Justin Reock:
Yeah. And this one, this trend has just continued that we’re seeing engineering managers ship more code. I think we commented on this one last one too. I mean, I tend to be a little bit more optimistic here. I mean, there’s multiple ways that you could read this data. Some people are like, “Oh, that means we only need engineering managers, right? We don’t need engineers anymore.”
No, that’s not the way to interpret this. But the player coach capability coming back I think is always a good thing. When leadership has a better idea of the ground truth of developer experience because they’re actually part of it, we can only expect better outcomes for developers. If developers have had an hour-long build time that’s been largely ignored by leadership, once leadership starts trying to ship code, they’re like, “Wait, this build takes an hour. We should do something about that.” I think that that’s actually a good thing. So thoughts here?
Brian Houck:
No, I think this is pretty straightforward. The barrier to entry for execution is going down. And even I think time was a big factor for engineering managers where they just didn’t have the time to write a lot of code. And the fact that you can now squeeze it into smaller intervals, I think not just engineering managers, but other disciplines are clearly writing more code. And I think it begs this interesting question on who is a developer in the future if everyone can do a little bit of development work. And I think it’s good, especially for engineering managers, to be exposed to more of the systems that their team has to use day-to-day because it also, not even just with AI tools, but having to wait for builds, having to go through code review processes. I think engineering managers being able to feel some of that pain will help improve other areas of developer experience as well.
Justin Reock:
That’s my key takeaway too. At least that’s what I’m sticking to. I like that. Okay, so speed, throughput. Yes, we are definitely seeing the needle move here and move consistently up. Time savings moving up, things like that. What is this doing to the counterweight of quality metrics? What is this doing actually then to the quality of our software? Well, this one is a bit alarming, and we’ve covered this in previous newsletters. The PR size has increased along the same period of study. Moved from an average of 42 lines of code per PR up to 72 lines of code per PR. There’s multiple reasons for this. I mean, I think on the one hand, AI just tends to write more verbose code.
It’s a nearest neighbor token distribution probability model, which means that it’s going to settle for average probabilities in most cases, which means it’ll settle for average code, code that it sees more often. And that’s typically not going to be the most elegant code, and it’ll probably need some more rewrite. I mean, we come from a time where you try to implement a use case using as little code as possible. Why? Because every extra line of code, every superficial line of code is possibly a bug, possibly a vulnerability, makes the code less portable, introduces more dependencies. So as much as possible, we would try for the smallest amount of code to achieve the same use case.
AI’s not necessarily going to do that because of the way that the token distribution and probability works. There’s a psychology in play here too. If we’re already dealing with downstream systems that may take a long time, 45-minute build, hour-long build, and I’ve got five features that I need to get out, and I had AI that can create those features for me almost instantly, will I create five separate PRs and wait five separate build pipeline sessions for that? Or will I create one PR with five features and I push it through the build time once, build cycle once? So there’s multiple factors in play, but I think that AI, I think this is one of those ones that we can correlate pretty well, especially when we start looking at some of the other metrics that we’ll get into in a moment. So what are your thoughts on this one?
Brian Houck:
I mean, this one, I’m going to be honest, this one scares me a little bit. what are the implications of this? I am a firm believer that smaller, more scope PRs are better for a whole host of reasons. They are easier to review. If something goes wrong, they are easier to revert. They’re easier to understand. And so the implications on cognitive debt change as PRs get bigger, we’re a lot more likely to rubber stamp them. They flow slower through our systems. Again, because not only are they slower to review, but no one wants to even start the review and like, “Oh, maybe in the future AI is doing all these reviews anyways.”
But we do still have to have code understanding. And so this is a place where we’re doing, not only are we doing more PRs, but each of them are bigger, I think is a potential quality concern. It’s a potential code understanding concern, and it’s definitely something to keep our eyes on.
Justin Reock:
Well, and this is another one that is not a linear progression. We’re seeing a curve begin to form here. And so no, this one concerns me as well, especially when we look at some of these other quality indicators and developer experience indicators that we’ll get to in a moment. So yeah, this is something to keep our eye on. I’m all for AI inducing the need for more engineers in the long run. I don’t want that to be because we’ve created a ton of technical debt that we now need engineers to come and unravel a couple of years from now.
Brian Houck:
Yeah. And I will just say I’ve looked a lot at things like PR size and what that does to code review activity, but not all the way to what is that correlation with reject failed rate. That would be definitely something interesting to dive into.
Brian Houck:
And so the last thing, and I know we should move on, but here is this really interesting thing on should we be giving agents the same best practices? Should we be encoding to those agents the same best practices that we ask for developers? And one of the reasons that I like tracking PR throughput, not as an individual measure of productivity, but as a measure of system flow, is that the way to game it aligns to the best practices. You break it up into a bunch of smaller PRs. And should that be part of the guidance that we are feeding into our system prompts for our agents? Probably.
Justin Reock:
I think that’s, especially if we start doing rubric-based validation loops and things like that to try to check better quality, that is a thing we can check for. Is this a good incremental delivery? But I think it’s always remembered and important too, that everything we tell an agent to do, it still treats as a suggestion. There’s no actual consequence for it not doing what we told it to do.
Brian Houck:
There’s actually this interesting flywheel effect. Whereas as agents are writing more code, they have to keep on reading that additional code in the future, which is all as context bloats and gets bigger, that’s token cost. And so as we allow the code to get bigger and bigger and bigger, then it needs to be. Obviously, if it’s all functional and it’s all delivering customer value, that’s great. But if it’s just sort of bloating, that directly goes into future token costs and not just one-time costs, but every time the agents have to read that additional code, you are burning tokens.
Justin Reock:
Yeah, that’s a really good point. So you could get into a bit of a vicious cycle there too. What about the impact to perceived quality in this case? In my opinion, this is one of the most interesting takeaways that we found here. This is a qualitative metric, but again, gathered across millions of data points. These are two trusted drivers from the developer experience index of code maintainability and change confidence. These are calculated on a top box Likert scale from a survey question. So the percentage of people who answer positively, so always or very often in the survey question answers about whether they feel comfortable modifying and changing the code in front of them and whether they trust that when they make changes and release them to production that they won’t break things. Typically, per organization, these metrics are very well correlated. People with a high code maintainability also tend to have a high change confidence.
But look at the psychology here. Look at what’s changed. So the code is technically more maintainable. Yeah, if I’ve got an agent or an assistant or something like that helping me understand what’s in front of me, helping me make modifications to it, then technically the code is more, quote-unquote, “maintainable.” But this change confidence score going down, actually intention with that metric now shows us that we don’t trust the outputs as much. I can understand the code that’s in front of me and I can modify the codes that’s in front of me, but I don’t trust what I’m pushing into production as much.
So these are drivers from our developer experience index. Yeah. So to me, this is concerning because who’s the best judge of whether I’ve got good, safe code in front of me? Typically, a software engineer. So for software engineer’s confidence in the code to go down, but to have more agency over changing it, I don’t know, this is troubling to me as well, especially when we start looking at this in relation to PR size increases and stuff. What is this viscerally? How do you react to this data?
Brian Houck:
I mean, I look at this and I think our definition of maintainability is changing. We can’t apply what we used to think of as code maintainability to a new world where it’s like, well, yeah, it’s maintainable because I’m not expected to do anything. And so I wonder if our sense of ownership is changing, and that’s what we’re seeing here. I do think I have a study coming out hopefully soon that in more depth explores how AI is changing a large variety of dimensions of code quality, including maintainability with things like readability, things like security, things like robustness, testability.
And in general, it does look like for now, humans are better than agents at writing code. Turns out we have expertise, we are better. Now it scales a lot better to let agents write a lot of code, but I think the thinking that, oh, we’re going to have AI write all this code and it’s going to be better than what we can have these high expertise humans doing. I think that’s at least for now broken. Although the fact that this signal is different, I think warrants us diving into what does maintainable even mean now.
Justin Reock:
I think that’s a really good point. Yeah. How do we redefine that? Just like we’re redefining some of the other aspects like redefining toil and things like that too. Pets versus cattle all over again, except now with code. Yeah, it’s changing a lot. This is a distribution of probability of perceived high quality, which actually then is staying relatively flat. We see it dropping amongst smaller, sorry, dropping against mid-sized organizations. Down here we see a little bit of a drop-off in perceived quality, but this one seems to be staying relatively flat. What is one interesting trend here though is that these larger organizations, this tends to be trending up. And I would probably call this out against what we already know and also what Dora has substantiated too, that structured rollouts, clear AI policies, things like that are very positively correlated with a less volatile rollout. And so for a larger organization where this is already part of the DNA, it might mean a slower rollout, but it also means less interruption to at least what we’re perceiving as good quality.
Brian Houck:
I have a much more cynical view of this, Justin, if we’re being honest. And that’s just at large organizations, I think the general perception of the baseline of where we started was probably lower. And so it has a lot to do with at large organizations, you have so many dependencies. It’s impossible to understand all of them. Larger code bases typically that are contributed to by a wider variety of teams. And again, that last study that I’m working on that I just referenced, we found that everyone views their code much better than the code of the people around them. And there’s sort of this K-Nearest Neighbors effect to that, I suspect. And so with large organizations, there’s just always been a little more skepticism. And so the bar to moving that up I think is lower. And so some of that may be not necessarily like, oh, AI is better in these use cases, but we started from a more skeptical place and being able to be more hands-on with the AI code than some partner team, 10,000 developers away. I think those dynamics might be playing in.
Justin Reock:
Now we’ve reported on this level of volatility before. This is a system metric. These are system metrics. And so this is based on rollbacks and failed releases. And each line on this chart represents a single company in this study. And obviously you want to be on the bottom side of this graph. You want your change failure rate to go down, but we are starting to see outliers up in the 3% range with this one. The previous data showed 2% topping out there. And again, against an industry benchmark of 4%. So if you have a 2% increase in change failure rate against an industry benchmark of 4%, you have the possibility of shipping 50% as many defects as you were before.
These folks that are moving up into the 3% range now are possibly shipping 75% more defects than they were shipping before. But this output still is highly volatile. What I always want to point out with this graph though is that this pattern has always existed. You’ve always seen this disparity in defect shipping based on so many other aspects of that organization’s software health and security posture and everything like that. What we haven’t seen is this level of amplitude. And we’ve said this before, that AI is going to amplify the good and the bad. So for this pattern to exist, what I really see in this graph now is just higher amplitude, which does mean more volatility.
Brian Houck:
This one is mind-blowing to me. I think to your point, AI does not automatically mean successful high quality code. I know we had a previous question on whether or not it’s fair to say that agents can write code just as well as any human, but not necessarily good at building software. And I actually do not think that’s true. I think there’s plenty of evidence that shows, particularly in areas of unique expertise, humans are definitely better than agents at writing code. I think for general use cases for boilerplate stuff, agents are better and something in between, there’s probably a mix. It depends on the human. But what this shows is there are a lot of factors at play that might be indicating whether or not a company can ship higher quality code or code that has more failures in it. And that’s again, how sophisticated is the usage, but also what are the kinds of problems you’re solving?
I suspect if we were to try to control this based on industry, based on languages used, age of code bases, we’d start seeing some interesting patterns there. But this notion that AI as a whole drives up failure rate or drives down failure rate just isn’t true. There are a lot of other things at play. And so how you are using the AI tools, how you are integrating it into the systems around you continues to be so important. And it also highlights why you can’t just look at one metric like, “Oh, throughput is up.” You have to combine lots of signals to get a full picture of what’s going on. And nothing highlights it to me as strongly as this slide right here.
Justin Reock:
I totally agree. And I just think this shows the disparity, the still uneven distribution of the ROI from these tools. But I think it’s also interesting that this is an artifact of the past. This volatility existed, just not anywhere near this amplitude. But no, I totally agree, Brian. We all need to measure this stuff so that we end up on the bottom of this graph where we want to be.
Brian Houck:
And also, I will note my researcher knit in general is all of these slides are quantitative. It’s just subjective for system is the difference, but it’s all quant data. We are all counting up numbers. None of it is inferring from open text responses or observational studies or things like that.
Justin Reock:
No, you’re right. Because that’s definitely some tribal stuff within DX. We kind of think about the perceptive subjective as qualitative.
Brian Houck:
Subjective for subjective is a different dimension. So sorry for being a pedantic researcher, but it’s like-
Justin Reock:
No, no, no, you’re right. I mean, this could be perceived qualitative because we tie it to a number, but yeah, that is totally fair. I’ve had that knit myself in the past, but now I’m fully absorbed into DX culture and this is how we talk about things. Okay. So yeah, volatility. And I want to make sure that we have time to address more audience questions. We have about 10 minutes left in the session and still some interesting points to go over. So for those who aren’t familiar with the Developer Experience Index, this is a very sophisticated qualitative metric that’s based on a weighted average of 14 different individual drivers. So those drivers like change confidence, code maintainability, make up this larger composite metric that we call the Developer Experience Index, which is again, an average of all 14 of those individual drivers together. The reason that I’m concerned with what we see here, which is about a 2% drop in DXI over the last two quarters, is that our data is very biased towards improvement.
If you’re a DX customer and we’re aggregating anonymized metrics out of these platforms, our customers tend to improve their DXI. They’re measuring this and they’re investing in developer experience. And so typically these numbers tend to trend up and historically they have. This is trending back down now, which is like, okay, well that’s probably not good, but when we start breaking apart the individual drivers, the factors that are part of the DXI, this is where I think that the real story begins. And Brian and I were talking in the pre-show, we’re like, “Wow, our biggest risk is not getting completely mired down in this one slide because we could probably do a whole session just on the implications of each of these individual drivers.” And we’ve only got about 10 minutes left in the session. So we’ll go through some of the highlights here, but I think that this is one of the most interesting pieces that we have, and this is how over this period of study that we’ve seen developer experience impacted.
Documentation. Through the roof improvement, let’s start with the bright spot. And I don’t think that that’s controversial at all. I think we are all, maybe I’m one of those weird engineers who actually never mind to document. I kind of like documenting, but I’m definitely a minority there and I recognize that. It was not always a thing that we love to focus on and put a lot of effort into, and we now have AI, can really help us with that. This I think should surprise no one and I think is non-controversial. But even with these wins that we see in these areas here at the top, these bright spots like better documentation, we already talked about code maintainability. I don’t know that we can correlate. These numbers like production debugging, dev environment are a little too small within an acceptable margin of error to necessarily cause a correlate with AI utilization, but they’re up.
But then here, all of these wins are eclipsed by about that 2% mark because the overall composite score has gone down 2%. And we look at these things like incremental delivery, which is I get to work on small incremental changes. We already addressed that with PR size. We see PR size bloating and we see perception of incremental delivery going down seriously. So I think we need to pay attention to these. And yeah, Alex’s point about amplifying negative practice. Totally, absolutely. But the point is that these numbers, the overall DXI has tended to trend up, at least in our dataset because of the inherent bias. And we’re seeing this go down because of a combination of features now. Brian, quick thoughts because we still got to get through costs.
Brian Houck:
Yeah, I think I mentioned this in the last webinar, Annie Vela has been doing some really interesting work on what she calls the Productivity-Experience Paradox. And this is not unique to the data we’re seeing. In her research, she’s seen that the old relationships of as you do more work, you feel more productive, those are breaking. And I think we’re now starting to get signal on why that may be true. The documentation one specifically is interesting because documentation is a means to an end. We use documentation to get answers to our questions. And great, now it’s easier to get answers to our questions. But since documentation now is also feeding the context that our agents are using to do their work, it really begs the question, something I’m looking into. I’m like, well, how do we assess the quality of that context, that documentation for agents? And I think that that becomes increasingly important moving forward.
Justin Reock:
I totally agree. And it is another one of those ones that maybe the context changes a little bit. We’re going to wrap this up and get to questions. Maybe the only incredible exponent that we’ve seen in this whole report is cost. Spend has gone up rapidly. This is about a 28X at the top here with the largest companies versus what the spend was a year ago. We’re seeing this move up across all sectors, but especially software development and IT. Here’s a nice bright spot is that we do see dev ramp up. So our time to 10th PR, which is not how long it takes to crank out 10 PRs. It’s when a new person is brought onto a team or a project, how long until they get their 10th successful PR. I think this is one that’s fair to correlate. Onboarding has gotten better and we’ve seen it consistently get better through better interpretation of documentation, the code that’s in front of you and that sort of thing.
Brian Houck:
And I have a recent ACMQ article about engineering thrive at Microsoft where we sort of prove why this metric is so important. And so I love seeing it go down. The only then remaining question is, does the value of this metric mean the same thing that it used to now that we’re seeing those barriers reduce and that requires some further study?
Justin Reock:
And as we’re onboarding less humans too. But this is another interesting signal. So with that increased exponential cost, right now here in Q2 2026, we’ve not seen this move the needle at the same degree with our innovation ratio, which is the percentage of time that engineers can spend working on new features as opposed to keeping the lights on in maintenance. This is data that comes from allocation within systems like Jira. So it does rely on tagging and there is a qualitative validator here as well. But directionally, we’ve not really seen innovation ratio go up. I mean, it’s gone up one percentage point basically. In other words, this is now we’re seeing engineers able to allocate 1% more of their time to working on new features.
I don’t want to make any strong conclusions here because I think we’re still in an inflection point where people are pulling stuff out of the backlog that may not be for whatever reason tagged as new value. But ultimately, ideally what we would want to see is that that cost going up roughly in proportion with the amount of innovation. But we can look left, look right. I don’t feel like software is twice as good as it was a year ago. I don’t think that software companies are twice as profitable as they were a year ago, except for AI companies. And so I think that this is another one of those ones that we just want to pay attention to. We want to make sure that ultimately for all this additional spend that we’re actually seeing this reflect and being able to ship new value for our customers.
Brian Houck:
And it goes back to a question from the very beginning. Where do the AI time savings go? And this says, will they get evenly distributed across innovation and keeping the lights on, or just general overhead? This is a place, as I previously mentioned, I’m doing a dedicated deep dive analysis, and so hopefully we’ll have something in the coming weeks.
Justin Reock:
Yeah, I’m really excited to see that research too, because I don’t think this tells the full picture, but I also think that at the very least, if our hypothesis is that AI is going to increase our capacity for shipping value and we’re looking for data points to say yes, no, or maybe, this one is not saying no, it’s saying maybe. It’s certainly not saying yes. And I think that again, so many of the savings, those six hours get absorbed, get eclipsed by what we know are other sources of time sinks, meeting heavy days, interruption frequency, still the two biggest culprits of time loss for engineers, as well as cumulative effects in the dev environment, build and test wait time, dev environment, toil, review wait time.
So until we can see these savings not eclipsed or absorbed by other areas of the SDLC, which by the way, we can apply agentic solutions to, we should map our value streams, we should find out where our toil and our bottlenecks and our friction truly exist. And if it makes sense to apply an AI solution to that, then we should to increase flow in the organization. So yeah, that’s why I don’t want to draw any strong conclusions right now in that this time savings and everything could just be absorbed by other areas of process that we can still improve.
Brian Houck:
Yeah, and this is one in a recent newsletter. I actually published a real world case study where an organization made a big effort to reduce unnecessary meetings, and they actually found that the PR throughput gains they got from doing that from the time they saved was bigger, in fact, about twice as big as the gains they got from using AI. And so meeting heavy, first principles still matter if you’re wasting a lot of time. Some meetings are incredibly valuable, but some meetings aren’t. And then if you’re wasting time in those meetings that aren’t, it doesn’t matter what tools you’re using, that time is still lit on fire.
Justin Reock:
I know we’re almost up on time, and I know we’ve answered some things along the way, but maybe pick a question or two and then we can wrap things up.
Brian Houck:
Yeah, so I think we definitely tried our best to answer questions as we went. In the end, there were a couple of questions I think that were really interesting around just general economics of at some point, is it actually cheaper to have humans writing code than agents? I saw a comment around tradable quality hypothesis, Martin Fowler, on if we’re spending this much time, if we’re spending this much money, can we correlate that to actual increase in revenue? And I don’t know that right now we can. I don’t know if we look at revenue growth of the S&P 500, that that follows the same cost increase curve. And I do think that there are going to be a lot of interesting articles and formal research papers from places like the National Bureau of Economics Research that look into that. But those are exactly the kinds of questions we should be answering.
And that’s sort of a theme of the DX Core 4 is we should be tying this to impact. It’s like we want to remove all of these barriers, remove all this friction, but if ultimately that doesn’t lead to more customer value, then what are we doing it for? And I think those are really important questions to answer.
Justin Reock:
And that economic absorption, that almost ouroboros pattern where it’s like, oh, the software industry’s blowing up. Why? Because AI companies are doing really well. Why? Because software companies are paying them a lot of money. Well, hold on. That’s like a redistribution. That’s not creating additional wealth across the entire spectrum of software companies. So I think it’s a really good point.
Brian Houck:
Yeah.
Justin Reock:
Brian, we could have gone hours as always, but we’ll have another one of those. And thanks.