Brittni Allen:
Hello. All right. Whoa, that was much louder than I thought I was going to be. Hi everyone. Welcome back from lunch. I hope everyone enjoyed the beautiful scenery, got some fresh air. I’m a little out of breath because I had to run back here because I was having such great conversations, and I hope you all did as well. For those of you who I haven’t met yet, my name’s Brittni Allen. I’m one of the strategic account directors here at DX and I work really close with our research team. Super excited for the afternoon breakout sessions. The ones that are going to be on this stage are going to be tactical, practitioner led, and full of things that you can actually take back to your team. So I think you all will find a lot of great notes in this, maybe some roadmap items and more things to head back to get started on.
Let me start by introducing Michael Redding, product director, and Jeff Davis, VP of Core Infrastructure from Indeed, up here on stage to tell us exactly how they were able to 2X productivity gains from AI by structured AI enablement training.
Jeff Davis:
Thank you. Thanks, Brittni. Hi everybody. Thanks for joining us today. As Brittni mentioned, my name is Jeff Davis. I’m a VP of engineering at Indeed. And today we’re going to talk to you about how we used structured in house training to drive adoption and more effective use of AI coding tools amongst software engineers.
In case you haven’t heard of us, at Indeed, we help people get jobs. And so if you need a better job or are looking for great candidates for a job, come and see us. This is us, me and Michael. Michael is on my team. He’s a principal product manager working on developer platforms for Indeed.
So a quick overview of the agenda. We’re going to talk about kind of where we started, the goals we set, the strategy that we used to get there, and then fill in some detail along the way and then briefly talk about what’s coming next.
Okay, so first, I want to take you way back in time to the beginning of last year, which is a long time ago in AI terms. At that time, end of 2024, January 2025, our DX survey told us that only about half of developer time was going toward new features, innovation, creating new value for the business. And 48% of developers’ time was going toward activities that we would consider to be overhead. So this is maintenance, upgrades, incident response, administrative overhead, things that are not the reason they got into the job in the first place, probably.
And at that time, end of 2024, beginning of 2025, we had been actively encouraging adoption of AI coding tools among engineers. We had Cody, Copilot, we had unlimited budgets, we had training, we had lots of leadership talk about using AI more. We were still hovering at around 25% of developers actively using AI tools on a weekly basis.
And so we laid out our AI powered developer productivity strategy for the next couple of years to try to correct the problems that we saw in the DX survey, to create more engineering throughput, more product velocity. And so we said, we’re going to do two things. First of all, we’re going to use AI to shrink the amount of time that developers spend doing overhead, not innovating, not creating new value. We’re going to automate, we’re going to reduce the workload there. We’re going to try to make the yellow part of the pie … Yes.
Jeff Davis:
Oh, there we go. So we’re going to try to reduce the size of that yellow piece of the pie. Here’s the real trick. Okay.
And then second, with the fraction of the time that developers have for innovating, creating new features, we think that we can use AI tools to make them more effective. We think that with AI, they’ll be able to generate more output per unit of time. And so, not only are we going to close the Pac-Man’s mouth, but we’re going to actually make the Pac-Man bigger and thereby double overall engineering product productivity over the next few years.
This presented us with a big challenge. As I mentioned a minute ago, at that time, we had been pushing AI coding tools and adoption of AI engineering practices. For almost a year, we were hovering at a frustratingly low 25-ish percent of people actually using those tools on a weekly basis. And so the question that we faced was, how can we get everybody to use the tools and to use them well?
So to spoil the ending, we did it. We got there. And this is a graph of late 2024 until almost now, the weekly active users for our coding tools. And we did this through a variety of efforts. Michael’s going to talk more about some of the details around how we did training and enablement, and the measurable results that we saw. And I think it’s worth calling out here that most of the adoption that we saw happened before the best models came out. Around Thanksgiving, Opus 4.5, Codex 5.1, those really started to change the game, but by then we were above 90% or somewhere in that range.
And so now I’m going to turn it over to Michael and he’s going to talk more about how we got there.
Michael Redding:
Awesome. Yeah. Thanks, Jeff. So in January of 2025, which feels somehow like it was a decade ago, we were basically here, right? We only had, like Jeff was saying, we only had Cody and Copilot available for us. And then we kind of committed … We started hearing from our developers actually that they were really enjoying this new tool called Cursor. We found people paying for cursor licenses out of their own pocket, which maybe you shouldn’t do in an enterprise organization, but that was really the vanguard of some of the more agentic, what we might call second generation tools. And so when we were looking at this, it’s like, okay, our current tool lineup’s not working. We need to add sort of a second layer here that’s definitely more agentic in nature. And so this is sort of our timeline of rolling things out.
And then we actually have phased out Cody and Copilot. So sort of that first generation of tools we’ve actually already fully retired. Our usage when we actually did retire them was down to about 3% of people. So our users actually abandoned them before we did. And so mostly where we are right now is in this sort of agentic IDE and agentic CLI tool environment.
Like Jeff said, in 2024, we were at 25% usage. So it was basically entirely Cody. And then now, right, with just our Gen three tools, our gen two tool, sorry, we’re all the way up to around 97%. And if you’re interested, our breakdown rate, we’re mostly Claude Code, a little bit Cursor, some Windsurf, and then down at the bottom, Amp. If you actually add up these percentages, it’s definitely more than 100%. So we also see that our devs are using multiple tools at the same time. And even more interestingly, within teams themselves, over half of our teams, the makeup of tools being used across people on their teams is actually three or more tools when you look at it at the team level. So some interesting little wrinkles that we weren’t necessarily expecting.
But yeah, we’re like at 97%. And you might be asking, how do we get there, which is a question we asked ourselves a lot too. And so what we started with is a program called AI Coding Ambassadors. Our original thought here was like, let’s just do a train the trainer model because we have 2000 some odd engineers at Indeed. At the time that we were committing to the strategy, we had three people working in our AI toolings-
Michael Redding:
We were committing to the strategy. We had three people working at our AI tooling space. So three to 2000 is quite the ratio to try to do really, really in depth training. And so yeah, we were like, “Let’s do a train the trainers.” We’ll find a few engineers per team. We’ll work with engineering leads to say, “Hey, who on your team is actually the most excited, the most interested in AI?” We were looking at usage patterns already. We identified folks. And we basically came up with a cohort of around 60 engineers all across our organization and all over the place in a whole bunch of different locales. And we basically did a bunch of hands-on training. We did webinars, we did workshops, we had peer sharing sessions, we had demos, we had them do some hacking projects and share it back to us. And they were very invested.
Anybody that was participating in this program, I think we had something like a 90% satisfaction rate with the program. And so we were like, “Great, this is a really good model.” When we look at the data from the AI Coding Ambassadors Program, we rolled this out from January through February of 2025. For the blue bar here, that’s the coding usage or what we would call power users, which is a now retired metric that we don’t use anymore. But at the time, their usage was so much higher than the people who were non-participants. And even within teams, right? So now the pink bar here is the members of their team, right? Not them, but the people who are on the team with the people who are the coding ambassadors. At first, we were like, “Wow, this train the trainers model really seems to be working.”
In that first month, we were seeing near the same usage in our power users metric. And then in February, it started dipping, but the top line was dipping too. So we’re like, “Okay, well, that’s fine.” And then in March, it started going down again the wrong direction. And then we started talking to folks and what we realized was while they were going through the training, they were bringing stuff back to their product teams and they were having those conversations. But as the program itself was ending, those types of conversations stopped happening on the teams, but the people who got the training directly, they were still at the high level, but everybody else on their team, the rest of their cohort, they were kind of declining.
And so when we talking to folks, when we were interviewing, what we realized was the key difference was actually having the hands-on training yourself, right? Not having somebody tell you, “Hey, here’s this cool training that I took and here’s the stuff I learned,” but actually having people take that training. And so we committed to a different program, which we called AI Coding Essentials, because I think for all of us that are practitioners, there’s about five vocabulary words that we can use, essentials, ambassadors, showcase, another fun one that we use a lot.
And basically this one was, how do we take the content from the coding ambassadors program that was really, really effective, but actually structured in such a way that all of our engineers could take it? Like I said, we have around 2000 engineers. Most of our structured training tended to be compliance in nature, right? So don’t click on phishing emails, make sure to follow our privacy policy. And so the idea of having sort of an actual learning, like in our actual learning environment, our structured learning environment have a training for engineers. I’m pretty sure this is the first and only one that we’ve done so far at scale that wasn’t something that was being mandated by some compliance regime somewhere, but we did.
We designed an entire course and then we also worked with our engineering leads across the organization to make sure that it was prioritized. When you sit down and do the math, 2000 engineers, eight hours of training over four to eight hours over four to five weeks, that’s about a three to four million dollar investment, right? Just in this program that’s not a JIRA ticket, that’s not making a code change. It was actual real structured time and making sure that that was actually prioritized. And then really focusing on sort of the engagement pattern of, this isn’t just a one and done, like let’s build the community, let’s have people talk about what they’re doing. We’re definitely not the experts here. There are other people that are out there in our engineering orgs that really are pushing the envelope even further than we are. How do we make space to elevate some of those stories as well?
And so I’ll pause here. I’m not going to read all these words, but if you do want to grab the QR code, this is our actual playbook that we followed. And this is really, I think the key things here is that this was like an all hands on deck effort. We actually partnered with learning architects in our learning and development organization. We hadn’t historically worked with them before, but they were great. They actually really helped us figure out how to structure our learning, what are the actual things that we’re trying to get across, what are our outcomes, what do we want people to actually take away with? And they actually helped our engineers that were developing the content, structure the content in such a way that it was actually really effective, right? So it wasn’t just, here’s a whole bunch of blog posts out there on the internet to read, here’s maybe a couple of YouTube videos, but it really was, “Okay, so for this piece of content, for this type of message, it’s a hands-on exercise. So let’s come up with some way to have a hands-on exercise that you can do.”
For this other, for using MCP, which was just coming out as we were developing this, it was like, “Okay, actually this probably is sort of some blog posts maybe from Anthropic’s website and maybe some YouTube videos that other people have filmed about MCP,” and really kind of structuring that multimodal form of content delivery. And I do want to zoom in a little bit here on the organizational targets and reporting, right? We never set a mandate that said, “You must use AI.” We did, however, set a very suggested, you should complete this target for all of our engineers for the training itself, because the idea that we had was, “Hey, we can’t force you to use these tools, but if you do start using them through this training and you find that you really like them, then you will use them.” So sort of the mandate and the target that we set was really around complete the training, not use the tools.
And so that, I think we really do believe that that one little degree of separation of an organizational mandate was a key part of this. And then I think the other bit to really focus on too is like the engagement. We really didn’t want it to be a one and done. About a month after the actual training program wrapped up, we had our fall hackathon. The theme for that hackathon was, I forget the specific wording, but it was basically AI all the things, right? And so the amount of engagement that we had coming off of this training program into that hackathon, having people share what they were building, what they were learning, and kind of keep that conversation going has been really great. I was looking the other day and in our main Slack channel where people share all their tips and tricks, we’ve had over a hundred in March, we had over a hundred unique community posts about something that somebody had been experimenting with, right?
So a hundred out of 2000, that’s a pretty high engagement from like a community management point of view. So it’s really even now, here we are six months after this training, really encouraging the continual sharing. During the training, right, we did see an improvement in our coding time metric. So when we think about software, like the actual SDLC, some folks might have seen the talk earlier this morning about sort of coding time is actually only about 14, 15% of the actual SDLC.
So for this training, we were basically just targeting how do you get more effective at coding time? So that was the primary metric that we wanted to look at for this training. And so this top bar, these are the people who did not complete the training. So even though it was mandated, we had about a 70% completion rate. So about 30% of our devs never finished the training. Their coding time basically hasn’t changed over time. It’s around somewhere between 20, 25 hours. For the people who did complete the training, we saw about a 35%, 36% reduction in their coding time.
And because we had around 70% complete, our overall reduction in coding time was about a 20% decrease. And so we were like, “Wow, this is actually a really good result. We’re really excited about this result, but can we make it sticky?” The short answer is yes. We did make it sticky because this is those same cohorts today, right? So again, top bar, the pink bar, these are the people who never completed training, right? The bottom bar, these are the people who completed training. At 97% AI usage, basically everybody is using AI, but this top bar, these are the people who didn’t actually have the hands-on training to get that effective use of training, right? So this cohort here, they’re using AI tools. We know they’re using Claude code, we know they’re using Cursor, we know they’re using Windsurf, just not as effectively as the cohort that actually did the training, right?
And that’s a five hour coding time, right? I don’t think we were even thinking we could get that. And as we see this going through, this is where Opus 45, this is where some of the more frontier models are coming in play, where it’s like taking the skills around context engineering, around effective context engineering, around effective MCP usage, combine that with really, really powerful frontier models that came out a couple months after the training completed and we’re able to see that drop, right?
So I think that was the thing that was really the most surprising for us and kind of fortuitous because we weren’t necessarily expecting the stepwise change in model capacity over between October when our training wrapped up and basically the first of the year as people were coming back from the holidays. And yeah, how have we made sure to keep it continuous? Like I said, community engagement I think is still really, really important. Our main AI for developer productivity Slack channel has around 2,100 people in it. Most interestingly, about 200 of those are not software engineers. They’re sort of other R&D roles like product, UX, program managers. There’s even a few like customer support reps in there. There’s a couple of folks from finance in there.
Michael Redding:
… customer support reps in there. There’s a couple of folks from finance in there. So we’re seeing sort of an interesting use of people who are using our channel, where we’re really sharing our tips and tricks. And really the tactic there is really around just community management, right? Almost like social media style, community management, like facilitate the conversation. If people get stuck, actually help them right then and there, don’t make them file a ticket. Have people who are working on things, share things there publicly.
One thing that we also did in partnership with our main enablement team is we spun up an actual recognition program. We called it AI showcase. And it was really about to incentivize the actual experimentation. So a lot of times it’s like the thousand flowers bloom. And so these were actually, in some cases, the winner of our AI showcase actually, there was a monetary prize associated with that. So the person that was judged to have sort of the best innovative use of AI.
We’re actually having an awards ceremony next month for internally and we’re bringing people from all over the world to Austin to actually participate in that award ceremony just as a way to kind of reinforce, “Hey, this is a learning process that we want people to take.” From a support ecosystem, after the coding forum ended from the surveys, our post-completion surveys, we had a surprisingly large number of people that didn’t want us to stop, which I’ve never in my life heard people say, “No, I’d like more training, please.” And so we started our monthly coding forum. That’s the fourth of the five words that we are allowed to use.
And these are just webinars on special topics. So when Anthropic released skills, the skill standard a couple months ago and marketplaces and plugins, that becomes our venue to kind of share more widely those newer patterns into our wider culture. And we average about 200, 250 people a month tuning into those webinars. And then we also hold office hours and workshops for individuals and teams that need help. So if you do have a question, what we do see is some folks, I’ve actually heard this from more than one person, if they’re a little too embarrassed to actually ask their question in a public forum, like having office hours that’s dedicated one-on-one time with an engineer.
There actually have been some really, really good discussions and really, really good learnings for us as a platform team coming out of that. And I’ve had more than one person tell me that they wouldn’t have even asked that question because they didn’t want to seem dumb and public. And in at least one of those cases, it actually was not a dumb and public situation at all. It was a, “Oh no, this is actually a problem with our platform that we fundamentally need to rethink how we’re doing something.” So it’s just like having these multiple touchpoints that aren’t just here’s a training, here’s a webinar, but making space for that individual one-on-one time.
And then like all of us as new tools, as new techniques come out, make sure that we’re constantly bringing in the tooling improvement. One thing that I forgot to mention earlier, but the training itself, we made sure that it was tool agnostic. So in our Agentic space, we used Cloud Code, Cursor, Amp, and Windsurf, and we made sure that any content that we created, whether it was a video or text or anything, we would always have examples of it in each of those, at least two of the actual tools that were available.
So that way it didn’t turn into, here’s how you use Cursor, right? Here’s how you use Windsurf. But it was really about those practices that become transferable across. I will say the one downside risk of that is when you work with these vendors and they want to offer enablement, right? They’re offering, “Here’s how you use Cursor, here’s how you use Amp, here’s how you use Windsurf.” And so we have actually not really gone down that path too deeply because we want to make sure that we keep that multi-tool mix going. And with that, I will turn it back over to Jeff for where we’re going next.
Jeff Davis:
Okay, thanks. Yeah. So just to wrap it up, we have a few follow-on plans for this year. First of all, we’re going to do it again. We recognize that you saw the data. We feel like that program was very successful. It had a lot of measurable impact on engineering output, and also AI changes very quickly. The landscape has changed a lot in six months since we put together the curriculum last time. And so now we’re going to do another round of this to help bring the big middle of the bell curve along into full Agentic coding and then drive adoption of orchestration as well.
We’re also exploring opportunities to use AI more interactively in the training process, and actually to have AI as a coach directly in sessions, in coding sessions with engineers. When we can spot opportunities where they’re doing something that’s not optimal, where there are opportunities for them to use the tools more effectively. And then also exploring different content delivery mechanisms, new kinds of hands-on workshops. Yeah. Yeah. Just iterating on this delivery mechanism, using our engineers to train the rest of the engineers. And I think that’s it. I think we’re onto Q&A. The batteries, something has failed here, but we’ll go to Q&A.
Brittni Allen:
Perfect. Okay. Well, there’s a lot of questions in the chat, so that was incredible. Excited to see where this is going to head. So maybe one of the clarifying questions the audience had is, how did you all calculate coding time in the first place? Was that self-reported data? Was that telemetry based in some way?
Michael Redding:
Yeah. So the way that we measure coding time, we use Jira. And so we actually do have a fairly … Well, I should say fairly opinionated. We do have a couple of opinionated workflows in Jira. And so what we calculate that as is from when a ticket is picked up by a developer to when the DIF is opened in GitLab.
Brittni Allen:
Amazing. Really creative. I feel like that’s a really creative way. It’s not something that we typically hear, so that’s amazing. Tell us a little bit more about how you’re leveraging these AI coding playbooks today. Are they now a part of onboarding when you guys hire new people or when people switch roles? Are you using them in more creative ways past just the structured enablement for people who are already in seat?
Michael Redding:
I will say, one of the things that has been really interesting is last year, I think the theme at Indeed was really around how do we enable and upskill developers in the SDLC? I think our big focus this year is really around the entirety of the product development life cycle. So how can we have our product managers and our UX folks get some of the same benefits? And I think a lot of these principles we’re digging into that space as well for our designers, our UX researchers, our program managers, and kind of taking this playbook and kind of trying to adapt it and apply it to what is actually … It’s a harder workflow to map because it’s not just one workflow in the same way that pick up a ticket, write a diff, ship it is, but using the same principles around enablement and figuring out what works and taking it there.
Brittni Allen:
Yeah. I know the audience is looking forward to seeing that QR code for all of those ones too, so sweet. Okay. So, outside of all of these things, some of the questions that are coming through are … Tell us more about why you ultimately determined to make some of this training and enablement more asynchronous rather than in person or even mandated. Some people are wondering if you have a mainly remote culture or … Maybe explain a little bit more of your thinking about why these specific elements were the right approach for you.
Jeff Davis:
Yeah, we do have a remote first culture. We have engineers all over the world in lots in the US, but also in Asia and Europe. So time zones are always a challenge. Getting together physically in person is not possible really. And so yeah, kind of meeting people where they are, making the content available for them to consume at a time that works well for them, their team, their workflow, their lives outside of work. I think that’s a lot of how we think about it.
Michael Redding:
Yeah. And I think the thing about the program in particular, we ran it over five weeks from late August to the end of September of last year. And we did make sure that within those time slots a week, we did have sort of open forum type, Zoom based, but open forum sessions that were synchronous. And what we were seeing is that as the weeks went on, the attendance and the synchronous sessions were actually dropping.
And so I actually do think that part of having the sort of hybrid approach of sort of asynchronous content and then synchronous place to ask questions actually did work really well for people to kind of feel engaged and to kind of get the understanding of like, oh, this is actually something that Indeed is making the space for us to do. And so I think it’s not an either or, I think it’s a yes and.
Brittni Allen:
Yeah, that makes sense. So something that I know from working with Indeed is how much you all prioritize psychological safety of your team members and making sure that they feel really safe and comfortable about the initiatives that you’re rolling out and the metrics that you’re tracking and all of these details. How much did that play into the types of enablement you ran or your overall strategy with this in general?
Jeff Davis:
I mean, I feel like you kind of spoke to that before. We carefully walked this line between actually like mandating tool use and just strongly encouraging, and ended up settling on very strongly encouraging people to complete the training, getting a lot of support from engineering leaders in every part of the business for echoing that message over and over again, but never actually going so far as to say like, “You’re going to start using the tools.” And yeah, I feel like it worked out pretty well.
Michael Redding:
Yeah. One thing that is interesting in the data, and I don’t have the breakout for what it means for coding time, but at least in terms of completion, we did have one engineering organization that took … I want to be careful when I say this. They didn’t want to say this is a mandate. They were very, very lax of like, “Hey, here’s a thing that’s being made available. You should like prioritize it maybe.” And that cohort was the cohort that had the lowest completion rate, right?
So even just the language that you use of sort of where does it fall on that spectrum did make a difference, right? So one of our engineering orgs that actually did have sort of the thou must language to complete, I think they have like an 80% completion rate. Part of this that I kind of glided over …
Michael Redding:
Part of this that I kind of glided over is actually having the tracking of completion available in a dashboard for managers so that way they could have those one-on-one conversations because it’s part of psychological safety,. It’s like, how do you make sure that you’re creating the space for people to have those conversations in a forum that isn’t public, that isn’t … That dashboard, I think we locked it down so only managers could see it. We didn’t want sort of public shaming, those types of maybe more negative approaches. And so I think there’s just a variety of tactics to take there, but really making it more about conversation unless top down, you must.
Jeff Davis:
And another thing I think is really important, we continue to talk to engineering leaders across the business about continuing to repeat this message about the importance of upskilling, of adopting new practices. Even in our most recent DX survey, like a month ago, we were seeing verbatims from engineers saying they just don’t have time to pick up the new skills, new tools. And so we find that we have to keep saying it over and over again like, “No, you totally do have the time. We need to carve out the time. This should be a very high priority.” And in spite of all the efforts that we went through through 2024 and 2025, we’re still seeing that reluctance from some people feeling maybe not entirely safe to invest in their own skill development.
Michael Redding:
Yeah.
Brittni Allen:
Super fascinating … Hello? Oh, there we go. Super fascinating stuff about what you guys were talking about with seeing the differences in the data for how teams approach this and the language that they used. On that note, one of the slides that you showed showed a jump from 25% to 50% adoption rates in the first week of January of 2025. Any idea of what led to that massive of a jump?
Michael Redding:
Yeah. It was less visible, but we also saw a little bit of a jump right after the holidays from 2025 to 2026, less noticeable because we were already at a high rate. And our best theory of the case is as people go off for the holidays, they’re just trying stuff out on their own time. They’ve got some downtime, it’s a holiday time, it’s vacation time, they’re racking up their bill with Anthropic or Google or whoever their providers for their personal projects. And then they come back and they’re, “Wow, this is really cool. We should do this.” And so we saw that in '24 to '25, that jump. From '25 into '26, we saw another jump with our usage around Claude Code in particular, which we think is similar effect of people using it on their holiday break. And so that’s our best theory of the case.
I do want to run a study just to try to really tease that out because we’re also seeing some interesting things from our non-software engineer cohort where people six months ago who were like, “I could never use the terminal. No, terminal bad.” And now some of them are our most prolific Claude Code users outside of the software engineering cohort. So there’s more that I would love to learn for that specific tool, but that’s our best guess.
Brittni Allen:
Yeah, that makes sense. So we talked about how you saw coding time improve. Well, really you saw reduction in coding time, which we’re saying is a success. How did you think about corresponding metrics to that? Did you look at metrics around quality or any other signal to validate that the full impact across the SDLC was actually positive?
Michael Redding:
Yeah. So we use the Core 4. I mean, this is the DX Conference, got to plug that. We set a benchmark target for our change failure rate, a do no harm target of about 4%. We haven’t gone over our do no harm target. When we look at our throughput, our discipline engineer, we’ve seen that go up as well. So I didn’t have that graph here because we can’t completely correlate it to just this program, but to all of our enablement efforts. At the start of 2025, our diffs per engineer, I think we jumped 40% year over year from 2025 to 2026.
Jeff Davis:
Yep.
Michael Redding:
So we know that was definitely not all of this program, but we’re kind of keeping track of those. And then even just in terms of like sentiment, which is really hard to measure, but just one of the things that I’m probably the most proud of, and I probably should have put it myself last self eval, is like our most curmudgeonly engineer in the entire organization that in the eight years that I’ve worked at Indeed he’s never had a single good thing to say about any of our internal tools, wrote a blog post that’s like, “Wow, I love Claude Code. This is amazing. My job is fun again.” And it’s like, “We did it. We did it. We did it. We can go home. We solved it.” And I think it’s just that kind of mix of qualitative and quantitative understanding that we really want to see.
Brittni Allen:
Jeff, it sounds like you have to redo a performance evaluation now.
Jeff Davis:
Yeah. He already got a pretty good rating, but we’ll see. And then also delivery lead time, that was another metric that we track and that we’ve seen a pretty significant decline as well over the past year.
Brittni Allen:
Nice. What about any insights into where this time saved is going? What are people reallocating their time towards?
Jeff Davis:
I mean, I think first and foremost, we’re seeing a 40 to 50% year over year increase in diffs per engineer. So they’re just doing more, there’s just more output. I don’t know. I don’t have a theory about where other time is going. Do you?
Michael Redding:
Yeah, that is probably the literal million dollar question. I do think one of the things that we are starting to see some concerns around is a bottleneck at the actual merger. So we use GitLab, so merge request, that part of the pipeline. So we’re starting to see, for a very, very long time, we had a very, very tight correlation between merge request opens and merge request resolutions, tracks very close to each other. And I think in the last two to three months, we’re starting to see those lines diverge a little bit. And so we’re definitely opening more than we’re resolving. And so that tells us that we’re maybe starting to see a bottleneck at a code review. That hasn’t completely manifested itself in our wall clock data yet, but qualitatively, we’re starting to hear that more and we’re starting to get some signals. So I would not be surprised if that started really turning up in our hard data where folks are spending more time in code review more than anything else right now.
Brittni Allen:
Yeah. Okay. Well, you all mentioned, you hinted that what’s next on the roadmap for you is redoing an enablement program but focused on agent first work. Tell us about your plans to also go back after the cohort who hasn’t yet adopted. Do you have any initiatives there? Do you have reasons why you think maybe they still have yet to adopt or what are you thinking there?
Michael Redding:
Yeah. So I think the thing that we’re the most interested in is that AI is coached. So at roughly 100% adoption of these tools, they are using these tools but maybe not as effectively. So if we’re able to have like the session data that exists and we are actively exploring DX’s new local demon that can get some of those transcripts as well, we should be able to start nudging those people who are using the tools but didn’t complete the training to kind of nudge them in the direction that they would have been nudged by taking the training. And so that’s probably the biggest thing.
I think there’s probably some other programs that we should do, some research that we probably need to do to understand, "Well, why didn’t you complete … " Some folks say, “Oh yeah, we didn’t have the time,” but other folks say, “Oh, I didn’t think I needed to learn this.” And so it’s kind of teasing that out, but I think the most concrete thing on our roadmap is that AI is coached.
Brittni Allen:
Yeah. Nice. Amazing. Okay. Well, any final thoughts, any final advice for the audience here?
Michael Redding:
It’ll be interesting to see what happens over the next year. I think a lot of what we have planned or a lot of what we did in 2025 was really reactionary based off of we were kind of behind the curve a little bit in terms of like where the agentic part of AI was going. We caught up and then we exceeded it. I think where frontier models are going, some of the things that we think are bottlenecks today, if Mythos is as good as everybody says it will be, some of those bottlenecks may not exist. And so I think some of our plans around training enablement are basically right now, what we know right now today, this is what we’re thinking. And already this morning Anthropic released Opus 4.7. It’s like, okay, well, let’s go rerun our benchmark and see what that means for our roadmap. And that’s just that continual need to constantly go back and kind of just see where the actual frontier models are.
Jeff Davis:
Yeah. And another thing I’m thinking about a lot now is I think we’re going to move the bottlenecks outside of the SDLC and I think we’re going to have to shift or increase our focus on training the rest of our R&D functions in the broader product development life cycle, all the way from like customer understanding through an industry kind of analysis through product ideation embedding and prototyping and then out to analysis in production. I think, yeah, we’re kind of thinking about the big outer loop as well and how we can apply some of these same techniques across the whole thing.
Brittni Allen:
So many roadmap items that I know everyone was able to gain from that. So thank you both so much for being here and for all of the lessons shared. Super excited to have you here with us today. Thank you so much.
For everyone else in the room-
Jeff Davis:
Thank you.