Abi Noda: Derek, great to have you back on the show for the second time. Excited to dig into all the latest research you’ve been doing. Thanks for your time today.
Derek DeBellis: Yeah. Oh, no, I’m thrilled to be here.
Abi Noda: So couple months ago, Dora released this new gen AI report. So I wanted to start. You were sharing with me just a little bit of context about how this report came together, some of the data coming from the last Dora study, and then some of the data being new. Just share with listeners the origin of the data collection process for this latest report.
Derek DeBellis: Yeah. So there’s multiple sources. Some of it’s coming from, I don’t know the exact number, but the 130-page Dora 2024 report. That’s great for some people, but we realized pretty rapidly that not everyone wants to read a 130-page report. That’s understandable. I can relate to that. So we started taking out some of the AI-specific nuggets from the AI sections in the Dora 2024 report.
We’re also working on a lot of blogs, just trying to get some content out there to people who might not be familiar with Dora, or just some bite-sized feedback or ideas. So we started combining that into this new AI report. And also we ran some additional secondary surveys to try to follow up on some hypotheses, try to close some things out, and that made it into there as well.
Abi Noda: I found the new AI report really digestible, really easy to just skim through. So for listeners who haven’t checked it out yet, would highly recommend going to Dora.dev and checking out that report.
Derek, in our last episode, we went really deep on survey development, validation, cognitive interviews, just the methodology around this kind of research. And I want to start there again with this latest research to give listeners some grounding, an insight into what all goes into this.
There were new constructs in the 2024 Dora report. What did your process look like for developing these?
Derek DeBellis: So this year, our method, it stayed relatively stable. But if we thought before that there wasn’t a lot of literature about some of the developer stuff, now that when you add AI into the equation, there’s even more of a dearth or vacuum of literature available to work with there. the challenge is how do you go from some abstract vague concept, as exciting as it may be, into something someone can reliably answer in a survey, and then something that you feel as if is capturing that initial vague underlying concept.
And to be relatively brief, We start forming our hypotheses. We start trying to understand the concepts that are underlying those hypotheses. Then we go about drafting survey questions, doing qualitative research to understand how people just naturally understand these questions, these concepts even before we even get to the questions. And then we go through and we just start really just iteratively changing small parts of the question. Take out this word through 10 iterations for some of the questions of just trying to get it to a level where people can easily comprehend the question, easily retrieve memories that are related to that question. We don’t want to… You might understand the question and then think, “I have nothing that I can attach in my memory that’s related to that.”
Then we try to understand how people make judgments about those memories and how people would naturally answer that question. So our answer choices are connected to how someone would naturally answer the question. We don’t want to force them into our schema for answering that. And then we have a question after all of that, and then we’d run the survey. And then the real challenge is through are we able to find internal validity and external validity? Essentially do these items group together in the ways we would expect to and relate to other constructs in the way you’d expect? For example, if job satisfaction was highly correlated with burnout, highly positively correlated with burnout, you might ask yourself if you’re measuring these things correctly because all the literature would point to and all the way people think about these would tell you pretty rapidly that they should be negatively correlated kind of thing.
Abi Noda: And what people don’t realize, we talked about this in the previous episode, is just how rigorous that process of iterating through a single question really is. We talked about how just slight differences in certain adjectives can seriously throw off the effect and the outcome of a single question. Any instances of that that are memorable to you in this last round of development?
Derek DeBellis: I think you’ve had a similar experience to this that you were sharing with me in this type of particular thing is just sometimes just one word in there. Some people will… Just the connotation of that word to even a subset of the people reading through it will just throw off the entire question. And even if it throws it off only for 15% of people or 10% of people, that’s enough noise to inject into the question to make it not be able to connect with things as well as you would’ve like.
Abi Noda: The one from our previous conversation, if I recall, was we ran into an interesting difference in terms of the interpretation of the word adequate versus sufficient.
Derek DeBellis: Yes.
Abi Noda: And it led us to actually looking at the differences in legal precedent because in law, those subtle differences in words actually carry meaning in court and litigation. So that was one particular example that was memorable.
Derek DeBellis: Yup, I remember you bringing that one up. And it’s tough too then who is right? Because the connotation that it has to the survey-taker probably matters more even if you knew the actual legalistic terminology behind it.
Abi Noda: I want to ask you about a few of the constructs or items in this 2024 report that I think are really interesting. The first one I want to focus on is flow. Now, I can’t remember if you had actually introduced that in 2023 or if that’s been revised for 2024. Explain maybe the process, and we can pull up the question here and read it a lot for listeners, but how did you arrive at the current method for how you’re measuring flow?
Derek DeBellis: Yeah, we had some prior art. They had done a lot of research that’s, I think, published how flow and focus, how all those fit together. And we wanted to leverage their work because they’ve already shown so many exciting relationships with that. I’ll just read it. “In the last three months, how often were you able to reach a high level of focus or achieve flow during development tasks?” And if I understand their logic behind it, it was that they noticed that people express flow in a lot of different ways. It’s a very multi-faceted construct and it means a lot of different things to different people. So they thought being general with this would actually be a better way to measure it than getting more specific.
Derek DeBellis: … with it. And they thought that flow already had a really strong connotation among developers that they would just put the word out here and let it connect to whatever their mental image is of flow.
Abi Noda: That’s really interesting. That brings up another fascinating problem is how concrete or abstract do you go? And it all boils down to what is the actual concept? And flow being something to your point that developers maybe have an internalized definition for themselves that can mean different things for different people, but still measure the concept that you’re trying to capture across the population. So that’s an interesting one.
Derek DeBellis: It is. I guess there’s a question. Is flow such a foundational and atomic element of how a developer thinks about things that we could just say flow kind of thing? Because I think flexible infrastructure, which is just the question on the website that’s above flow, that’s a multifaceted construct. If you just asked, “How flexible is your infrastructure?” the reliability across time for that question would probably be really bad.
Abi Noda: So speaking of reliability, something I think newer in the 2024 report on this AI report that you did was measure time. And actually, I retract my words because I know Dora has historically measured things like percentage of time spent on unplanned work, things like that. But in the latest research, you specifically measured two things pertaining to time. One is percentage of time spent on toilsome work, which is defined as repetitive or manual work, and then also percentage of time spent on valuable work
Abi Noda: Tell me about your journey in developing and iterating and refining these two items. Then I have a lot more questions about just the challenges of measuring time.
Derek DeBellis: So these questions are really… They’re tough for me because I don’t think if you’re looking at the validity of does this connect to actual the amount of time someone’s doing this? So when someone says an answer to how much time you spend doing valuable work and they say 40%, I don’t actually think that if we had some special clock that understood valuable time, that you would find it was 40%.
Abi Noda: Why not?
Derek DeBellis: I think for a couple of reasons we said in the last three months as a timeframe for it. So the ability to actually quantify in your head and tag what part of this was valuable and then add it up and then divide it by the total amount of time you were at work probably can’t happen, I’m guessing.
Derek DeBellis: What matters for me though is that a person who says 10% would reliably be spending less valuable time than a person who says 70% or a person that says 30%. I don’t know about 15%. Maybe at that point we’re starting to get that the granularity isn’t quite right and that’s important. And also I think we talked about this last time. Sometimes a question might not be getting at what it’s exactly asking at, but if someone says that they’re spending 0% of their time or 10% of their time doing valuable work, that’s really diagnostic of what it’s like for them to do work.
Abi Noda: Yeah, there’s meaning. Yeah, that’s a signal.
Derek DeBellis: Exactly.
Abi Noda: Because interestingly, this question actually has two calculations. One is the estimate of time, but two is the classification of work as valuable or not, which is also inherently subjective. But again, would still be a very difficult… It would always be subjective in any perfect system. But again, I think one of the questions I have is, well, how else would you measure this, right? In a perfect world, how would this be measured, right?
Derek DeBellis: Well, if I had all the log data and then I had survey data attached to it on a daily basis and I asked the person, “How much time did you spend doing?” Or I could have experiential surveys like, “Hey, are you doing valuable work right now?” And then you connect their log data to that signal or something like that, and then you could see what valuable work looks like in behavior. Then you could quantify that across a year or something like that maybe, or just use the experiential survey data.
Abi Noda: Yeah. And just for listeners, when you say experiential, you’re talking about experience sampling where you’re randomly sampling people’s day?
Derek DeBellis: Exactly.
Abi Noda: Extrapolating from it. Yeah, sure, give listeners a little bit more of a primer on how that would work just for their knowledge.
Derek DeBellis: Yeah. So I haven’t done it too often, but-
Abi Noda: I think it’s hard in practical organizational settings, right? Hard.
Derek DeBellis: Yeah.
Abi Noda: It’s hard to pull off.
Derek DeBellis: I just think of some of happiness research that came out of Yale recently. Not that this is the only place that does it. I know some internal teams did it and reported on it for the flow, what we were talking about. But essentially you might get a text message or something that pops up in your browser that pretty much would ask you about the exact moment in time kind of thing. It’s a lot easier to answer about an exact moment in time than three months usually. And all they ask, the questions are going to be asking you about, what you’re doing at that particular moment and what your experience is like in that particular moment. And then hopefully after doing that across time, I think they usually, it tends to be, what I’m familiar with is the cross time. You get to see some really interesting patterns and hopefully connected to what that person’s doing in that exact moment kind of thing.
Abi Noda: So back to this topic of measuring time.
Derek DeBellis: Yeah.
Abi Noda: So first of all, have you then come across personally or Dora the report, have you guys faced criticism internally, externally around these measurements of time in particular? Has that been a discussion you’ve been a part of? And if so, how have you approached that conversation?
Derek DeBellis: Yeah, so it comes up. I’d say it’s about, I don’t know the exact number, it’s back to the assigning percentages to things. But some people don’t bring it up at all, which is almost as equally as worrisome as when people do bring it up. Some people take it on face value like, “Oh, you surveyed time.” And I was like, “No.” And then some people just take that it’s just impossible that you’re actually capturing anything here. Of course, I guess my response to that would be, “Well, if it wasn’t capturing anything, it wouldn’t have such strong relationships to everything else.” I agree that you’re right, that it’s probably not capturing exactly what it’s asking, but the fact that it’s so connected to job satisfaction, productivity, flow says to me that it’s not just pure noise. If it was pure noise, it wouldn’t relate to anything else. It would be like injecting a random number generator into, I don’t know, a column in our survey data. It’s definitely not that.
So that gets back to that more of the external validity. Does this relate to the world in a way you would expect it would? And nine times out of 10 it does, and one time out of 10 it doesn’t. I think, not to get past this, but I think we’ll have some questions about how it relates to AI later, possible.
Abi Noda: And what do you think are the best practices around this? Because measuring time is important. Time is really the best connection we have to, for example, money. There’s basic things like you use a three-month, a 90-day reference period for the question. Obviously all the best practices we’ve talked about today and on our previous podcast episode around item development, really important. What are other things that even given more resources you think should go into, not just even the measurement, but operationalizing it?
Derek DeBellis: I think it comes down to the question you’re trying to answer and what you’re trying to improve or make better. And I guess it’s going to depend on the granularity then. So if you for some reason need to know as close to actual numbers as possible, I wouldn’t recommend doing in the last 90 days. The closer you can get in the last five hours or what you’re doing right now kind of thing, the better off you are. That accuracy for that number is more important.
Now, if you just want to have an idea of changes across time, say there’s, I don’t know, say there’s a… You want valuable time to go up. Maybe I agree that it’s a loose definition, but you want your employees to feel as if they’re doing more valuable work across time. Maybe something like this is general enough. So maybe if you find mean at time point 1 is 15% and mean at time point 2 is 37% of time doing valuable work, you might say to yourself, “Well, it might not actually be 37%, but I think what we just did had an impact.” And if that’s the scope of your challenge and the scope of your goals and employee retention goes up, employee satisfaction goes up, delivery of products gets better, your product’s quality goes up, I’d say you’re onto something.
Abi Noda: That makes sense. Related question, what nomenclature do you use nowadays when you talk about survey data? Nicole Forsgren and I were just talking about this and I was actually able to propose a terminology that was accepted into a doc we were writing together.
Derek DeBellis: Oh, cool.
Abi Noda: But I’m curious what yours is today. For example, do you object to calling server data subjective data wholesale?
Derek DeBellis: It’s funny because this comes up often. People will call it qualitative data that I’m talking to, and I’ll be sitting next to a qualitative researcher and I’m like, “I don’t think this person would call it qualitative data.” I could see where that’s coming from. I treat it as in the way I analyze it. I analyze it like it’s observational data-
Derek DeBellis: … with the recognition that it’s just a snapshot in time and it’s based on the perceptions of the respondents. So given that it’s based on the perceptions of respondents, the subjectivity thing doesn’t bother me too much.
Abi Noda: Okay. Let me give you my counter argument to that.
Derek DeBellis: Okay.
Abi Noda: So if I asked you how old are you?
Derek DeBellis: Good point, yeah.
Abi Noda: Right. Would it be fair for me to say that’s subjective?
Derek DeBellis: There is one study where with P-hacking to show if you change some things, that someone who listens to When I’m 64 by The Beatles gets older.
Abi Noda: That’s incredible.
Derek DeBellis: So maybe-
Abi Noda: Okay. Well, there’s the counter argument to my example.
Derek DeBellis: I did dig deep for that one.
Abi Noda: Well, let’s move on to talking about the findings from this research. there were I think two or three interesting findings we want to talk about and discuss. And the first one I have, we haven’t talked about this before, is one of the headliners from the report is, I’ll just read what it says, “Productivity is likely to increase by approximately 2.1% when an individual’s AI adoption is increased by 25%.” There’s a lot there. In the headlines, productivity gains are being talked about in very simple terms like, “Oh, you’ll be twice as productive or 30% more productive.” So can you translate your findings in various ways? Simple terms, what does this mean? What kind of productivity gains are we seeing right now?
Derek DeBellis: So I think it’s worth noting that data is from May or June 2024. It just took that long to get to this point. We’ve seen it in subsequent data, really similar effect sizes. But I think our goal with this data is to essentially use the survey data, whatever we want to call that, to use the survey data. I’m still thinking about the subjectivity part and everything like that. It’s a good… But to use the survey data to essentially create two identical people where everything about their background, their roles, their organizations is the same. So we have identical, we have two possible worlds that are the same in every way imaginable. And the only thing we do is on one of those possible worlds, we turn up how much AI they’re using a little bit. And that’s the only thing we change.
That’s the goal. We try to create a virtual experiment. And what we find is when you turn up AI by 25%, that’s an arbitrary number, we could have picked 50, 100, we just wanted a number that didn’t seem like it would be that much of an effort. Increasing AI by 25% would be spending an extra 15, 20 minutes a day using AI or something like that properly.
Abi Noda: Yeah, okay. That’s what I was going to ask. What does that translate to in concrete terms? So okay, about 20 minutes additional of AI usage per day translates to about 2.5, was it?
Derek DeBellis: 2.1.
Abi Noda: 2.1%, which means what does that to you? And you don’t have to even go off the data, but just riffing on it, what does that translate to just overall in terms of what’s possible?
Derek DeBellis: Yeah. So it’s 2.5% in a latent construct of productivity.
Abi Noda: That wasn’t layman’s terms enough.
Derek DeBellis: I don’t think that means for an individual earth-shattering productivity gains.
Abi Noda: Yeah, but give me a number. Come on. You got to put it. I want to hear a number.
Derek DeBellis: Well, I guess it would be in terms of what? Line-
Abi Noda: Here we go. For that statement, I read a lot productivity. Was that based on that? That was a multi-item, self-reported productivity construct, right?
Derek DeBellis: Yes.
Abi Noda: Okay. And remind us is what were the main factors that went into that?
Derek DeBellis: Yeah. So we were inspired a few years ago by some of the work… What’s that? Slow Productivity? I don’t know if you’re familiar with that book.
Abi Noda: Is that Cal Newport?
Derek DeBellis: Yes.
Abi Noda: Yeah.
Derek DeBellis: Yep. And the idea behind that is that productivity for developers and knowledge workers isn’t how many keys you type. How many times do I press the space bar? How many times do I press tab or something like that? Probably not a great measure of productivity for knowledge workers just all up generally. So our productivity question tries to get at if they’re creating value. And so we ask, we have four items. One item actually got dropped from the factor. So I’m able to do my work in the most effective way as possible. I am productive at work. There’s just a basic productivity is an atom kind of thing. People have some sense. And my work creates value. Those are the three indicators that go into our productivity factor.
Abi Noda: And so going back to the question of what’s possible, if we just take a simple 25%, correlates to 2.1% productivity gains, and if you assume starting at zero, you have a max. So someone going from no AI usage to about 90 minutes of AI usage a day would be about 10% more productive along-
Derek DeBellis: Yeah.
Abi Noda: … those lines? Yeah, okay. And that makes sense. I think that would align with similar research we’ve done and data we’ve seen. And that’s different, as you said, from some of the more systematic, or sorry, more throughput-based measures and outcomes that we’ll talk about in a minute.
Derek DeBellis: Definitely. I think what’s also interesting to think about that even if we just say we keep it at the 2.1% and the 25% increase in AI reliance, if you multiply that across a thousand developers or a hundred developers even, that maybe the 2.1% for an individual isn’t that huge. They just feel, “I can handle some of this toil while I wrote those four lines of code pretty fast. Oh, I was able to look through the documentation a lot quicker than normal.” But across an organization, that could be a dramatic impact for if every one of your developers has a 2 to 10% productivity gain, that’s not nothing. That’s far from nothing.
Abi Noda: That’s a lot of time or something.
Derek DeBellis: It’s a lot of something, yes. Hopefully, quality and value and customers that are satisfied, and then hopefully that’s aligned with revenue.
Abi Noda: Right, absolutely. So something else that we were talking about earlier is not necessarily, you have an interesting hypothesis around. You saw time doing valuable work go down. So explain to the listeners, why is this surprising or perplexing? And what are some of the hypotheses around it?
Derek DeBellis: So I’m looking over here because I have the results right here so I can just… Job satisfaction went up. Productivity went up. Flow went up with AI usage. So turn AI up in this possible world, those go up. But what we see is time doing valuable work goes down. So one of the promises, at least on marketing material for AI, is toilsome work, say bye. Time doing valuable work and creative work, that’s your future. That’s all you’re going to do now. Our data seems to suggest time doing toilsome work seems, who knows, but probably not really that affected by, remember, June 2024, and your time doing valuable work goes down or the percentage of time you report doing valuable work goes down.
But that’s also even more confusing because we know time doing valuable work is highly correlated with job satisfaction, productivity and flow, all of which were benefited by AI. So that leads us, if we just looked at any one of those findings, it might be really easy to talk about, but if you’re watching a movie and you explain the whole movie based on one scene and your explanation, contradicts every other scene in the movie, that’s not a really good explanation of the movie. So we have to think about the whole data pattern. But our main takeaway from that is, well, maybe it’s not taking away from the amount of valuable work you’re doing. It’s just expedited how rapidly you’re able to get it done.
Abi Noda: Yeah, it’s really interesting. So last finding I wanted to discuss today, which I think was one of the headlines from the report, is that you actually saw software delivery performance that you’re mentally impacted by increased AI usage. And what’s really interesting, so there were two findings. You saw a negative 1.5% change in throughput And you also saw, and this is less surprising, is a 7.5% decrease in stability. Talk through these outcomes. I think that throughput one is particularly remarkable because that’s such a conventional way people are trying to actually show the positive lift from AI currently.
Derek DeBellis: Yeah. And I know a lot of people are collecting similar measures on this and having similar results. So if it was just stability, it would be a pretty easy story because we’ve heard a lot of developers say, “If you have too much trust and over-reliance on AI and you’re shaping huge changes, it’s going to affect your stability of your delivery. You’re going to have more change failures. You’re going to have more rework.” And especially if you don’t have good feedback processes like automated testing and even version control involved in all this. It’s just going to amplify those maladies and just make things worse.
But now that we have throughput down too, and also considering that code complexity went down with AI, documentation quality went up, code review speed got faster with AI, approval speed, these are all things that help the software delivery process. And I think three ways you can look at it, the easy one is just be like it’s a learning curve. People just started incorporating AI into this flow. We haven’t figured out how to take those individual gains and translate them into delivery gains. And I think there’s some truth to that in terms of constraints. If we think about this, and maybe in a theory of constraints kind of way, an analogy would be if you were, I don’t know, a tomato soup company and all of a sudden you had 5,000 more X times the amount of cans you could create, that doesn’t mean necessarily you have that many tomatoes to fill those cans. It’s like you need to have understand the constraints everywhere in the process.
Abi Noda: We were talking earlier about what are the biggest questions that leaders are asking, organizations are wanting to know. one being the question of what can we expect with AI? And we talked about this a little bit earlier in terms of that 10% increase, but I think we should double click on that. When you’re asked today what can we expect, what’s your guidance today on that question?
Derek DeBellis: I guess in a lot of ways, this is where Dora puts all its effort into. The report in a way is just saying, “Hey, here’s what to expect,” in the sense of maybe these percentages getting back to they might 2.1% on a latent construct. Who really… It’s hard to actually quantify that in something concrete, but relative to something like user-centricity, it’s about 1/8 the impact of having a user-centric organization kind of thing. So relative to other latent factors and things like that, you can start to understand, prioritize.
But I do think that there’s a lot of benefits and I think that’s clear in this data and subsequent survey data that we have and research well beyond ours that there are a lot of benefits to be had. But with each place that you’re implementing AI, there’s the possibility of byproducts that are unexpected and there’s the challenge of thinking about it holistically, not as just something that can fit in to particular areas or just get thrown against the wall and everything’s better kind of thing.
And also, and just talking with people from the qualitative side of things, leadership that has inaccurate estimates can really put a lot of pressure on employees. I’ve talked to one quote that stuck out to me was someone told me, “I’m expected to do twice as much work with half the resources and half the time because I have this magical tool.” And obviously they’re struggling to do that.
Abi Noda: To live up to that, yeah.
Derek DeBellis: Yeah. And you can see there’s some more survey data that came out by someone else, and I can try to find the link so you can maybe reference it, but leaders and developers or individual contributors have very different notions of AI’s impact. And we saw that in our data too. So if you look at the attitudinal data, leaders think people are getting a lot more out of it than the actual employees that are using the tool. Not that they think they’re not getting anything out of it, but not as much as the leaders think they’re getting out of it.
Abi Noda: Really interesting. And what’s your guidance today on measuring all of this? And of course, repurposing some of the approaches that you utilize for the support is one avenue. Obviously I’m in this business as well, helping organizations, but what’s your perspective on this? And I think in particular, given the finding and observations around impact on things like throughput, which I think were pretty surprising.
Derek DeBellis: Yeah. So the high-level guidance would just be if your organization has worked really hard to establish goals, some sense of where you’re trying to go, hopefully your measures are operationalized in a way that they’re aligned with those goals, that they’re a pretty full picture and a multifaceted picture of what your goals are. So you’re measuring what you actually care about. And so I’m not terribly prescriptive about what those are. It’s just they should be really aligned with the organization’s goals because a lot of times it’s easy to suddenly think lines of code written is now the new success measure or how many times you accept.
Abi Noda: It doesn’t.
Derek DeBellis: Yeah, that’s a means to an end in my opinion. Maybe if adopt AI adoptions one of your organization’s goals, that makes sense to me. But AI adoption is the first step. The next thing is how do you make the AI adoption effective and useful kind of thing. So make sure your measures are aligned with your goals. Multifaceted, I think the space framework does a great job of trying to say, if you just look at one facet, you’re going to be able to game it and you’re going to miss some of the important guardrails around it. I would listen to that podcast that you had at a space. I was nervous to come on after I listen to that. It’s so good.
Abi Noda: Yeah, Peggy’s great.
Derek DeBellis: And then from there, I think it’s about creating a feedback loop. It’s about actively measuring and then when you have the data, when the data you have it, it gets incorporated into your plan and your strategy, it’s not a vanity metric, you’re going to find discrepancies between where you want to be and where you are unless you have bad measures. If the measures say you’re perfect, they’re not the right measures probably. So you’re going to find that discrepancy between where you want to be and where you are. And that discrepancy should fuel research. It should fuel initiatives. It should fuel strategy. And if that’s not part of the feedback loop of these measures and revisiting them across time, I don’t know how useful they are if they’re not actionable or at least something that it’s built into your process that it prods action.
Abi Noda: I think every time something new comes along and folks are scrambling the measure, you see the same mistakes being repeated all over again. So to your point, one of those mistakes is just measuring what’s readily available. So yeah, things like acceptance rate. Oh, that’s a metric GitHub provides us. So that’s now suddenly this important metric. But to your point, I would completely agree that that’s a means to an end, then you see a regression back to crude output measures like throughput. Even though I would say largely similar to your findings from your latest research, we find very hit or miss relationships between output and AI usage. And that’s not even factoring in the amount of control that you’re putting into your data analysis. When we do a really rudimentary cross-sectional with no controls, so just, hey, just segmenting different metrics by level AI usage, it’s not uncommon to see that the cohort that’s using it more has higher throughput, but what that misses is are the people who are just more prolific, output-oriented developers also just using AI more because they just are more early adopters. So completely data that’s interesting but not necessarily… A lot of what we see in the headlines I think is not really valid scientific data, right?
Derek DeBellis: Yeah, I completely agree. That gets to data is like it’s a shadow on the wall. You don’t really know where it’s coming from. It just takes a form and you’re trying your best what generated that shadow kind of thing. It’s like a Plato’s cave type thing.
Abi Noda: Yeah. Well, Derek, always a lot of fun to have these conversations, and thank you for your time today to come on the show. Super insightful conversation. I’m sure lots of listeners will find this valuable.