Brian Houck (00:01.784)
Welcome to the Engineering Enablement Podcast. I’m your host, Brian Houck. Today I’m joined by Andre Meyer, a senior researcher, lecturer, and software engineering practitioner whose work focuses on the human side of software development. His work explores developer experience, flow, and many aspects of productivity.
His latest paper, Too Many DevX Metrics, Too Little Guidance is out now in ACMQ. It looks at how teams can choose metrics that help them understand and improve their experience. With AI changing how we build software, we’re going to be discussing today what’s worth measuring, what makes the work rewarding, and whether we’re automating the right things to begin with. On a personal note, I first met Andre when he was doing an internship with Microsoft Research working on a paper called Today Was a Good Day. That paper was foundational in how I think about developer productivity and understanding and measuring sort of developer experience broadly. I think I could say that that paper sort of kicked off my entire career.
And so it is with tremendous pleasure that I introduce my good friend and peer researcher, Andre Meyer. Andre, welcome to the show.
Andre (01:31.434)
Hi Brian, thanks so much for having me. It’s always super exciting to talk to you about good developer experience.
Brian Houck (01:38.816)
Awesome. So let’s let’s dive into your new paper. Your paper opens with an engineering leader being asked, you know, we’ve invested in AI, but has it helped? And leaders today frequently have dashboards, but they rarely have clear answers from those dashboards. Like, why is it such a difficult question to answer? Like, you know, has AI helped? Like, has our investment in AI helped?
Andre (02:08.426)
Yeah, from talking to developers and building the data set around the compass, which by the way includes metrics from around forty companies, we’ve noticed that most dashboards are only covering AI use and AI output. So instead of actual outcomes and value delivered. So they would look at metrics around AI adoption, daily active users, acceptance rates, in some cases even lines of code or token spent. And those are really easy to collect.
Which is why they are so commonly used, but it only shows if AI is being used. And it doesn’t really show how it helps to create more value or even true return on investments. And also we’ve noticed that they use a lot of sort of classic output related measures like throughput and number of pull requests and deployment frequency and things like that. And they were already a challenge before AI.
To like compare between teams and divisions and so on, but this got even more ambiguous with AI. The problem is that they were designed for a world where producing code was really costly and now it’s almost free, it’s super cheap. And so those metrics inflate a lot and they’re also very easy to game. And shipping more doesn’t really mean that you’re creating anything of value, right? There is one positive thing that I’ve seen though.
Brian Houck (03:27.437)
Yeah.
Andre (03:31.296)
From like AI measures, which is that most companies still capture qualitative feedback around developer experience and the impact that AI has, with things such as satisfaction and friction and AI time savings.
Brian Houck (03:45.336)
So that’s really interesting. Sort of the idea that whether or not AI is helping is a difficult question to answer because the easiest metrics to measure are the ones that tell us sort of the least about outcomes. And to try to help us reason about measures better, in your paper, you mapped out more than 120 different metrics, as you say, across over 40 companies.
And you used that to create what you called the DevX Metrics Compass. Could you briefly explain what the Compass is and sort of why you decided to create it?
Andre (04:23.626)
So as you’ve mentioned, we’ve looked into all the measures that we could find in the engineering space, so of what companies are actually applying, but also from the research space. And we’ve seen that there are a lot of metrics and we’ve started to collect them in a systematic way, and we’ve gathered the 120 or more measures, and we’ve also seen that there are more and more frameworks popping up, such as like the space framework and Dora and Ang Drive and so on.
But it’s super hard to really navigate all those frameworks and all those metrics and really identify which metric does really matter for my own team, for my context, or for my goals. And it’s especially hard to get started with defects measurement if you have no metrics at all or new measurement system at all, no dashboards. So the compass really helps you to navigate this fragmented landscape by exploring metrics, by comparing or benchmarking against other companies that are maybe similar to you and also by allowing you to receive an actionable report at the end to sort of know where to get started and what to do next.
Brian Houck (05:35.086)
For those who have not seen the DevX Metrics Compass, I like cannot encourage you enough. It is beautiful and just like awe-inspiring how many different metrics are sort of mapped together to different frameworks. And it’s like it’s a lot of fun to go and dive into. And so as you were creating it, as you were diving into all of these metrics in use across the industry, what surprised you most about what companies choose to measure?
Andre (06:06.132)
Before I answer one quick addition to my previous response, I wanted to give a shout out to also my collaborators on the compass briefly, Thomas Fritz, Gail Murphy, and Patrick Meyer, who helped me with all of that and setting it up and giving feedback and so on. And to come back to your question about what surprised me the most is that most of those metrics are focusing around development-related activities, and almost none of the metrics around collaboration. And challenge around that is that developers spend maybe 30, 40, 50% of their time in collaboration and communication, but with those dashboards, we don’t really understand how this part is impacting their work-lifes and their developer experience.
Brian Houck (06:53.26)
I think collaboration is part of the very definition of developer productivity. It is one of the dimensions of space and has been often the hardest to understand because collaboration is incredibly important to delivery and innovation, but it can also be an incredibly large source of toil. and I’m curious, like is there one collaboration metric or a couple of collaboration metrics that you’ve seen there, like, that’s a really cool way to look at it.
Andre (07:25.026)
Not that I’ve seen, to be honest, and and there are only very few collaboration metrics that we actually have been able to find. Exactly. That’s an opportunity for researchers, for DX and many others to contribute. I think that some good measures would evolve around finding a balance between collaboration versus individual work time, meeting quality, meetings being scattered throughout the day, but also some of the metrics that we have seen, like focus or flow and interruptions, but also yet they are really hard to measure, but they are also very hard to act upon oftentimes. And one of my more preferred ones are also to look into after work emails or communication, because that’s often a sign that something is amiss.
Brian Houck (08:14.326)
Which is like a huge shout out to the idea of measuring sort of meeting like clustering. And do you have sufficiently large gaps between meetings to do deep focused work? Found that that can be substantially more predictive of how much work devs get done than just like the total number of meetings. So as as you look across all these metrics, I suspect you know you you found that even very similar companies choose wildly different metrics. And, you know, how should a listener to today’s episode think about, you know, deciding like what’s worth borrowing versus not?
Andre (09:02.4)
That’s a fantastic question. And I want to start with an example from comparing Google to Microsoft, because they are both in the data set and they both publicly shared between 20 and 24 metrics, yet they only have around six measures that are shared among the two. And of course, there’s the caveat that we can only compare what they publicly shared, what we know that what what they thought was mature enough and so on to actually share but it also, if you think about it, makes sense to not just use a fixed set of metrics and apply to your company, but to actually tailor it to your own needs, to your measurement goals or to your initiatives. So maybe in the beginning you might want to start with more generic measures if you’re new to DevX or you want to start observing DevX in some ways. But as at some point, when you start to mature more and more, you wanna look more into specific ones, maybe around a certain friction that you identified or that you wanna tackle. and then you wanna create specific measures around that particular friction to see if your initiatives or your rollout actually makes developers’ experience and work lives better.
Brian Houck (12:01.174)
In the paper, you say that having more metrics doesn’t equal more insights. Does your compass run the risk of exacerbating that problem?
Andre (12:13.45)
That’s a great question, Brian. And I think it may, because starting with the compass and seeing the 120 plus metrics could also be a bit overwhelming. And you might just want to start to select everything that remotely sounds interesting. This is why we recommend defects leaders or engineering leads or even team leads who want to start capturing metrics to take a step back first and think about what is the goal that I want to achieve with my defects measurement. Is it to increase what value or is it to increase quality or developer enablement and things like that? And based on that start to filter down into the compass and start to choose metrics that are really specific to the objectives that you have.
Brian Houck (13:03.928)
So you had sort of hinted already that there’s maybe a little bit of selection bias at play and that your data set captures what companies publicly report measuring. How do you distinguish a real gap in measurement in something, say like collaboration, versus something that companies simply don’t want to talk about publicly?
Andre (13:30.176)
And I do think that most of the gaps that we found or visualize in the compass are real with some caveats and of course we can only capture what they publicly share before or what they admitted, like what they admit to. And I would assume that they basically share metrics that are mature enough or also they might omit some sensitive topics around like burnout or attrition and things like that.
So, what we are doing is we are triangulating the gaps with other signals. And one of those measures is that we have conducted interviews with 20 plus engineering leaders in the space to talk to them about their measurement practice. And some of them are also included in the compass. And they basically confirmed that the pop most used metrics are also the ones that they publicly disclosed. And those that those are basically the ones that are most actionable to them. There is some nuance though, like some composite or higher level metrics that often look very nice, like in a paper representation. They are often really hard to act upon and might be eventually removed.
The other thing that we are doing is, we are looking at research as well. So we’re not looking only at what companies share, but also what researchers found in industry papers, but also more scientific ones. And there we see that there is a trend of looking way more at well-being-related measures such as cognitive load and stress and burnout and so on in research compared to the industry. And then finally, as soon as more and more people are using the compass, we can use anonymous usage data to see what people are searching for and what they don’t necessarily find. One early finding is that some people started to look for metrics around joy and meaning, which is not something we’ve really seen in any other paper or shared blog post yet.
Brian Houck (15:40.128)
That is like so heartwarming that the way that like you’re seeing the compass actually be used is to find some of these more humanistic measures. And so I see sort of multiple sources of tension in the compass where practitioners might be more reluctant to talk about metrics that have not been fully vetted and you know they’re they’re still in progress, versus researchers, we want to see the cutting edge metrics to like try to help refine them and and think about them. And I’m curious, do you have this notion of not only which metrics, you know, may or may not be harder to act on, but how difficult are they to implement in practice?
Andre (16:34.358)
To admit, I haven’t really talked to them too much about measures that are actionable or not, other than like a few occasional inputs that they have given us. Very often I think it depends a bit on how abstract the measure is and how it is actually analyzed. So if it’s let’s say a flow measure, because this was a concrete example that I’ve seen at a company, looking at flow from a very specific team level perspective, or even a few couple of teams, this is usually still relatively helpful because they can say we have a small initiative, let’s say we group meetings and we don’t scatter them throughout the day. Does this increase our flow? On a larger scale, more and more company a couple of companies mentioned that it was really hard for them to make this actionable because people’s work is so individual. And what makes, in that case, good flow or what improves their ability to focus was just super hard to generalize and come up with initiatives that work for everyone the same way.
Brian Houck (17:46.382)
So that’s that’s very interesting, but I could imagine a world where even if conceptually a metric is easy to sort of action on, you know, it it helps guide decisions, from a technical standpoint, it is very hard to measure because of the required instrumentation and things of that nature. And was that part of sort of coming up with a compass or thinking about metrics, like looking at the feasibility of implementing some of these things from a technical standpoint?
Andre (18:16.924)
Not necessarily. I think we wanted to display what is out there as accurately as possible, but we did see besides some of the higher level or harder to measure ones that we’ve only seen at basically the bigger companies that have entired FX teams and good instrumentation to to try this out. the other thing that we have seen is that outcome measures or value-based measures are really, really rare.
And hard to get right. And we also prompted people to talk a bit more about that in the sense of is this important to you and how would you want to measure it? And a lot really said that they struggle very a lot in finding good definitions or good designs for metrics. And then they resort to basically relatively simplistic ones, like on the customer side, maybe on things such as deployment frequency, or if customers reported bugs, or if they’re happy on their promoter score, or on a business side mostly around AI spend and revenue. And they are not really satisfied with that status quo. But the key problem there is that they really have to not just look into engineering data, but also triangulate data from project management software or customer support data and finances and so on, which is really hard even to get access to from like a privacy perspective sometimes or from a technical point of view.
Brian Houck (19:46.496)
Yeah, so then like let’s let’s get specific. If an engineering organization wanted to measure sort of customer or business value for a specific engineering initiative, like what metrics should they be looking at?
Andre (20:01.92)
So in the compass, we try to be very neutral in not making specific recommendations, but I can share you my personal point of view basically. I think while having to admit that it’s really hard and it would be an amazing research project to do at some point, I think it would be interesting to look at it from an innovation side of view or innovation point of view where we look at how much time can developers actually spend implementing new features and pair that with user adoption data or customer satisfaction feedback? so releasing this particular new feature that lots of customers ask for, does this really make people happier? does customer retention increase and things like that? And I think around that, we could also look into what the Microsoft Engineering Thrive framework does ID to customer time. Like how long does it really take us to to ship innovation? Because if you get customer feedback on what would be valuable to them in their daily lives or making their job safer or making their using the software more fun and it takes a year to ship, they will have forgotten already about it or switched the product.
Brian Houck (21:21.184)
I love that. And I think especially as organizations start trying to look beyond just the SDLC to the broader PDLC, things like time spent on new innovation and the ultimately the velocity of that new innovation will become incredibly important and will matter to lots of different disciplines, not just software engineering.
So you argue that a healthy measurement system includes the concept of retiring metrics. Like what tells you that a metric has outlived its usefulness? Even if leadership may still like seeing it.
Andre (22:04.022)
It’s amazing that you’re bringing this question up, Brian. Thank you so much for that. So there is this myth around creating defects measurement once and be done with it. And many treat it as simple project and not a continuous practice. But the reality is that problems are changing, tools are changing, teams are changing, and so on. So if we just create something once and don’t evolve it, that’s bad. But also if we just keep on adding metrics that’s bad too, because some metrics might no longer be serving a specific purpose. And if we have too many metrics, there will be a lot of noise or waste and dashboards will be cluttered and it will be hard to know what is really relevant or what even changed. And also I think metrics that have been used for a long time have a bigger chance of being used, misused for gaming. And might be used as a target at some point, for example, for reviewing people or rewarding people. Once people start to learn about a specific metric being a target to greater success, they might start to optimize for them. So that’s basically Goodhardt’s law, right?
And what I would recommend companies to look for is whether a metric is still actively being used. So is somebody actually monitoring it and using it? Did this impact any decision in the past few months or not. If not, you can probably get rid of it.
Is it actionable? is it something we actually know what to act on if a metric gets better or worse? And then also a question around should we keep monitoring it? because it might have solved a problem in the past because of an in certain initiative or a tool or a process that we’ve rolled out.
But if this is something that we have established well, in theory we could remove it from the dashboard. And if this eventually becomes a friction again, our quarterly survey, for example, would raise that again. And then finally, I think we can should also ask ourselves the question, do we still trust the measure or are there any signs of like gaming it or being misused in benchmarking and things like that?
Brian Houck (24:25.102)
So that’s really interesting 'cause again there are sources of tension here where it’s you don’t want a million metrics 'cause if you focus on everything you’ll but you also need to be able to see if you’ve regressed. You know, there might be a problem. So let’s take an example, time to first pull request. You know, how long it takes a new hire to merge their first PR. And maybe in a given organization, it’s really bad. It takes many, many weeks. They focus on it. They bring it down to, you know, within within a few days. I could imagine a world where there’s this desire to say, okay, we’ve solved this problem, but stop measuring it.
But then it gets worse over time and no one sees it because that measure has been retired. And so how do you balance that tension? And is your your answer just we’ll we’ll get it in this survey?
Andre (25:19.894)
I think it really depends on the specific measure and again the goals and so on that you’re pursuing. And I do think there should be a base set of metrics that you know how to apply them, you know how to review them, and you’re really looking at it over the long time. almost almost like guardrail measures that that you just observe for a while. And I think those you shouldn’t change and throw out a retire. But we are seeing some companies or also in the interviews, this was this was a topic of discussion where people said, Yeah, actually we realized that nobody has ever looked at this metric anymore, but it just keeps being there. So just a simple review once a year or so can help to see if a metric is still relevant or if it still serves a purpose. But I do agree that maybe a baseline of metrics should always be around.
Brian Houck (26:18.19)
I remember when we first released space, people would eagerly show me their dashboards that would have every dimension of space and would have multiple metrics under each of those dimensions. And even though in the paper we explicitly said, like, no, no, no, don’t measure every dimension, like choose maybe three that are highly relevant to you, focus on those, and then move to meet the pressing needs of the moment. I’m curious, from your experience looking into this, when thinking about you know having this set of baseline metrics, like what’s the right number? How many is too few? How many is too many? What do you think? What have you seen?
Andre (27:55.209)
That’s a very interesting question, Brian. I think there is not really a a good answer to that because it it’s very individual to your measurement maturity, for example, or your ability to act on those measures, or if you’re splitting certain measures or showing certain measures to only a specific subset of users and so on. So I can’t really give you an exact number, but I’m also curious to hear what you have been doing.
Or what you would recommend to to companies?
Brian Houck (28:26.318)
So I think from from my experience, you need certainly more than one or two, because metrics have to act as checks and balances. And so you want these guardrails. And again, I mentioned, you know, if you focus on everything, you’ll accomplish nothing. And so I think sort of three to five top-level metrics is what I have seen makes sense for a lot of engineering teams
But I think of sort of a taxonomy of metrics or a hierarchy of metrics where you might have three to five top-level metrics.
But then you have lots of data points underneath that help you understand those patterns. And like there you might have dozens of them, but those are like the drill-down details for actually guiding the initiatives. And that sort of makes me think, you know, if you looked at the metrics compass, did you see other engineering leaders or other mad scientists like me thinking of sort of hierarchies of metrics?
Andre (29:24.328)
Actually no, and I’m surprised by that because I think it makes a lot of sense. I mean, we did talk a bit about abstractions and having some very specific measures that only serve as a backup essentially or as a additional context to another metric. but I do think it can be valuable to start with a couple of high level ones that those are probably also the ones that you will not retire at any point and you will keep on measuring. And then if you need more specifics, you can drill
into those metrics to see other telemetry metrics or even to see qualitative inputs from interviews or self-reports and things like that.
Brian Houck (30:04.28)
I mean, I think activity metrics are a good example of some of the things that I think are sort of at that lower diagnostic level. And we had opened this conversation talking about, you know, how do you measure the ROI of AI? And as you mentioned, so many engineering leaders jump to activity metrics because they’re they’re easy to collect. And I think those are really useful. It is interesting to be able to explain how we do our work, but
Sort of returning back to that question then, if if you were giving recommendations to engineering leaders about specific measures of AI ROI beyond just activity metrics, what do you think you might recommend?
Andre (30:50.238)
I think there are a couple of steps that I would recommend engineering leaders to take. And the first one would be to define the objective to think about why would they want to measure it and what do they want to optimize for. So is it for improving defects? Is it to create more features or values, or is it to improve quality? Because this will shape the the metrics that they end up selecting in the end. And then secondly, I think I would ask them to or recommend them to pair telemetry data with self-reports. Because telemetry tells you a lot about what changed, but not really why. And there is sort of this common myth that automated data tells you the whole story, but I really think it is wrong. Because we do need qualitative feedback and self-reports to help us interpret and contextualize the telemetry data. And if you look at Google and Microsoft and their systems, for example.
They also run service. And even though they would have very elaborated telemetry systems and good teams that can do proper statistics and so on, but they still want this additional layer of qualitative inputs. And so in the on the qualitative side, I think I would look into the typical things that we already have done before AI, such as satisfaction, ease of delivery, overall time savings, productivity, and things like that.
And then of course always pair it with a qualitative feedback on friction points. and taking those friction points serious and actually acting on them and analyzing them. And on the telemetry side, I think I would look for a established set of solid metrics that are measured across several dimensions, so not just output or activity or pull requests or lines of code and and all of that.
And also measures that are observed over time. so not just looking at them on like a single as a single value or single snapshot, but looking at trends and relative changes and and the like. So ideally creating a baseline of some sort. and I think
Brian Houck (33:05.014)
Is it I’m I’m curious on on that, is there a dimension that you think gets missed most often?
Andre (33:16.128)
So we don’t we have we’ve already talked about value-based metrics that are very missed a lot and the collaboration aspects. But also I think quality remains very hard to measure and the impact on quality. And but I think there are a couple of metrics that are helpful in that regard, such as yeah, rolling back releases afterwards and failures that are found after releases and and so on.
Brian Houck (33:24.738)
Yes, of course.
Brian Houck (33:43.948)
I mean, change failure rate is something that I’m really interested in looking at those patterns relative to AI adoption. And proving the point on why looking across dimensions is so necessary, we see across you hundreds of organizations, change failure rate doesn’t seem to be getting better as we’re using AI more. And like that’s something we need to keep our eyes on.
So shifting gears a little bit, as I mentioned in the intro, when we first met, you were working on your good day study. Could you just explain very briefly to the audience what that study was, what the questions you were trying to answer?
Andre (34:23.562)
Yeah. So this was a project that I’ve done in one of my internships at Microsoft Research together with Tom Zimmerman and Chris Byrd and Earl Barr as well. He was a visiting professor at the time. And that’s also where I’ve met you. And we wanted to better understand what makes developers’ work days good and what makes them bad. because potentially this better understanding could help us to better understand the
Very individual factors and reasons that actually make it they good or bad, and it could help us to fix the processes and tools that can help us to optimize happiness. At the end, happy developers solve problems better, they are more motivated, retention is better, they have less sick days, as you have shown in a study, and they will just do better work overall.
Brian Houck (35:17.538)
And unfortunately, what we’re seeing is this tension now with AI between developer sentiment and sort of the amount of work you know we’re producing. Annivella calls it the productivity experience paradox. And and I find that just like a very interesting phenomenon that is terrifying in in many ways. So you were mentioning
Sort of this idea of mixed methods, right? Like don’t rely only on telemetry, don’t rely only on surveys, but when you can blend them, that that’s even better. And when with the Good Day study, before the survey, you added yet another research type where you did observational work. You actually like went and sat with developers. And I’m curious, what did you learn from that that you wouldn’t have ever been able to see in the telemetry itself?
Andre (36:15.028)
Yeah, it’s so much fun to reflect back on the work that I’ve done initially when I started my research in the space. And the very first study that I’ve done were those observational studies. And it was super weird actually because I had to sit behind developers for several hours. I had to sit very quietly to not interrupt them and to hope that they forget about me while trying to really grasp what they were doing, the tasks they were working on, and the cut context and so on.
Brian Houck (36:37.706)
Mm, of course.
Andre (36:43.732)
And it was even weirder for the developers because they had somebody who took notes of everything they were doing, all their activities and bio breaks and meetings. but it was an interesting experience to really understand especially the crazy amounts of context switching that impacts developers’ work. And we later did some studies with automated tool usage data, and there you get a lot of very rich data around.
Activity switches. So is a person switching between the ID and the browser and the chat. But also they are not all bad, right? So if you’re switching from the ID to the browser and read up documentation, this might not be a costly context switch. But from just automated tool usage data, you wouldn’t know about that. So when we’ve done observations, we were able to distinguish between the different tasks that developers were working on.
And this sort of gave us this more nuanced understanding of the cost of task switching. And we’ve seen there that some task switches are, of course, very expensive, especially when developers have to rebuild their mental model of a task and we’re talking about very different aspects and topics of work or even work-unrelated things. But there were also tasks that developers reported as not being costly.
So for example, when they were waiting for a build to complete, or nowadays it would probably be an agent to complete the work, they would switch to email or do a quick code review and then later report that this was actually a very good use of their time and not a costly context switch. But this is something we would have never learned from just telemetry data.
Brian Houck (38:24.142)
Yeah, they’re like maximizing the gaps between the core thing they’re trying to accomplish. So now AI is completely changing the way in which developers work and task switching, you know, this idea that we’re all going to be managers of agents and so we’re constantly switching bit between tasks. I’m curious, if you were to rerun this study today, what do you think would change around?
Andre (38:28.585)
Yes.
Brian Houck (38:52.44)
How like what makes a developer feel like this was a good day versus this was not a good day?
Andre (38:59.488)
So one of the core findings of that study was that developers said that on a good work day, they were able to create something of value, which means they were able to make progress and work on something that is meaningful. And I think this hasn’t changed. The other thing that we’ve also seen is that on good days they spend they are able to spend time efficiently.
So this means they have very few blockers and no infrastructure issues and few interruptions and things like that. Sadly, this also has not changed yet. And I could argue that AI in some cases even creates more interruptions. And then finally, they also said that on good days they spend more time coding and they have more time to actually write code, which is pretty cool. On good days, this was around 30 minutes of their time.
Yet it wasn’t really about the act of typing code or like really writing it down. It was again really about being able to build something that benefits somebody else at some point. And this is also why many of the developers we are talking to today with AI, they are still pretty happy, even though AI essentially takes away that part of their job, which is the writing of the code.
But as long as they remain in the driver’s seat and can choose what to build and design a software or design a solution, they are oftentimes still pretty happy about the creation part of their work.
Brian Houck (40:38.732)
Now I I do think that for many developers, having a sort of deep understanding of how the system works was also like a a a sense of joy. And they liked to get deep into the technical details. And you know, you you say that as we move to sort of higher levels of abstraction, you’re still dictating what is what it is we’re trying to accomplish, but maybe have less hands on saying the way in which it gets accomplished. Like, does that run the risk of sort of complicating this good day, bad day relationship? And and exacerbating what Peggy’s story calls sort of like intent and cognitive debt as well.
Andre (41:21.026)
I would say yes or no. So yes, there is this switch of from writing code to orchestrating agents more or supervising agents more and then reviewing code as well. but I think it really depends on whether the developer can decide what they delegate and what they still wanna keep close to their rest and whether they still understand the result in the end. And I think it’s a loss if people get or developers get.
Bigger and more frequent pull requests that they need to review and they have a harder and harder time to understand the code base that they don’t really understand at all because of intent and cognitive depth and so on. But it’s also a win when they remain able to set the intent and own the key decisions, and AI basically just removes the tedious part of writing code. And I think the reality is probably somewhere in the middle.
Brian Houck (42:15.318)
Mm-hmm. So we’ve talked quite a bit about collaboration so far. And you know, in in your Good Day paper, you sort of like look at that tension a little bit between collaboration can be incredibly helpful, particularly around things like planning and spec generation, but it’s also a source of interruptions. And I know a lot of your work has looked into sort of focus time and flow state and things of that nature. And so
In practical terms, like how should we be thinking about protecting our focus time without treating all collaboration as sort of interruption and toil?
Andre (42:57.728)
Yeah. Yeah, that’s a fantastic question. And the short answer is it is very individual how you can find a balance between the two essentially. but meetings and interruptions being the number one impediment to productive work is something we’ve been hearing in all the studies that we’ve run so far. but we have to remind ourselves that even on good days in our study, we’ve shown that people still spend about a third of their time collaborating. So collaborating is not necessarily bad, but it just depends on how much of it and when you have to do it. And then of course it depends on the quality. And developers talked quite a bit about meetings being bad in general, but when you then push them a bit to explain in more detail, they talk mostly about the timing of meetings.
So if they fragment their the their the workday, leaving very little time for actually focused coding, especially. And also when people are not prepared or when meetings are not moderated well and things like that. And then there are also specific project phases where it’s more common to actually have more collaboration, such as around kickoff of a project, or towards the end when you need to fine-tune certain things and so on. This is a nuance that most developers understand. So they don’t really just want to get rid of all the meetings, but they want to structure their workday a little bit more.
And I think it’s in the end really about, as you mentioned, protecting for focused work, where maybe for an hour or two you can just focus on your own things and sort of have your focus head-on. And then you switch to collaboration, maybe you have a daily stand-up, and then you have a couple of follow-up meetings, and then you can go into focus again, where it’s like not super split up and split out. And we’re seeing that very few.
Teams actually talk about when do we want to spend time for collaboration and when do we want to spend time for focus. And when you just when it just naturally happens, everything is fragmented and everything is scattered across the the week. And this is especially hard for individuals that are more introverted. When they have to switch between their collaboration head and focus head all the time and and
Brian Houck (45:15.938)
Mm-hmm.
Andre (45:22.076)
rarely have a few hours where they can just focus on their own work. Yes. Or like batching of emails and chat messages and so on. And I mean one thing that we hear every now and then is that companies roll out no meeting days, for example. But I don’t really think this works very well in practice because on most days you will still need to ask the occasional question or maybe want to do a quick sync. So I think what works much better is if teams say, okay, let’s for example start our day with focused work, then do a daily stand-up, then do a bit of collaboration, and then maybe in the afternoon we also, all of us again block some time for focused work, really like structuring collaboration time, focus time together and not on an individual basis.
Brian Houck (46:11.948)
Mm-hmm. And and consistently across days, not just like one given day day of the week.
Andre (46:17.352)
Yeah, then just if I then have a question, but I know that my team will be around later for a collaboration time, I might be willing to wait for half an hour or an hour to ask that question instead of just dropping by my colleague and and interrupting them all the time.
Brian Houck (46:40.472)
So sort of final question for you, Andre. If a team wanted to make more of their work days quote unquote good, you know, over the next month, what’s one change that you would recommend for them to try? And how would you tell whether it helped both the developers themselves, but also ultimately did it result in good outcomes for the customer?
Andre (47:10.198)
So my number one suggestion would be to optimize for focused versus collaboration work. But another thing I have been thinking of more recently is that peer review load is increasing a lot. And for developers, it’s a huge challenge to deal with this growing review load. And either they just rubber stamp reviews or they do it properly, which stresses them, increases cognitive load, and so on.
Brian Houck (47:26.786)
Mm-hmm.
Andre (47:40.243)
So, what I would do is I would gather with the team and agree on something like a PR etiquette for AI assisted work, basically saying, okay, you as an author, you own the change. So this means you actually still need to read and understand your AI-generated change before asking anyone else to spend time on reviewing your potential AI slop. And then the author might also need to rate something like the risk level or like a confidence score or something because if it’s a low risk change or like high confidence in that change, we can maybe just trust our tests. So we may not even need to ask someone else to or like another human to review it and we can just rely on an AI review. But for the high risk ones or the low confidence changes, we need to ask someone else to to review, even not just for finding bugs, but also to spread knowledge and to ensure that things such as committive debt are not growing, right? So for those we should probably I would recommend the team to then think about or agree on what is a size of a PR that is still reviewable and reasonable. And also what is a good review turnaround time.
Brian Houck (49:00.43)
So if I understand, it’s like a mix of both having the time to do deep focused work and then having clearly agreed to sort of accountabilities and boundaries on where everyone’s responsibilities lie on that. And you think like doing those two things will lead to to better days?
Andre (49:22.28)
Absolutely. Yeah. I think if we can reduce the number one thing that we still keep hearing in the age of AI, which are meetings and too many interruptions, that’s an actionable and simple thing to to work on and and explore. And the other thing really is the growing cognitive load that developers keep talking about and the growing burnout and stress that they need to tackle. And I think managing this a bit better or finding ways, very specific, team specific ways to to deal with that better, is something that can have short term positive impacts already.
Brian Houck (50:02.648)
Awesome, I think that’s a a great place to leave it. I’d like to remind the audience, too many DevX metrics, too little guidance is the paper out now. ACMQ could not encourage people to read it anymore. It is phenomenal. Andre, thank you so much for joining us.
Andre (50:21.75)
It was my pleasure. Thanks so much for the good discussion, Brian.
Brian Houck (50:25.806)
Cheers.
