Brian Houck (00:01.69)
Welcome to the Engineering Enablement Podcast. I’m your host, Brian Houck, and I am so excited for today’s guest. We have joining us the Max Kanat-Alexander. Max is Executive Distinguished Engineer at Capital One, where he focuses on improving developer productivity for one of the largest engineering organizations in the industry. Over his career that spans Capital One, Google, LinkedIn, and more.
Max has worn a lot of different hats. He’s been an engineer, an architect, an author, an open source contributor, a researcher, and an executive leader. And it’s these different hats that give him this incredibly diverse and rich perspective that I’m really excited to dive into. On a very personal note, I will say that if I’m ever at an event, that Max is also at. He is the first person I make a beeline towards to talk to because he always has something interesting to say. And it’s for that reason I’m so thrilled to have him on the podcast today. Max, thank you so much for joining us.
Max Kanat-Alexander (01:06.348)
Wow, thank you so much, Brian.
Brian Houck (01:09.178)
So just to to kick us off, could you share a little bit of context about your role at Capital One and what your team focuses on?
Max Kanat-Alexander (01:18.232)
Sure. And I also want to say that, Brian, you’re also the first person that I look for at an event and find a beeline for and have interesting conversations with. So we are in good company with each other, I think. Yeah, so I help lead a part of developer experience at Capital One. I am a very senior engineering IC at the company. I am the technical lead for AI-assisted software development, for programming languages, libraries, and frameworks, and also for our internal system of quantitative and qualitative metrics that we use to understand and improve developer experience and to help teams improve their own developer experience through data.
Brian Houck (01:59.365)
Yeah, like we could have an entire episode just diving into metrics. But I will resist that temptation for now. We’ll see where the the conversation takes us.
So to dive in on the AI side of things, is like every time I blink, the world is different. And everything is moving so fast. And it’s hard enough for me as a researcher to keep up with, but for someone who is tasked with building developer platforms, developer tools, it is even more challenging. And I’m curious, given how fast everything is moving, what
Can engineering organizations be doing today that will still be relevant, still be paying dividends three years from now?
Max Kanat-Alexander (02:41.59)
So one of the things that’s wild about the current moment is that both everything has changed and nothing has changed. So there is a huge transformation going on with AI, with how software development works in the world. And yet the core fundamentals of the software development lifecycle fundamentally haven’t changed at all. And what you do is you have to base your outlook.
In the fundamental principles of software development. If you go back, and we’ve known those for for decades, you know, we do studies over and over and over and they they show us many of the same things about about flow and cognitive load and cycle time, you know, and all all those different things. And
There’s different opportunities to optimize those things nowadays than there used to be, especially on the left side of the software development lifecycle, the point where you’re really coding actively. Because we have so much more insight into that thanks to coding agents. And also, that has been so transformed, the most transformed, probably of all parts of the software development lifecycle today, due to coding agents. You still have to care a lot about the fundamental tooling.
And the fundamental ability to get work done in your development lifecycle. So, just as an example, if your company’s workflows involve frontline engineers spending a lot of time doing things that aren’t coding, no amount of AI is helping them. If your CI pipelines don’t work, the AI is not helping them.
If there is a vast and complex human review process before they can even start to do work, your AI isn’t helping them.
Brian Houck (04:30.082)
And in fact, in some cases, maybe making it worse, I assume. I think like flaky tests, like we always should have cared about flaky tests. But now AI, since it’s making execution so much faster and easier, it’s like if I’m buried under sort of sub-optimal signal about my validation, it’s just like AI is actually exacerbating that.
Max Kanat-Alexander (04:52.502)
A hundred percent. The number one thing that I also agree with that you’ll hear every person who’s an expert in this field say is that AI is an amplifier. And it will amplify everything bad about your software development lifecycle and it will amplify everything good about your software development lifecycle. And the people who are getting the most gain out of it are people who already had a solid software development lifecycle inside of their company. And if you have problems in the software development lifecycle, every one of them will be dramatically magnified by AI.
Brian Houck (05:17.678)
So I love that. It’s you know, the things that we should be investing in now or the things that we always should have been investing in. And it’s just like the fundamentals still matter. And it’s you know, you said something that was that’s really interesting to me, and I keep coming back to it 'cause it’s a theme in my own work. It’s like developers only spend about 14% of their day writing code. You know, this is even before AI. And so it’s all of that other stuff still needs to to to be optimized for and will still be relevant as we move forward. But should acknowledge the world is changing. And you know, for that 14%, it’s get it’s getting evolved dramatically. And as AI gets more, capable, we’ll see, you know, it it touching on different parts of the developer workflow. And so, like, what skills do you think engineers need to be improving today to help make sure they’re still relevant in three years and five years?
Max Kanat-Alexander (06:16.12)
Such a good question. I think the balance of skills required to be a software engineer is significantly changing. and I I don’t think there’s any skill that is eliminated. I just think it’s the importance of different sets of skills is changing. You know, the first time that I used a coding agent,
I was like, wow, this is great for me. And I’m really, really worried about people with less experience understanding what to do here. Because the reason it was great for me was as a person who becomes more senior as an individual contributor, your job becomes how do I express in English clearly my desires to other people or my intentions or designs or and how do I provide feedback to other people in English?
And so all of a sudden I had a tool that was like, I just do that and it writes code for me. So for me, that was like my daily bread and butter. But that’s not a skill that anybody learns in school. And it’s a skill that usually takes many years to gain as an engineer. Like the understanding the clarity of how to express requirements and how to provide feedback and how to sort of like manage an intern. Like that, we can argue about at skill levels, but I think the conceptual model you can think of is that it’s an intern, right? Like it might be an intern that’s very skilled. but or it might be a junior or senior engineer you know, people have different different mental models for that, but that that skill set is really changing. On the other hand, the skill set of like I am a deep expert in algorithms or data structures, or like I am a deep expert in programming language X.
Still has value because you need to be able to understand what you’re shipping and at a certain level you will still be better today than the agent. But those sorts of things that are purely based in effectively things that could be reproduced as rote knowledge. Like things that could be reproduced as like if you had read all the docs you would know all the things. the AI will eventually get better than you at.
That skill set is really helpful as I think an enhancement today to your ability to be an engineer, but it’s becoming less and less relevant to have that deep, deep expertise in like fiddly bits, I would say.
Compared to like, how do I understand the domain that I work in? How do I define problems in the space that I work in? How do I clearly express my results? And also, still really relevant, how do I think about software design conceptually? Let me give a concrete example from today. I was working with a coding agent and it had done a wrong design, but it had but it had done it, it had output like 700 lines of code for that wrong design. And I only figured it out because I saw something that looked off, and then I asked it about that one thing that looked off, and it was like, yes, you’re right. My whole design is wrong. Great. I don’t have to read the other 700 lines. So like figuring out that skill of: What’s the little thread that I need to pull on and not have to go through that whole like, I now I just read a thousand lines of code that I didn’t need to read because it turns out the whole thing was wrong.
Brian Houck (09:57.233)
There’s a lot of really interesting stuff there to unpack. And you know, on on one hand, a lot of what you’re talking about sort of implies that the lines between roles could in fact blur. It’s like we no longer need some of that deep domain technical expertise. It really is about how do we decide what the right things to build are. But at the very end there, like you talk about effectively failure patterns and being able to recognize failure patterns. And this is like one of the things that I’m a little concerned about is, you know, it’s more senior engineers have seen more failure patterns. And so if the AI spits out 700 lines of code, you’d be like, wait a second, something here doesn’t look right. But it says we no longer have necessarily the same deep technical expertise. Do you worry that we’re gonna be less attuned to being able to detect those small failure patterns?
Max Kanat-Alexander (10:52.49)
Yes. And in fact, a lot of the time it is having gotten that deep technical expertise that walks you up to the level of being able to spot those failure patterns. I am much more able to spot those failure patterns in languages that I am much more familiar with. So my point is that your entry-level skills, like the skills that you need to enter with as a software engineer, have changed.
Max Kanat-Alexander (11:17.902)
but the way that but you can you can still up level as a software engineer by continuing to get deeper and deeper technical knowledge. And and that’s always been true to a large degree. Like you don’t need to understand how the compiler works in order to write a program. You don’t need to understand how the CPU works in order to be a programmer today. But understanding that makes you a better programmer. It it really does help you. Like, and when you go through that process and you learn it, it makes you a better programmer. You don’t need to know more than one programming language to write a computer program. But knowing more than one language makes you a dramatically better computer programmer. And so I think that all those things are still true. It’s just maybe the sequence and the importance of each thing as far as like immediate ability to produce results and also to produce like reasonably good quality results.
The order in which I would sequence those things is maybe a little different today.
Brian Houck (12:12.132)
I totally agree and you know, showing my age a little bit, but it’s like, I’m thankful that I learned how to code in C, you know, like even before C, because it does change the way that I actually think about how software works. and you know, like even just like having to manage my own memories. Like that’s a crazy idea nowadays. So sort of moving moving on a little bit, but but related, it’s lots of discussion about
You know, the new code review bottleneck. Now I will posit that as an industry we’ve always sort of sucked at code reviews. Like we’ve never been great at following well-known code review best practices. And to your earlier point, AI is an amplifier. Now we just have this big, spotlight being shown on it. But I am curious as we think about the bottlenecks around code review.
Lots of people are trying to address it in in different ways. Things like, we’ll just have AI automate all of our code reviews. I’m just curious, like what you’ve seen and how you think about the code review bottleneck.
Max Kanat-Alexander (13:17.186)
Wow, this is a great question. So for context for your listeners, I am also the author of Google’s code review guidelines, at least the original author of the content that is currently Google’s code review guidelines. I mean with many other contributors, but I was the primary author and maintainer. And so I have have lots and lots of thoughts about code review, and I have in my life done over fifteen thousand code reviews of real real PRs, like not like rubber stamps, like actual code reviews where like I left comments, there were I have done 15,000 of those. Here’s the problem that we face. There is some quality bar that we do not know as an industry how to hold without human review. I believe this is still true.
I’m not arguing that fact. There is an aspect of software quality that we do not know how to hold without human review, because there is a core part of software design that involves subjectivity. It involves the ability to understand the system. So, like when the definition of simple for code is easy to read, understand, and correctly modify.
Those first two are entirely subjective human factors, easy to read and understand. And you can argue, like, well, we don’t need to read the code because the AI will read the code, but I will tell you, the AI has the exact same problem that humans do with reading the code. If you name things wrong, the AI can get very confused and do the wrong thing. It is much harder for the AI to modify a poorly structured code base, and it is much more likely for it to do the wrong thing in that poorly structured code base. And it’s even more dangerous in some ways, because with the amount of code that AI can generate and with how convincing it can be about its own correctness, when it tells you that it did the right thing, it is tempting to believe that the code is right when it has built on top of a poor foundation and thus is wrong. Now
That’s the real reason we do code review, by the way. It’s not to catch bugs. And anybody who thinks, well, I’m gonna have human reviewers catch bugs, I cannot tell you the number of studies in the world that have shown, even before AI, how bad human reviewers are at catching bugs. Like you should always have had tests doing that or static analysis doing that.
Some sort of automation should always have been finding whether or not there were defects in your system. Not to mention because that’s the only way to do that at scale. Like other people are gonna build on this. If you have hundreds of engineers building on the system, you can’t rely on every human having understood all the possible consequences and interactions of your system.
So then what’s left is really this maintainability bar and this the like I and this and this soft this or this bar of does the design match the intent, including the architect’s future intent about the system, sometimes which is not just unwritten, but hard to write. Like you’re in like if you’re the architect of a system, very often you’re quite busy writing code. And it’s not like you have all the time in the world to be like, I need to write down every single future idea that I have about the potential direction of this thing. Because you have to make a trade-off of your own time of like how much time am I gonna actually go and review PRs? How much time am I gonna go actually write code? And how much of the time am I gonna like sit here and write out every single thought that comes into my mind about exactly how I want the system to continue to be designed? And then two.
As systems become more and more complex, and all systems that serve humans well will eventually grow to a certain degree of complexity. As those systems grow to that greater level of complexity, it can be harder and harder to be sure that the human intent, like the product intent, matches the code intent of the entire system. And it’s very hard to accomplish that today without some form of human guardrails. So this leaves us with a few interesting questions though.
And the first question is, does every code base need that discipline?
What if your code base had very few consequences for getting it wrong, could be rewritten in a day, and there were no there was there weren’t other parts of the business or a lot of people that depended on it to be to keep behaving the same way. Like there wasn’t a lot of change management involved in it.
Brian Houck (18:07.62)
Yeah. What is the risk? Like what is the risk of things going wrong here?
Max Kanat-Alexander (18:12.406)
Of things going wrong with this code base, yeah. And and and what is the cost of replacement? What is the cost of fixing the risk, right? If that number is close to zero, you probably don’t need code reviews at all. And and I think very often companies fail to differentiate their code bases in this way and people sometimes fail to differentiate quality. You know, even see my in my first book, Code Simplicity, there’s a rule about quality. And I I don’t remember the exact way that I phrased it, but it’s the rule is basically the quality required of the system is proportional to the length of time and the amount of people that it will help in the future.
Brian Houck (18:54.564)
That’s interesting. What is the impact of this to your end user, right?
Max Kanat-Alexander (19:00.918)
Yeah, and also it’s about sustainability. Like quality is fundamentally about sustainability over time. And if the sustainability impact of your code base over time is close to nil, you do not need quality. Like I do not do a great job of software design in scripts that I’m going to throw away in five minutes. Right?
Brian Houck (19:33.168)
But if you’re running enterprise software that like is on a 10 year refresh cycle, then it’s like, well, no, like by definition, what quality means there is very different.
Max Kanat-Alexander (19:43.439)
Totally. So there’s this spectrum. And I think people often they just want to paint everything with a broad brush because they don’t want to like involve judgment in it. But I feel really feel we’re at a place in an industry where we need to start involving judgment. And and so this judgment comes, I think, in two other places as well. one of them is, and this is probably one of my most controversial takes, and I haven’t tested it out, so I really want to have somebody test it out and tell me what happens. I don’t think every person needs code review.
Brian Houck (20:15.138)
As in not every person needs their code reviewed.
Max Kanat-Alexander (20:17.794)
Not every person needs their code reviewed in every circumstance.
Brian Houck (20:20.654)
Okay, because they like they’ve achieved a level of trust?
Max Kanat-Alexander (20:25.01)
They have achieved they have achieved a level of trust and ownership and responsibility that it would be better to let them fix their thing afterwards when they realize that they did the wrong design and unblock them to do that and you will actually have a better quality code base.
Brian Houck (20:43.115)
So I have now my somewhat controversial take, and I’d like to to hear your expert opinion of this, is we are in incredibly strong agreement that code review is about so much more than than defect detection, right? Code review has never been the right way to detect defects at at scale.
Now, where I go a little bit further is I think code reviews are an incredible sort of culture team building activity where it’s in addition to looking at, you know, evaluating the maintainability of the code, it’s a way for junior engineers to see more senior engineers’ ways of doing things. So it’s a way of transferring knowledge. It’s a way of learning what my peers are working on, which is incredibly important. And so when I hear, like, well, there are some engineers who have proven themselves so adept, or their code is so easily throwawayable. Like in either case, they’ve they’ve proven that they don’t need review. Sort of excusing them from the review process deprives everyone else from what they learn by reviewing their code. And like I’m curious, like how you think about that, if that makes even any sense.
Max Kanat-Alexander (21:55.179)
It is a great question and it is the question that people have asked me when I have brought this up before. The role of code review as mentorship is extremely important. I have never in my life made anybody a better software engineer in any way other than reviewing their code.
There is no amount of talking to people that I have done that has ever made them better at the core fundamentals of software engineering. I can help people with their careers, I can help people with their communication skills, I can help people with their problem discovery, I can help them with all kinds of other things. No other way that I have ever found to improve people’s core software engineering skills than doing code reviews.
Brian Houck (22:33.168)
But is the reverse true? Do they get better by reviewing your code as well?
Brian Houck (22:39.75)
'Cause you get to see the way like firsthand the way you are approaching all the problems.
Max Kanat-Alexander (22:44.472)
People claim this to me, I do not believe that it is worth the time. I don’t believe that is it is the right trade-off for a business to do that. I I believe that they will still get the same value by so like and the kind of trust that I’m talking about is like
Max Kanat-Alexander (23:03.028)
I’m the architect of this code base. Like, I might even be its original author. I have proven myself to the some my more senior ICs than me that I can be trusted with software design in general. That like I care deeply about software design, I care deeply about the developer experience, the people working on my code base. I I care about engineering excellence and and engineering rigor in all aspects. I care about observability, every single thing. I care about all those things. And I agree to accept responsibility and accountability for the production consequences of my changes. and I can be trusted. And I will review the code of other people who have not achieved that trust.
Max Kanat-Alexander (23:46.743)
Or that been granted that power on this code base. And to be clear, I think it is a pair of person and code base. I don’t think like I should not be trusted. Like I am a very experienced software engineer, but there are many code bases in the world that I should not be trusted to commit to without review because I do not understand the architectural intent of those codebases. I mean, I write op I write PRs into open source projects that get rejected because people are like, you found the right problem, but that’s not the right solution. And I’m like, great, I’m so glad you reviewed my code. I learned something.
Brian Houck (24:19.024)
Yeah. Yeah, and to your point sort of about like the very definition of quality, is you know, regardless of how skilled you are as a developer, if you’re making a change into the Windows kernel, it’s like, well, that should probably be reviewed because of the specific code base that it is going into and the ramifications of getting it wrong. Tim Bozarth at Microsoft says like, this isn’t a code review problem, it’s a confidence problem. And I think your answer really speaks to that. Is like, where do we have confidence and like where do we need a little more confidence?
Max Kanat-Alexander (24:53.272)
Totally. And and the the one other comment that I want to make about about your statement about do I do we learn things from reviewing others’ code? Maybe we do, but I’ve also been in this really weird situation where I because of the like the requirement for to review everybody’s code, where like there’s only two people on a team. One of them is actually the sole and only author of the thing. And I’m reviewing their code, but the value of my feedback is probably not sufficient to justify the time barrier that I’m putting on the organization. And yet I still have to like I felt really awkward. I can think of a specific situation that I was in at Google where I was reviewing this code base. There is, and and the thing that I don’t have an answer to, and the one of the reasons why like I want people to experiment with this, there is a bus factor argument to be had, which is that like at least that sort of understood that code base. So if that other engineer vanished, then I would be able to take it over. But you know what’s weird? When you are only the code reviewer of a thing and you don’t work on it, you don’t actually understand it. Like I like there’s a claim that I could have taken over that code base.
I don’t know that I would have been dramatically more qualified to take over that code base than any other person coming out of the woodwork and looking at it, because I’m only reviewing mostly individual changes in isolation, and I’m not the person with the architectural thought. Like I would kind of have to have ramped up on the whole thing myself anyway, even though I had reviewed every single change.
Brian Houck (26:14.66)
Yeah, and like even as I am saying like my own point, I also think like I I don’t know that there’s any magic in reviewing a PR, like being able to block a PR from completing so that I can review it versus me just going and reading it once it’s already been merged. Right? Like if if I can’t meaningfully contribute to the review, then like I could read it at any point in time and probably still get the same benefit.
Max Kanat-Alexander (26:40.13)
And there was one last thing that I wanted to say about code review, which is that I think not all PRs need review. So third there’s this third access. And so like at you know at Meta, for example, they have like a risk scoring system that they can run on PRs. And I Yeah. Yeah, the diff risk score. And I think that like if you could come up with a system where you’re like I feel super confident about this PR, especially on this low-risk code base or like on this maybe medium risk code base. I think this doesn’t need human review. I have a logical argument for why this category of thing doesn’t need human review. Let’s skip it.
Brian Houck (27:18.532)
Yeah, especially if you could make it sort of a deterministic score as opposed to having like an LLM. They’re like, what do I think? But like yes, I I do again, what is our level of confidence? And some of that is based on, as you earli you mentioned earlier, it’s like what’s the ramifications of getting it wrong? And sort of to sort of like move on a little bit, but in a similar vein, I’ve heard you talk about a potential vicious cycle that engineering teams can get in. And I’m just curious if you could explain a little bit about what that even is and what can be done to prevent falling into a vicious cycle.
Max Kanat-Alexander (27:57.017)
Great. So the vicious cycle of AI-driven software development is this. You start off with a code base that’s very hard for the AI to work with. It has poor structure, it has unclear tools, it does non-standard things. The AI can’t run tests or it has very few tests. The AI doesn’t really understand how to run it, even to manually test it, things like this. And you start putting through changes with an AI agent.
You then have a weak code review process, maybe just a very rubber stampy code review process or no code review process at all. and what happens is the AI writes bad code, sometimes even code that doesn’t work. I know it might look good, it might be convincing. It’s very hard for anybody, just even conceptually. Conceptually, it’s hard for any possible thing to write good things on top of bad code.
And so it gets worse. And then the next time the AI has a harder time and it slows down. And the biggest danger is, and occasionally you hear these like crazy stories where people are like, I got called in to take over like a three million line code base and I deleted two million lines, because it was all like slop. It was like all completely unnecessary slop that had nothing to do with producing the result, and everybody was just confused about why it was failing. I mean, I think the nightmare scenario is you know, you’re in some situation that’s like an existential threat to your business because of a code base you don’t understand and now can’t maintain. And that’s like the that’s like the very, very bottom of the vicious cycle. and then I think the the second half of your question is w how do we not do that?
Brian Houck (29:38.372)
Yeah, what what do we do?
Max Kanat-Alexander (29:41.099)
Yeah. So the the thing we need to do is once again the same thing we always need to do. We need to care about the developer experience and the quality of our code bases. The great news is the AI can help you. You just have to direct it in that direction. If you don’t have tests, have it write tests. But if your code base is untestable, there’s gonna be like a little back and forth work you’re gonna have to do. You have it write tests, and then if it writes tests that are entirely like I assert that the code is implemented the way it’s implemented, like I mock out the code and then I assert that the mock returns the thing that the mock returns, like your code base is untestable. Like the agent will attempt to follow all of your instructions, which is also a whole fun other topic.
Even if your instructions are impossible. And so it will write quote unquote tests. And if you see tests that are like so you have to have some judgment. There so if there is no experience and no understanding of testing, and no experience and no understanding of code quality, the first thing you need to do is you need to start maybe having a conversation with your agent about what are good tests. It will tell you it knows. Like, like what what are good testing practices? how would I be able to spot a bad test?
you know, go ask any AI. Any AI made in the last two, three years will be able to answer all these questions perfectly well. And you know, educate yourself. So the first step is educate yourself if you if you aren’t familiar with these concepts. Make it maybe even make it do some drills with you, make it show you some bad things, and then have it explain to you why they’re bad. and then you can so then you have it generate tests.
And then be like, okay, well, obviously the code base is untestable. So then I have to fix the problems with the code base that are untestable. So let’s refactor the thing to make it testable. You could very often you can literally just say, please refactor this to make it testable. Nowadays with the with the most modern models, and it’ll do fine. it might not do it on its own, it won’t do it on its own though, because it’s gonna try to narrowly follow your instructions sort of within the framework that you have.
Max Kanat-Alexander (31:47.073)
And you can work your way up through this. And then it’s like, I don’t know how to run your tests. And I’m like, maybe I haven’t documented that. Maybe it’s not in the agents.md file. maybe I’m using some weird, super weird, non-standard way. Maybe I should move to the standard way so that the agent has stops having trouble running the tests. and you can
Work your way up with this with the agent. The agent can help you. It’s it’s really actually the literally the thing that the agent is the best at is refactoring. Of all tasks in the world, the agent is unbelievable at refactoring. I speak to that as a person who’s done more refactoring than most people in the world. Like it it has it is off the charts at refactoring as long as you point it in that direction. And then once you get there, you can see, hey, this is good. And the only caution that I’ll give people is that you should always try to do this in service of making forward progress on your product goals. You should not just be like, now I’m gonna spend two months refactoring this code base for the sake of refactoring this code base. You should be like, I have a goal, and I’m gonna need to write a feature. And this feature is gonna touch this part of the code base. Okay, what do I need to do beforehand in order to be sure that both humans and agents can do good work here? Okay, I need to refactor, I need to write tests. So we’re gonna do that first. We’re gonna do that as if and if you’re in an environment where there’s PR reviews, you’re do it as separate PRs first. if you’re like, I need to fix something about the developer experience of this of this code base, I need to add some pre-commit hooks, I need to set up the linters, I I need to improve how the test runner functions, you should do that upfront.
Like that should be your first step. You should send that off as separate PRs. And then you should refactor just the parts of the code base that are relevant to your feature work, not go off and like on a giant boondoggle of I’m gonna go fix the whole world. Because also all quality has a purpose. And quality is only relevant to accomplishing that purpose. And if you don’t do like all of your refactorings and all of your developer experience work in the direction of a concrete objective, you will do it wrong.
Brian Houck (33:56.081)
The laws of quality relativity, right? It’s all like it’s all about relative to the task or or impact you you’re going for.
So you said a lot there that that I thought was incredibly interesting, Max. And it echoes a research project that I’m working on right now where we’re looking at what are all of the dimensions of of code quality, like the fundamental dimensions. And we’ve talked about a lot of them. Is it maintainable? Is it readable? And one of them is is it testable? Like what is the testability of this code? And one of the things we found is there is no relationship and maybe even a slightly negative relationship between the testability of your code and the frequency of your AI use. And so you’re not just like using AI doesn’t suddenly magically make your code more testable, but it’s, you know, if people follow your advice, use AI to teach you how to make your your code more testable. I would love to see like, well does that shift the paradigm?
Max Kanat-Alexander (34:53.632)
Yes. Yes. And I mean I can tell you that in my own code bases, it absolutely does. And and and in fact, that quality, like there’s many aspects of quality. So I I have long believed that actually code structure and software design is actually even more important than testing. But like that’s not a place that the industry is necessarily at for the most part. The industry has sort of realized mostly that testing is important.
Brian Houck (36:09.2)
Like so we’ve talked a lot about I mean like first principles. And it’s again, as AI is an amplifier, it makes first principles more important. But you know, do you have good code review practices? Do you have good testing practices? Have you thought about all of the parts of your workflow and like having efficient, smooth CIA, you know, CD systems, things of that nature? And I just think like as teams are rushing to adopt new AI tools, new AI workflows, like AI adoption.
Pretty sort of ubiquitous across the industry, but there are definitely levels of sophistication. It’s like, all right, well, we’re using coding assistants as sort of like a pair programmer versus like, we have multi-agentic workflows. And I’m just thinking, as we think about all of these foundational things that need to be in place that should have always been in place, but now it is more important as we go further into the AI maturity curve. It’s like, are there specific signals that leaders should be looking at to evaluate like are we even really ready for sort of the most advanced AI world?
An interesting thing about fixing software development lifecycles is that it’s different than if you built them the right way in the first place. It is much, much harder to fix them than it is to build them the right way in the first place. Like, I have sometimes said that if you gave me 500 people in three years that I could reproduce Google’s development environment, but that I could not do that in almost any company in the world today because it would take me three times as much resources and time to fix.
There’s a fact about this that’s always been true. And when you come into an organization that has bad software development lifecycle practices, you start at the right and you move left.
Brian Houck (37:53.552)
Yeah.
Max Kanat-Alexander (38:16.652)
When you are starting a new project that you want to have good software development lifecycles, you start on the left and you move right. So like, you know, like you like with a new project, you like, okay, do I have a great linter? Do I have testing, have I thought through my software design process? Have I do I have you know, like how what’s my code review system like, right? When you c if you come in and you’re in a and you’re in a s an environment that has nothing like that, you start off like, okay, how do I make production stop falling over all the time?
how do I how do I make incidents better? Like like usually that’s like one of the like when I I see this all the time with senior engineers who come into companies that don’t have a history of engineering rigor. They go, okay, the first thing I’m gonna do is I need to fix the incident management process because that’s the thing that’s taking up the engineer’s time, that’s the thing that’s affecting the company the most. And after that we need to fix observability and monitoring, and you can like work your way back, and then we need to fix like the rollout process. Okay, and then we need to fix like the checks that happen right before deployment. Like what are all the checks that happen right at the end of the right right before deployment pipeline?
As you can see you’re just moving further and further and further left, and you’re moving rigor further and further and further left. Your readiness signal is how far left rigor is. and that’s also gonna determine, I think, the benefit that you get.
Max Kanat-Alexander (39:39.637)
From AI tools. I think that is probably the number one signal of whether or not you are going to get the benefit, is how far left rigor is. So, at the very least, if you do not have controls that prevent dangerous things from getting out into production, you will probably have a bad time with AI adoption.
Because if you’re at that point, you probably also don’t have an effective code review process. You probably also probably don’t also don’t have great testing practices. And that’s an organization in which I would be hesitant to do any significant roll out of AI at all. And where I might be like, we’re gonna constrain AI usage to only our most senior and experienced developers purely because of the risk to our environment from us basically giving people a tool that allows them to move at great velocity with low quality.
Brian Houck (40:33.008)
This is like something I I’m gonna be thinking about this for the rest of the day. It’s like this is like I I love that answer. It’s just like how far left have you been able to to sort of like solve your big pain points? And that’s like a signal of your your, you know, sort of rigor maturity, I guess.
Max Kanat-Alexander (40:51.446)
Yeah. And to be clear, I I am very bullish on AI and I I really want people to adopt AI, but I also think that people need to look at the business that they have and not headlines.
In some cases, the AI will produce better code than people currently produce. And so you’re you’re gonna have a good time with that aspect of it, but I have yet to see a single study show that when you give people with a bad software development lifecycle AI that quality improves, I have seen only the opposite.
Brian Houck (41:31.143)
Yeah, I mean, even in my own work, it’s like I am not seeing that AI dramatically improves code quality. In fact, it might make it slightly worse. Cause it’s if you’re building on sort of a shaky foundation. Even when I look at things like across the industry, how is you know change failure rate moving relative to AI usage? It’s like, well, it doesn’t necessarily seem to be getting a lot better. And that’s not to say to your point, like I am also an AI optimist. Like I use AI
very thoroughly every day. I can’t imagine having that taken away from me at this point in time. But that doesn’t stop this hype cycle where it’s like I also recognize we’re not suddenly, you know, 2x, much less 10x more productive as a result of it. Because it’s about so much more than just just the writing code, the typing those letters. so that like leads me to to a
Another question, you know, as as a leading practitioner, what is it that you wished that researchers like myself would be spending more time studying right now?
Max Kanat-Alexander (42:41.122)
The biggest open question that I think we have in the world right now today is how are we gonna make new senior engineers?
Max Kanat-Alexander (42:50.792)
I I do not know the answer to that problem. Like and and I and I think in a way, like what are successful practices that cause junior engineers to become senior engineers is not a very deeply or well studied field, actually. It has always been sort of like a magical thing that just occurs and we’ve just been like, Well, I guess some people make it and some people don’t.
and I and you know, like, well, I guess we do mentorship and apprenticeship and somehow that happens. And even within companies, it’s invisible. Like, even like, you know, when when you go through every company, when you go through like the performance management process, you know, people always talk about like, well, are there our senior engineers mentoring or more junior engineers? Let’s talk about mentorship. Somebody we somebody always wants to talk about mentorship because it is so critical and important, but like it’s also fuzzy. So it’s like, what do we do?
Brian Houck (43:38.961)
Yep. It’s like every panel I am on, I am asked this question and it’s like to your point, like no one has a great answer right now. Where it’s like, well, if we everyone recognizes in you know, in 10 years, we’re still gonna need software engineers and we’re gonna need senior ones. Again, like we talked about recognizing failure patterns, like that comes from experience. And it’s like, well, if you want senior engineers in 10 years, guess what has to happen? You have to hire junior engineers today. Yet we’ve seen that, like, you know, that’s something that a lot of organizations are reluctant to do. Love it. I I agree, there’s like sort of organizational dynamics research that needs to be done in this space. please.
Max Kanat-Alexander (44:19.574)
I believe in the world there is a future where many, many more people are software engineers at some level than ever in the past. I believe that in the world there are a number of citizen developers that dwarf the number of professional software engineers now. I do not believe, as some people have claimed, that professional software engineering is vanishing from the world. in fact, I believe the opposite. I believe that
the amount of engineering expertise required is proportional to the rate of change occurring and that AI enables the world around you to change at a rate never before possible. And as a result, you will need more engineers with more expertise in order to cope with that rate of change that occurs with the environment around you and in order to keep up in the business world as well. But a lot of that change is going to occur because we will give hundreds of millions of people the ability to write software. How?
Do we understand and improve their experience?
Brian Houck (45:28.57)
That is a phenomenal like we could have an entire discussion on this thread. Where it’s like, I love the take that we’re not gonna have less software engineers, we’re gonna have more. It’s not even that the lines between roles are blurring, no, we’re actually expanding the lines of what it means to be a software engineer. And if all of our dev tools are sort of built around this notion of professionally trained developers is like, well, what is the developer experience for the more hobbyist builder? Or I don’t even want to say hobbyist, but like the builder who doesn’t come from a software engineering background. That’s that’s another great area of research.
Max, with that, I think we are at time. I cannot thank you enough. This was such a fun conversation. And again, like thank you so much for for joining us today.
Max Kanat-Alexander (46:21.77)
Absolutely. Completely my pleasure.
Brian Houck (46:25.392)
Thank you everyone. Cheers.