One Medical's customer-driven approach to improving DevEx

Abi: Jason, thanks so much for coming on the show. Really excited to chat with you.

Jason: Very excited to be here. Thanks for having me.

Abi: Well, I want to start by asking you about the renaming of your team from Engineering Efficiency to Engineering Experience. Tell us how this happened.

Jason: So I’ll back up a little bit. I joined this team and I joined One Medical last October, and so I had seen a posting for an engineering efficiency team. Sounded really cool. I’d been doing platform work before that, but hadn’t really gotten a chance to dive deeper into what felt like a lot more developer experience type work. So I was really excited about the team, was excited that it had obviously already been created, so I wouldn’t necessarily have to come in and advocate for the creation of a team like this. So I came in, the team had been formed maybe four or five months before, but had only gotten a PM attached in the last month or two. And it was cool that I came in right as the team was already starting a chartering exercise. So who are we, what do we do, what do we want to do, things like that. So that was really cool. Got to come in and see some of the things that had already been put together, but also help shape a few things as well.

And so, obviously, they had named the team Engineering Efficiency, I think, wanting to focus on efficiencies. But I think as we were going through the chartering exercise and looking at the way that a couple other teams at One Medical operate, which is to focus on the end experience of a person. So One Medical sees patients in clinics, and so a lot of our teams are focused on that patient experience, that visit experience, even the clinician experience because we’re writing software for them too. And so as it came out more and more, we thought, well, efficiency is a great thing to shoot for, but I think what we really care about is the experience, the development experience, of these engineers around us. And so decided to make the case for let’s shift efficiency to experience to better reflect. Naming’s always hard, but we felt like the experience reflected that better.

Abi: It’s interesting, so many developer experience teams that I meet with seem to have recently gone through or are going through this charter exercise. It partly seems like it’s because this is just still such undefined new territory. But I mean, what was your experience like joining the team right about when it was going through that exercise? What’s your advice for other folks going through that process?

Jason: It was really interesting because obviously the team had been formed out of folks that had either been on infrastructure teams previously or maybe even some product teams or even front end platform teams. And so part of it was just pulling what the folks on the team had previously been doing and wanted to do or felt strongly about doing, and trying to put that into a more cohesive unit. I think what I’ve noticed teams like this can struggle with is being an everything team because it is like, oh, well, it’s the development experience. That’s the whole reason why we’re here. So it could be anything and everything. So finding a way to both be pretty broad about like, hey, we’re here to drive simplicity or efficient experiences while also adding enough clarity to say, okay, we’re not focused on that. We’re not focused on that yet also.

Abi: I think it’ll be helpful to listeners to just give some definitions here. For folks who are currently coming from or working on an infra or platform team, in your view, what’s the difference in terms of focus, scope, and mindset between a more typical platform team and a developer experience team?

Jason: Yeah, definitely. I think a lot of what we, in that chartering exercise, came up with was really where we sit in the org. And I think even visually with application or domain teams on the top and infrastructure teams, what we actually call foundation teams at One Medical I think is also a great term for the bottom of that, and then we sit in the middle. So it’s not just about getting into, say, the AWS console and doing things there, but it’s also not about only being in the application and doing things there, but really finding that middle ground of how can we make development easier and pave some roads, make some easier paths for people there, letting traditional teams like infrastructure focus more on the, okay, we’re actually going to really make this a efficient cloud platform and things like that.

Abi: So a little bit more of an overall development experience focus as opposed to focusing on a specific tool set or platform, it sounds like.

Jason: Right.

Abi: I want to ask you, you already touched on this a little bit how One Medical, as a company, focuses really closely on patient and clinician experience. These are your customers of course. To you, how has this, and I know we’ll touch on this more later, but broadly speaking, how has this inspired the way your team thinks about developer experience? How does how you approach the customer experience translate to how you think about developer experience?

Jason: No, I think, for me, really seeing the way that we approach this patient space and clinician space in a very, what we call human-centered technology powered way, I think is great. Let’s do the exact same with developers. It’s obviously a little bit more complicated than that, but I think, again, there’s this pattern of experience teams at One Medical that really take the time to map out their process, their experience, the steps that they go through from checking into the desk, to meeting with the doctor, to getting labs done, to exiting the office and looking for waste and improvement all along that path. It’s like, let’s do that with developers, let’s get in. And it’s a huge space, so we’re still in the process of doing that. We haven’t mapped it fully out. But really getting into sometimes the minutiae of how are people putting up PRs? Let’s watch people, let’s watch them step through this process and really see what’s happening here.

Abi: What is this, and I know we’re going to talk about the full methodology later. Where, in terms of the point in of your developer experience team, is this happening, this mapping out of the developer process? I mean, is this happening as you’re reforming your new charter and trying to redefine where you need to focus? And is this a months-long process or is this a one-day thing and is this being led by PMs? Share more about this process of mapping out the developer journey.

Jason: It’s been a mix. I think, like I said, it’s an incredibly complex journey, and so I think we knew even spending one whole day to try and map it all out isn’t going to get it. So I think, over time, as we’ve hit bigger milestones that we want to plan for, we choose an area that we think, hey, this really deserves more attention. And then we can tackle a problem more specifically and then start to expand. So the team, as I was joining, was already really focused on the local development experience around our monolith. And so that was the, okay, let’s really try and map out these experiences in this process really well. Knowing, okay, at some point, we’re going to get to continuous deployment and how the deployment process works, but let’s not pay attention too much to that right now really helps scope down so that we’re not just completely overwhelmed. So we do a bit of both. Let’s do it over time, but then also try and carve out, okay, let’s spend a bigger chunk to talk about this in particular.

Abi: Makes sense. I’d love to know what does the developer population look like at One Medical? As you were thinking about mapping out the developer journey, looking for opportunities, are you doing this, you mentioned local development. I mean, are you doing this across a set of different personas, whether they’re mobile developers or cloud development?

Jason: So I think, overall, we’ve probably got I think it’s somewhere around 150 engineers. I could be a little off on that. But we’ve got a very large monolith that affects a really large portion of that. And so it makes sense that even before I came in, that was a pretty obvious initial, okay, let’s start with this because it’s got some of the widest impact. But then we also, we very clearly know and call out, hey, we’re not really addressing our mobile developers right now which, to be honest, sucks. We know that, hey, given the size of our team and just the effort involved, we can’t do much there. But over time, we’re trying to identify, we’ve got departments that are more patient facing and we’ve got departments that are more clinical software facing. So we’ll use some of that to identify, okay, what improvement are we doing that’s going to have a bigger impact on this side of engineering versus the other side of engineering, or hopefully both.

Abi: What have you found to be the hardest part of doing this mapping exercise? I imagine figuring out just how you’re going to break the problem up, both from the persona standpoint, but then when you’re actually doing the mapping, how do you break down the process is probably pretty challenging. Then I imagine actually turning this information into something that’s useful for your own teams. So I’m just speculating here, but what’s been the biggest challenge or hardest thing to get right in your view?

Jason: I think it’s been daunting to figure out at what granularity we really want to go down to. I’ve watched some of the folks on my team, like I said, put up a pull in a request and it’s like, yeah, there’s a lot of steps here. Not all of them are right for, oh, we can really improve this. But I think it’s just daunting to think about there are a lot of very small steps that happen, and then starting to understand there’s probably a lot of small variants even in how all of these different people do it. One of the folks on my team uses Vim really heavily. She might be one of the few, and we’ve got a lot of folks using VS Code, a lot of folks using RubyMine. So it’s even thinking about, okay, how does someone commit and then move to a pull request could be fairly different even though it’s really the same outcome. So I think just the daunting, okay, what do we really want to focus on here and what do we think would make the most impact because it would be the most shared part of that experience.

Abi: Makes sense. Well, I’d love to transition into talking about career stuff because before One Medical, you were the VP of platform at another company. I think career transition progression in the DevEx platform space is still so new, it’s a new topic, and so it’s interesting to cover. One of the changes for you was going from a VP role previously to more line manager role now. Share a little bit about your experience going through that change, the reasons around it, et cetera.

Jason: It was a really, really interesting change for me for sure. I’d grown up in that previous org really just from an engineer all the way up to a VP, which was super fun, lots of growth, lots of breadth of growth. I spent some time on the product engineering side leading teams. And then as I was discovering my love for all things platform, shifted into that more leadership role on the platform side, and it was great. But certainly one of the things I tell people about, even management and then certainly moving up into leadership as you get further and further removed from where the work is sometimes really happening, and it’s a much more indirect leadership, which I think is a fun challenge in its own, but it is very different.

And so while it was, again, a lot of fun to lead quality and infrastructure and core platform and all of those combined, there were a number of things that I was like, oh yeah, I can’t have as direct impact over some of these particular things. So as I was leaving that company, certainly one of the things that I wrestled with was, I think there’s this cultural thing about moving up into the right constantly with your title, with your salary, with your career. So there was, I think being honest, a lot of myself that was like, oh, I don’t know how I feel about moving, quote unquote, backwards in my career. Even taking a director role, it was just hard to get out of that mindset of this feels like going backwards. But in truth, there’s only so many leadership roles also.

Statistically, it can’t happen for everyone to get to a VP role and then just stay there forever. That’s just not going to happen. I think there’s a lot of great conversation around manager to IC and back again, but I think even the leadership back to line management, I don’t know that’s something we talk enough about. But I found this opening at One Medical I think was just really drawn to like, oh, wow, this is a company that’s already invested in this being a team. And I was like, I think this will really give me a chance to dive much, much deeper into the depths of developer experience and give me a lot more opportunity to be hands-on, well, as hands-on as managers are.

Abi: I really appreciated your point about how there’s conversation about the transition from manager roles back to IC roles, but not as much about upper leadership back to line manager. It’s awesome to hear your story and hear how it’s been a refreshing and invigorating transition for you. And you just also touched on how one of the things that attracted you about One Medical was that they already had this committed investment and belief around developer experience and having a team around it. I know at your previous role, you tried or did at least get a team, a DevEx team, to spot up. I mean, maybe share with listeners, I mean, I’m reading between the lines that I’m guessing it was harder to get that buy-in at your previous job. Maybe share advice with listeners what was just your experience going through that.

Jason: It was a little difficult to get that spun up previously. As your company is going through all of that growth, I think it falls into that same bucket of technical debt where everybody’s aware that this is something we should address, but priorities come in, the market changes, all of those different things put pressure on creating a team that I think, at least where we’re today, isn’t as obvious for folks. I think leadership certainly recognizes the need for efficiency. I’ve heard that repeated in several different companies now about, hey, we’re getting really big and it feels like things are going slower.

And sometimes there’s things to back that up and sometimes there’s not, but it makes sense from just a system’s standpoint. Things are going to slow down, but there’s still that how do we actually address that? And I think because of the way that orgs are typically set up, there’s not just an obvious like, oh, here’s exactly how we’re going to do that. So it takes some convincing that, hey, let’s pull often your maybe most tenured or senior engineers because they know the most about what’s broken in order to really focus them on changing things in your process.

Abi: I love that insight and the challenge of spinning up DevEx teams in real companies with pressures of the market and business. It’s always interesting to hear about platform engineering and developer experience are still new. They’re not widely accepted everywhere as an investment that needs to be made. And I’m curious, now that you’re on your second go-round with this, you mentioned in book by the way at the beginning of the conversation, I would love for you to share thoughts on that, but what are some of the distinct ways at a high level in which the way you’re thinking about these problems has changed since your previous job?

Jason: I’ve really taken what feels weird to maybe say at first, a very customer service approach to thinking about how we do developer experience. Customer service can bring up a lot of different thoughts of IT support or even just regular support that we all know of getting on the phone. You’re like, that doesn’t sound fun.

Abi: A call center.

Jason: I don’t want to create just another call center team within engineering that just does nothing but field problems and fix things, but a couple different influences coming in. So it was actually Jasmine James on one of your previous podcast episodes, talked about, it was the Disney book about their own customer service ethos. I read that and just immediately saw so many connections to this is how we can think about what are developers are doing right. Thinking about waiting in lines or various things like that that it’s like, oh yeah, these are great one-to-one comparisons. And so, really digging into, okay, how can we take a much more empathetic, really customer service minded approach to what our developers are actually doing and how they care about their own experience, and then how that translates into how efficient they can actually be after that.

Abi: Were there any other parts of that book, the book around how Disney approaches customers, that you feel were particularly relevant to your work?

Jason: Yeah. One of the things that I really enjoyed out of that book was Disney’s approach to creating magic. To them, well, and maybe to really magicians in general, magic isn’t magic. It’s just a lot of very intentional steps used to create this joy and experience on the other side. And I really fell in love with that concept because I was like, I’ve never met anyone who says that their development experience is magical. That’s just not something that people say. But what if we could do that? What if we could really get people to think, dang, how cool is that? Right?

I was actually talking with some folks recently and things like getting really quick ephemeral preview test environments. They’re like, wait, how did that happen? It really does almost feel like magic, even though on the backend, it was just a very intentional we set this up on the backend and this was something that we can provide to developers. But I think this notion of how are we creating magic, which I think is joyful, which I think really goes to that overall sentiment of what you’re doing for your developers.

Abi: I love that. That is an inspiring north star. I was just laughing because I feel like the only time I’ve heard magical experience pertaining to development tools is from product marketing, from dev tool vendors.

Jason: Trying to sell.

‍Abi: But certainly you hear, certainly much more of the opposite is typically the reality for developers. Whenever I speak to someone like you who’s moved from a previous role, I love asking them what their biggest mistakes and lessons were. And when we were chatting earlier, you mentioned that one reflection is that you feel, in your previous role, you may have fallen into the trap of overly focusing on the DORA metrics a little bit. Can you share a little bit more about that experience?

Jason: So I think I’ve fallen into probably a pattern of a lot of other folks really falling in love with the DevOps movement and really getting into things like DevOps handbook, accelerate, all of those other things. I think a lot of my experience with platform did start with a very infrastructure-minded and then DevOps approach. And so the DORA metrics were awesome. We’re like, sweet, let’s figure out how to do this and figure out how we can use GitHub and Jira to get it all in the system. We got it into Looker so anybody could take a look at it. And I think certainly one of the biggest positives there was I think that gave us the ammunition to really focus heavily on continuous deployment and say, this needs to be one of our biggest focuses, not only because of how it affects the metrics like we can use the metrics to measure it, but how I think this really goes to, I don’t know if I probably would’ve said developer experience at that time, but I was thinking, this hurts developers to not be able to deploy quickly.

And so what are we really doing to people to say, cool, I know you’re done, but if you could wait a couple of days before that gets to production, that would be great. So I think that was a big positive. But I think, over time, it just started to lose some of its signal, became a little bit more noisy because I think you’ve got things like cycle time that just don’t always fit in with, say, other infrastructure or platform teams or even just regular product teams that just go through ups and downs of, hey, we’re really busy on this one thing and it’s going to affect the metrics in a particular way, or we’re heads down on this other thing.

And so there’s always that context that you have to bring to the metrics and the metrics can’t be understood completely on their own like what is a five-day cycle time really mean? Is that good? Is that bad? For one team, it could be fantastic. For another team, it could be horrible. So coming into One Medical, I really haven’t been focused on too many real particular metrics. We’ve been focusing a lot more on the surveys and sentiment and using that approach rather than going, say, directly to something like DORA.

‍Abi: Well, that piece you mentioned about how you can cycle time in five days, is that good? Is that bad? I was, again, laughing in my head a little. Just had a conversation for this podcast that’ll go out in a week or so with some folks at Google who talked about the exact same thing, how they were tracking all these different objective metrics, but literally, these metrics don’t mean anything if you can’t say whether they’re actually good or not good. And they talked about how the only way to actually know if they’re good or bad is to go talk to people and get more of that qualitative side, the human side of the experience. And so what you shared resonates. And, of course, later in the show, we’re going to talk a little bit more about the different methods you’re using to get that feedback. 

Before going there, something that you mentioned you’re still wrestling with today is ownership. Share with listeners what you mean by that and what the tricky hairy problem around it is.

Jason: I think if I was paid in the amount of times that I said ownership over the last five or six years, I would be paid a lot. So I think, going back to my previous company, certainly one of the things that we dealt with as we were going through really heavy growth and then working with a primary monolith was, okay, how do we manage the maintenance that needs to continue and how do we manage potentially pulling out services out of the monolith and going that direction? But there was this question of, okay, as we start to pull things out or as we create a user service and then set it to the side, who actually owns that? If it breaks, who’s on the hook for helping resolve it? And I think the difficult thing is often the business is moving so fast and teams shift domains or focuses that you end up creating things and then you take away the support completely.

I’ve seen that in a couple different instances where you’re like, cool, I’ve got this service and absolutely no team that would really prioritize fixing it or just continuing to maintain it. And so it’s certainly more in that socio-technical side of problems, but we’ve found that it really does contribute to higher and higher cognitive load. You’ve got engineers that are like, oh, this thing is breaking, but I don’t know who knows anything about it to talk to them. Or maybe you do know who it is, but they’re like, hey, I’m super heads down on this number one P minus one problem and I can’t really help you right now, so you’re just going to have to figure it out. And so like you said, still struggling with that. I think, unfortunately, don’t have any silver bullets, but if any listener is struggling with that know that it’s a problem.

‍Abi: No silver bullet answers from my side either. So definitely a topic that we should continue to explore. Just a few minutes ago, you touched on how you’ve moved away from metrics and focusing now more on the qualitative side and really understanding the customer. One way you do that is this practice, and I may butcher the pronunciation, you call it Gemba, right? And this is really cool. I mean, I’ve talked to other teams that have done some shadowing, some user interviews, I understand, developers, but this practice of Gemba sounds like a bigger, more official methodology that goes beyond just developer experience at One Medical. So can you start by sharing a little bit about the origins of this practice and then how you’re applying it today?

Jason: For sure. Gemba was not something that I’d heard of before coming to One Medical, and I’m actually not sure I can probably point to the exact origins at One Medical, but it is something that, company-wide, really, from day one, in orientation is something that’s really emphasized. And the way that it shows up at One Medical is we run these primary care clinics, we’ve got patients seeing doctors on a regular basis. And again, we’re very focused on that human-centered, really empathetic experience. And so Gemba is what we set up as a whole company that we encourage people on a regular basis to actually go shadow that experience or some small part of that experience, obviously with consent of both the doctor and a patient if they’re involved. But it’s really cool because it really forces that see the effect of what you’re doing.

We’re writing software that patients are using, we’re writing software that clinicians are using, but let’s go see how that is actually being used. So backing up, Gemba is a Japanese term for the actual place. So it’s the actual place where the work is done. So let’s go to where the work is actually being done, look at it, see it, observe how it’s being done, going back to that process mapping. It’s not just sitting in a back room thinking, oh yeah, the customer checks in and then they go sit down and they do this. It’s actually watching, recording, looking for those waste and improvements there.

Abi: Well, I’m actually Japanese, so now I feel like an idiot for… Can you share a little bit more about how this is actually tactically implemented? I mean, can you share an example of one such initiative or session, if you will, where you’re actually doing this on your team?

Jason: So obviously, we, on the team, are encouraged to do, won’t say actual Gemba, but Gemba but with patients. But my team really took this like, hey, our customers are actually our engineers, so let’s go do Gemba with our engineers. So we keep a signup of, hey, if you’re willing to have someone shadow you for an hour or so, let us know. We’re also in the course of collecting surveys, and if we notice maybe someone’s having real trouble with something in particular, we use that as an opportunity like, hey, can I book some time with you and actually watch you experiencing this issue or error? But then we approach it in the same way.

Let me watch you through the steps of actually creating a pull request from like, show me what Git commands you’re writing on the command line, show me how you open up Chrome and go to GitHub and create it from there, or show me how you create it in a different way. And then looking for those, oh, yeah, you do a lot of different steps in order to open this PR. Let’s talk about like, I think we actually know some easier ways to do it. So we can get some localized improvements there, but then we can also start to think, okay, how can we, as a team, take some of these broader themes and start to work on them as a more first class initiative of the team?

Abi: I love this a lot. A follow-up question I have is, so sounds like you’re able to offer some localized suggestions as you mentioned, but then there’s a lot of probably takeaways and research findings that you’ve come to an understanding of what’s the process on the backend or the follow-up, how and where do you guys log and report this stuff? Where does it get triaged? And is this a PM that’s doing this or are these just happening led by your engineers or EMs?

Jason: It’s a mix. So we do have a PM on the team that I think primarily collects a lot of this stuff, but certainly we’ve got some really senior folks on the team that after a Gemba session, they’re like, hey, I noticed one part of our development script that is really tripping people up. And it’s pretty quick. I’d say maybe 75, 80% of the time, maybe more, we almost immediately create a Jira ticket and pull it to the board and somebody works on it. Because it’s often, it’s little things here and there. And then we reserve a few of those for, hey, what if we do something a little bigger here?

But a lot of times it’s just, well, hey, let’s just pull it into the board and we’re doing Kanban versus Scrum, so let’s just do it right now. And then we make sure to ping that developer and say, hey, I think we’ve actually fixed a little bit of this. Do you want to chat again or do you want to try it again? And then if it goes to a larger, we’ll announce all of engineering, hey, if you’ve been struggling with this problem, here’s how you can resolve it, which is cool.

Abi: It’s so cool to hear this because I think one of the things that probably gives people hesitance to do these types of very personal user research or shadowing sessions is I think the assumption that they’re going to find things that they can’t fix or do anything about. And in your case, sounds like this surfaces lots of paper cuts that you’re able to plug right into your backlog. Was that surprising to you a little bit? Did you go with the expectation that you’d find things that were bigger and more systemic?

Jason: That’s a good question. I think I had, maybe not necessarily, a pessimistic view, but the view that I do think there are a lot of paper cuts and a lot of smaller things that, despite us asking for feedback, a lot just really aren’t going to surface. People are just like, oh yeah, I know that’s a thing, and so I’m not going to say anything about it. So I think maybe hope or hypothesis was like, we’re actually going to uncover a lot of little things here and there. I think in terms of the larger systemic ones, just putting more and more themes together and noticing, hey, we keep making adjustments to our development script, let’s really think about how we’re doing this overall because there could be something that we’re missing. And so we’ve got folks that certainly will bring us their own ideas for solutions. Hey, what if you try this? We’re like, let me think about that. But if we start to see more and more people struggling with that, then let’s carve out some work on here.

Abi: Well, again, I love the way you guys are approaching this, and I think this example is one I’ll personally point a lot of people in the future. In addition to the Gemba practice, the practice of shadowing and your general customer focus approach, that you also run a set of surveys to give you continuous insights into pain points developers are experiencing. First, I want to just go into exactly what these different surveys are. Then I want to go into the backstory a little bit more. So, to start with, one of the surveys that you run is a weekly Slack poll. So tell listeners how you design this, what it’s for, how it works.

Jason: This is really our way of trying to get a pulse on, over time, what are people really struggling with, and hopefully giving folks a more immediate way to give us feedback. And so it’s a Slack poll on Fridays. I think we do it at 1:00 PM Central or so, so about the middle of the day on Friday. It’s one question. What affected your development experience this week? And we throw in a number of base responses like issues with local development, problems with flaky tests, various things that we probably know are things. And so giving people just easy like, oh yeah, I’ll just click a few of these and say which of these affected me, but obviously also a way for people to say something else affected them like, hey, I keep forgetting that I need to put dash dash local on this one command. It keeps tripping me up. And that’s really cool because we can look for spikes over time.

For instance, there was this one problem. We had seen this come in in a number of different channels because I don’t think anyone knew how to solve it, but engineers were really annoyed that they had to log into the AWS SSO command line what felt like constantly to them. They’re like, I do this multiple times a day. It never seems to be logged in. I don’t get it. And we had gotten that question, and I think, honestly, we had punted it to infrastructure at first because I don’t think we can affect the setting, and the infrastructure punted it back to us. Yeah, no, we can’t change it. And it’s turns out there’s a finite limit in AWS as a whole like, I guess we can’t do it. And we actually closed the ticket. But over time, we kept seeing this come up.

And so through the poll, we were like, let’s go pick somebody and go figure out, okay, when is this really happening? And so, one of those ad hoc Gemba sessions we’re like, oh, it’s actually happening every time someone wants to start the monolith server locally. And we’re like, I don’t know if we need to do that. That doesn’t seem right. And so, again, quick ticket, let’s figure out where people are having to log in just because they’re starting the server or realize, oh yeah, that makes sense for when it’s in staging and production it’s starting out, it needs to go fetch some stuff, but locally, you really should be good. And so it was a really pretty quick fix to let’s just remove that from this critical path of starting up the local server. Then via the weekly poll, we could see, oh, we had this spike of SSO is really getting in my way to basically zero after that. We haven’t gotten that complaint almost at all. So that was a really cool win for us I think.

‍Abi: You mentioned that you are able to track the trends in the survey month over month. Can you provide a little more detail into what you’re actually measuring? Is it the frequency in which certain things are being selected as pain points or is there other ways you’re quantifying this data?

Jason: It’s a pretty simple just frequency right now. And even, I mean, given obviously the number of responses we get week to week, which is only maybe some small percentage of folks, we start to aggregate it on a more month to month basis and see that, hey, problems with local development seem really high still, okay, let’s figure out what we’re not doing well there, or the SSO issue dropping off. But then, sometimes we’ll be surprised by lately flaky tests seem to really be coming up in the rankings of frequency, so okay, cool. Let’s think about what efforts we could really be doing there and if there is anything that we could do there.

Abi: Moving on to your other survey, so in addition to this weekly continuous poll, you run a quarterly NPS survey. Share with listeners how this survey works.

Jason: So this is fairly new for us. We’ve only run it twice. The second instance was actually just a week or two ago. But this came about as wanting to look for overall sentiment. We’re getting some of the week-to-week changes, but how are people feeling overall? So using that more, I guess, eNPS, employee NPS style like, hey, how much would you recommend your development experience at One Medical? And then we also tack on a question, if you had a magic wand, what would you change about your development experience? Which I’ll talk about actually how we think we might want to change that. But what’s been cool so far is people will just tell us, hey, get rid of this, or add more services, or I want a one click development environment.

So that’s been really cool certainly to see the overall sentiment, which is actually very low, but I think what we expected to be honest, but a lot of pretty neat insights into if we just say, hey, what do you want different? It’s been really interesting to see what people come up with. And so our second iteration just a couple of weeks ago, a couple different challenges because we actually had less participation, but a better score, but also a couple new themes and then a couple of existing themes. So I think we weren’t confident like, oh man, this is really going to help us immediately. We’re hoping for that trend over time certainly. But it’s given us some interesting insights so far.

‍Abi: Well, first of all, I was laughing with if you had a magic wand because, oh, this is that Disney influence coming into play again. But I mean, one thing I would share is sounds like you’re early on in this journey with the larger quarterly survey, and in that conversation I was referring to earlier with Google, they actually talked about how it took them a long time to establish a quarterly survey program and grow it in terms of participation, value, buy-in from the rest of the organization. So it definitely seems like it’s a bit of a journey. I want to ask you, well, first of all, a little more tactical detail. I mean, you mentioned sentiment was low. What’s the thing you’re measuring specific? Is it that eNPS around the developer experience and you were saying, and when you say low, earlier we talked about how, just because you have a number, is it good or bad. In this case, how are you coming to view it as bad?

Jason: I think the first NPS score that we had in Q1 of this year was maybe minus 30 or around minus 30 or so. The most recent one was, I think it was minus 20. So still, I guess, low from the NPS way of thinking about that you’ve got more detractors than the others. But I think our hypothesis going in, we asked everyone on the team, hey, do you want to pick what you think the score is going to be? And everyone had a pretty low hypothesis, a pretty low guess. And I don’t know exactly where that comes from. Maybe just as developers, we know developers to think that everything can be better. There’s always something that’s maybe wrong, but I think we were still surprised at how many positives really, really came in. A couple of people in their magic wand said, actually, I don’t think I would change that much, which is cool.

Abi: Well, I can tell you I do a lot of research on the industry data around developer sentiment, and one thing I would say is that the benchmarks for NPS that exists for customer experiences I think don’t necessarily apply to developer sentiment. So while you called your score low in absolute terms, without knowing the specifics, I would imagine that your scores really aren’t that low compared to the industry data and averages out there.

Jason: And that’s where too, I think, we’re not hanging too much on that individual score. I think we care more about certainly the direction over the next couple quarters and then understanding a baseline because maybe this is actually a really good baseline depending on how you think about it. But also looking at, we can see the lower detractor score combined with particular magic wand feedback. It’s all anonymous, but it still gives you that, okay, someone really rated us low and they said the biggest thing that they care about is a one click development environment. So that’s having a really big effect on their score maybe. But then looking at someone with a seven or an eight, and they say, oh, I wish we had less flaky tests. So maybe the flaky tests are a problem, but to them it’s not really contributing as much to an overall bad experience.

Abi: This last point you just made is so interesting because it’s not just the existence of problems, it’s the magnitude of those problems, which is usually partly a reflection of that person’s expectations, their previous experiences, their seniority, their role, the tasks they work on. So we talked about like oh, you have to contextualize metrics earlier. And the survey data is the same way, but it almost comes with this context that you have. It’s the opposite. Comes baked in with context and you have to reverse engineer it to understand why something is coming in the way it is. I want to ask you, I think it maybe obvious to you, but you started with one survey, you have two surveys now. If someone else were thinking about surveying, would you advocate them doing both? Why do both of the surveys simultaneously?

Jason: That’s a great question. I was just talking about this idea with my PM this week because I think one thing that we’re concerned about is that survey fatigue. Are we getting a drop off in responses because of that? Are we getting a drop off in responses because it’s summer and more people are just on PTO? What’s really contributing to that? So I think we are trying to be careful not to overload that. If I had to give advice, I’m still a big fan of the weekly. I think it’s, despite it being more frequent, I think our hypothesis and hope at least is that it’s a regular feedback opportunity that people know about. They see it every week.

They don’t necessarily have to participate every week, but I think they know it’s there. I think even the quarterly one, we’ve only done it twice. I don’t think people are as used to it yet. They don’t know that it’s coming again the next quarter, so they’re not really, I think, thinking enough about it. Whereas, hopefully the weekly is that like, oh yeah, I really did have an issue. It’s Thursday. I’m going to remember tomorrow to put this in the poll.

Abi: You’ve touched on participation rates, survey fatigue stuff already, but taking a step back, in general, what are the biggest challenges with surveys in your experience so far?

Jason: I think something you alluded to earlier, which is the impact of a particular response. We might be getting a particular response for, say, testing with better data. That seems to come up often, but really figuring out, okay, we get one of those responses every week. Is that indicative of a smaller problem that’s localized either to maybe one engineer or just one team, or is that indicative of something bigger, but not enough people are really jumping onto that to say, hey, I have that same issue. And then I think just the granularity of it. We started with a problems with local development as an option, which I think at first sounded good because we were working on local development. Now, over time, one of the things I’m thinking is that feels very broad now and there’s a lot less responses. Is that indicative of this not being granular enough or is it a result of us actually doing things to improve and so it’s not as big of a problem? So I think still how do we really understand the data and the granularity that it gives us.

Abi: That second problem that you’re touching on here is super interesting. I mean, naming comes into play, how do you break down the problems that you’re asking about the right things and the right way at the right level of detail where it’s people can respond, but then it’s actionable for your team to interpret? It’s really a fun problem, as I’m sure you’re finding, so fun to hear about your approach to it.

Jason, it’s been great hearing about your experiences at One Medical, and the work you’re doing, and the culture there around being focused on the customer in general. Thanks so much for coming on the show. I think listeners are really going to enjoy this one.

Jason: Thanks so much for having me. It was a great conversation.