Building an internal developer platform at CVS Health

Abi Noda: Jim, so great to have you on the show today. Really excited to dive in.

Jim Beyers: Thanks for having me. I’ve been really looking forward to being here, big fan of the show.

Abi Noda: So, you’ve been at CVS for just about a year now, and I’m really excited to dive into the entire journey so far. I want to start at the beginning. You were hired in to work on developer experience, so I’d love to better understand the context. Why were you actually brought in, and who is it at CVS that was championing this effort?

Jim Beyers: I’m fortunate enough to have been able to work for a tremendous leader more than once. Amaresh Siva is the senior vice president of digital engineering at CVS. I’ve worked for him in a past life, so to speak. When he reached out and talked to me a little bit about what the state was at CVS with respect to how much opportunity there was to have impact, it was incredibly compelling. And so, having worked on similar problem set with him before when we worked at Target together, he was very familiar, I guess you could say, with what we’ve learned together during that time. And we were looking to do something very similar at CVS essentially when he explained, like I said, what the problem space is, which to be more specifically just how enormous a company it is, how many different technologies and tools, ways that people are doing things, and where they’re at in their transformation in terms of SDLC and team maturity, moving to a product mindset, et cetera. It’s just such an exciting time to jump in and be a part of that.

Abi Noda: A lot of new DevEx leaders or people trying to get these types of initiatives off the ground run into challenges around getting buy-in from the business, getting buy-in from the c-suite, figuring out how positioning this type of work in a way that aligns with the rest of the business. If I’m understanding what you described earlier correctly, is your work developer experience aligned with the rest of the business because it fits within the overall transformation initiative, or how does it align?

Jim Beyers: I’d say that, and I feel like a big part of it is building goodwill through small wins that are enabling without the end product state. So, you don’t have to build a full-featured paths replacement in order to build that goodwill and show progress. And so, the small wins or even some of the big wins that we’ve had to be able to just take toil and distractions off people’s plates, I mean, this is maybe a silly example but we’ve got a host of applications that are running directly on Bare Metal. And so, some members of my team are just taking the time to containerize that stuff up so that we can hand it back to the business instead of it having it be like, oh, we don’t want to make changes on that because if we do, I have to go train someone how to go do this in this nonstandard way.

And that’s helping so that obviously a lot of work is unfortunately sometimes tit-for-tat. And so, doing some of these things opportunistically help later I think, when you’re say, “Hey, we want you to do these things because it’s going to help you.” It doesn’t get then as perceived as a distraction. I’m not sure that that’s answering your question.

Abi Noda: No, that makes sense. I want to ask you, and for listeners we are going to get down into the nuts and bolts of what you’ve been working on, but I want to ask you at what level of the organization is developer experience in your work talked about and visible? How important do you feel like the organization views your work? Some organizations I know of, it’s coming from the CEO or CIO, other organizations it’s more of a little bit of a bottoms up effort that’s being bootstrapped and fighting for that visibility. So, how is developer experience viewed and talked about at CVS?

Jim Beyers: I’m lucky to say that when I was interviewed, the other SVPs that I talked to, they all knew that this was the reason I was coming in. First time I talked to from the data team an SVP over on the more business side and retail space, and they both were excited about the prospect of not having to have their teams be so concerned with the entire gambit when they’re just trying to deploy a container. That’s incredibly rewarding, and obviously it doesn’t feel so much as Sisyphus, just if you know that you have other people that are going to be championing what you’re doing versus the alternative which is, hey, we have this thing and we want everyone to do these things differently. And then everyone’s like, “How dare you. No, why would we do these changes?”

Let’s just say it can be a lot easier if you’ve got that tops down support obviously. So, I’d say it’s pretty consistent across the executives there at CVS. I can’t speak for everybody for sure, but at least in the digital engineering arm as part of our annual OKRs we’ve been talking about, measuring cycle time, ensuring that we’ve got better visibility on DORA metrics, that’s one of our top OKRs for digital, which is really honestly very exciting.

Abi Noda: Shifting a little bit now, we’re going to get into all that you’ve been doing for this past year, team building, defining strategy, winning the hearts and minds of colleagues. But before we do that, I’d love for you to share with listeners what it is, just high level that you’re actually trying to do. What’s your vision? What are you trying to build and achieve?

Jim Beyers: The long-term goal will be that we’ve built a container specific paths for engineers, starting with digital, maybe ending with digital. They bring their container, we’ll take care of the rest essentially. So, I think Kelsey Hightower has said it best, at least in my opinion, shifting EML left was a mistake. I feel like the amount of time that folks have to spend understanding all the intricate aspects of software defined infrastructure is time that could be spent in another way. And that said, I don’t want to give folks Kubernetes, I want to give them a deployment surface that allows them to run their applications and not have to care about load balancers, and config, and secrets, and everything else. We want to build this in such a way that there’s a UI, there’s an API, and there’s ACLI, and so essentially all of these capabilities can be leveraged in different ways.

So, someone wants to come and write a top Terraform provider to deploy their apps, go for it, that’s great. But maintaining those common patterns of usability, that similar experience is super important so that then behind the scenes we can do all this other work for them, which is ensuring that we’ve got fine grain security access, engineers don’t have access to credentials so that they have direct access to production data. All of your SOC compliance stuff basically you get for free, which is pretty exciting. And similarly, I believe that’s true for PCI and other compliance also. So, compliance ultimately just comes for free. People don’t need to care about any of these things, you can eliminate all these different meanings.

So, to answer your question more succinctly, a container paths for the company starting specifically with GCP, that’s the digitals cloud that we use, but also leveraging on-prem resources so that we can shift workloads and save money as appropriately, or do placement so that it’s appropriate latency for dependencies, et cetera. And as simple as just redeploying, I want to be in the data center, I want to be in GCP, and then you don’t need to know how to interface with both.

Abi Noda: We’ll talk more about how you’re looking to achieve this strategy, because I think it’s really interesting the way you’re thinking about it, but we’ll talk more about that when we go into your strategy. One of the first things that you had to do in your new role is build a team. So, how did you think about what kind of team you needed to build then, how did you go about actually getting the people together?

Jim Beyers: One of the things that’s been interesting has been, like I mentioned before, I think all the other transformations that we’re going through, and CVS is a agile software development shop. We’re still figuring out, I think in my opinion, what long-lived technical products look like. And so, my first job was to observe and listen, and try and learn as much as I could because I’ve never worked in such an enormous company, there’s just so much going on. And so, that took me a little while, but what I quickly realized was is that there was a lot of opportunity for us to drive down on just inefficiency of the ways that we were working by migrating towards more specific long live product teams. We moved from more like a DevOps, developer and ops teams, into more long live product teams.

So, high level there is a cloud resources IaaS team where we’re building templates so that we can have consistent management of how we’re interacting with those tools. There’s a team that’s managing specifically how we deploy API proxies, that actually now is going to be consistent across the rest of the company, which is pretty exciting. There’s a team that manages how we are deploying and managing our Kafka clusters, et cetera, instead of these bits and pieces across the board. Because I feel like it’s important for people to have not only the agency and the deep understanding of those products, but then also having them be the ones that are defining essentially what we’re going to work on next, which is important. So that was a pretty big step, a big change for folks, and the great news and honestly very rewarding to see is how happy people seem to be now that we’ve gone through that, that’s been pretty exciting.

Abi Noda: What’s the composition of these teams? For example, are there product managers on each of these project teams? What’s the makeup of the engineering expertise? Are these people with deep cloud expertise, more product engineers?

Jim Beyers: It’s a mixed bag. So, that was another fun thing that I got to learn after joining, which has been great, is my team is the only team in digital that have product folks reporting inside of a technical team, and they’re wonderful. We’ve been also being more focused on aligning to specific rubric around job roles and whatnot. So, essentially we’re going on a three or a four in a box model, if you’re familiar. So we have architects, we have obviously engineering managers, high level ICs, principal engineers, and then product folks. And on some of these long-lived product teams, like I was talking about, we don’t have all four and that’s okay, but ultimately between that working group they’re responsible for identifying what’s the roadmap for their product or product area, how are they going to prioritize those things.

And then ensuring that things are stable and reliable. And then also being involved in the broader discussion and conversations about what’s the long-term strategy that we also want to be building for so that they can have an eye and a lens on what’s important right now, but also can we make the right decisions so that they’re not misaligned to where we want to go.

Abi Noda: I was able to gather from your description, but for listeners, just to clarify, what is the four in one box model?

Jim Beyers: Oh, sorry. One in the box model means it’s not one person that’s deciding, you need to build consensus or at worst disagree and commit amongst in high level IC, the engineering manager, the product person, and the architect. And sometimes it’s three in the box, sometimes it’s two in the box, depending on maybe there’s not an architect assigned to that particular area, or maybe there’s not a product person. But the idea is rather than having one person like single-threaded ownership specifically only to one person, it’s a consensus driven model so that we are building things together rather than just Jim comes in and says, “Do this.”

Abi Noda: That makes sense. You mentioned being fortunate to have product people within your teams. I want to ask you, is that different than your previous experience? For example, at Target, was this a new experience for you to be overseeing product managers? And maybe share a little bit about what you’ve learned from that.

Jim Beyers: We did have product folks at Target. The interesting thing is when I came back, if I remember correctly there was only one or maybe two in all of infrastructure. It may have been more than that, I hope that that’s not the case because then I feel bad about it. But regardless, the ones that we had were terrific. I was also very fortunate because I was able to work with the person by the name of Amanda Engleman, who was our product person for TAP, which was the Target application platform, who was just transformational and we wouldn’t have been able to do it without her. But we had to go source and build the case that we needed product people on this technical role. And so, in that sense that was just additional work that we had to do.

Versus in this case, because I’m lucky enough to work for Amaresh, he already had me set up with people when I joined, so I was able to hit the ground running. And product’s a very important thing for people being able to prioritize and do long-term strategic planning, and without them I think we’d be hamstrung to be able to move as fast as we are.

Abi Noda: So, in the end how large is your team today, and what’s the thinking around future headcount?

Jim Beyers: We’re about 250 people between DevEx and digital security. That’s a mix of both FTEs and contractors. We’re making some investments right now in what we’re calling a seed team essentially, to bootstrap this paths in partnership with the other teams of course. But we talked about this a lot, and the executive director I mentioned earlier over that particular team, we learned about trying to involve everybody at the get-go from our Target experience, and that was not successful until John Engleman, and James Westover, and Dan Woods were the guys effectively that started TAP together. And if they wouldn’t have done it, I don’t know how long it would’ve taken. We’re going to use a similar approach, which is start something and then have it become emergent. I’m hoping because of the fact that in some ways we’ve already started down that path that we’ll be able to include teams along the way, but that’s where we’re going to be growing the most, I think is on that seed team to start with.

Abi Noda: Let’s shift into the strategy. So you get hired in, you don’t probably have a crystal clear idea of what you’re doing yet. What was the journey to define what you were going to do, and not only define it but communicate it, evangelize it, and get people bought in?

Jim Beyers: The fun thing about this is some of my friends that I worked with on this before, largely maybe in some ways probably should take way more credit than I do, for the work that we did at Target are also off doing this again someplace else. And so, I was lucky enough to beg, borrow and steal from my friend Dan Woods, that’s over at Maersk now trying to do something similar, and Westover, who’s also trying to do something similar with Dan at Maersk, to put together a strategy paper. And so I did that, and our team bought in on some of what we’re going to do. And then essentially reality hit, there was so much things that we needed to get through first before moving things along.

Fast-forward a few months, I brought in Mike Amundson, who’s our distinguished engineer for DevX, and he was able to let’s just say put a much better point on what I had been trying to do previously and put together a strategy paper and our platform tenets to get the whole rest of the team bought in, and way more successful. We had to work through removing some of those distractions of, like I said before, a lot of how we were structured before was less than ideal. Some of the things that we were working towards we needed to just stop doing, and now I think we’re in a much better space to be able to succeed, which is going to be good.

Abi Noda: Share with listeners the strategy paper, sounds like it’s been effective, went through iterations, collaborations with folks even outside of your company. What is contained in this strategy paper? What’s the structure? How long is it?

Jim Beyers: Yeah, so it’s a work in progress is the beginning and the honest truth obviously. I think we’re going to figure out what we’re going to start with and I think that’s going to be bring your container and we’ll take care of the rest. And along with that it’s going to need to come with load balancing, and config, and secrets, and it outlines a lot of the specifics of what that looks like. Additionally, it outlines some of the other requirements that I think probably a lot of folks maybe don’t think about, which is in order to be as successful and able to do this, you’re going to need to have things like an application registry. And we feel like that application registry is going to need to be able to be so dynamic that it probably can’t have a third party, let’s say heavyweight name behind it, if that makes sense.

We need to be able to have ties into our accounting systems, and alerting systems, and everything else. That app registry, given the frequency of deploys is going to need to be much more dynamic so that it can make appropriate placement decisions, and have better understanding of the ontology so that we can build … Anyway. So there’s all this, I don’t want to call it extraneous, but the stuff that folks don’t think about. And so, there’s a lot of pieces about that. Like things, for example, like applications need to be globally namespaced. If you don’t have naming conventions and you don’t have strict strong opinions about how these things are done, you’re going to find out a couple months into it that like, oh my God, this isn’t going to work and we have to start over again.

The other thing I really want to hammer in on is, is Michael was started with writing our platform tenets, and so really we started by rallying people around what those things were before getting into the specifics. Because having those foundational things are so important so that when people are thinking about what the solutions are that they’re grounding themselves in the how and why of what we’re doing, we’re only building for 80%. We’re not going to be looking to figure out how teams are managing SAP. SAP’s great, HANA’s great, but it’s not going to run on the platform. And self-service is full service. If we build something and it only works up to a certain point and then somebody has to figure out the rest, it’s not truly self-service. Our dev environments are a prod. Our dev environments can never go down.

Our interface to people being able to build stuff is just as much production as cvs.com is to our customers, and I think we talked about this before, platforms are forever. Once people move on to this feature rich paths, we can’t ask them to re-platform again. It’s got to live forever, and we need to be able to make it such a way that it’s adaptable to changes under the covers. If Kubernetes isn’t the compute runtime of the future, or it isn’t in two or three years, that shouldn’t make any difference. We should be able to swap it out and have our teams keep working.

Abi Noda: Related to this, there seems to be a number of different ways that different companies are tackling this problem space. Of course, Spotify Backstage is one platform. There’s a whole bunch of different SaaS providers doing similar types of things. Can you share your perspective on the evaluation of these different paths and options, and why you’re set on a slightly different path than what you see as the main benefits?

Jim Beyers: Just to say it out loud, we love Backstage. Backstage is great. I think Backstage is a terrific tool for exactly what it offers, and I was lucky enough to meet Lee from Spotify and talk to him about this a couple months ago. Lovely person, and was a great conversation. We have a couple different instances of Backstage at CVS, and like I said, I think it’s terrific for what it does. What it doesn’t do is provide those higher level order guardrails that I think are really required in order to completely remove toil, because while making it easy for people to provision infrastructure is definitely better than just go figure it out, absolutely, I don’t think it’s enough for a huge percentage of the population of the engineers that we have with all the work that we have to do to ask them to do those things and still deliver on all the work that we need to do as a company.

Shifting YAML left was a mistake. This is not the same, but it still largely is stitching things together and you aren’t going to have the same strong opinions loosely held and common patterns of usability that would essentially obviate the compliance stuff I was talking about before. Or maybe it would, but I still feel like there’s still enough room for interpretation that it’s going to potentially leave openings for, let’s say bad patterns.

Abi Noda: And so in the end, maybe you could share from the developer experience itself, platform that you’re trying to build, how would it be like for a developer? I mean, walk me through a day in the life of setting up a new application, making a change, deploying that application, monitoring that application. What would that look like through what you’re envisioning building?

Jim Beyers: The best part about this is I’ve seen it done, and so that’s so exciting about it. The very first day, so Dan Woods, when he wrote his paper at Target talked about the midnight developer. And so, he went on to say like, “Nobody, please don’t do work at midnight.” But the idea is no meetings, no muss, no fuss. The first day that you show up at work, you should have the entitlements that you’ve got access to your source code manager, we use GitHub, I think everyone should use GitHub, get access to GitHub, and once you have access to GitHub you’ve got access to your paths. And probably not deploy to prod, but you should be able to bootstrap an application and have it running on some computers moments after joining the company, and that’s where we want to be. And then having essentially an opinionated set of things that are easy to have access to.

Maybe you don’t want to give self-service, easy onboarding access to a cloud resource that you have the potential for having a million-dollar bill, probably not that, but for those things that it’s pretty difficult to do a big damage on, like a database or streaming events platform, those things should all be just plug and play. Simple, easy. I haven’t used Heroku very much lately, but Heroku I think is such a great model of they provide you with access to these infrastructure resources, even for free if you’re just a personal user, you spin stuff up incredibly easy using these conventions, and then when you’re done it goes away. Under the covers as an infrastructure offering, you can save the company tons of money but you’re also saving or enabling by removing all of that toil and increasing the developer productivity by just allowing them to do things without having to go and have a meeting, submit a ticket, et cetera, et cetera. That’s what we want to do.

Abi Noda: You brought up Heroku, there’s a number of open source platforms out there that are modeled after Heroku. In terms of what you’re setting out to build, are you leveraging any major third party components, or is it primarily going to be homegrown?

Jim Beyers: I think that remains to be seen. I don’t know for sure. My opinion on this might not be popular is some things make sense. So, we love HashiCorp Vault. It’s wonderful. I don’t know why anyone would use anything else. And that applies to a couple of other handfuls of things, which maybe won’t go the laundry list, but once you start to get to a certain scale I don’t know that there’s things out there that fit necessarily because of the heterogeneity of us as a larger corporation. I mean, we’re in three different public clouds. We have seven different, I think on-prem data centers. Potentially, ideally, I would hope that someday we’ve got a similar experience for our 9,000 stores. That’s a ways out there. I think that there isn’t anybody out there that’s in a similar boat as us, if that makes sense. And so, probably we’re going to have to build it ourselves. Hopefully there’s stuff that we can give back and that we can share. I’m skeptical, that’s what I’m trying to say.

Abi Noda: I want to now talk about how you’re dealing with your customers, and by customers I mean developers, leaders within your organization, how you’re evangelizing what you’re doing, how you’re having these conversations, getting folks on board. How has that journey been like for you? How much skepticism, how much pushback have you encountered thus far?

Jim Beyers: I am anxious to see how it happens once we have something rolled out. So, our goal is to have something in Q1 of next year we can start onboarding folks to. The thing that I learned in my past life is when we’re ready to do that, I really want to find folks that are excited and that desperately want to get on board, because we want a different approach and mandated that we were going to turn off other providers, and that everyone thou shalt must migrate to the paths when we were at Target. And it was awful. I mean, it was just terrible. And of course it was coupled with the fact that maybe we did that too soon and we had some problems, people waving fingers and screaming at us and saying, “How dare you,” and, “You won’t be successful and you’re going to ruin the company,” and all this other ridiculous stuff, and it was terrible.

And so, I learned a lot from that about winning the hearts and minds is so important. And if those folks see other people being successful and you taking care of the problems that they also don’t like, I don’t think that the large vast majority of them desperately want to manage those infrastructure resources and eventually will move over, which is exactly what happened. The folks that were our biggest detractors, for the most part I would say became big champions, and it was lovely to see over the course of a couple years later, people when they were joining being like, “Oh my God, I wish we had this everywhere. My first day and I can write and deploy code.” So, I feel like the good news is at my peer level and a lot of the directors, they’re all well aware and they’re excited about what we’re going to do. And the fact that we’ve had the success before is going to be helpful, but we’re going to start with the folks that are dying to get on board.

Abi Noda: And what’s the scale of this? I mean, how many constituents or teams are there out there that are eventually hopefully going to be using your platform instead of what they’re doing today?

Jim Beyers: I can’t say for sure about the potential for long-term. All I can say for sure, we’re targeting for digital engineers that are running in GCP and on-prem, which is a community of about 2,000 folks. I think that there are roughly around 1,200 to 1,400 ICs that would be potentially be candidates to be on platform. I don’t think that there’s a lot of teams that are running things that are applications that are “off-platform.” We don’t run SAP and digital for example what I’m talking about. My hope is that we can build it in such a way that it’s consumable for other partners, and so it becomes the defacto option for engineers at CVS to run containers on. That remains to be seen, and I’m fortunate enough to be working with partners over in another side of the organization that are in another cloud, that are doing something similar, and we’re joined at the hip but things are somewhat different. So, I’m hoping we work together to figure out how to bring some of these patterns, because that level of scale is really exciting.

Abi Noda: You’re bringing up this other group with an organization that’s focused on similar problems. Can you share more about just again, why are there two different groups tackling these similar sets of problems? And how are you working together with those colleagues to tackle the similar problems?

Jim Beyers: Largely there’s three large groups inside of what we call DDaT. So, DDaT is digital, data analytics, and technology, and digital is obviously our organization. The technology group is the biggest group in the company. They’re largely in on-prem and also in Azure. And so, given the fact that we have different clouds, we have different folks racing to enable teams to be able to deploy obviously. And so, Doug Safford’s my peer over in technology and IT, wonderful human being, we talk all the time, we’re doing great stuff, we’re working on stuff together and we’re trying to figure out how we can share. It’s because we are serving different customers, I think that’s a big part of it.

Either of us were trying to take a bite out of everything it would just be too much. And so honestly, I would be thrilled if I was able to solve for this for GCP and he was able to solve for this for Azure. That’s success. If we don’t build something that’s ubiquitous, that’s fine. If we do, great, but if we don’t, that’s okay too. And then also there’s the data team, which is also in GCP. I think there’s some potential opportunity for us to partner as well, but also there’s stuff that just won’t be on platform and that’s okay.

Abi Noda: As you go on in your journey, and the work you’ve done so far, what’s been your approach for sharing progress, reporting on progress, conveying impact and value, what’s been your approach to date and what do you see as being your approach going forward?

Given the amount of money that we’re investing to be able to do this sort of thing, that’s an area that I’m learning to be better at this year in terms of being able to report essentially migration, stats, and adoption metrics, and other metrics, like we talked about being careful to shy away from things that can be misperceived. So, effectively we’re honing in mostly on TORA metrics, I think we talked about earlier, and trying to just showcase that with a higher degree and frequency of smaller, more of them deployments, we can have a higher degree of quality and fewer incidents.

Abi Noda: Jim, thanks so much for sharing your journey thus far. I’m excited to follow your progress and hear more in the future. Thanks so much for coming on the show today.

Jim Beyers:

Thanks so much for having me.

Abi Noda: Thank you so much for listening to this week’s episode. As always, you can find detailed show notes and other content at our website, getdx.com. If you enjoyed this episode, please subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Please also consider rating our show since this helps more listeners discover our podcast. Thanks again, and I’ll see you in the next episode.