How developer experience impacts databases like MongoDB

Abi: Tara, thanks so much for coming on the show today. Super excited to chat with you.

Tara: Oh, I’m very excited to be here. I love nerding out about this stuff.

Abi: Well, where I wanted to begin today was actually the story of MongoDB, and specifically the point in time in which you were hired and brought on. I’d love to just understand where MongoDB was on its journey and how it came to be that you joined?

Tara: That’s a great story. I joined, it’s coming up almost a year. I joined in May of last year, and one of the things that I discovered about MongoDB that was really exciting was more than any other company I’ve worked for, with the exception of Google, Google’s special, the amount of investment that MongoDB puts into continuous integration into having strong test culture into really focusing on developer productivity as a key element of engineering culture, it was just tremendous. I had never seen anything like that, because it’s expensive. It’s a lot of resources, it’s a lot of tools, costs and subscription or hosting subscriptions. And so, when Mark Porter reached out to me and was talking about the opportunity, it was not that they needed to start investing in developer productivity, but that they wanted to up-level and do even more.

And so, there was the two main product areas. There’s what we call the core products, which are what we think of MongoDB, where you could download it and spin it up on your own servers. But then, there’s also, and that’s one organization, and there’s another organization that’s all about the hosted offerings. And those operated in some ways, they’re very distinct problems. And so, they evolved development infrastructure slightly differently. So, part of what I was presented with as an opportunity was to identify ways to unify key elements of that. Now, we don’t want to say that everybody has to do the exact same thing, but are there ways for us to up-level so that MongoDB can continue to scale as an organization, improve developer experience internally, and our velocity, which would hopefully also lead to improve velocity for our product deliverables and improve quality. The whole nine yards about why developer productivity is so important. That way I was like, how can I say no?

Abi: That’s great. Well, it sounds like MongoDB already had this preexistent belief and commitment toward investing in developer productivity, but what was new around the time you joined was this vision of unifying some of the pieces to put in place a platform team. So, would love to know, was there already a platform team when you joined or were you brought in to help start one?

Tara: There was already a platform team when I started, but it was pretty nascent as a team. And so, the first thing I did was to try and stay quiet, and learn, and get to know people, and basically go on a listening tour, and get a better sense of who they were and what they were trying to do. Within this first, what, nine months now, going on 10 months, definitely have already identified some opportunities for change that we’ve started to make, which we will continue. And one of the high level propositions that I’ve been selling to my team is, change is actually both scary but more exciting, and we should get comfortable with it.

I tend to push the boundaries of things when it comes to development infrastructure, probably to the chagrin of my engineers at times. But, they’re amazing, and are usually pretty game to try things, and something else I appreciate, they’re very willing to give me feedback when they think I’m being a little bit crazy. So, it’s an amazing group of folks with an amazing R&D organization as a whole. Like I said, the culture around developer productivity as a value at Mongo is just tremendous. I really enjoy my job.

Abi: I wanted to ask you, platform engineering and platform teams is being treated like a new thing in our industry right now.

Tara: Oh, my goodness.

Abi: There’s a lot of buzz around it. But, you’ve been doing this a really long time. I’m curious, what’s your perspective? How is this similar or different to what’s already existed in the past? What’s your view on all the buzz around platform engineering?

Tara: And it cracks me up, because you’re right, it’s just branding. And I’ll tell you something even funnier to me which is, I started at a company called Borland at my very first job, which did developer tools, databases and compilers. And this was very old school. The release cycle was two to three years for a new product and very labor-intensive and lots of actual physical product. This was not even CD yet. This was floppy disks, and you bought a box with software, it came with paper manuals. There was no internet yet, or no worldwide web internet yet. And then, moving on to Netscape. And then, Netscape introduced a new challenge which is, now this is the beginning of the worldwide web and web-based technologies, but it was pre Linux. And so, it was all of these different operating systems and chip sets and the challenges of trying to maintain a single source base in all of that.

And over time we got away from that. Linux became the defacto standard, and cloud you just provision a VM instance of whatever, and was an AWS user, and then I was at Google for a while, and then I come to Mongo and lo and behold, I have to worry about platforms again. It’s like, this is not just Linux, there’s also Mac and Windows and mainframes. And it was like, it’s 30 years later, I have the same problem, which is how to have a single source base that we’re able to successfully and quickly maintain and get feedback on from a development perspective.

And so, it really truly, the same problems. There’s just different, maybe there’s different flavors around it, but fundamentally it’s all the same thing. And whether you call it platform engineering, or if you have a DevOps oriented engineering organization, or you call it infrastructure engineering, or you go back to the old days when we called it build release team, it’s all the same. It’s really, whatever it takes to get code from developer to customer as fast as possible with as high a quality as possible, and just what are the challenges? And it just depends on who your customers are and what you’re selling them. You just, it’s details.

Abi: Well, I love your take on it and the experience you shared saying, 20, 30 years later, it’s like the same problem again. That’s a similar perspective I’ve heard from others as well, so I really appreciate that. I want to shift into talking about the developer platform, so to speak at MongoDB, and more so the developer experience at MongoDB, which I think is such an interesting topic, because the majority of listeners of this show, I think are accustomed to working on cloud software. And, I guess, MongoDB is cloud software, at least in some respects.

Tara: Absolutely.

Abi: But, I think listeners are far less familiar with developing and deploying a database technology as a product. So, I want to start by asking you something we were chatting about before the show, was just what it involves to build and test a database. So, would love to have you just walk us through what that looks like at Mongo?

Tara: Well, I want to say that this is the first time I personally have been responsible for ensuring the success of a database. Even when I was at Borland that had database products, I worked on the compiler team. So, coming to Mongo it’s like, I don’t really don’t know that much about the low levels of databases, and was new to document DB, and no SQL as well. But, what I did know and it was confirmed was that, it’s hard to have, to support the development of not just a database, but one that’s intended to be distributed with sharding and replication. And it’s a product that also has a lower failure tolerance. Customers who are using our products, we’re storing their data. And as anybody knows, your business, no matter what your business is, ultimately your business is your data behind the scenes. And so, the level of correctness that we need to ensure is very high.

And so, when I was first talking to Mark, and when I went through interviews, people were like, well, we have this Evergreen-CI system and I’m like, really? You all have your own homegrown system in this day and age? I’m like, okay, all right. But, I’m not sure I would’ve made that choice thinking to myself, of course, they don’t need to do that. And then, I start to learn, when I get to Mongo, I start to learn the size and scale of the challenge. I’m like, oh my goodness, I cannot see any commercial system that I can think of that would really survive this level of abuse. We’re talking hundreds of thousands of compute hours daily to build in and more importantly test this product on all of those different operating system variations. It is a tremendous amount of effort, and I have a team of very smart, wonderful people who not only keep that thing going, but continue to refine it and add onto it.

And it is actually, I think a differentiator, that lets MongoDB as an organization, do what it does. I don’t think they could do it without Evergreen. But, it’s also frightening, the level of importance then that we have on this system. So, one of the changes we’ve made is we’ve actually, we now have, I brought in more operational oriented folks to help us really do resilience and robustness at production-like levels, because that’s the level of importance that this internal only tool really has, and it’s been a fascinating process. So, I ate a whole bunch of crow, as they say, and I was like, oh, you don’t need that. Oh yeah, we do. We really do.

Abi: I’m sure I would’ve had the same perspective going in, so I completely understand. Would love to really dial into this, maybe from the developer’s point of view. So, you hire a new engineer at MongoDB. How do they get up and running? Do they develop on Macs or PCs? How do you get started as a developer there?

Tara: I think the default, I think most folks have MacBooks, but we are not limited to that. There are definitely folks who have their Linux boxes, either a workstation or a laptop, and I think there’s the stray Windows developer here and there. But, I think from what most developers have, hands on keyboard I think is mostly Mac these days. But, given the cross-platform nature of our products, our development infrastructure is, we have a lot of investment in having available resources in Macs, Intel, and ARM, M1 and M2, that’s all the different variations, power PC and Z series IBM mainframes.

I was like, wait, mainframes are still a thing, what? I had not been aware until I got to Mongo. I was like, oh yeah, IBM there’s, here we go. So, that part has been really interesting too, because as a developer, you’re certainly not going to be developing locally on all of that. So, how a big part of ensuring the developer experience is as performant as it can be, is to try and get them feedback as soon as possible. You made this change and it might look good on your computer, but you should go check this out.

Abi: So, what does that experience look like from the developer’s point of view? So, I assume start a, let’s call it a feature branch, maybe you guys use Git, or do you have a bespoke version control as well? So, you check out a branch, write some code, how do you validate that?

Tara: So, it actually, you just about proprietary, we don’t have a proprietary configuration management system, but a lot of our processes predate the use of GitHub. And so, now when we think about GitHub, we think about, oh, well you create a branch or you fork off, and then you pull a request in and there’s all this great infrastructure for it. Evergreen predates our use of that, may predate some of these features in Git actually, now that I think about, Evergreen’s been around for a long time. And so, we actually, in addition to supporting a Git style workflow and testing by PR, you also have the ability to generate a patch, and even before you do a commit, you can send off a patch and have that be pre-validated to make sure that it’s worth even putting that PR in. Because, it is expensive to do those tests just because of the sheer volume of tests that we try and do.

So we’re, again, trying to ensure that all possible ways of assisting the developer in getting their work done as quickly as possible are supported. Now, if you ask me again, I keep thinking, well, maybe we can go to a peer PR based workflow, but as with, knowing whether or not we should use Evergreen or switch to a more commercial system, continuing to have an open mind and try to really understand, hey, you know what? Actually this is serving our needs in useful ways. We should keep this going. Everything is always in constant state of introspection.

‍Abi: So, Evergreen, you have an Evergreen server running, developer has a patch and submits it to this CICD system. What exactly happens from that point on?

Tara: Well, depending on, so in Evergreen, and by the way, Evergreen is open source, you can go to Evergreen-CI and GitHub and have a peak if you’re interested. It’s written in Go. And I don’t know that there’s a lot of users of it. It’s a pretty specialized thing to build and test distributed database systems. But, in addition to being able, you go in, you set a project, you pointed out a repo, you define what behaviors you want that project to do as far as the different types of tests. There’s also a CLI, this is very helpful when you’re doing those patch builds that I mentioned before. So, you can basically batch up a job and send it off to Evergreen to do. You can also, we actually wrote a plugin where you can hook up a GitHub action, and based on your GitHub workflow, it will kick off an Evergreen action behind the scenes for you.

For example, if you’re using a GitHub workflow and you have something were you need them to test on the Mac, that’s not a default runner that’s available in the GitHub ecosystem. So, you can kick it to Evergreen. And so, it really kind of depends. It’s like you set up your project to support the needs that you have for the Evergreen that you’re developing, and then Evergreen manages all of the resource orchestration and then the task management. And then, we actually spend a lot of time organizing how the output of those tasks are brought back to the developer for debug ability and having smart log scanning and other types of features. It’s a constant battle to try and optimize that stuff.

‍Abi: Yes, certainly. Sounds like you mentioned to me there’s a ton of processing that happens to test across all the different platforms that MongoDB runs on. What does that feedback loop or turnaround time look like for a chain a patch or command? How long does it take to actually validate for a developer?

Tara: Well, it depends. It could be minutes or hours and depending on the type of thing. So, for example, sometimes we’re resource constrained. The beautiful thing about using cloud services is you can spin up as many things as you can bear paying for other types of hosts, we call them static hosts. We’re limited. We don’t have the ability to arbitrarily expand out. So, that gets queued up and those queues can become quite long. So, the actual execution might not take that long, but just waiting your turn to get access to that very limited hardware becomes a challenge. And so, then that starts the discussion of, well, when do we need to do what kind of testing under worth circumstances? And kind of thinking not just technically, but also taking advantage of other types of discussion like coverage and test strategy optimizations to think about how instead of just constantly paying and paying and paying for increased capacity, just trying to be smarter about test dependency management or other types of ways that we might shorten that amount of resource need.

So, where there’s even discussions about bringing in AI to really analyze some of this output to give guidance on when do we need to do what type of testing. Certainly if you look at a build system like Basil, one of the beautiful things about Basil is, it actually has pretty good ways of telling you whether or not something needs to be tested. But, Basil really works the best when it’s a monolithic repo. It’s a little bit harder challenge if you’re talking about 10 or 12 different GitHub, sub, smaller repos. So, everything’s a trade-off. I think that’s the important lesson that, no matter what there is, you’re never going to have the perfect solution. 80% is a really good outcome.

Abi: Definitely a game of trade-offs. I love this journey of what the developer experience looks like for developing and testing. I’m curious then, again, as someone who has never done database development, what does it look like after that? What’s the release process? For example, are there multiple stages that a change goes through? How do you put together releases and what does that process look like? Especially since now I know you have a downloadable product as well as a cloud product.

Tara: We do. We have staging environments even for those. So, they can be spun up and people can poke at them, people can grab them and install them locally or install them on a virtual instance. And then, we have, if you think about Atlas, which is our premier hosted solution for MongoDB, Atlas is consuming the bits that are otherwise released as RPM packages or tarballs. And so, that process also has multiple forms of staging into that. And we try to dog food. I think one of the cool things that I love to talk about is the fact that, Evergreen at one point had the largest known sharded MongoDB installation, because of all the log data that it generated. So, the amount of dog food that we do was pretty tremendous. And we recently actually switched from being a self-hosted to an Atlas consumer, but we’re still using MongoDB under the covers.

So, that’s an element of testing as well, and in production like environments. So, there’s a lot there. And there’s also, we have a whole infrastructure around how we do performance testing, where we grab bits, and then subject them to different types of abuses to make sure that we’re keeping a thumb on latency related challenges. And my team also supports the infrastructure for that. Evergreen is used to execute it, but there’s actually a whole framework that’s used to wrangle how you use those types of resources.

Because, even if you’re using AWS or its cloud resources, you need to have single tenancy, because its performance. So, that again, changes the type of resource orchestration we have to be concerned about. So, the variability of our development environment is tremendous. And the goal is that for our Evergreen ecosystem that most of the complication around coordinating all of that is hidden. The developer just can rely on, all right, I made a PR, or I submitted a patch, and now I know all the right things are going to happen, and eventually I’m going to get some really good feedback about what I just did.

Abi: Well, a lot of companies are struggling to find ways of abstracting Kubernetes from their developers, but this sounds like way more that you’re dealing with from an infrastructure standpoint. I got really excited when you brought up the performance testing infrastructure. I would love to learn a little bit more about that. What goes into performance testing a patch or release of MongoDB?

Tara: Well, I’ll be frank, I don’t know all of the specifics. It’s an area where I’m still learning. But, what I can tell you is, again, going back to the level of investment that Mongo makes around infrastructure. So, my team is responsible for the actual infrastructure and orchestration of how the tests are run so that Evergreen can then do what it does. There’s a whole other team, actually a number of teams that are responsible for creating the test, and this is very challenging. And then, not only that, understanding how to analyze the test results, because there can be a ton of nuance about interpreting the results that you get. And there’s another aspect. So, there’s a developer productivity team that I run, which is kind of the uniform ground layer of the infrastructure. But then, there are product oriented developer productivity teams that are more specialists for their product areas.

And they’ll build tools that based on Evergreen results, are either for correctness tests or for performance tests will automatically create bug tickets. And then, there’s tooling around how to manage that. So, both for our cloud hosted and deployable services, so that there’s this really rich involved ecosystem that is multifaceted. And so, making all of those things work together has been a really interesting challenge as well. But, for performance testing, we’re thinking about all of this, like P90, P50, and all these other data sciencey types of approaches and statistical analysis that are always really interesting. And then, you’re trying to design your test approaches in ways that will cover the maximum amount of potential user use cases, customer use cases, so that hopefully they never are surprised by suddenly seeing a, after a long upgrade, that now they’ve got a degradation and they’re unhappy. We’re always fighting to find ways to make sure that doesn’t happen, which can inform, can cause changes in the infrastructure strategy.

Abi: Absolutely. You mentioned more of these tools that are worked on internally to support other developers, which brought up a question. This is something that we talk about a lot on this show, but approximately what percentage of your, well, MongoDB’s engineering organization would you say is focused on the enablement aspect, like tooling or process, whatever it is, but infrastructure to support engineering work?

Tara: So, I would say roughly between eight and 10 percent-ish of, now, that’s not all my team, but that’s people who, and maybe actually it’s probably 10 to 12% of people who work, focus, who are engineers who focus on non-product facing, but rather infrastructure or developer enablement tooling. So, that’s what I meant, other than Google, because Google’s special, part of the reason that Google is special is because it’s Google. So, the big FAANG companies, Google has a five nine system. And so, you can understand percentage-wise, there’s probably more people working on Google’s infrastructure than are in engineering at MongoDB. But, at MongoDB size, you don’t often see that level of investment. And so, that was a thing that excited me so much to realize, wow, no, Mongo takes this very seriously.

‍Abi: It definitely speaks to the culture around developer productivity as you’ve been articulating. I wanted to ask you, my takeaway so far about developer experience and infrastructure for databases is that it’s hard, which is what you said at the beginning, but I’m curious, as you’ve worked in this domain, what are some lessons you feel like that can be taken away for non database companies? What are things that you’ll take away when you go work on something else next that you’ve learned from your experience working on databases?

Tara: Well, if anything, I think being at MongoDB has reinforced my view that a lot of companies give up too quickly and do not see the value, the business value proposition in doing this level of investment. And to be clear, and we didn’t really touch on this up until now, but I do want to call out, developer productivity is not just about tools and services, it’s about culture. It’s about how you communicate, it’s about how you do bug tracking, it’s about how you do documentation, and what is your developer environment? It’s all of those things. And any company can benefit from having, putting investment, you get an, the ROI on it is very measurable.

And so, the TPM organization that we have here at MongoDB, we partner with very closely, so that it’s the tooling in conjunction with processes, and communication, and documentation are coming together. And that’s another element that I just, every day I’m just impressed by. It’s not perfect. No company is ever going to get this perfect, but again, that 70, 80% you’re doing really well. And I think we are so far past where other companies could be if they chose to be. And if we can do it with the complexity that we have with the type of product that we have, there is no excuse for other companies not to adopt the same approach.

Abi: Well, I’m thankful that you brought up culture. Because, I know we were chatting before and you mentioned one of the biggest things you’re focused on is changing or influencing culture and how the organization thinks. You talked about some of the concrete pieces that are involved in that and how you’re working with the TPM organization, but you also mentioned to me that there’s real challenges with doing this at a 10-year-old company. So, would love to know, what are some of the challenging aspects of affecting change with when it comes to culture?

Tara: Well, rule number one, engineers hate change. And so, there’s a couple of things. One, you get habituated to something. And even if that thing could be replaced by something better, you have to evaluate whether or not the cost of disruption is actually worth it. So, that’s one thing is to understand, you think that you can replace it with something better, but can you quantify it? Can you demonstrate that value, so that it’s worth a developer to go through the learning curve to change? So, that’s something that’s really important for any platform engineering or dev organization to focus on. And the other challenge that you often run into is, and every company I’ve been at has elements of this, which is the build versus buy. Most engineering teams, you’re going to reach for the build, you’re an engineer, it’s your toolkit, let’s build it.

And at the time, and Evergreen is a great example. Evergreen started to be developed when there really weren’t any other options that met the need. And to be honest, I’m not sure there’s still other options. But, there are other surrounding tools where, why are we still maintaining that? Maybe we could replace that with just something that we get from a vendor or download from a reputable open source community. And again, you have to evaluate, is the benefit that you get from changing worth the disruption cost? And every single time you want to think about some of those things, you have to have those conversations, and you have to prepare yourself and have your arguments and your data ready in order to do that because again, engineers hate change and they need to get their job done.

So, really understanding not just this is a good thing for us to do, but within the context of this is a good thing for us to do. In the long run, it’s going to be better for our business, and it’s actually going to be better for you personally. Please trust me with all of this data that I’ve accumulated on your behalf. Again, 30 years, I’m having the exact same conversations. Just the details are different.

Abi: That’s funny. Well, you earlier described the types of things that are encapsulated in culture, but as you joined MongoDB and started to identify culture as an opportunity, what were some of those very specific opportunities you saw that you felt like there was high ROI and trying to impact?

Tara: So, I think the one that’s coming to mind the most is, because there was already a culture around investing and testing, and that is held high. I think building on that and in identifying collaborators to up-level testing, it’s like, hey, let’s explore ways that we can make your test turnaround time go down. That is the opposite of a cold open. Any developer, you tell them like, hey, you’re doing all these tests. These tests are important to you. Let’s see if we can make them happen faster. That kind of thing is fantastic. Historically, it’s been harder to get developers interested in writing tests in the first place. I am not going to name names, but I was at a company where I had the VP of engineering look me dead in the eye and say with a straight face, how can my developers write tests they haven’t finished implementing yet?

I’m like, oh dear, we’re going to have that conversation. The fact that it never even occurred to me to have this conversation at Mongo is, it’s a blessing. I give thanks. So, I think that’s probably one of the biggest opportunities. Similarly, different types of tests. And then, also, people are interested in understanding the value of automation around developer productivity. They see the value, and the developers within the developer productivity organization are often frustrated, because these are hard problems.

How to get 600, 700, 800 developers going in a direction is never an easy task. But, at the same time, there is I think a culture of respect around the value that those teams bring. And so, when those folks come and have ideas they’re engaged with, so it’s like, ah, that’s just the dev prod people, don’t worry about them. That is not a thing that I see at MongoDB. Or if it happens, it happens where I’m definitely not seeing it, because I’d have words. But, that idea that there’s value in investing in exploring new tools and new opportunities, that’s also a way that we can build on to make things better. But, telling them, hey, you can’t use a ID anymore. That’s something I try to avoid.

Abi: That’s a tough one for sure, or trying to get people to switch from BIM to Emacs or something like that.

Tara: No, we don’t need Emacs. We already have an operating system.

Abi: One of the things you said to me earlier was that, this quote, I wrote it down, how you ship is constrained by how your code is organized. How do you see that playing out at MongoDB, and is that something you’re trying to change the way the code is organized to change the way engineers ship?

Tara: That’s one of those things where my team is less involved with that particular conversation, but it is a conversation I see happening. MongoDB has been around for a long time. A lot of the code structure has been around for a long time. And so, there are very smart engineers that are now looking like, you know what, maybe we should rethink how this is organized. Could we break some of these things out into more logical grouping so that we have more efficiency and agility around touching this bit of functionality versus that functionality? And, I think that’s happening in a variety of engineering teams.

And then, that starts to make you think about, what should your internal API strategy be? What’s your dependency management? And I think there’s, again, that constant low grade recognition. And I call it low grade just because usually we talk about feature development. But, there is that constant low grade recognition of like, hey, there might be an opportunity to reduce technical debt to reduce the cost of adoption, or the reduce the cost of interoperability between these two things if we were to restructure in this way. And again, that comes back to that culture of always looking for opportunities to create improvements in all aspects of our product offerings.

Abi: Those types of restructurings can be such a big chunk of work to bite off, and sometimes a little bit risky too. I was having a conversation with head of developer productivity at Notion, and they were also thinking about, it was something that was on the radar, felt big and risky, but they definitely saw this opportunity to completely change, split apart how the code base was organized to unlock much higher levels of productivity. So, I’m curious, how do you at MongoDB, is this something that’s going to be multi-year or are there low hanging opportunities for this?

Tara: I think the answer to that is, yes. I’m a firm believer that, as much as possible you do things incrementally. Anytime you try to boil the ocean, you’re going to be in a world of pain. Or as one person said, anytime you boil the ocean, there are going to be a lot of dead fish. So, we don’t like dead fish, they’re smelly. So, incremental improvements, one, are less disruptive on a case by case basis. And then, two, allow you to be able to evaluate, are we going in the right direction? So, are there teams that have identified low-hanging fruit? Almost certainly. Like I said, I’m not directly as involved in some of those conversations, but you think about the safest piece of software is the one that is already in pretty decent shape and you don’t touch as much as possible. Change is what introduces risk by its very nature.

So, separating out maybe if there’s a single module that actually constitutes two very divergent levels of churn, were you can break the part out that doesn’t need a lot of churn, and that’s just sitting there and available, and now you have all of the churn in a much more constrained way, that’s an easy win. And again, you do it incrementally over time and make sure, does that work? Maybe it ends up not working. Maybe there’s a reason that, from an optimization standpoint, we’ve now introduced latency that we weren’t anticipating just by having that tiny little like API interface. And then, the things like distributed databases that have highly performant requirements, sometimes stuff like that will come up. So, I think that this is going to be an endless review.

And as we bring in new types of products and services into the overall ecosystem, it may be that something that was great for this set of products is now actually not as good as we think, now that we’re trying to bring in this new technology. And if the history tells us anything is that, tech is constantly evolving, constantly changing. And the more you can try and future-proof your code structure, organization architecture, API strategy, et cetera, that makes it adaptable, the better off you’re going to be.

Abi: Got it. One of the things we were talking about earlier is this, the amount of investment in internal tooling and infrastructure at MongoDB. And MongoDB has been doing this for a really long time, and now you’re focusing on this as well. So, one question I have around that is, what’s the culture like around building for internal users? What’s the relationship like between these teams building the internal developer tools and their, I’m going to call them customers, internal developers. Are there product managers? How are they getting feedback? What’s that culture look like and how is it evolving?

Tara: There’s product managers, there’s project managers, and our R&D organization is widely distributed. So, we’re in Slack. We also have a culture of Slack bots. We have a strong culture, we have a strong writing culture. Probably one of the strongest I’ve seen. It’s almost intimidating for how we propose ideas, very Jira centric. There’s a lot of Jira workflows that the TPMs build in. So, it’s every facet of the engineering culture has a pretty strong element of intentionality around it for how we can try and track all of the different moving pieces. Now, again, going back to there’s no system that’s ever perfect. There’s a lot of initiatives that we’re having right now to try and make improvements, because we’re always striving to do better. And so, that’s another aspect of something that we’ll probably never feel a 100 on. But even the fact that we’re willing to invest that time, I think is indicative of the positivity around this idea of developer experience as being a key value.

Abi: Again, I speak to so many leaders in similar roles as you and I have to say that, but the culture you have at MongoDB, the buy-in, the investment, the focus around developer productivity, I think a lot of other leaders would be really grateful to have the same circumstances at their companies as well. So, that’s awesome. I wanted to conclude by asking you about Evergreen-CI. And we talked about how it’s open source, but I’m curious around the story behind that or the strategy behind that. As you mentioned, you’re not sure how widely applicable Evergreen-CI is for other companies. So, just curious, is that more of a building and public thing or something that MongoDB’s customers appreciate or a lot of other companies using it? What’s the vision behind that?

Tara: Well, it being open source predated my arrival, but what I understand was the thought that, it might be beneficial for MongoDB customers in particular. One of the things, I think there’s two things that actually that Evergreen does that is pretty unique to our circumstance that maybe are not as relevant to other users. And this is not just a philosophical approach. But, because we have that cross-platform challenge and that needs, and again going back to the 30 years ago, I can’t believe we’re having this problem again. There’s that platform variant strategy and the amount of effort to try and optimize those needs across the various different product operating system chip set matrices, it does that very well and at scale. And then, the other thing that it does that’s really interesting that I think is maybe not as necessary for other folks, or they may not see the value, or not be wanting to pay for the value, is the ability to interface with those resources as part of the debugging process.

So, you can actually, if you see a problem and you have your little MacBook and you see a problem, oh, something showed up on Windows and that Linux variant and this other thing over here, you can actually get access to a development environment, a spun up dev environment that gives you that debugability very easily. But, the resource management for that is challenging and probably not worth most commercial systems. And the security element of it is, of course, also challenging. You have to manage that. And so, I think if someone really has a truly complex system, distributed system that they would need something like that, then maybe Evergreen will be something that they be interested in, that multiplicity, and that variability, and the willingness to pay for that type of functionality. Because, at the end of the day, that’s all stuff that we have to pay for in the system. So, I don’t want to talk about my hosting bill.

But, it really is an example of, I think also a team that has, as is the R&D organization, that knows what they want and values the opportunity so much that we’re willing to sustain and constantly try to improve that environment. Whereas, an off the shelf type of solution, you can only customize it so far, and it’s going back on my preferred goal of buy first and only build if you have to. When it comes to what I see us able to achieve with not just Evergreen, but some of the surrounding customizations that we have, it is actually in fact in support of something so much more than just straight build test CI. It’s truly that build test CI element as it fits in this broader culture. And that’s the thing that I think is truly unique about it, is that it’s really tailor-made to something that is the full three pillars of tooling environment process that we talk about with developer productivity.

Abi: I love that. I love the way you characterize that. And thanks so much for taking me behind the scenes with Evergreen-CI, and the developer experience at MongoDB, and how you arrived, and the culture of developer productivity at MongoDB. I think a lot of listeners will be really inspired by this and interested in just the under workings of database development. Tara, thanks again so much for coming on the show. Really enjoyed this conversation.

Tara: Thanks for having me, and thanks for letting me brag about my team for a while. They’re a really amazing group of folks who do some truly stupendous work.