Abi Noda: Well, Gilad and Amy, awesome to have you back for a second episode to continue covering your story. Thanks so much for your time today.
Amy Yuan: Good to be here.
Abi Noda: So last time on the podcast, we talked a lot about the shifts you’re making, how your developer productivity initiative came to be, how you’re thinking about measurement. Today, we want to talk about how you actually execute, and you guys suggested this. You said, “Hey, we should talk about operational excellence,” and it took me a minute to understand what you actually meant by operational excellence. I’ve come to learn what you mean by operational excellence is really around communication and how do you engage with the rest of the company. But would love to hear from both of you, how would you define what you mean by operational excellence?
Amy Yuan: Gilad, why don’t you go ahead. You are the PM.
Gilad Turbahn: All right, so if you think about it, we’re starting from the baseline of what does it take in order for the developers in the company to really trust us. That’s actually the baseline and in order to develop that level of trust, the very basic thing you need to do is say you’re going to do something. Make sure you do it at the time you said you’ll do it, and make sure that you listen to the feedback.
So when we think of operational excellence, the very baseline of that trust is to say, “Okay, this is a set of products and we treat ourselves like a production service.” We want to make sure that everybody knows, hey, we can trust any other product that you buy off the shelf. We can trust that if we need our IDE to work, it’ll work. We trust that if we need our cloud workspace to work, it’ll work, like electricity. And if not, if there’s an issue, we know exactly who to go to. We know that they’ll respond, that they’ll listen. And we know that if there’s a bigger issue, they’ll fix it. And we know that they care. So I don’t need to worry about that. I can as a developer, focus on my work. That is actually our baseline for operational excellence.
Now Amy, I think you can talk about how did you put that into motion with the organization itself.
Amy Yuan: Right. Because behind the communication you need to back it up by actually high reliability, which goes back to treating it like a production service. That means we actually have SLA, we have dashboard for operational excellence, we have operational review, we have post-mortem. And really needs to become the team’s DNA that we have real customers. They’re not paying customers, but these are customers that use our service. And that is so ingrained into everyone, it goes into not just running the services itself, goes into documentation as well. Through documentation also like production documentation.
Abi Noda: Before getting more into what you’re actually doing. Both of you guys are so passionate about this and so rigorous in the way you’re thinking about this. I’d love to get your views on both based on past experiences as well as talking with other peers in the industry. What do you think is the biggest challenge or pitfall in terms of executing in the way you were talking about?
Amy Yuan: When we talk to our peers, we often hear about two big obstacles. One is to get leadership support to really align the entire organization in believing this is the right priority. Because if team A wants to do something and depends on team B, team B doesn’t think it’s important, it would never get done. And then the second obstacle, a lot of times when you’re pushing a new thing, is adoption. Even though it is markedly better experience, it still takes a lot of convincing of developers to change that behavior. That’s a baby duck syndrome, right? You are married to the first idea you use or the first whatever tool you use, even though it’s not ideal, it’s very hard to convince them to switch. So those are the two challenges we hear all the time. And these are definitely things that we need to work on ourselves as well.
Gilad Turbahn: I’ll add two more. Amy touched on one of them, I’ll touch on it in a slightly different way. If you think of what we do from a product life cycle perspective, especially when it comes to migrations or adoption, you really do have all the different stages that you have with any other product. You have the early adopters, you have the chasm that you need to jump. You have the early mass majority, late mass majority. One of the things we’re dealing with right now is the laggards for some of the products. And when you’re dealing with laggards, your strategy needs to change.
Before what we used is a lot of carrots, now we’re starting to use a little more stick to a certain extent. We’re using a little more tops down by having directors reach out to their teams, as opposed to trying to show just the benefit of stuff. We’re putting more firm dates to say, “Hey, this is when this system is getting shut off.” Which is stuff that we definitely wouldn’t have done before, depends on the system. So that’s one thing that we’re absolutely doing.
The other thing is we talk about this operational excellence. The level of trust varies over time and is varied in different pockets as well. So we have some areas of the company that already got to a point where they’re like, “Okay, these guys are on it.” We’ve got other parts of the company that still think we’re a bunch of bozos. Fine. Then some of the work that we’re trying to do is to make sure we cater to all of them. We go and talk to each one of the teams separately. We do a lot of the listening there. And it’s not a one-size-fits-all and that was a key part of the learning. We had to make sure we know what are the specific challenges each team faces, and that we’re showing that we’re listening and then we’re jumping on them. And especially saying, “What are we not doing?”
I’ll mention roadmaps for one second and then we can dive into it later. The goal of a roadmap, especially in internal tools, is not just to show, here’s what I’m working on right now. It’s also to show, here’s what we chose not to prioritize right now. Doesn’t mean it’s not important, it means that there are other things that we see are more important right now. And when you share that with customers and you can get their inputs, then you’re developing that level of trust of, “Oh, they’re thinking about it. There’s a line of thinking behind it, there’s a rationale. Maybe I can convince them to switch this item with that item.” But at least you have that bidirectional communication. That’s how we establish the trust with the operational excellence and how we work day to day.
Abi Noda: So we’ve talked about what the overall goals of operational excellence are and some of the biggest challenges and pitfalls. You guys at Snowflake are doing so much to execute in a thoughtful and deliberate manner. When we were talking a couple of weeks ago, midway through the conversation, I just thought to myself, wow, you guys are doing so much. Can you create a doc that just lists it all? So myself and maybe people in the world can fully appreciate what you’re doing. I would love for you both. We’ll dive more into communication, goal setting, all those kinds of things. But in broad strokes, maybe give an overview or a list of all the different things that go into your rhythm of business and playbook.
Amy Yuan: I would probably summarize the rhythm into five different things. One is that we have the weekly newsletter and biweekly part all hands where we meet all the engineering team in person. Second is every quarter we have a survey and that result is super critical in our planning and prioritization. The third one is we do lots of interviews at different levels, and GM do it, the MTL do it, I do it as well. And the first one Gilad mentioned is the road show that we do at all hands at the different teams, so that we can reach that team, give very customized information and get feedback. And the last one is again, drive adoption. And it’s top down and bottoms up with different approaches depending on the life cycle of where we are, and also vary sometimes by team as well.
Abi Noda: There’s so much work that goes into this and we’ll dive deeper into specific examples. But overall, how are you finding the time? Who does all this work? Is it the product managers? Do you have admin assistants, program managers? How do you actually do all of this customer engagement?
Gilad Turbahn: The real answer? Us, all of us. There’s no definition in engineering systems saying this is the PM’s job or the TPM’s job or the TL’s job, and let’s throw it over the wall. We’re all busy, right? We’ve got a ton of stuff.
Amy Yuan: Yeah, it is basically part of the rhythm, part of the DNA. Be super intentional. A lot of things are built into our rhythm now. Every week we have ops review and we’ll also review communications and feedback as well. So once you do it as part of your daily, weekly, monthly or quarterly rhythm, it’s built into the plan and everyone does it.
Abi Noda: Amy, you had asked me in our last conversation, do other companies do this? Because in our view, it’s very necessary, right? I wanted to ask you to expand on that. I mean, in your view, what would happen if you didn’t do all of this?
Amy Yuan: So Gilad was there last year, so I can share definitely the before and after picture. And I have done similar things in my previous jobs as well. I think if you don’t spend the time really to build that connection with customer, build trust and then have that close feedback loop. Is that customer will feel not heard, or worse, abandoned. They just don’t know if anyone is working on this, if things are getting better. In the case when you’re building a new solution and you’re probably not spending as much time working on the old solution, which people are still using. Without that customer engagement, people will lose trust. Lose trust that things are getting better, lose trust that anyone actually cares, lose trust that this team actually knows what they’re doing.
So a lot of that bring the customer along on your journey to have them feel like they are stakeholder, that they have a say, have an influence in how things will get better. And then have autonomy over priority of things. That goes a super long way in making sure again, we’re working on the right thing. If we working in a silo, I think engineers tend to work on shiny new things and also tend to work on things that in the end actually might not move the needle in terms of customer experience. Even though in their mind they build this beautiful new infrastructure, beautiful new tooling. So that close connection to customer again, goes back to treating internal tooling like a product. And that sort of culture shift, mind shift is very important to the success of our service and our team and then to be really trusted by our customers.
Abi Noda: I want to start diving more deeply into how you do different aspects of customer engagement. Something you shared with me a couple of weeks ago that I thought was really insightful, was you mentioned that you try to plug into existing communication channels just like you plug into existing developer workflows. Say more what you mean by this in examples of how you’re doing this.
Amy Yuan: Okay, I’ll give you two examples. One is when Gilad and I went to visit our developer center in Berlin and Warsaw, we can see because of the time zone difference, a lot of the breathing that we do that we thought was global actually didn’t really work for them. We presented a product all hands that’s in U.S. time zone. They actually don’t watch that and I don’t watch the recording either. So a lot of updates that we thought we’re given to everybody was actually not landing with them. Instead, they have local rhythms, the Berlin office, they have a Monday, 30 minutes stand up site-wide every single day, every single week, and that was actually the best way to engage with the whole site, 150 engineers. So the next time when we had a survey, when we have an update, we actually use that forum. That was super effective.
And the other thing is, part all hands was attended by, supposedly open to thousands of people, but not everyone attends. People mostly attend their monthly all hands in their team group, and that is usually between 50 to hundred people. And that is actually the most effective way to engage at more of a local level. So that’s what we started doing. So in the last two quarters we attended about 12 different team all hands. And then before we went to the team all hands we actually talked to the key influencers in those teams to understand their scenarios, their user journey, their are pain points. So that when we go to the all hands, it’s actually a very tailored message. We’ll say for this product, “It’s a great life changer for this team and this is a current adoption. And this is what other early adopters saying about their experience. And these are the laggards that we love for you to join the party and give us feedback so that we can make all of you life much easier.”
And at the end of the all hands, we always do call to action again, to summarize the places where you can get help, the new documentation, how to check adoption, what things have gotten better, how you can give us feedback. Just always keep that dialogue going. And then every one or two quarters, then we give a fresh update as well.
Gilad Turbahn: Amy mentioned going to the separate teams. It’s something that Amy, I believe you’re the one who ended up bringing this to the table to begin with and then we all jumped on it. We’re like, “Hey, there’s a communication vehicle, let’s use it.” I think we are almost underselling that piece. That is such a critical piece in order to change how our customers think about us. When we’re talking about bringing tailored content, it means looking at the data for that specific organization and seeing, how many developers migrated from system A to system B, in this specific organization? Where are they at compared to the rest of the company? Where are they at compared to other similarly sized groups?
There’s the aspect of having the listening session specifically for them, and then showing how what they’re asking for fits or doesn’t fit within the broader needs of the organization. It’s making sure that we bring testimonials specifically from that specific group so things can resonate with the people in the room.
Amy Yuan: Yeah, and also in those meeting, I think the thing that resonates with them is that we are very data driven. Again, we show all the data from operational metrics, user journey, user experience metrics, output metrics, and then the quarterly survey metrics so that they know we actually understand their journey, their experience. So that again goes back to building that trust and confidence.
Abi Noda: What are some of the more notable questions or feedback, even criticisms you’ve gotten in these types of presentations? I’m really curious to know what do you get back when you’re in front of folks and sharing updates on your work and the data asking for things? What do you get back?
Amy Yuan: With any migration there are always two camps of supporters or detractors. So based on migration I think at Snowflake or and any company that we have talked to, the decision is always controversial at the beginning because basically it’s not a tool that works for all the programming languages equally well. And again, going back to the baby duck syndrome, people are married to something they’re very familiar with. And when you’re doing the migration, something always not going to be quite on par. And even when you’re done, you probably still have work to do to make it better. So then in this journey we always have people who are supporters or detractors, and it’s always a journey to make sure the detractors also feel heard, that we’re not just steamrolling them, that we will make their experience better on par. And this is just again, a prioritization exercise, so we get that quite a bit.
Abi Noda: How do you influence and convert those detractors?
Amy Yuan: I think a couple of things. Again, the detractors, they’re developers too. They do understand it takes time to build something and it takes time to perfect something as well. So a lot of times they actually help us in various different ways. So for example, in the phase of migration we want to improve build time. But improving build time is not just in the tooling, it’s also in the product itself. So there are people who work on making the product boot faster, making the unit test, IT test, integration test, the infrastructure to be better, to be more lightweight, to share some of the loading things so that we can be more efficient. So those are the people at the beginning they might not believe in based on migration, but then they are actually part of the bigger effort. And then they actually help the end-to-end output time.
We talked about in the last podcast that the transformation is not possible without army of volunteers across the company. It’s not just our work at all. We have tons of volunteers and those are the people that really help move the needle so quickly in six months, because they are experts in that domain, and they believe in the mission and the whole company is aligned in that priority. So once they again are part of the journey and they see the effort, they see the result, and then it’s easier to bring them along on the journey with us as well.
Gilad Turbahn: I’ll bring another example from a slightly different world, specifically on the PM interactions. There’s a big debate in the industry, do PMs need to be technical, yes, no. What are the advantages, disadvantages? In this space, the answer is 100%, absolutely yes, but here’s how rubber meets the road. The way to convert detractors or laggards is to have somebody who’s a PM and understands how to ask questions and understands how to identify underlying pain points. But at the same time can launch the right environment and experience the pain with that person, and really go through all the steps and replicate and have the discussion back and forth. The folks on my team, I’m very lucky to have folks that can absolutely do that, and that enables you to establish that trust that okay, it’s not some PM that just wants to ask me a bunch of questions and then he or she has no idea what that actually means in practice. No, they will go and they will try to recreate it or reproduce it, and they’ll try to see do other people in other sides of the org face the same thing.
That’s how we jumped on a bunch of stuff. One of the examples was the experience of using C++ inside the IDE. Right now that’s a top concern area. So one of the PMs on my team just went and interviewed a bunch of folks, tried to recreate exactly what they’re facing. Saw where the IDE sync times were very problematic and that’s how it enabled folks to say, “Okay, they’re not hand-waving. They know exactly what we’re going through. They’ll probably solve it.” So that’s another way to bring the laggards or the detractors.
Abi Noda: Another insight you shared in a previous conversation of ours was the advice of using insiders as your messengers. Elaborate on what you mean by this, and examples of how you’re doing that.
Amy Yuan: So a lot of times developers have a default higher trust level with people who are on the same team, because they feel like they understand their lived experience. Versus someone who is outside the team, central team, for example, like a yes team, to tell them use this and use that. So this is why it’s very important for us to build the CAB, the customer advisory board. And these are the people who are early adopters who are so eager to make things better, they’re eager to try beta things and give us feedback. And again, we had so many volunteers and they saw how sausage was made.
During the two quarters when we had so many volunteers embedded into making the cloud workspace better, into making CI better. All of the feedback they had later on was, “Now I have so much more appreciation about the complexity, both in the technical space as well as in the organizational space.” And then they become very natural passengers going back to their team to, “Hey, this is something actually I worked on,” or, “This something I used and I can tell you this is much better.” And going back to the example of the bling team, how they’re a little bit disconnected because of time zone difference, so they had earlier adopters as well. So their early adopters then used their Monday scrum to again demo cloud workspace to show them how this had improved his productivity by 2X or 3X. And again, after that demo we’ll see adoption for that site spike.
Gilad Turbahn: I’ll build on what Amy just said. It’s not always people with a specific title that are the influencers. It’s not necessarily the TL or the EM or the director. In many cases it’s the person who’s been there for a bit of time or just… One of the things we did when we went to Berlin, when we went to Warsaw, when we talked to the teams in the U.S. is to go and try to identify those people trusted by the teams who are those influencers and tried to spend a lot of time with them. We knew that that investment of time, especially if we are able to convert a detractor who’s an influencer into a supporter, and there are multiple ways of doing that. Anything from a lot of chocolate and bribes, no, I’m kidding. Anything from listening and showing how the sausage is made, like Amy just said. And talking about where we act right now and giving a true representation of the current reality without making it prettier or putting lipstick on the pig. And also showing what the future can look like, there are a lot of different ways.
And once we’ve spent that time, they don’t necessarily get converted into huge fans, but they can get converted to people who trust you enough to get their team to listen when you’re trying to push a different change, a different enhancement. So we use a lot of those as well embedded in the teams, and that’s both the early adopters and also getting from say early majority to late majority, use those folks in order to get other people to try because they trust those folks on the teams.
Abi Noda: I really like the insight you shared around how when you spend time on all hands or site visits, that you go in there and deliberately try to identify and build relationships with the influencers. I think that’s such a key step. You can’t do this type of customer engagement. You can’t communicate through the insiders unless you first identify who those insiders are. And I want to also double click a little bit on the customer advisory board. You just brought this up. Can you share with other folks who might be interested in doing something similar, how does that work? Who’s in the customer advisory board? How do you get people in or out of it? And then what’s the cadence of meeting? What are the activities, responsibilities? How does the customer advisory board actually function?
Gilad Turbahn: Sure. So let’s talk about the concept at large and then talk about how it works at Snowflake. So the concept of a customer advisory board is something well known in the industry and PMs fell in love with it a long time ago. The idea is very simple, in order for you to get a good picture of different types of customers at different stages of life. And get all of them together to talk to each other so we can understand the underlying pain points. And also share some of our longer term thinking or specific designs, for all sake of the matter. It’s the two-way communication with that trusted group of advisors. What you usually do is you identify folks from different parts, in our case of the company. When you’re talking about external products, different types of customers of different sizes.
So you get a diverse group, usually you vet to them to make sure there are people who are vocal and want to communicate. But are not necessarily just your promoters, you absolutely want detractors in the room and you want the balance. And then you need to do a really good job at moderating that forum such that you can get stuff out of it.
To give examples of what you do in that forum. It’s anything from let us share our roadmap or our upcoming quarterly plan. Tell us what you guys think. Let us show you a design of the new pre-commit report that we’re generating. Let us show you the design, tell us whether this is going to solve the issues that you have. What’s top of mind for you? There’s some input questions, so what’s top of mind. Sharing the insights from the survey and trying to get those types of information. So that’s how it works and what exists.
Specifically in Snowflake we’re actually revamping our CAB right now. The investment is both for a monthly meeting, it’s usually 45 minutes once a month, always the same. I think it’s like the third Thursday every month, just so it’ll be stuck on the calendar and everybody knows when it usually is. And the idea is to bring folks from the different disciplines and have the conversation in the room. We also use those folks offline. We have the CAB forum specifically on Slack where we can reach out to them and say, “Hey, we came up with this new concept,” or new migration plan or new communication, “Can you review it?” And then we get engagement from folks reviewing it offline.
How do you get the members? It’s anything from, we did a customer interview and based on the conversation we had we’re like, “This is the right type of person to bring into the CAB.” They get nominated by the directors on the team, because they might be the right influencers or the right folks to communicate back and forth or just the directors want these guys to represent the teams in front of us. So we’ve seen both work.
And then finally the entire goal is, it’s informal. There’s no, you’ve got to attend every single meeting, you’ve got to be… No. The idea is these are people who care and want to make the company better, and therefore they will come and give us those insights. We give them insights internally to what we do and how we operate. So they do get purview into the sausage making, to a certain extent, and the thinking. They get early access to stuff. We’re playing around with an LLM bot right now for documentation. Those guys helped us dive deep and figure out what works and what doesn’t work. That’s how we leverage it for hands-on stuff as well. That’s the idea behind the CAB.
Abi Noda: Well actually several times today you’ve brought up the Berlin and Warsaw development centers. I know fairly recently you actually visited these folks in person. You traveled to Europe to visit these sites. Tell me more about this and why it was important to go visit in person.
Amy Yuan: Go ahead, Gilad.
Gilad Turbahn: So let’s first talk about the importance of being there and not just for us, not just for me and Amy. We had additional folks travel from the U.S. to those offices and when they travel, we had the folks on the performance team, our performance team travel. But when they travel, they don’t just represent the performance team, they represent our entire broad team focused on developer productivity. They will listen to things. Sure, they’ll have the specific meetings, but they will listen to things outside their purview because we want to keep that constant touch.
When we went there, the idea, so we went there earlier than when most of the team went. We sent folks after we did in order to establish that basic trust and in order to get folks to know that there are faces behind this organization that’s responsible to doing work. But that’s why the follow-up was so important. For instance, when we sent our tech lead specifically on the CI side, he did a whole listening session to things outside of the CI, just to make sure that we really show, hey, here’s how we’re acting on the stuff that you said. That constant touch makes people feel like they’re not on the other side of the world and they’re not forgotten.
I will still reach out to the developers that I met during that trip to make sure that what they’re getting is still the right type of service, and we’ll do all those follow-ups. But you can’t replace the in-person. Wherever we can, we absolutely need to be there in-person. Because out of sight, out of mind, that’s usually the feeling. That’s what we’re trying to avoid. So giving that airtime, being there in-person really made a difference for us. We can see it. Survey participation rat, the fact that I now have the relationships in the Berlin and Warsaw offices made us have much higher participation rates. We know the rhythms, we knew how to reach out. That worked extremely well.
Amy Yuan: And the other part is really center sometimes has different challenges. Berlin and Warsaw, they’re not as close to our build one in North America, not as close to our weeping gateway. When we visit, for example, the Toronto office, we feel that experience is more on par with the rest of the U.S., both in terms of support and in terms of infrastructure. So within those sites, for us, we saw a few things. The time zone difference definitely becomes a lot clearer when you are there. As they end that day, that’s when the U.S. teams come online, you get flooded with Slack messages and email.
So then you really need to make sure we give them local support for the questions they face. And also how we can communicate and work more asynchronously so it doesn’t become a big burden. Because when you’re there, you understand how that time zone difference is such a drain on work-life balance a lot better. As well as any challenge they have around their local office setup, ISP, how that creates latency. And also just local teams, the areas they own, sometimes it’s a bit different from the U.S. that can change their experience as well. So that understanding creates empathy, creates connection, creates trust. So that’s why we went.
Abi Noda: We’ve talked a lot about how you are engaging with developers, the folks who use your products. I want to ask you about managing up. Ultimately executives are the ones funding all of your work in your organization, so how do you approach that?
Amy Yuan: I think for us, we were very fortunate. A lot of that was actually built in, which was how you and I were hired, how both of us joined the organization as well. The difference I think, from what I see in Snowflake and in other places is a very engineering-driven company. Our co-founders, Benoit and Thierry to this day, they still write a lot of code. So Benoit likes to say last year when their productivity had so many challenges, he stopped writing code and he was sad. And this year after so many improvements, things are so much better and he’s more prolific writing code himself again and he’s much happier. So with the leadership being so hands-on and understanding that lived experience of developers, of course they’re going to prioritize the work.
And also they understand that the work here is a multiplier in terms of impact and efficiency for the entire company. That’s why even when I go to again, the all hands and talk to the union directors of product teams are under pressure to produce features, they say their productivity is P zero for the entire company. And that’s why when our CTO had the call for volunteers, when we started the type faster initiative, all the teams they volunteered. And they didn’t give a volunteer who is a college grad who didn’t know how things work, they gave principal engineers who are veterans, who know everything inside out, who can really move the needle fast. And that’s what made it possible, is that leadership support and the entire company being aligned on the same mission.
Gilad Turbahn: I’ll add a different side of managing up that honestly, Amy is a master of. We talked about trust before, so I’ll mention trust again. What do executives usually tend to care about? They care about bringing the right folks in that care enough, that are competent enough, and they can get stuff done such that the executive doesn’t need to worry about it anymore. If there’s a question, they will know how to answer it. If there’s a challenge, they’ll know how to deal with it such that they can focus, the executives can focus their attention somewhere else. That’s what executives usually tend to want.
Our job throughout the last, I don’t know, six, eight months, was to get to a point when there’s enough trust that if a meeting goes shorter, not longer, it’s because these guys know what they’re doing, let them run. They showed us the plans, makes sense. Go. That’s where we wanted to get to. In order to do that, you need to do a bunch of stuff. First of all, you need to be on top of really truly understanding the space. Understanding the industry and how other companies compare. Understanding your own data and what does it actually mean. Going deep into the metrics, showing that you’re always on top of stuff. And every time that there’s a question about whether we should double down on CI right now or invest in taking all the company offline for a day and do a dockathon, we always have a thoughtful answer. One that takes into account multiple lines of thinking such that again, we have that trust.
We do our monthly meetings with our CEO, with Sridhar and the entire ELT to review the progress of this investment. And in those meetings, a lot of the work doesn’t happen in the meetings. A lot of the work happens in the two weeks before where the entire team is working on making sure we compile the right level of data. And we don’t see it as a burden, we see it as a way to get us to be on track and be accountable. And we see it as a way to make sure it stays top of mind for our leadership, but the team is protected and they can focus on doing the actual day-to-day work. So sometimes we end up taking some of the brunt of scrambling to get the right slide in place, but the idea is to manage up so there’s enough trust so the team can focus. Instead of building all that trust, they can focus on doing the work as much as possible.
Abi Noda: Final topic I want to cover today is how you set goals in roadmap. This is something I’m seeing a lot of different companies have different approaches. It’s always a challenge. First thing I want to ask you, what is the cadence in which you are putting something out there saying, “Here’s what we’re doing, here’s what we’re going to…”? Is it happening on a quarterly cadence similar to your survey or on a different cadence?
Gilad Turbahn: So our planning at Snowflake is quarterly and there’s also annual planning. We’re actually in the midst of the annual planning right now, and then we do quarterly planning to be more granular. The way we set goals is a combination of things. First of all, there’s always, so we use OKRs right now and there’s always the long-term goal in there, our north star, which right now at least part of it is around the developer sentiment. Because we want to make sure that we do measure and see the change of that over time. We’re actually experimenting with a slightly new way of doing road mapping right now and especially prioritization. We’re using something called thematic prioritization.
If you think about it, the idea is very simple. If you focus on a ton of different projects, you’ll end up peanut buttering yourself and not really move the needle enough for developers. So instead of focusing at the project level, you think of the main themes or swim lanes if you will, the that can make a difference for developers. If I’ll pick on, let me pick on Bazel for one second, or rather on our inner loop development. We have the Bazel migration, that’s a big part of what we’re doing right now. We have support for teams outside of the monorepo that I mentioned before. We have investments in the IDE. We have investments in making Bazel, top-notch product within Snowflake and we have a bunch of others. If I try to invest in all of those themes, it won’t fit, it just won’t fit or we won’t move the needle enough.
So what we said to the teams, “Tell us what are the up to three themes that you want to invest in this quarter, and what metric will get moved by investing in those themes?” And everything that I just said has nothing to do with the actual projects. Only when you land on the needles you want to move and those themes, then you say, “Okay, now stack-rank the projects in each one of those themes and decide which ones to do first.” But that’s a matter of what fits based on the amount of time you want to invest. If you said, “70% of our time needs to be invested in finishing the migration to Bazel.” Then that’s how you pick the items in that bucket.
We’re currently experimenting with this approach, it’s something that a lot of companies in the industry did. What it really helped us do is make sure that we all sing to the same tune to a certain extent, and all the teams bring the themes to the table. And we’re still trying to iron out kinks. And that’s the other big thing that I want people to at least know. There’s no one-size-fits-all. There’s no one solution that’s going to work for everyone. Thematic prioritization is going to work for 80% of the teams and be completely irrelevant for 20%. The idea is for us to be flexible enough so at the end of the day, we plan with goals in mind, with customer impact in mind, but to give the teams the flexibility so they can focus on doing the work. We’re also learning how to do that right now, but that’s the mindset we’re going into this with.
Amy Yuan: Yeah, I would say for planning and execution, it’s really important to be both tactical and strategic. In the CI space as an example, that’s a long term, we need to rewrite the infrastructure. But if that takes a year, we need people to see incremental progress. So we build trust. So that’s why when they do the quarterly planning, it’s always a balance of both. Now we are working on the second repo after the monorepo. Our architect, Nathan, when he works with that repo owner is again, is we need to work together. You need to give us volunteers that we work together as one team, because us as an outsider to the repo will not be able to really move the needle very fast. We need to work together as one team and also we need to have two goals, short-term goal and long-term goal. Every quarter we need to show developer how things are improving, want to show that in the satisfaction survey, we want to show operational metrics. And also long-term, this is where we want you guys to be. We need to have both.
Gilad Turbahn: One last thing I really want to add, because I think it’s critical to understand. Our job is not to define all the different processes and make sure everybody follows them to the tee. Our job is to paint with brushstrokes what the direction is and make sure we’re all focused on customer engagement and customer satisfaction and customer value. And to bring in really strong, really talented people who care. And that’s what we have on this team. That’s what at least I’m really proud of. I know Amy is really proud of that as well. Our TLs care, our EMs care, our ICs care and are involved in this and they get a voice in the room. So we get to paint those broad brushstrokes of thematic prioritization, but then the team is doing the work. And as long as the mindset is not just churning out features, but rather bringing customer value and being able to measure that you’ve moved the needle for customers. Go ahead guys. Push as much as you can. You’ve got the floor. That’s the approach.
Abi Noda: And one question I have for you is when you’re publishing these themes, sharing these out with the organization. You mentioned that you not only discuss the themes, but here’s the metric that these themes are going to impact. Is that all you’re saying? Are you sharing, “Here is what we’re going to impact”? Are you saying, “We are going to move this score by this much”? And the reason I ask is this is an area I see teams getting a little hung up on sometimes. Do you commit and put the metric out there or is this more talking about, this is the metric we’re trying to impact to rationalize the investment?
Amy Yuan: Yeah, I think it depends on where we are. So for example, in the last three quarters for monorepo, where we had built up more context we do commit to quarterly goal. And every month when we do review with ELT, we’ll actually see if we’re on track for the quarter, whether we’re going to hit it at the end of the quarter. So we are now talking about the second largest repo where we don’t yet know what is possible. So we’re just talking about the metrics we’re going to do, we have not set a goal. But once we have an idea, we will absolutely set a goal.
Gilad Turbahn: It’s actually a part of the process. We set a target, you know the term date for date? That’s part of what we’re doing. We’re saying, “Okay, we don’t know yet, but we will know by November 15th what can be accomplished within a quarter, within two.” That’s the line of thinking. We started off by setting very aggressive goals that weren’t realistic in some cases, and we’ve skewed towards, “Okay, let us assess the work. Let us do the benchmarking. Let us figure out what’s doable, and then push as hard as possible and set goals for it.” But I do want to remind people who are listening, metrics are not the goal. Metrics are a way to measure whether you’ve generated enough impact. If you’ve generated other, the metric didn’t move enough, but a different metric did move, that could be great as well. Things changed throughout a quarter or throughout whatever period it is. The metric is a way to make us accountable and see that things are changing, but it’s not what we’re chasing. We’re chasing customer impact.
Abi Noda: Yeah. I love your advice about, you mentioned in the past you maybe just set goals to set goals and they weren’t really achievable. So I think the takeaway here is that it’s situationally dependent. If you know confidently how and if you can move a metric and they’re confident setting a goal, do so. But if you don’t know enough yet, it’s okay to say that and say, “Here’s what we’re exploring to understand the problem better,” and then set a more confident goal. I also want to ask you, tell me about the long-term versus quarterly planning. So I think I have a pretty good sense of how you’re setting themes and metrics and areas on a quarterly cadence and then reporting on those. What’s different about the annual planning that you do? What is the annual plan? What does it look like?
Gilad Turbahn: The annual plan focuses on what is the north star metric, what is our mission and vision, and then what are the main areas that we truly want to move the needle on, and what are the ones we’re saying we won’t get to. And it’s really important to define what will we not get to or what will we say is not something we want to invest in right now. It helps the team focus, it helps them know where do we think that most of our time needs to be spent. It doesn’t capture everything, to be clear, but it captures a lot of the big investments. And it allows us to funnel resources to right problems and away from other problems, because we’ve got, as we said, we’ve got a really strong and talented team. We don’t want to peanut butter them. We want to focus them on the right stuff.
So the annual plan is a way for us to get buy-in from the company’s leaders of, here are the big investment areas and here are the ones we won’t tackle. And once we have that, we can align the team to be more invested on those areas, to hire more folks for the right areas, to move people from one team to the next, to put more PM attention or TPM attention or data science attention on those areas. So that’s how it’s used, to inform. It does not replace the quarterly planning, the quarterly planning is where we actually say, “Here are the work items. Here who’s assigned to what. Here’s how much capacity we have.” It is not there to replace it, but rather it’s there to set a baseline and make sure we’re all rowing in the same direction. And now let’s figure out the tactics.
Abi Noda: And then all of this planning, the artifacts that you create in the process, how are you then distilling that down to share? I assume, for example, with the entire organization. You’re sharing something that says, “Here are the themes for this quarter,” for example. What does that look like? And then are you going into more detail with certain executives? What are the different formats and zoom in, zoom out levels of this information that you’re sharing out with others?
Amy Yuan: We share them with many different forums at different levels, and actually repeat ourselves quite a lot. The annual plan is not just for our team, it’s for the entire company. So that is properly shared and everyone get a chance to weigh in and review. So next week for example, the extending LT will review plans across all the different teams and that will be sent via email, shared also in part of all hands with the entire company, not just ourselves about everything. And then with our stakeholders and then the teams, we also plan to do the next round of roadshow to share for next quarter, for next year, this is what we plan to do, and this is where I need your help, and this is where we’ll help you to move your key metrics. And then in that process we will refine. Once we set a goal, it’s not always absolutely set in stone. In the quarter there is also some flex as well. When things change, we’ll always prioritize something that come in last minute as well.
Gilad Turbahn: The important piece is the two-way communication and knowing that even though, sure, we’re the chief repetition officers, we repeat the message again and again. We’ve got different audiences. We want the team to be bought into it and we want to listen. For instance, we have a forum with the TLs to make sure that they go through and are aligned with where do we think we need to invest, both on the quarterly level and also at the yearly level. Is this the right stuff? We’ll reach out to the CAB, as we mentioned before, the customer advisory board to get their insights on it. We’ll engage at different levels and sometimes it’s a slides presentation and sometimes it’s a Google Doc, and sometimes it’s a Jira set of items. It depends on the audience. It really does.
But the idea is, it’s not us publishing something and the world needs to cope, it’s a two-way street. It’s a discussion. Things can change throughout the quarter, throughout the year, and it’s totally fine. We want to have a baseline, so we make our… Well, I’ll put it this way. We want our decisions to be explicit and deliberate. It doesn’t mean that they can’t change. It means that there needs to be thought behind it and we’ve made a decision together, and now we’re running as fast as we can and as diligently as we can with the highest quality that we can to fulfill that. And if new information comes in, by all means let’s discuss. Let’s change our plans, let’s change direction. We’re flexible enough to do that as long as we’re deliberate and explicit at what we’re trying to do. And everybody is aware such that they’re not surprised, and that brings back the trust factor that we talked about.
Abi Noda: Gilad and Amy, thanks for your time again today. Always a pleasure to learn about your journey at Snowflake and all the insights you’ve been able to share with others. Thanks so much for the time and really enjoyed this conversation.
Gilad Turbahn: Thank you very much.