The value of having a PM on a platform team

Abi: It’s interesting to have you on the show because we typically talk with engineering leaders or engineers who work in platform or enablement or DevX teams, but you’re a product manager, which is a bit unique. I’m curious to know, how did you end up working in platform versus a customer-facing function?

Jelmer: That’s a good question. And of course, there’s also a lot of people around me with great engineers and that’s an interesting mix to that end. So, when I joined Picnic about five, six years ago, we started out with one big team of engineers and we didn’t really have products or product managers to that end. And then we started formalizing them, splitting off from this huge team. And I initially was focused around payments and finance. As we were moving to Germany, so it’s our second country next to the Netherlands, this really was a challenge for us where we had to accept payments, which is very different because in Europe everything is very localized and we had to change a lot of things.

For myself, I had to really get to the architecture of how we do things and go to these nitty gritty details there and also how we very quickly also realized that you shouldn’t mess with people’s money. So, if you charge them twice or you don’t refund the money, people get upset. So, it also made us really focus on quality and how we release things and we went more towards, let’s say, continuously releasing for multiple releases on a given day. And really this continuous deployment of model, and over time, sparked my interest a lot and I started doing more and more things around it, more research in my own time to see how can we just improve as a team and also working with other of our platform teams within Picnic.

And then after we internally restructured things a little bit, then I jumped ship a bit and became PM for platform teams. And to that sense the first one, as well, and that made a nice challenge to see where we can introduce, bring management more in other, for a platform teams.

Well, thanks for sharing that. And I’m curious, what’s the scope of your platform team today then?

So we have a few platform teams. So we have one around Java platforms, I think, around tools and libraries for most of our backend engineers, as most of our backend services are in Java. We have a Python platform, some backend services, but a lot of things for data science, a lot of more sort of offline like scheduled jobs that are written in Python. We have a lot of analysts working on our company who are building things in Python. And so really giving them the tools to accelerate there. We have one around analytics, which is, let’s say kafka pipelines, giving tools for our customer analytics, but also for internal analytics around what happens in our warehouses, what happens, let’s say, you’re in deliveries for all the screens and things that you see. We’re building up one around QA. So, for around quality, how do we do our testing, but also again, giving the tools to everyone within Picnic, making sure we use everyone’s knowledge around that and then related teams. So around security, digital workspace, but also infrastructure and networking.

Well, that’s a lot of different responsibilities. How many other PMs are there within this group other than yourself?

So we are now at, I believe four. So four other PMs. And definitely looking to increase that. But it’s a bit of a struggle finding the right one. So whilst we are hiring PMs going through the hiring process, in many cases we have not found good ones there. In a few cases, we were able to hire more directly. And this is very often still, people with an engineering background, but then moving towards product. So having a bit of a different route than for myself where I actually came from the other side, if you will.

Well, that brings up a useful topic because I’ve definitely spoken with other engineering leaders who’ve kind of talked about the difficulty of recruiting and hiring PMs for platform teams. So I’m curious, what’s been the experience for you guys in trying to go out and recruit and hire? What’s been the biggest challenge in terms of finding people?

Well, like many companies, finding good people in tech is just a big challenge in itself. We are getting bigger. So we have about 250 people in tech at the moment. So it’s definitely already changing compared to a couple years ago. So there you were really this small startup, which you’re fighting with the bigger ones.

What helped us a lot is really just, let’s say, speed is a massive difference. If you can iterate, this multiple browser, you can basically be able to respond faster than these big organizations, which may take one to two weeks to sort of respond to your interview route. If you can already cut that down to one to two days, then you’re just faster to give them an offer for the ones that you want to hire.

Now, we are slowly transitioning into a mode, which is actually pretty cool to see where people are actually coming from larger or from other more well known organizations are actually now applying at us. Because they actually want to go back again to a little bit something smaller, again, a little bit something a bit more dynamic than… But at the same time, bring so much experience with them, which helps us a lot making the next step in our organization as well. Very few people in our organization have worked in organization larger than we are currently at. So this is also… CEO also makes this sort of analogy from time to time. This is the largest company I’ve ever run.

I’ve never done this before. But from time to time getting, people that did have that experience does help. And I think that gives a very sort of healthy mix compared with more sort of junior hires, where we have an internal tech academy as well to really get them to up to speed and really sort of have them, giving them the learning opportunities, which allows you to hire people where other companies might not sort of take that path. So you’re hiring a lot for potential, which is harder and it doesn’t always pan out, but if it does then it works very well.

Well, that makes sense. And those sound like great strategies for competing with maybe larger, more established companies that can maybe pay more. I’m curious to know more specifically about the challenge of finding people of the right profile to be a platform PM specifically. And I think you touched on this earlier, when you said you were maybe looking at pulling people more from engineering backgrounds. So how are you specifically, finding people who can be successful as platform PMs?

Yeah. So you have a few traits that you would require in any PM. So one of the key things that were I think PMs in our platform teams can provide a lot of value is very much focused around communication collaboration. Where we have amazing engineers who love to build the next level of their domain, but are not always looking into way. I don’t want to spend my day talking to some all these other teams, but I want to build this next thing, which is super, that they’re doing it, but it also needs to be used and also needs to be… You also need to get the experience of all the other products within your company. So one of the things where I really feel that they can provide a lot of values around that communication collaboration, but that’s very similar to any PM position.

In here, I do require them to have a bit more technical knowledge. You need to understand what your team is actually building. And the level is a little bit… It’s a bit harder to get into, whereas maybe for… Of course, if you look at UX and UI, that’s also a very sort of expertise to have but it’s a bit easier to grasp sort of what that means for your customers. But here you do require them to really understand what are you building. So if you are on a Java platform team, you want them to be able to at least read, also write a little bit of Java to understand, “Hey, if you are giving the tools for others to maybe do a certain migration, what does it actually entail? What the questions are they sort of asking? What do you need to sort of help them with?” So, there is definitely something you also the interview process want to… I want to touch upon to see to what extent they can do that.

A challenge we do find is that because we have quite a broad range of these platform teams, it’s very hard to have one strategy where you’re just looking for a generic PM that could to work in any of those teams. So in whilst the conceptually is quite nice because they are then more flexible, et cetera. I think in practice, we do seem to hire more specifically per team to that end. So around QA, you want to have somebody with, let’s say some preferably with some QA background or around analytics. It’s very helpful to understand what does it mean for analysts or in data warehouses or maybe managing data guidelines, which is very different from, for example, infrastructure.

Do you think someone like a platform PM has to be technical then? Do they have to have a technical background or do you think you can be successful without one?

I think it can be without. But you will not have an easy done. So, it does require you to do a lot of research into what does it mean to build things. And if you have never written a single line of code, there’s quite a journey you need to go through. But that’s said, it’s definitely not impossible. And I think a lot of the other things still definitely apply. Doing user research, figuring out their challenges, be able to translate that into actionable things actually work on and iterate is what you can do in any team.

Yeah, that makes sense. Well, I was recently speaking with Will Larson, who you may know. He is writing a book on infrastructure teams called Infrastructure Engineeringand he mentioned that he didn’t have PMs on his infra team. And he said, part of the challenge is it’s very difficult for PMs to maybe build a career as a PM in platform or infra, because a lot of the things you work on are a little bit invisible. They don’t directly impact revenue. They don’t directly impact customers. They’re a little internal. So I’m curious, do you agree with that sort of opinion from Will or has it been different in your case?

I can see his point. What I would see… What I would think about is building a career for PM and being fairly invisible, I think is across all products to that end. I do think we tend to over generalize or think about, well, maybe… Very stereotypically, developers do not care about UX. Therefore, you need to have somebody translating that and working on that and having some clear specifications for them to work on. I think in reality by the way, that is not true. I think a lot of engineers do care about that, but that maybe therefore more invisible if you’re talking about more technical products. I do think they can still provide a lot of value. So of course, I’d like to see it as you’re B2B2C type sort of model. So we’re building internally for other companies or other products for them to be successful. Which means that your metrics is completely different because we are… Or metrics is not, in many cases is not about revenue growth, but it’s more about developer productivity, which has their own metrics to then figure out what do they need to become successful.

And it becomes a little bit harder to explain maybe to your board, “Why are you there? What is your fit?” But I think it’s still very helpful to think about what is the value that you are building as a product and are you spending your people in the right places to get the most value out of it. We shouldn’t just build things for sake of it. And I think that’s slowly or… Yeah. You see a little bit of change where I think in the past, we have seen, infrastructure teams as in being centralized and “Hey, we just put it there because it’s easy to put everything in a single place.” Transitioning into what value can they add and how can they give the tools to other teams to be successful or moving more towards this sales service model, if you will.

Well, I love that insight around maybe your metrics are different rather than revenue growth being kind of your KPI that you have that B2B2C model where developer productivity, that internal sort of enablement is your impact on the business. We’ll come back around to maybe talking about how you actually measure and track that, but kind of continuing to talk about the role of PMs on platform teams. I’m curious. So not every platform team outside of Picnic has a PM, so I’m curious. What’s your opinion on whether they should have one? What do you think these teams are missing that currently don’t have platform PMs?

I like the question it’s where, in some cases you could get away with, let’s say, a tech lead or an engineering manager covering a little bit of both sides of it. What I think makes it just you as a team much stronger is having these two sides of the coin in there. So having one hand, strong engineers working on the product, but at the same time, keep doing that research, keep doing that… Figuring out what are the struggles of the other product teams are. And I think to that end, you can be much more effective and improve, get much more value out of it. And in some cases the platform team can sometimes be seen as these people where they just define the standards and other people just need to adopt it or something.

Whereas, but I think what you can do as a PM is really sort of one hand communicate. Well, why is important we are doing something, right? Because in some cases you do have to, especially if you think about security or maybe if we’re moving towards more future proof tech, but at the same time also, translating those challenges that other teams have to your engineers and your platform teams. And I think it is hard for people to understand how their product is used across many teams. So in our case, we’re very lucky that we have a very high retention rate. And we have a lot of people in our platform teams that have been with us for quite some time, which means that they know at least a few people in every team to talk to them and have that relationship.

But the other challenge is then having newer joiners joining that because they’re missing the context, they’re missing… Let’s say they, understand how maybe this is used in one to one or two teams, but then you’re solving the problem for those instead of trying to solve for at least 80% of the dev teams within your organization. In our case, we have around 30 product teams in total. But imagine if you have maybe 50 or a hundred, that becomes… There’s sort of to impossible for, to get that full picture. I think they’re having somebody at least dedicated on this. And call it PM, call it a different sort of title, I don’t mind. But I think it’s important that you do this work. And if it means that you, as an engineer manager doing that is all great. I mean, to that extent, it’s more role in a team than just let’s say the title to the net.

Well, I love that explanation and certainly in talking to other platform teams and enablement teams, this sort of muscle of understanding the customer and doing the research is something that all these teams face challenges with. So, I think it makes sense that sort of the PM role can be a function that’s devoted to that and brings excellence to that craft. I’m curious, you touched on this in your previous statements, but you previously told me about a pretty interesting approach you guys developed for collecting feedback about new versions of libraries that your team releases. I’m curious, what was the problem you were having? What prompted you guys to develop this approach?

Yeah. So some of our platform teams, they develop libraries for other apps to use. So around… For example, Java Python or analytics, and you are building that, but you also need to understand, does it actually work, right? So maybe if you are releasing a new feature or you’re deprecating something, what does that sort of mean for other teams? And in the past, we sometimes maybe took the approach where we said, “Well, this is only one day effort for you to migrate or it’s sort of to…” Adoption of this new framework is super straightforward. But then of course, if you are the expert and you know everything and all the intricate details, then yes, for you that maybe just a single day or maybe just an hour to do that.

But for other teams, who’ve maybe not even heard of this before, or maybe they have not, the experience or the time to really look at that, that might be very different. So maybe where it’s for you, one hour. Maybe for your other team is maybe a five day effort to figure out what they need to do and how they need to do that. So, of course, on one end, you can improve by having better guides or documentation around these things. But on the other side, we want to collect also the struggles that other teams were having. So what we do when we release a new library is that we have, we use a tool called Renovate, which basically automatically creates PRS at other products that for automatic upgrades.

So we use that for all libraries, but also for our internal libraries. “Hey, there’s new version out there, which creates DPR runs all your CI and checks if this is actually working.” But after that, you want to collect, how easy was this process, right? Was it, for example, build was green. You click the button, you ship it, everything is just fine. Or was there actually a lot of struggles to get the adoption of this new version? And one way to do that is after PR merger after release, you can have very easy slack bolt saying to the developer who did this, “Hey, how easy was this upgrade and was this okay? Was it neutral or was it actually pretty bad? And for sort of those cases, can you tell us a little bit more?” He can relate this to the actual version or the actual upgrade that you need to do.

Maybe this was just the batch version release from, I don’t know, version 14.0 to 14.1 or maybe actually they were lagging behind because they were busy with other priorities and they were actually upgrading from version 35 to 40 or whatever. And then suddenly actually that became a horrible experience because they now suddenly had to figure out all these change logs, looked okay, they released it. Everything sort of broke. People were running around, everything was in fire. And they had a terrible experience, which then also can have significant impacts on the goals. And let’s say the objective as a company, right?

So in our case, we have a lot of operations because we do our delivery. But then if we impact them and people cannot actually deliver groceries, that is very painful, right. People are sitting around unsure what to do. There’s a lot of people you’re impacting very directly and it’s very expensive to do so. So you want to get a feedback loop, going to understand what that means and taking the understanding from your context. So this may be easy to do is not always the case for everyone in your organization.

Well, it’s such a cool approach to gathering feedback in sort of a seamless way. I’m curious. What do you specifically… What kinds of feedback do you specifically ask for? Is it just like an open text box? Are you just asking for any feedback? I’m just curious what kind of insights you’re able to capture.

So it’s very straightforward. We didn’t want to make huge surveys out of it. Because especially if you are doing a fair amount of these upgrades then. We give you three buttons. It’s was it good, was it neutral, was it bad? And optional, you can add in free form text field, you can can add some more information. We can improve that in the future at the moment, this is good enough. Because you already get enough qualitative feedback to do so. We do also do more quantitative feedback, but for that is more of a structured survey that we’re running within our company where we… Across our developers, across all the roles, but also our analysts working a lot with our Python tooling, ask them what on very aspects their input. And that is more quantitative, analysis with a few more qualitative aspects to it.

So I think you kind of always need both, but with very easy unstructured feedback it’s often that you can already get a lot of value out of it. Doesn’t always need to be, let’s say the step page form that you need to fill out because people will also not do it. You need to make things easy. If it’s not, then people won’t do it.

Totally. And I’m curious, you started kind of giving a privy of this, but what is this sort of larger survey that produces quantitative measurements? I’m curious, what your guys’ approaches to that. How often do you run it? What are some of the maybe key questions or measurements you capture through it?

Yeah. So we run it twice a year, which we… The inspiration came a bit of the state of that DevOps. So we call it the state of Picnic DevOps report. How are we currently doing with our company around this? So run is two times a year, and then it’s an anonymous survey, so everyone can participate. We encourage them to participate. It’s not mandatory. If you don’t want to, you don’t have to. But voicing your opinion, most people find valuable. And we see very high participation rates from there. We gather a few insights and looking at what also the other more known reports are doing such as, say the DevOps, but you also have from, jet range and overflowing, you have a few others who do these bigger surveys across the state of tech.

So we focused around a few survey areas, so on one end around goals. So how aware are you as a team of your goals and do you actually feel that you are doing well and actually meeting them or not? And from there, we move slowly towards actually building things. So around knowledge and documentation, around the implementation of things, how do you do change approval? So let’s say, the process around PRs and feedback, and this is not only about the speed of these change approvals, but also do you feel that you always are comfortable providing feedback and do you actually also get constructive feedback? Because we rely on it for high quality software. But if you don’t feel that you can speak your mind, then something is really not going well around quality, around how do you deploy, but also what are sort of your levels of testing that you’re actually working with?

How do you… What sort of your deployment strategies do you… But also as an engineer, for example, question is, do you know how to deploy and do you know how to… Is it deployment? Would you consider that as being straightforward? Do you also know how to roll back of things going, are going wrong? But then also around the observability, once you actually put something live, do you get proactive? Let’s say notifications, if something is going wrong, do you feel that you have the right tools to diagnose things? Do you notice it before maybe your customers are reporting issues? When we got started, the way how we to know when things were going bad, a couple of years ago, is that basically your customer success will basically start falling or we had… We even had a channel for this, I think, which basically man everything’s down, things are not working.

We need to fix this ASAP and now much better state, but even then, how are you able to do this much quicker than maybe your other teams, do you notice? And can you basically act upon this in a sort of a timely manner? So there’s basically this whole DevOps cycle and per item sort of look at it and how well are we doing it? So we do ask you also your role. So maybe a job engineer or machine learning or data science or an analyst or a product manager or a tech lead. And also sort of what… What roughly your domain you’re working and then you can slice and dice all these answers that they’re giving them. And then you see that in some areas teams are doing really well, maybe in a certain area, let’s say for example, around quality, but maybe in other areas, this is more challenging.

So an example was that we saw around quality that our data designs and machine learning engineers were having a bit of a hard time because not only do you need to make sure that your services are actually working as appropriately, but you also actually need to have the right quality of your model. And of your data set that you’re actually reading from. So they actually hadn’t… They have more aspects that you need to take into account and not only releasing your features and making sure that your services is running as fast. And with high up time and higher, let’s say SLAs, et cetera. So that gave us a lot of insight into their thoughts. And now from there, the question is how do you go forward? Because in some cases, things are not going well, right?

And it is anonymous. So we cannot send them a message to clarify our answers. What you can do from there is start organizing sessions around it. So, “Hey, we’re seeing in this domain area, it’s not going well, shout out to everyone. Who wants to participate in a session to talk about this?” And then it’s still optional for them. So if they don’t feel like they want to talk about it, they don’t need to join, but if they do, they can voice their opinion, give bit more color to it. You can have a very fruitful discussion around it just to see how you can help them as best as possible. And hopefully then over time, you see nice trends in there. We haven’t run this for many years yet. So I think the trends we still need to… We don’t know yet, but at least we feel that we’re providing value to that.

How did you design this survey? Was this inspired by the annual state of DevOps report survey? Is that where a lot of the kind of questions were kind of pulled from or was this very much done just in house?

So, we definitely used a few of them as inspiration, but we did develop at least the questions in house. We just also ask others from other teams, their input, just check, hey, in terms of completeness. But also do they feel these are direct questions. And in some cases, especially in the beginning, I think we phrased some of the questions a bit too quickly as being more opinionated. So you introduce bias very quickly. So there we had to do a few iterations on it to make sure that we are covering it well and not biasing it too much.

Is this a program that’s owned by the platform team or is this coming from your CTO? Who’s owning this sort of state of the DevOps thing?

That would be me. So I would basically do this. I improve my HTML Caesar skills again and combine it with my data analysis. So I am actually an analyst, originally. So working with the data, et cetera, is definitely something I like to do. But also help me here to put that in there and also make it public in the end. So what we in the end published was in our… We have an internal thing that we deploy where we now have the sort of website hosted that you can take a look at it. So it’s completely open with some of the insights that we took out of it. But then also all the raw data is still available to everyone.

So if they feel the conclusion would be actually different, they can sort of dive in and they could also slice and dice across also the different roles. But sort of a nice to blow dashboard where you can do all the nice things around it. So this is something I started doing where I feel that I think as a PM is a great angle to do right. You want to understand, a bit of the state of things are doing and you can actually improve things and what should be your areas of focus.

Awesome. Well, I’d love to learn more about all the tools you’ve felt and the approach around this at some point. But kind of coming back around, I’d love to ask you more about the transition that your organization and your team went through. You mentioned to me before this call that, it’s been this kind of ongoing effort to transition your platform teams from pure engineering teams to actual product teams. So yeah. I’m just curious what have been the biggest challenges of integrating PMs into infra and how has it changed maybe how these teams work and what they’re focused on?

So maybe as a star, disclaimer, I guess, we’re definitely not there yet. And we’re definitely not… I think we can always keep improving things. The state that we were in before is that there was a lot of good things happening and a lot of more senior engineers or engineers who were with us for a longer time who worked on various products. They wanted to focus more on the foundational elements and reduce on one hand duplication. But also on the other hand, actually push the state of tech that we’re using, deprecating things we should get rid of and improve things or adopts in new technologies. So over time, historically, and a lot of these engineers with huge passion for these things, sort of moving to platform teams.

One thing that was quite hard is if you would ask any of them, “Do you have a roadmap of things to do?” And they will list things that they’re working on, but not so much very similar. So what is sort of the goal that you’re trying to solve here and very similar. And that doesn’t mean that we’re not providing value, but in some cases, they could have maybe provided more value for the other prior product teams within our organization. So it’s then more sort of a trade of time in that regard.

So now with a few of our first PMs there, working together with team all coming from different backgrounds of both from a engineering background, but also from a more traditionally PM background. You see that they are really focused and also building up that community of users and building out that shared understanding, where do we sort of want to go? And what’s a bit of sort of also the vision for the product that we want to go, what are our clear goals that we want to achieve and actually does it actually fit within the challenges that we have as an organization, right? An example is with which you see many companies is they want to maybe reduce the time for onboarding. But if you are… Which can be a struggle if you’re onboarding something new. But if your strategy is not to grow quickly and hire a lot of new people, then it’s still… Yes, it’s painful for these few hires that you’re getting in, but that’s maybe a bit of a shame of the time that you’re spending, because you’re not getting the most value out of it.

Maybe you should actually focus on the productivity of your current engineers, because maybe you already have quite a few and every 0.1% improvement you’re doing there has a huge impact on the overall, I think. And the third one is also the experience of your developer. So it’s not only about, does this make you able to build something faster, right? So also sometimes if you… Yes, maybe it doesn’t take too much time to, I don’t know, click these couple of buttons, but if everyone is incredibly annoyed by doing a certain task, because they have to deploy things or whatever, then that can be sometimes very easy gain that you can do. And just increasing your happiness in your organization, which helps retention, which helps in the end also productively, right? If it’s more of a joy to build things, there’s enough research that shows that engineers are also more effective in their jobs, even though the absolute time save is maybe not so much.

Well, I love that story. And also the distinctions there are at the end around. Well, just being deliberate about helping enable developers and not just thinking about maybe time inefficiencies, but also just the actual developer experience itself. Well, Jelmer, I really appreciate having you on the podcast today and really enjoyed this conversation. Thanks so much for being on the show.

Thanks so much for the invite. Love being here.