Justin Reock:
Welcome to this week’s episode of the Engineering Enablement Podcast. I’m your host, Justin Reock, and this week I’m joined by Eleanor Millman, Senior Staff Product Manager of Platform Engineering and Mina Tawadrous, Associate Director of Platform Engineering at SiriusXM. And we’re going to talk about how SiriusXM uses data to enhance their internal product thinking. So before we get started, Mina, tell us a little bit more about your role and background, and then we’ll do the same with Eleanor and then we’ll get into the questions.
Mina Tawadrous:
So I lead product over here in platform engineering at SiriusXM. I actually started off as a game developer coding some games back in the day. Got into the entrepreneurship space and then I fell in love with product management. So I’ve been doing product management for 15 years now. I started off in consumer products, so education, retail, healthcare, finance. And then about halfway through my career, I fell in love with developer tooling. So we started working a lot more in the developer tooling space on security tools and on other productivity tools. And that’s where I learned about the wonderful world of platform engineering and that’s what’s led me to where I am today.
Justin Reock:
That’s interesting. Eleanor, how about yourself?
Eleanor Millman:
Similar to Mina, I started out in a more technical area. I actually studied physics in school and I was a Android developer, a little bit of Java and Python for a few years as well. But I noticed that as much as I enjoyed building tech, I even more enjoyed thinking about people. So thinking about the developers and what made life easy or hard for them. And so I transitioned into product management and there, I did a little bit of non-technical products, but mostly I focused in my career on technical products. So I’ve been at VMware, Google, and I’ve really realized over the years that I like thinking about developer tooling and infrastructure and just ways to make developers’ lives better and reduce their pain points. So now I’m at SiriusXM. I’ve been here for almost three years and I joined the platform engineering org, built up the product team for platform engineering, spent some time promoting internal AI use and I’m still here helping SiriusXM developers have an easier time of it.
Justin Reock:
No, that’s great and thank you both for joining us today. I’m really eager to dive into this subject and learn from both of you. So today we’re talking about what I think is a really common crisis problem in platform engineering. Specifically, you’ve got five teams, there’s limited resources, you’ve got a million requests coming in. How are you deciding what to prioritize?
Eleanor Millman:
So that was actually the first problem I tackled. I was the first product manager in platform engineering. I was hired by Jared Wilensky, who actually was on the show a few years ago. And when I came in, it was very clear that my engineering counterparts were doing such a great job of prioritization but as the needs grew of our users, of our stakeholders, we were supporting folks over, of course, the whole SDLC, 700-plus developers, what they were doing for prioritization was not scaling. And so they really needed a more robust method of prioritization that, of course, they felt very comfortable prioritizing at the team level, but once you get over five teams at the whole platform engineering org and beyond, that’s where we really needed something sturdier, broader, as you will.
So we looked around and there was just nothing that existed at the time for platform engineering. And as probably most folks here know, platform engineering is really a non-traditional product space. I feel like in most product spaces, most products, you can generally look at revenue as a way to prioritize. But as we know, platform engineering is twice removed from revenue because we build things for the people who build things that eventually generate revenue. So it’s a complex space and we had to improvise in the end on how to prioritize.
Mina Tawadrous:
I think when Eleanor brought me in, I think at the time I was working at a consumer-facing group and we have end users outside the organization, they pay money for our products. So we had a crutch almost, right? You’re looking at revenue, you’re looking at ARR, you have all these metrics that have been built for years. So then when Eleanor showed me the world of what they were building, I was shocked at the framework, that I think I should go into a little bit, about how relational each item is to each other and why that’s needed in internal, whether you have these different stakeholders, but you don’t have a bottom line metric that you’re aiming towards. It was a nice surprise to see the level of maturity that Eleanor was able to bring the team on in terms of prioritization with an internal products team.
Justin Reock:
And I think you can make that same observation about just developer experience in general, right? It’s like, it’s hard sometimes to connect the dots to revenue when really what you’re doing is making more frictionless systems and you’re encouraging throughput for the business, but it is hard sometimes to really quantify that in terms of revenue. Have you found a good way to justify that up to executives? Do you feel like there is unification up at the top level of leadership that what you’re doing is directly tied to throughput in the end?
Eleanor Millman:
Well, yes. So this framework, we worked hard to put in all the different ways we could be impactful as platform engineering and looking at all the things that leadership did care about. We always knew, I would say it’s fairly obvious as a platform team that folks really care about development speed. Of course, execs want features to be delivered faster. We also had runtime reliability factored in because of course we want end users, and in our case at SiriusXM, that’s listeners, we want end users to have a fabulous experience, so low latency, few errors, et cetera. And then of course, cost reduction and security and compliance, we knew all those mattered to stakeholders and we were able to find a way to work that in.
Over time, we actually fully figured out other things we needed as platform engineering to be more successful, such as being more data driven, making ourselves more efficient, and really building the trust of our users. And so by mixing all of these impact factors into this prioritization framework, and then we were able to tune it, we changed the weights depending on what signals we were getting from leadership. Maybe one quarter, the company OKRs were really focused on this huge launch coming up. So we really had to increase development speed while keeping quality high. And then a few quarters later, cost reduction was a focus once we had hit our launch, we had all this traffic running through our AWS platform. So this framework allowed us to be responsive in a pretty detailed way based on leadership signals.
Mina Tawadrous:
And I think what I want to add here, Eleanor, where I think it really stood out to or where it really helped is a lot of these cross-team efforts, and I think when you have a global store for a priority, the reality is like I’ve worked in startups. If I’m working on a five person team, we know what we’re building. We have a pretty good idea, we could align the five individuals. I think what was difficult when I joined platform engineering and where the framework really helped is, okay, now there’s a product or a piece of work that involves two teams. How do we decide that this is actually globally important for platform engineering? And Eleanor, I think that the weights and what you added there was something that I found really valuable join-in to help with those cross-team conflicts.
Eleanor Millman:
Yeah, that’s a great point, Mina. I can throw in an example, we have our DevEx team working to make local iteration easier for developers. And we have our delivery team working to make the pipelines run really well. And in both cases, to finish their projects, they need help from our cloud foundation team to set up the needed AWS permissions. And before this framework, it was really unclear which projects should be prioritized and of course our cloud foundation team has its own projects. This framework allowed us to align on the org level as well as on the team level. And it just made things a little more clear really to everyone in the org and platform engineering about what we wanted to work on and why.
Justin Reock:
So let’s dig into that framework a little bit. you started with that standard off the shelf RICE framework, for our audience that’s the reach, impact, confidence, and effort model. Why did you have to evolve past that? Why wasn’t that framework enough for the high performing platform team that you were trying to build?
Eleanor Millman:
We started with RICE. It’s such a common prioritization framework in product management, but immediately it didn’t feel right to us. One thing is that we had urgency for a lot of our work. And let me be clear, as a product manager, always, you should try not to use urgency as a reason to prioritize. You really always want to look at impact and building the most valuable things for users. But sometimes there are real, urgent, deadlines. Like for us, we had this enormous company-wide launch in December 2023. We couldn’t ignore that. Or sometimes you have a vendor contract that’s up and you cannot use a tool past the end of the contract and you have to do whatever you got to do to mitigate that issue.
So RICE doesn’t have urgency, so we put urgency in. And then secondly, impact is part of RICE. Fine, we kept that there, but that’s where I would say we did most of the changes or we maybe fleshed that out, what does impact mean and for which users? And I mentioned earlier, basically we started out with the four impact factors of developer speed, runtime reliability, cost reduction, and security and compliance. Those were quote-unquote, “the obvious ones”, I’d say. And then over time, we realized we did really need that, having platform engineering be data driven, having platform engineering be more efficient and helping our users trust us because that helps us long term.
So we really iterated bit by bit, just as I would recommend for any product development, we iterated on this framework to just make it work for us. And so we had the impact factors in. We weighted them based on what signals we were getting from the company and our users, and made a few tweaks to the formula along the way. And in the end, we’ve ended up using this formula for almost three years at this point, very much changing it as time goes by. And we’ve prioritized, I’ve been saying over 200 projects, it’s probably even over 250 projects or something, 300 at this point. So it’s really lasted the test of time, but with significant changes over the last few years.
Justin Reock:
And maybe let’s talk about that iterative process a little bit. So you mentioned these impact factors, how have these specific factors, like could you give us an example of maybe… Okay, I see a very like high quality score in one of these areas. How does that help you then prioritize the next area of improvement for the platform?
Eleanor Millman:
So all of these factors, as I said, we weight them. And for example, I mentioned how we needed to reduce costs at one point. We wanted to focus on cost reduction. Well, if we increase the weight of the cost reduction impact factor, something else has to go down. And at least at first when you look at all the factors, you feel like you don’t want to give anything up. This is where actually really being data driven helped us. So we used DX Snapshots and so when we’re figuring out what to reduce, we were able to see that we had a very high quality score, especially relative to other industry benchmarks. And so we felt comfortable reducing our runtime reliability weighting, not to say that we don’t care about runtime reliability, but just that we had confidence that we already had pretty high quality software systems and we looked at another metric from DX that showed us that we were having really few significant incidents, and thus we could reduce runtime reliability weighting to give more weight to the cost.
Another example would be leading up to the really, really big launch. We built up technical debt within platform engineering because we were so focused on increasing developer speed. After the launch, we wanted to take some time and make ourselves more efficient because if we’re more efficient as platform engineering, we can help our users more. And so for instance, we were able to see that code review, scores were not as high as we would like for platform engineering. And so we were able to prioritize, so more heavily weight platform engineering efficiency and then prioritize some code review projects so we could build up more automated tests and other ways to help code review go more smoothly within platform engineering. So it really helped us prioritize what we thought we needed to prioritize and ignore, at least temporarily ties what we thought we needed to prioritize and ignore, at least temporarily ignore some areas where we were doing well.
Mina Tawadrous:
And I thought that the cool thing, if I could recollect back to when those decisions were made, I think that’s one thing in an organization when you have just a lot of these weights are assigned by our leaders or worked with through our leadership because they have a good idea of what the weights should be this quarter for business objective reasons. So I think in a typical world, often what happens is a leader might say, “This is the quarter that we are going to focus on developer speed or this is the quarter we’re going to focus on cost.” It’s very different when we then ask that leader and work with them to actually assign a weight to it.
Because if you up the weight of cost, that does mean there’s going to be other work that doesn’t get done as well. We come from a world where in product management, we used to do these things called like we used to stack rank priorities. We used to sit executives in a room back in our consultant days and force them to choose, where are you going to put this sticky? Is it going to be up here? Is it going to be down here?
And I think the weighting or being forced to actually add a numeric value to one of these dimensions acts as the same kind of force and mechanism to not just to say the thing, but to actually see what that impact would be for your roadmap if you decide to change the weight of the importance on one of these factors versus not. So I thought it was a very interesting way to take what we do day to day of executive leadership through words and turn it to something very actionable that would immediately change the roadmap for the quarter if they decided to pull that weight up or down.
Eleanor Millman:
It may sound hard when we set a stakeholder or a leader, like you have to give us the exact percentage weight that you want to weight cost or developer speed, something like that. What we found, this was part of the iteration that we would say, “Let’s put a weight in. Let’s choose values for all parts of this prioritization formula. And then let’s see how the 70 projects we have in queue right now, how they line up.” And early on, we would definitely look at the ordering and saying, “You know what? This does not feel right to us. Our gut feels like project B should actually be above project A.” And then we would talk about it. It really generated a lot of great conversations and sometimes we would realize, okay, we forgot a whole impact factor.
Crap, we actually really want to be data driven and we didn’t have a data driven impact factor at the time. Okay, we actually need to take into account the value itself of being data driven as platform engineering. Or maybe we just had mis-estimated that actually there’s a whole nother group of users we weren’t really thinking about who would benefit from this project and we need to take that into account. And lastly, sometimes we would talk about it and think, “You know what? Actually, this formula is really leading us to realize the true value of the ordering we have, and that’s the right one.” I mentioned this because a lot of times, early on when I introduce this to folks, I hear criticism of it’s really heavyweight, it’s really numerical and I totally agree. The onboarding cost at the time was not great.
We’ll get into later how onboarding costs to this framework has been reduced with Mina’s work, but early on folks would say it’s heavyweight, takes time to onboard, totally agree. You really have to learn it. But the good news is that a lot of it for us was aligning with our gut and having these really good conversations and landing in a place that felt good to everyone where it felt like what was in their gut could be accurately described by what was in the spreadsheet in the formula.
Justin Reock:
So it sounds like this framework has allowed you to balance that need for the autonomy that you have as a leader who wants to be able to improve the platform, but not have to ask permission for everything, but also maintain alignment between the needs of the business. So now you have this framework that allows you to scale that prioritization that we’ve been talking about, almost like a high quality score in one area gives you permission to then move and focus on another area of the platform without having to take that through multiple layers of approvals. Is that a fair statement?
Eleanor Millman:
Yeah, So I would say that the instinct at first when we think about prioritization and platform engineering is to really simplify it. Let’s lump everything together into one or two values, like Rice when you start out with it, that original prioritization framework. But the issue with that is that it’s very, very hard to quantify. And now that we’ve teased out various impact factors and weights for them and ratings for each project within, it then gives you a place to put data. So exactly as you said, we had a place where we could say, “We’ve got this quality score, high quality score that empowers us to make this decision.” Jared, a leader of our org could then use that in case we were getting pushback from higher folks higher up. He could say, “Look, this is why we’re making the decisions we’re making.”
Mina Tawadrous:
It’s the conversations that happen as well to create these. It’s not just the store. I want to be clear about that. It’s the same way if you think about point in stories. It’s not the fact that you’re assigning points to anything, it’s the conversations you have to get there. It’s not necessarily just that we were able to rank this project a 19.2. It is more so that the team had a conversation about the value in each of these dimensions and had an idea of the assumptions that they were actually leading into around that project. So the conversations behind the stores, I would argue, also led into what you’re mentioning, the feeling of alignment comes out in these conversations.
Justin Reock:
It becomes a forcing function to go deeper into the analysis and make that more of a collaborative exercise. I think that’s really interesting. So Eleanor, you hinted to this already, that there was a way that we simplified some of the onboarding onto this process. And in a pre-show, we talked about something that I thought was really, really unique, which was this concept of assumptions as code. So being able to… A lot of times, the many, many assumptions that we have in an organization are hidden amongst various communication channels, but you’ve actually come up with a framework where you can treat this as code. Mina, I would love to hear first from you about how you came up with this and how you put it to practice. And then Eleanor, certainly how has this helped to enhance the way that you’re able to think about product and platform?
Mina Tawadrous:
So it’s interesting. So the way this came up, I’ll say, is in using the framework, we get into these conversations around like, okay, how do we go ahead and assign a value to development speed? First, we have to decide how big is our user base for that product initiative. And right away, on one of the initiatives that I remember storing, I walked in with like, this is going to be obvious. We have the metrics. It’s in our intelligence platform where we’re using DX. We know exactly how many users can feel that pain point. This is just going to be me showing up a dashboard and saying, “Hey, look, there’s 350 users that match this pain.” You could see where I’m leaning into. It was not that easy, right? All of a sudden you have these conversations, you realize actually I came in with an assumption that is very different than the other five engineers that all have different assumptions as well. And it reminds me of this study. I’m not sure if you heard the tappers and listeners experiments.
Mina Tawadrous:
It talks about the cognitive bias of the curse of knowledge, this idea that if you have certain knowledge, you overestimate the amount of knowledge that others have in that same domain. So this experiment was an oldish experiment in the '90s. What they would do is they split users into two groups, tappers and listeners, and a tapper was told to think of a tune, a well-known tune like happy birthday song, and they had to go ahead and tap it on the board, and it’s playing in their head. They know the melody, they’re tapping it out, and they’re told to estimate how likely the listener, not knowing the melody, would get it right. Tappers estimated on average that, okay, got the melody in my head, I’m tapping out all the signals, the listeners will get it right 50% of the time. That’s that cognitive bias.
In actuality, listeners were getting it right to 2.5% of the time. And I think of that story, and I think exactly that’s how we work in product. In product, I work with a specific user persona or user type for three months. I feel like I’ve sent enough signals out there. My engineers have been enough interviews. We’ve all aligned at the same assumptions that we think that everyone is aligned with the same assumption, and we are totally wrong often about how that assumption is actually shared, not just within the team, but I think more dramatically across other teams that haven’t been a part of those same user research sessions or those same interviews. So like walking into the framework and seeing this play out and seeing how assumptions are always often misaligned between product teams, we hit this challenge with the framework.
And the way we resolved it using the framework is these like 45-minute conversations that were super fun, super healthy, but also 45 minutes. And the challenge of that is like, we are getting to these assumptions, but we’re not sharing it out to any of the other teams. Our delivery team comes up with finally, “Hey, there’s actually 300 users of this user type.” But now how likely is that going to actually transfer to the foundation team or another team that isn’t in those same conversations? So that’s where this idea came from. I realized that you could break assumptions down into three faulty states, I would say. There’s assumptions that can be in conflict, right? So that’s when two product teams or two engineers have assumptions that are different from each other. There’s assumptions that are invisible. That’s when a team might not even know that an assumption already exists from another team.
And then those assumptions are stale. And this happens often when you’re working on a product area, maybe six months ago and you fall in love with this product area and six months down the line, you pick it up again and you’re biasing towards stuff you learned six months ago. There’s a lot of… Lots change in six months. So that’s when your assumptions become stale. And what we realized is all three of those problems actually are solved in code, right? You can actually solve conflicts in code. You can go ahead and codify the date of an assumption, you figure out how stale things are if we codify that assumption. And also again, by storing assumptions as code, literally in a repository, you make it more visible. So in practical terms, what we did is we took the prioritization framework and we started adding our assumptions into essential repositories.
So looked at all the user interviews we’ve conducted, as soon as they’re finished, we go ahead and submit it as code into repository, all the data sources. So we have over six different data sources linked into this machine that pumps out different assumptions. But the key here, I would argue is that it’s all human in the loop enforced, right? We’re using this as a recall mechanism, not a judgment mechanism. So that’s where we store all of our assumptions in. And as we prioritize work, we have used and leveraged different agentic AI tools to use the recall system to help prioritize work and help also go through these assumption conflicts with each other.
Eleanor Millman:
And I’ll mention too, so already what Mina built worked really well when we would be scoring for the prioritization framework and realized we had two folks who disagreed an d they would talk and they would ideally align on something when there was already enough data to have that debate. One other thing I’ve really enjoyed about this framework and then Mina’s solution was that sometimes we would debate and folks really could not agree. We had this assumptions conflict that we couldn’t resolve. And so that, to me as a product manager says, “Go do a little bit of user research.” Now, obviously if it’s a hugely important project that you’re going to spend months and months and months on, I advise you to go do some user interviews, like really maybe make it a little more formal, do your due diligence if your assumptions are going to tank the project if you’ve got them wrong.
But there’s a lot between doing a bunch of user interviews and nothing. And we luckily had those tools at our disposal. I’ve already mentioned if you’re doing a bunch of user surveys, like for instance, with DX snapshots, you have that data available. Another thing that we did sometimes was to send out a more targeted survey to a specific user group. For instance, we had an observability-as-code library and we maybe thought it was being used. We didn’t know how much. We were trying to decide if it was worth the effort to keep maintaining it or not, so actually using DX, we were able to find the right user group in general to look at. We sent a targeted survey understanding how frequently folks were using this library, because we figured if they’re looking at it daily or if they’re looking at the dashboards that it produces daily or weekly, probably pretty valuable. If they’re looking at it monthly or not at all, not that useful. We were able to see that 50% of our respondents were using it at least weekly, and that allowed us to feel good about continuing to put in the effort to maintain this observability-as-code library.
All that’s to say is that with the framework, when we were disagreeing on values, we could go out and do a bit of user research. We could also, sometimes we would just send out a few Slack messages to engineers with a couple questions, and within an hour or two, we’d usually have at least a little bit of data to question or assert that our hypotheses were correct. We would do all this work previously, but now, once Mina put forth this idea of, well, let’s actually codify the answers, the data that we’re hearing for these various assumptions, then we’re able to take this research, and now we have a place to put it and we can reuse it as opposed to having to redo a research study every time we think we might need that data.
Justin Reock:
The as-code aspect of these assumptions is code, and I think what makes this so unique is that you took these things and you actually built a coded framework that you could store in a repository, that you could run some analysis on, but that most importantly, you could then use this in a very transparent way, to drive human discussion, human analysis really, for a lot of conflict resolution around trying to achieve the same goal. It reminds me in some ways, if you’re familiar with the evaporating cloud model, the thinking process and the theory of constraints.
It sounds like this is like almost an evolved version of that, where you have everyone trying to achieve the same goal, two people thinking that they need two different things to achieve that goal, and how do we tear down the assumption and figure out how can we achieve what we need to achieve with everybody getting what they feel like they want. That’s what it’s reminding me of, but it seems like you’ve taken that and put a whole systems approach behind that. I find that really fascinating.
Eleanor Millman:
I kind of like this fractal view in the sense that, as platform engineering, we, of course, want to help development teams standardize on tooling that we don’t want team A and team B to be building the same tool and duplicate. For the prioritization framework and the assumptions-as-code system that Mina’s developed, same thing. We’re kind of preventing our platform engineering engineering teams from not duplicating effort around learning about users.
Justin Reock:
Now, you’ve got a lot of data, it’s codified, have you been able to find AI applications that have been able to assist with some of this analysis?
Mina Tawadrous:
I do believe that this is enabled by some of the AI application we use. In actuality, this is something that we run anytime someone uses an agentic coding tool, as an example, or just a code generation tool. The way that most of our users are using this today is they are, for example, building a product brief or a PRD or they’re creating the description of their story right now through Epic, so if they’re using the coding bots to go ahead and help this generation, it is actually checking this backend repository for those assumptions. It’s informing the user, “Hey, these assumptions exist in a terminal in this case,” and then it’s asking the user to validate it right there within the terminal, and then it’s actually generated in the PRs from the interaction that that user, developer, or product manager has.
The reason why I think this works really well with AI as well is data is expensive. Finding data, doing surveys is expensive, and historically, what we’ve done is we’ve done really good research for our biggest bets, but by having everything in this repository and by leveraging AI on product instantiation, we can actually store smaller initiatives as well. Your smaller stories also run through this prioritization analyst and recommendations are provided on maybe what tier of value this work gets put in, and because it’s so cheap and easy to run, it’s still useful to a degree for even the smallest initiative. It’s helped us scale, which was very hard to do without AI around this, because we don’t want to replace judgment, human judgment. We’re putting the human at the judgment level and at the assumption validation level.
What we want to do is make sure the recall level is helped by or assisted by AI. That’s where AI is in the mix. It’s that, “Okay, you’ve written a pretty interesting Epic or a pretty interesting product brief here. Just to let you know, here are some assumptions that already exist in the organization. Do you agree with them?” That’s where AI can really help, because otherwise, asking users to go ahead, read through every data source, scrape every system on every initiative just probably does not pay out compared to the economic cost of doing that research. That’s where we really see AI leveraging in the system.
Eleanor Millman:
Before AI, before this assumptions-as-code framework, after we developed the original prioritization framework, which, by the way, we really only used it for projects that we thought were about four developer weeks of effort or larger, just because there was a time cost to prioritizing each project. I did work with another product manager on our team to try to do prioritization of the backlog of it happened to be our software delivery team, and there was a lot of value in my mind to doing that. These are smaller backlog paths but there were many of them. I think most teams usually have a significant backlog of a bit lower level or shorter tasks, so figuring out the highest value ones for our users would have been great, but it was really laborious at that time to figure out the assumptions for each of these backlog tasks.
We abandoned that effort at the time because it was just too time-consuming, and so I’m really excited for the way that we can use what Mina’s developed along with AI. We can do it a lot more quickly. I’ll mention also, Mina did something that I thought was great, very pragmatic. He added a very pragmatic feature. I’m going to pause for a second to say, years ago, Mina and I actually met at a place called Pivotal Labs. It doesn’t exist anymore. Their three values I loved, and I’m going to quote them because it’s relevant for now, the values were do the right thing, do what works, and then be kind. I mentioned this because, of course, as a product manager, yes, if we have infinite time, let’s please do the right thing, let’s do the right user research, let’s review each assumption for every prioritization effort we do.
Let’s give it all of our focus because this is so important, but also, we got to do what works. Realistically, we, as product managers, but even more engineers and engineering managers who have many other responsibilities in their jobs, they’re not going to have time to do a half an hour review of the assumptions perhaps
Eleanor Millman:
I really like that Mina has built in different options, which I’ll let him explain, that really lets folks sometimes do the right thing and sometimes do what works, but either way bring more prioritization and assumption knowledges in than we would have before.
Mina Tawadrous:
There really are different personas or different user types even using a tool like this. I think there’s three modes, two to three game modes, but there is like a rapid prioritization where you could just have the tool go ahead and recommend a store for you based on the rubric, based on the OKRs of the quarter, and based on, again, prior assumptions about the user base, et cetera. It’s like a 30-second plug-and-play, “Hey, you just wrote a thing in Jira, here’s what it actually equates to.” But then there is like standard mode, which is more conversational. You’re actually working with an agentic conversational AI tool to kind of ask questions and where we’ve seen the most value, because when engineer leads are going through that conversational standard mode, they’re being asked by the AI tool. “Oh, okay. It looks like we are addressing this type of incident. I can make a guess that the incident has probably occurred at least two times a month. That’s what the industry standard is. Do you agree?”
Of course, I usually encourage the engineer or manager statute go and check their own data, I think that conversational standard mode is where we’re seeing most of the value today, and then there is like a deep-dive mode where if you’re someone that’s really interested in the product thinking and you want to guide maybe a new initiative, maybe this is the first time we picked this user group, maybe this is something that we are pretty sure we don’t have any internal assumptions about already, there’s like a 15-minute mode that would really dig in and make sure that we are generating the correct assumptions and making sure that we are analyzing what assumptions already exist and we’re solving all those conflicts a little bit deeper than some of the other two modes, because in plat eng, we do have different types of work that we would run through this and different type of users as well that have different needs from this tool.
Justin Reock:
A lot of what we’ve talked about here is scale, using technology and cultural process to allow more and more efficient use of the platform, and we spoke earlier about now we have this need and, really, the industry is seeing this shift moving from the term specifically software developer towards just builder. How has that distinction been important for both of you and what do you think the implications are in terms of scaling your platform?
Eleanor Millman:
Well, I’d say it’s something that’s been on my mind quite a lot. As a product manager, by the way, I certainly work to empathize with my users no matter who they are, and as I’ve said, developers have always been one of the user types that interests me most. This builder persona is especially interesting because it’s personal for me and for Mina in the sense that I am not a developer, I am a product manager, but I am what I like to call tech-forward. I’m very comfortable with tech. I do have some experience coding. Other examples that I’ve seen are we’ve got like a marketing strategy guy who has no coding experience but is just obsessed with AI and really loves diving in and figuring out how to build stuff. I’ve encountered another person who was one of the highest users in our chatbot, and I went to talk to him again, not a developer in any way, but he was just building this Python app by asking ChatGPT questions. This was before more of the agentic AI came out.
All that’s to say is I really believe that in platform engineering, we have to cover these users. Platform engineering, it was very easy. It used to be people who build apps, our developers, or our users. Now, people who build apps are more than our developers, and we need to find ways to help them because they will proceed either way. Most people really want to have a golden path, follow security best practices, et cetera, et cetera. Let’s give them that opportunity.
Mina Tawadrous:
I can add, it’s always validated. I think plat eng is an interesting place where we like to see what’s thrown on the rest of the industry and then we kind of replicate and build our worlds around it, like who are the tenants of building… Who are the people that are going to live in these little apartments that we have in our platforms? I think I read a stat recently around the GitHub, GitHub, year over year, had a 25% increase of users just this last year. I find it hard to believe that we’ve built 25% more technical software engineer roles and organizations, so who are these users? Who is going into GitHub today and building these repos at a quarter of a higher rate than ever before? I think what we are seeing is this democratization of building, and this has been covered kind of everywhere. I think it would be kind of a shame to have this growth happen outside of organizations, not replicate that same sort of safe space within organizations. And I think platform engineering is going to probably take a lot of lessons from the open source world of what it means to allow for different types of contributors that are not traditionally the maintainer of that repository or that app or that idea. But what would it be like if we actually started allowing for different user types? And that’s a fun but challenging space because let’s not downplay the amount of effort and energy that we’ve gone through to make sure that software engineers know security. They know InfoSec and PII rules. That’s a lot of work. So the answer is not to open the floodgates and let anyone push anything anywhere. But it is also, I think at this point about enabling business value while also mitigating risk the same way that we’ve looked at everything else.
Eleanor Millman:
We would be remiss if we didn’t call out the challenges to platform engineering. And Mina’s already mentioned the security, privacy concerns. Of course, also there’s supportability. Suddenly you have all these folks who really don’t know much about software development, developing software. If we’re not staffed in platform engineering to support these folks, especially across an entire company. So we have to find better ways to support folks like this. And also there’s concerns such as maintainability. Someone builds a great app, maybe they get a nice hundreds of people at the company are using it. Who knows? And then what happens if they go to another job? Sometimes platform engineering might be asked to take on support, maintenance of this particular internal app. That’s a little nerve-racking too. \We need to put the thought in to address all of these concerns that are coming out of platform engineering.
But I think that they are all solvable. I’m really excited, just for example, on the support issue. As we know, these agentic coding assistants can read from things like skills. If we just do a good job of documenting security best practices that we want to be used at our company and put it in a GitHub repo, that of course can be turned into a doc site, you can either have the human, the builder, the non-developer builder, they can read through the doc site and learn it, or they can even point their coding assistant at that. And that coding assistant will probably follow non-deterministically, but probably follow those security recommendations, those privacy recommendations, scalability recommendations, whatever it may be. So I’m really hoping that we can take all that knowledge that exists in many developers’ minds and try to get it out in front.
And lastly, I’m very passionate on this subject. I’ll point out that if we do have our more senior developers, our InfoSec folks documenting best practices, it won’t just help the non-developer builders. I’ve been a junior developer and boy have I appreciated it when I have had access to documentation, especially customized to my company of what I should be doing. So I think that a rising tide lifts all ships. If we make our platform easier to use for everyone, smoother onboarding, better documented, it’ll support far more than just the non-developer builders.
Justin Reock:
So it sounds like although scale has always been a challenge and you’ve worked on things and all of a sudden we’ve thrown accelerant, you’ve added the number of people now who need to be able to take advantage of this platform, you’ve also found ways where AI can help augment the work that you’re doing and help solve some of these challenges as well. So hopefully we’ll hit that nice balance and that’ll help kind of navigate kind of this new shift. I’d love to hear, we talked about prioritization, we’ve talked about that framework for prioritization. What are you prioritizing for 2026 at Sirius in terms of your roadmap for platform and for culture?
Mina Tawadrous:
So it’s funny, I think a lot of the things we talked about today make it not obvious, but you kind of see where we’re going with the roadmap. So I think what we’ve seen is a lot more building going on. So I think the bottleneck is no longer being able to build a tool, it’s also discovering that it already exists. So investments in internal developer platforms to ensure that there is an idea of what exists already, the ownership, et cetera. Those investments will continue going forward at SiriusXM. And I imagine this to continue in other platform engineering orgs, but really increasing the knob of value that we get from discovering what already exists within a system and also aligning ownership to individuals so that we could figure out who owns those things. I think as we kind of alluded to, I think that’s going to become more and more important this year.
So we have a lot of work in the books around kind of discovering those things. Also, if you think about it as well, like now that we’ve discovered, “Hey, there’s a really awesome tool being built by this team,” we’re going to be building a lot of work to make it easier to contribute to each other’s projects, to break down these silos and make sure that there’s actual tools that get added into the platform to make it easier, to contribute to something that already exists. Because the cost to build something on your own is actually quite low now, any sort of barrier or friction to contribute to an existing project needs to be kind of removed. So we’re looking at ways to remove that friction. And kind of I think what’s on everyone’s mind right now, and this is an interesting one is like, we’ve seen a high throughput this past quarter, like a high PR throughput.
DX has a great kind of metric around true throughput that we use as well. It’s gone up. And that’s great on one side, but I think what we’re obviously looking at is like, well, now you’ve got a bunch of, a lot more contributors and if our vision comes true around different types of contributors, I think there’s going to be efforts on making sure that the bottleneck does not stay closed on, “Okay, this is great code, but now how do we review the code?” So code reviewers things to make code review easier, things to make code review faster, more automated, and maybe possibly a deal with some of the more basic ones and help prevent a lot of the PR bottleneck. I think that the PR space is something that we’re investing in this time around as well because of all these part A, faster engineer throughput that we’ve seen, as well as a higher volume of users that are able to contribute as well.
Eleanor Millman:
So we in platform engineering definitely are thinking quite a bit about AI because we want to make sure that developers and non-developers, if they want to use AI and feel like it’s valuable to them, we want to make it easy for them. We’ve certainly done some work for supporting AI early adopters, but from a learning standpoint, they’re great. Almost the definition I would say of an early adopter, especially on AI, is they’re so hungry and excited for AI that they just go figure it out and we have to unblock them with tokens and permissions, whatever. But one focus that I’ve especially been thinking about is, well, what about the majority that we’ve got a lot of developers who just have so much on their plates and they want to learn more about AI or they want to get started at least, but they’ve just got a lot pulling on their time.
So I think that we, in platform engineering, need to think about how to enable these majority users, not the early adopters. So things like just really, really clear onboarding instructions. We really care about telemetry from them. So we’ve done work though instead of having to set various variables so we get the right telemetry, let’s find a very smooth onboarding process so we get telemetry no matter what, so that they’re set up to hit the ground running no matter what. Also, a few basic enablement sessions. People learn in lots of different ways. I was a high school physics teacher years ago, and I certainly learned this then, that some people just want to be handed a doc and they get to read it on their own. Other people like to be shown, even if it’s simple, like, “Let me show you how to install this agentic coding tool. Let me walk you through the steps. Let me show you one or two examples,” and suddenly they feel unblocked.
So we’re looking for ways to help that majority and to segue into, of course, that topic that we were just covering, that this AI enablement, of course will help those non-developer builders as well. I would say the last focus very related to this is understanding more about who these citizen developers are, these non-developer builders. However, we want to call it, we’ve got to standardize clearly on a term at some point in the industry. But as we talk in platform engineering about how to support these folks, a lot depends on what they want to build. If they want to build fully functionized, productionized apps, that is a big ask and that will require one type of guardrails and help whatever. Or some assumptions or hypotheses we have is, maybe they don’t care about internet ingress, maybe they only care about egress. Maybe they don’t actually need internet access at all.
Another one would be which internal tools that have data do they need to connect to? Certain internal tools. We got to lock those down. Others, maybe there’s less of a risk if they want to access them. So we’re doing some user research, talking to these builders, seeing what they want to build or what they’ve already tried to build. And lastly, I’ll say, figuring again how much they need to scale it. Another assumption, hypothesis, whatever, is that maybe they only want to share with up to five folks on their team or build a POC that’s shown to 10 stakeholders and that’s that. That is way easier from a scalability standpoint and availability standpoint than something that will go to external users like our SiriusXM listeners or that will be scaled to a thousand internal users. I mean, they’re just fundamentally different requirements. So we’re really spending time understanding what these folks need.
Justin Reock:
Thank you so much for both of your perspectives. It sounds like you’ve done a lot of forward-thinking about problems that I feel like a lot of platform engineers are actually just starting to run into today. So I think there are going to be some really great takeaways for our audience, some really great perspectives on how to deal with kind of this kind of new shift and these new challenges that are going to have to be dealt with. So Mina, Eleanor, thank you so much for taking the time today and coming on the podcast.
Eleanor Millman:
Well, thank you so much for having us. It was a pleasure.
Mina Tawadrous:
Thanks.