Kelly Anne Pipe:
Thanks so much and thanks everybody for joining us today. Conference has been going great so far.
Nicole Scribner:
And a big thanks to our DX friends for inviting Vanguard to the stage to share our journey.
Kelly Anne Pipe:
So as was mentioned, we, at Vanguard, are putting together a maturity model for our teams and we’re calling it Augmented, Accelerated, Autonomized. Later, I’ll get into what that actually means. First I want to start with, in one word, how would you describe your org’s AI adoption so far? Think on that. I’m going to pass it to Nicole.
Nicole Scribner:
Hang on. We’ll come back to this slide. Keep those words in your head. I’m Nicole Scribner. I’m a director in our chief technology office in engineering, enablement, and advancement. And I’m going to walk through the leadership lens of this journey today.
Kelly Anne Pipe:
And I’m Kelly Ann Pipe. I’m the head of developer experience at Vanguard. I work directly with our engineering teams, and I’m really in the trenches kind of building out the framework, helping them implement it, running the pilots, and catching all the data that comes in.
Nicole Scribner:
For those who may not be familiar with Vanguard, we are a financial services company. We are headquartered outside of Philadelphia, Pennsylvania. Our mission statement is on the slide behind me, and we have 20,000 crew members who fully believe and embrace this mission. To take a stand for all investors, to treat them fairly, and to give them their best chance for investment success. And we have 20,000 crew members, which we call our employees, rallying around this mission for 50 million plus clients.
Kelly Anne Pipe:
So we asked before we started, in one word, how would you describe your org’s AI adoptions so far? We asked some others before we came here, and we got kind of these responses. A little bit all over the place. The biggest thing was that it’s accelerating. We got, “It’s awesome, it’s overconfident, strong, risky, weirdly slow and fast.” Any of these sound familiar?
Here’s the thing. A year ago, our AI reality at Vanguard was this. We had rolled out Copilot for engineers. There were some pockets of brilliance. We had some power users doing really incredible things. We had our PMs not really using AI at all. They were writing requirements the same way they always had. And then we had our designers and our QA who were also not really touching AI tools for their products. They were still doing their manual testing at the end of the sprints. We had an AI engineering initiative, not an AI product team initiative, and that’s a massive difference.
Nicole Scribner:
A really important distinction, Kelly Anne. And although we love that our engineers report that they’re 30% faster, there is a big world to the left of the engineer. We’ve heard a little bit about that world today in some of these other talks. We have problem in solution discovery, and we have requirements in design that aren’t moving at the pace of the engineers. And to the right of the engineer, we still have some slow test processes. And so we are not getting to production and getting that value to clients as quickly as we want. So because we are focused and had been focused on the engineers, it’s wonderful that they are more efficient, but we’re not seeing the overall improvement and cycle time that we had hoped.
Kelly Anne Pipe:
This is what we’re calling the engineering bubble, where the engineer is using their AI, they’re finishing the things they need to finish in half the time, but they had been waiting for the PMs to give them the stories. They’re taking three days to a week to make the stories. The designer has to handcraft every wire frame. The engineer’s knocking through that backlog real fast, and then there’s no more stories coming up, at least no high quality stories. And then over here, after we’ve written the code, our QA has to go through it. It has to go through all the test processes. So this engineering bubble is when your organization creates AI transformation with, “We bought everybody Copilot licenses.” When your AI strategy is just an engineering tool rollout, it’s not about adoption rates just for the developers. We need to look at what’s happening with all of the roles.
Nicole Scribner:
As a leader, and most leaders, I love data, I love metrics. I love seeing the metric that our engineers are 30% more efficient. But the metric I love the most is in this black box, the output metric. It’s about delivering value to our clients as quickly as we can, and that’s what we’re focused on. So yes, as leaders, we ask questions about cycle time, we ask questions about productivity, but if we only have one input metric on the engineer trying to drive improvements in cycle time, we’re not going to see that ROI that we hope for and our story falls apart.
So we’ve been asking this question, how do we help engineers code faster? And what we’ve realized is we need to ask a new question. How do we enable our full stack cross-functional product teams to go faster? And for Vanguard, that’s 800 product teams across the enterprise. And so now that we have this new question, we’ve also developed a new aspirational goal for these 800 product teams, five times faster by 2030. What if all 800 teams could go five times faster?
Kelly Anne Pipe:
And not by adding more headcount, not by outsourcing, but by really embedding AI across the whole product development lifecycle. Dude, to do that, we need a map.
Nicole Scribner:
And that’s the question that led to the map. How are these 800 product teams going to know what to do? What behaviors should they emulate to get to five times faster? Kelly Anne, help us out.
Kelly Anne Pipe:
So our map is an AI driven product team maturity model. It’s a lot of words. We have three levels, six dimensions, and it applies to the whole team, not just to engineering. Now, this is a work in progress. When we first started the presentation, it wasn’t the three A’s, it was called something different. But it has always been the three dimensions. We are running this past our IT leaders, we’re running this past our product teams, we’re running this past everybody involved in the process to really make sure that it’s matching what we want to do.
Nicole Scribner:
And as our pilot teams get more familiar-
Kelly Anne Pipe:
It’s matching what we want to do.
Nicole Scribner:
And as our pilot teams get more familiar, we will refine and we will continue to refine this model to meet the needs of our product teams.
Kelly Anne Pipe:
So this is the three stages, Augmented, Accelerated, and Autonomized. Augmented is where you’re establishing those sustainable practices with some measurable productivity gains while you’re managing risks appropriately. So every role is beginning to use AI consistently for some things. 20 to 30% of the tests are getting some AI assistance and we’ve got some shared standards that are being established. AI is basically a helpful tool. Accelerated is when AI moves from a tool to a real strategic differentiator. We’ve got not just every role using AI, but there’s role specific AI practices that are embedded. We’ve got 30 to 50% faster delivery. Autonomized is where it’s really redefining what’s possible. 60 to 80% of the routine work is going to be agent driven. We have two to three times capacity expansion with the same amount of headcount.
Nicole Scribner:
So in addition to these maturity levels, we have six dimensions of maturity that we have outlined. And I’m not going to read the descriptions on the slide, but we are going to deep dive into each one of these today and share with you some of the behaviors that we hope our 800 product teams will adopt. And we hypothesize that these behaviors are what teams need to improve. So why don’t we jump in to the first dimension?
Kelly Anne Pipe:
Yep.
Nicole Scribner:
All right. This is my favorite dimension, partly because it addresses everyone on the product team. This is no longer just about the engineer. So this first dimension is AI powered delivery products. So the engineers are still leveraging AI here to generate code, to generate their docs, to generate tests, but we’re also getting our product managers and designers and other roles into the mix here with tools. Think of a world where the product manager is out in problem discovery, talking to stakeholders and clients, has their client notes, and AI helps those product managers translate those notes into workable requirements that then the engineers can take and implement. That is the world that we want to live in.
As teams get more advanced in their maturity, AI becomes the default. We see teams experimenting more and we hope to see at least a 30% improvement from ideation to delivery to production for our clients. When we get to an Autonomized state, this is where it gets really exciting. We still have our product managers working directly with the clients, ideating, understanding their needs, but then we have agents in the mix and agents can then take that information and orchestrate from requirements, hopefully through to production. We don’t expect to hit an Autonomized date this year, but it certainly is a north star for us.
Kelly Anne Pipe:
So you can have the best AI tools in the world, but if your code base isn’t ready for agents, they’re going to struggle. So Dimension 2 is around an AI ready code base. At the Augmented level, this looks like the basics. Every repo having a comprehensive README, an agent’s MD or a Claude MD file or whatever the next iteration of that might be. You’ve got linting enforced, you’ve got some unit test coverage above it 70%. CI/CD is running on all commits. Your agents can navigate and understand the code base. At Accelerated, you’ve got fast feedback loops. CI/CD runs in under 10 minutes. Code coverage is above 85%. You’ve got comprehensive API docs, architecture decision records, security scanning integrated. Agents can more quickly iterate through the code base because they get fast signals that their work is correct. At Autonomized, they’re generating complete deliverables from specs.
They self-test. They autonomously refactor, update dependencies, and improve code quality. Your code bases are really agent ready at this Autonomized stage. And the key insight here is that the code base is really the interface between your team and the AI agents. If that interface is bad, everything downstream suffers, no matter how smart the agent is. Raise your hand if your code bases, if most of your code bases, most of your repos have Agent MD or Claude MD in most of the repos. Okay. I see some hands raised. Now raise your hand if throughout your whole code base, you don’t even have a README in every single repo. I see a couple of different hands raised there. Everybody’s at a different stage of how ready their code bases are. And across 800 product teams, we have way more than 800 repos. We’ve got a huge code base. So we have quite a ways to go even to really hit the first level of this maturity model.
Nicole Scribner:
Yeah. And Kelly, and that’s definitely been a theme today in getting our documentation ready for AI and agents. I think we’ve heard that through every talk.
Kelly Anne Pipe:
The third dimension is around how much real work are agents actually doing. This is making sure that agents are not just auto completing, but they’re actually implementing things. The key shift here is that we’re shifting from agents are assisting me to, I am now an orchestrator of agents. That doesn’t mean the human role disappears. The human role elevates. We are in the early days here. We’re leveraging some agents that are provided by our SaaS partners and learning as we go to see how we can get these agents in an orchestration across the whole PDLC. Now, this dimension is about what happens after you ship. Dimension 4 is AI Augmented operations. Can AI help you monitor, troubleshoot, and heal production systems? So in Augmented, that’s the monitoring and troubleshooting part of it. You’ve got some basic monitoring and learning. AI is helping you generate those postmortems when things inevitably go wrong.
In Accelerated, maybe there’s much fewer disruptions and much fewer need to develop those postmortems because you’ve got some predictive monitoring. You’ve got automated remediation for 30% of your incidents. By the time you get hit Autonomized, your systems are self-healing and 70% of incidents are auto-remediated. You have near zero unplanned downtime and continuous optimization. Sounds like a dream. Your teams can stop firefighting and start innovating.
Nicole Scribner:
All right. Dimension number five, team autonomy and enablement. And we won’t, again, go through all of these bullets, but we definitely want to double click on the behavior related to dependencies. Kelly Anne’s been talking a little about agents. We know agents are fast. They’re superfast. They’re faster than humans. And you know what’s not fast? When we have to wait a month for an approval to use an API from another team, or we have to wait two weeks to get approval from security to this new application, or we have to wait to get an environment stand up. When agents enter the picture into that ecosystem, it stops them in their tracks. And what may have been tolerable for the human, now again, it just stops us. And so we are really focused at Vanguard in trying to mitigate dependencies through a variety of levers.
We want to analyze where we can across our 800 product teams, understand what commonalities might exist and what’s slowing people down. Is it security? Is it a vendor, et cetera? And we want to tackle those problems now together as an enterprise. And we’ve created an internal tool called our wait time analyzer that’s analyzing all of this workflow data and not only giving enterprise insights, it’s giving insights into our subdivisions who may have local issues that we don’t experience at the enterprise level and that we don’t necessarily address in our chief technology office. So dependencies are so critical to get them under control and to mitigate them.
So again, dependencies that are tolerable at human speed become the bottleneck at agent speed. And when agents can implement a feature in ours, every human speed dependency becomes very painfully visible. And that’s when we started saying, “Dependencies are really defects.” All right. Our last dimension, last but not least, responsible AI. And this may seem counterintuitive because I think when we hear about responsible AI, people may think about governance and privacy and equate that to slow, a break, something that’s in product team’s way and doesn’t allow them to move forward quickly. And sometimes we hear that we’d be a lot faster if compliance would just get out of the way, but we’ve actually started to find the opposite to be true. At this Augmented state, at this early level of maturity, we’re really focused on our crew having all the training that they need around our AI policies, around guardrails, around the audibility expectations, so they have what they need to build.
They can build without fear, they can work within those guardrails. At an Accelerated state, automated controls are doing a lot of the heavy lifting. We’re thinking about things like data classification for sensitive data. We don’t want that sensitive data to ever reach our AI tools, and AI generated code is auto scanned. We’re testing for bias and fairness. In an Augmented state, which again is our greatest maturity level here, we again don’t anticipate to hit this this year, but this is where we actually can leverage AI to help us with our endeavors.
Kelly Anne Pipe:
And some of the guardrails is one… The guardrails around responsible AI, it’s one of the first things we put into place. We actually have an AI SDK at Vanguard that was one of the first things we brought in so that our teams have a safe place first to experiment with AI and now to begin building things off of AI in a place where they know that we’re going to make sure that the Vanguard stays safe.
Nicole Scribner:
And getting to that state of continuous monitoring and self-healing is definitely a place that we want to go. All right. So we talked about these six dimensions and some of the behaviors.
Kelly Anne Pipe:
It’s a great framework.
Nicole Scribner:
What’s that?
Kelly Anne Pipe:
It’s a great framework.
Nicole Scribner:
It is a really great framework and we feel really great about it. We also think we know the tools that teams are going to adopt to help them achieve adoption of these behaviors, but like many others that we’ve heard today, I think Jennifer mentioned it this morning and our friends from Mercari talked about this as well in their talk. It’s not the tooling that we’re worried about, it’s the behavior change. And we think that is truly-
Nicole Scribner:
… that we’re worried about, it’s the behavior change. And we think that is truly one of our biggest barriers to adoption. Changing hearts and minds is often much more difficult than bringing in new tools. And so as a leader, I don’t know if you can all relate with this. I hear what you see here on the slide. Crew saying, “Well, you’re telling me that we are going to use AI to automate routine tasks. What does that mean about my job? Do I still have one? Why should I adopt and skill up if it’s going to take my job?” And what they’re not hearing is, “Hey, we want your role to evolve. We want you to focus on ideation and strategic thinking.” And really, the fear that creeps in is really is what preventing adoption in some cases. And we as leaders need to help our crew and our employees understand through example that their roles are not eliminating.
They’re going to evolve just like we have over the years when technology has changed. And the more examples that we can show our crew of where this is happening successfully across our 800 product teams, we feel that that will help us start to slowly overcome this fear.
Kelly Anne Pipe:
The second problem that we have is the measurement problem. The easy metrics are misleading and we have to remind our crew that we’re not actually interested in those easy metrics and remind the leaders that we’re not actually interested in those easy metrics. We are not interested in the number of lines of code that are generated by AI any more than we’re interested in the number of lines an engineer put into a PR.
Nicole Scribner:
That’s right.
Kelly Anne Pipe:
We’d love to say how much time was saved per developer, but trying to isolate that is very difficult. We have to make sure that any measurement we do of the success of this is multi-layered. We can track adoption by are people actually using the tools? And then we need to also track, is the use of those tools actually helping processes change? And then we really need to track, is those processes changing driving the cycle time down? Is it driving higher quality? Is it driving more value? The point of doing all of this, especially at Vanguard, is to deliver more value to our clients, to help more people reach their financial goals, and to be responsible stewards of the Vanguard. And we have to make sure that any investment we put into AI really comes out with that outcome. Shorter cycle time with higher quality and more value.
Nicole Scribner:
Okay. Another audience poll. What is your number one blocker to scaling adoption? And two different questions here. Two options. Number one, is it people? So things like leadership adoption and buy-in, skills, behavior change, culture. How many people feel people might be their number one blocker? Okay. A few hands. All right. How about structure being your number one blocker? This is things like tooling. This is things like measurement, dependencies. I love seeing this.
Kelly Anne Pipe:
Some people don’t even have their hand raised for either. So I’m going to be finding you all and seeing why you don’t have any blockers and what lessons you might have for us.
Nicole Scribner:
We have to pend our slides to include that. So when we ask this question internally, depending on who’s in the audience, the answer will vary. If we have a group of product managers, typically we hear them say skills is their blocker. If we have engineering, they’re talking to us about new and different tooling. And when we have the leadership team in the room, there’s always a lot of debate about measurement, whether we have the right measures and whether we’re set up for success to measure. Now, the great part about the framework and the dimensions is now that we have a common vocabulary to talk with each other about the blockers and we can pinpoint these behaviors to really start to dig into what is holding us back in a more specific way.
Kelly Anne Pipe:
We’re going to quickly touch on some of the lessons we’ve learned as we’ve begun to roll this maturity model out at Vanguard. And then there should be some time for some Q&A. So lesson one is we need to meet our personas where they work. Listening to the last talk, and I loved hearing that some of their PMs were hopping into Claude Code and doing great things at the CLI. We’ve seen that with some of our PMs. Some of our PMs are like just pulling down terminal, hopping up terminal, pulling down Claude Code, making their PRDs. Love that for them. All 800 of our product managers are not successfully doing that. All 800 of our designers are not successfully popping open terminal and feeling real comfortable going right into Claude Code. We need to really meet our personas where they are. Oops. Not every tool is right for every person.
Nicole Scribner:
All right. Second lesson, and again, I think this is a lesson we’ve heard quite a bit today at this conference. Tooling is the easy part. Changing hearts and minds is the more difficult thing. It takes time to grow at scale, and we have to keep with the same lessons and the transparency around behavior change. One thing we recently did at Vanguard, which was a huge success is we had one day where we brought our engineers together for a… And a training or a conference that we called Prompt Intel. And this was a subset, maybe about 10 or so teams out of the 800, that showcased what they’re doing with AI and specific tools, how they’re making the testing process better, how they are building better designs. And we want to do more of that. The more all of our roles on our 800 product teams can see the success stories and see people leaning in and blurring lines for existing roles, the more successful that we’ll be in our journey.
Kelly Anne Pipe:
Lesson three, our agent speed is exposing a lot of organizational debt. So here’s some sample cycle time. Our agents are driving implementations in two days, but their design review is taking four days. And then that API that you needed to onboard to isn’t giving you the onboarding for three days. And then the security review queue is all backed up and you can’t get your stuff reviewed for five days. And then finally, deployment of approval is going to take two days. This is not actually how it happens right now at Vanguard, but it’s just an example of how the processes that are in place aren’t matching the speed of AI implementation. Agent is fast, the organization is slow. So the way that we’ve always worked is now the thing that’s holding us back. And we need to think of how can we utilize AI to improve these processes where we’re still having the same high quality standard faster.
Nicole Scribner:
And the last lesson for today, embrace responsible AI. It is an accelerator, not an impediment. It is not a break. Feels counterintuitive. Again, we have to think differently about responsible AI and governance and privacy. It’s there to protect us and to protect our clients. And we have to continue to share that the idea of we can bypass this to move faster is really not the right sentiment because we’re finding the opposite. If you invest in responsible AI early, like policies, scanning, et cetera, you are able to move faster, you’re able to move safer, and you can feel good about the quality of what you’re delivering into our production environment.
Kelly Anne Pipe:
Let’s talk about what’s coming up ahead.
Nicole Scribner:
What’s ahead? Okay. So a few key things. I’m sure there’s more than these, but these are the things that are top of mind. The delivery lead or the delivery manager role is shifting and is changing. Today, a delivery lead manages a team of people. Tomorrow, we will have agents in the mix and our delivery leads and managers will need to think about that interplay between the humans and the agents from everything from workforce planning to resource allocation, quality oversight, workflow. What does it mean to have both working together? What can we expect, how to plan? We’ve heard it’s really hard today to plan a year in advance. So we really have to think about what that means and determine how we can train our leaders to be thinking about this one.
Kelly Anne Pipe:
Second is that the measurement’s going to catch up. It’s not there yet. There’s no perfect AI ROI number. If you know one, please come and tell me. But there are leading indicators for sure. For us, as people progress through this maturity model, as our cycle time trends, hopefully up down for cycle time up for quality. And as our quality metrics go up, together that can tell a compelling story. So not quite one AI measurement, but together the numbers will begin to tell the story.
Nicole Scribner:
All right. And last but not least, 5X is ambitious and we want to be setting a high bar. Will all 800 product teams get there? Probably not. Will they improve their cycle time? We bet that they will to some extent, but the teams that are furthest along are already seeing that adopt some of these behaviors are already seeing two to three times faster cycle times. And the gap between AI mature and AI immature continues to grow. And so we can’t wait. We have to act and we have to set that bar high now.
Kelly Anne Pipe:
I just want you to think is, what’s one thing that you might do differently after this talk or after any of the talks you’ve heard today? Whatever insights that you’re gaining today is going to be the most valuable thing for you because it’s your team, it’s your company, it’s the way that you work. Our maturity model is probably not going to work for every company at this conference, but you can probably glean some insights and maybe build your own maturity model that will work just for your company.
Nicole Scribner:
And if we reflect on some of the other talks, there definitely are similarities that I think we can take away and reflect on and figure out how we can embed them into our processes back at our companies. There’s one thing that you take away today. I think it’s this on this slide. This AI transformation is not just about engineering. Be treated as an engineering initiative, probably won’t be successful. It’s a product team initiative. We need to get all roles involved. We need to get product managers, designers, and continue to have our engineers to get more efficient as well because when we optimize all the links in the chain, that’s when we get faster, that’s when our cycle time will decrease, and that’s when we will deliver to our clients faster in a production environment.
Kelly Anne Pipe:
That’s it. Thank you so much. We have some time for Q&A.
Moderator:
Yeah, you’re perfect. Yeah. Thank you. That was a really interesting framework. I like a lot of how that’s going to be able to guide different types of industries. We have some really good questions from the audience I want to get into. So you touched on this a little bit, but a lot of what you call out in sort of AI readiness seems independent of actually using AI tools. So did you increase investments in platform and in platform engineering and how are you prioritizing the boring work of getting the infrastructure ready?
Nicole Scribner:
I think-
Moderator:
… work of getting the infrastructure ready.
Nicole Scribner:
I think what we’re seeing with AI is some of the things that maybe we haven’t addressed, things like certain types of technical debt are definitely exacerbated in this type world. And we need to go back to some of our best practices around testing and requirements and make sure that they are visible for AI to absorb so it gets the full context.
We have not specifically hired more people or consultants to help us solve these problems. We’re looking to work within the product teams and we are trying to maintain a state where 80% of teams’ work is around feature work that directly benefits our clients and about 20% or maybe 15%, 20% is around non-feature work and it’s cleaning up some of those ills of the past that we are now facing. We’re also hoping that AI can help us accelerate and change some of those ills more quickly in this new environment.
Kelly Anne Pipe:
We’re actually encouraging, if you are coming to the engineering teams as the major tech effort, everybody needs to change X thing in their code base for this new role. Come to the teams with a solution possibly generated by AI or tell teams how they can utilize Claude to help them change their node version. All that kind of boring stuff, some of it can be automated.
Moderator:
Oh, that’s really interesting. It’s kind of a meta perspective, I guess. Cool. All right. So on dimension number six, can you dive in a little bit more about the audit trails, like specifically what’s included in those audit trails?
Nicole Scribner:
Yeah. This responsible AI is dimension number six, and that is the one we are really just getting started with at Vanguard. And the emphasis right now is on the training and giving everybody the foundational tools to work within AI. I think in a more mature state, we’re working towards a world where we have AI to help us continually scan for defects, to understand the health of our production environments. I don’t know that we have … I think the tool stack there is open for what we use, but that’s the idea is how can AI help our crew maintain the health of a product or an application and help us reduce the BAU. Kelly Anne, would you add anything to that?
Kelly Anne Pipe:
Yeah. As we begin to roll out any of these tools, any of these best practices, we sit in the CTO. We’re also working really closely with our security partners as we begin to roll things out to ensure that what audits, what best practices should we be putting in place as these new tools come out, as these new practices come out, so that we are making sure everything stays secure. And if we have these things in place today, what would the future look like? And how can we not just be as secure as we are today, but more secure in the future?
Nicole Scribner:
Yeah. And being part of financial services, our security bar at Vanguard is very high.
Kelly Anne Pipe:
Very high. Very, very high.
Nicole Scribner:
As we take great care in making sure that we have all the proper checks and balances in place.
Kelly Anne Pipe:
There’s restrictions right now in what you can use AI for. We don’t allow production data to go through AI until we are absolutely sure that the critical guardrails are in place to enable that kind of stuff.
Moderator:
Interesting. Okay. Lot to think about there actually. How far along are your teams on this maturity map right now?
Kelly Anne Pipe:
So I think there’s three stages and six things.
Nicole Scribner:
Six dimensions. Yeah.
Kelly Anne Pipe:
Thank you, six dimensions. I think it depends on the team. We do have some pilot teams that hopped on the train real fast and intentionally, we got a senior developer, a real innovative product manager, and we put a tiger team on this and had them run real fast and see what they could go with. So they were able to come up with a lot of success.
A couple teams adopted some of their things that they had done and are seeing some similar success. We have some teams where just, hey, the tech lead really loves AI and is really clever and innovative and just kind of crack the code in the way that, that team operates. So we have pockets of brilliance. I think overall, most teams are across the six are kind of in that beginning phase still, maybe moving towards the middle.
Nicole Scribner:
We’d like to see a lot of the teams get to that first phase this year and see some frontier teams be out in kind of the middle phase. We brought in Claude, I think August, September of last year, and that’s where some of these learnings came from. We were trying to have product managers go in and work in a way that engineers did, and that didn’t always work. And so I think we really truly started thinking about the different personas probably in the fourth quarter of last year and testing and learning with some frontier teams. And I think we feel that we have enough learnings now to go out and test with a broader set of product teams.
Moderator:
No, that’s good. And it sounds like there’s even an organic or even kind of grassroots component too. You find the heroes and replicate what they’re doing.
Nicole Scribner:
Yes, that’s right. That’s right.
Moderator:
How does Vanguard ensure cycle time measurement is consistent across 800 different product teams? Do you have any guardrails in place specifically to prevent Goodhart’s, gamification, that kind of thing?
Kelly Anne Pipe:
We do have an ongoing conversation with our product teams to try and avoid gamification. I don’t think there’s a perfect way to do it. You give somebody a metric, they’re going to be like, “I’m going to gamify that until I hit the metric.” We do try to avoid that by trying to avoid over-anchoring to any single thing.
So if your cycle time is really low, you’re delivering things really fast, but all the things you deliver break constantly, that’s not successful. If you’re delivering really fast, but none of it’s getting to the client because it’s not what the client needed and your CSAT scores are still down, then that’s also not successful. So trying to avoid people just running at one of the metrics by making sure the other metrics are paid attention to too.
Nicole Scribner:
I’m just going to add on to that, Kelly Anne. We do have a great deal of senior leadership support on cycle time. It is an IT-wide objective to reduce cycle time this year. Every one of our subdivisions is at a slightly different place in their journey. And these are 800 global teams and some have more aggressive goals, but having that kind of IT lens and our CIO saying, “This is important,” really helps the 800 product teams rally. And it’s not perfect, and I think our data is definitely directional and there’s certainly outliers and we’re constantly looking at the hygiene of the data to see if we can get better and more accurate.
We also aspire to try to better measure end-to-end the product development life cycle. We’ve done a lot of great measurement on SDLC and the engineer roles and we know how to do that, but when we start with problem discovery, we haven’t quite figured that out yet. And so we’re looking to expand how we’re thinking about cycle time to start at the beginning of the process.
Moderator:
Got you. Yeah. So that sort of concept to cash or ID to value like I’m hearing several times.
Nicole Scribner:
That’s right.
Moderator:
I wish we had time for more questions. There’s some really good questions in here, but as always, I would encourage you to go find Kelly Anne and Nicole in the audience. Thank you so much.
Nicole Scribner:
Thank you.
Kelly Anne Pipe:
Thanks for having us.
Nicole Scribner:
Thank you so much.
Moderator:
Thank you very much.
Nicole Scribner:
Thank you so much.
Moderator:
Yeah, absolutely.
Kelly Anne Pipe:
Thank you.