How Pfizer scaled to 1,000+ engineers

As it grapples with massive scaling, Pfizer has added a principal engineer layer and a platform to drive and measure developer satisfaction.

There’s no doubt that Pfizer and its vaccine have played a starring role in our collective, global pandemic story. Behind the scenes, the pharmaceutical company was already in the process of growing from 100 to 1,000 tech workers, relying on a lot of software outsourcing, across about 200 countries, and supporting about 88,000 employees. And while it found itself another accidental tech company, Pfizer was still operating in a risk-averse, highly regulated environment, with nine products with sales of over a billion a year.

Now that it has time to be less reactionary and more strategic, how has Pfizer adopted a more strategic approach to long-term growth?

Jamie Hook, director and DevOps lead at Pfizer, spoke with Laura Tacho, CTO at DX, at IT Revolution’s recent Enterprise Technology Leadership Summit about Pfizer’s recent journey to develop an engineering culture centered on developer experience (DevEX). He talked about how over the last six months Pfizer has evolved its delivery model, enabled its new principal engineers and measured it all — aligning toward the company’s goal of changing 1 billion lives per year by 2027.

A startup no longer

Strange as it sounds for a Fortune 50 company, until recently, Pfizer’s engineering organization worked like a startup. “We fostered and created a great culture in the early years of our team, bringing brand-new, crazy ideas like open source and agile methodologies to a large, risk-averse corporation,” Hook said. “As its growth accelerated, it got harder and harder to maintain the spirit of exploration and technical excellence.”

Pfizer engineering was already experiencing rapid growth with increased demand and scope, but found release cycles getting longer and tedious PowerPoint status updates becoming more frequent.

Then COVID-19 hit.

“The past few years have been busy for Pfizer. We’ve had to deliver results across all of our business lines in extremely compressed timelines,” Hook said. “To meet these demands, the best engineers and practices were put front and center. During this we were able to launch sites and solutions to multiple global markets at the same time.”

But that pressure and purpose that drove everyone during the first year or two of the pandemic was simply not sustainable. Developer burnout and turnover became understandably high, which meant Pfizer was losing skills, knowledge and culture at a high rate. And data breaches and security attacks were more sophisticated than ever.

The past year became a necessary time of reflection for Pfizer.

“The fact that our teams were able to rapidly respond to change and deliver high-quality results has opened the conversation to evaluating the way we work,” Hook said. Referring to the vendor partners and outsourced engineers that make up a large portion of the company’s technical staff, he said, “We want to ride this wave of understanding of our business partners and take the best parts of our delivery model to the wider organization.”

In support of that ambitious companywide goal, engineering created its own mission statement:

“We are the industry-leading digital organization, working at the cutting edge of technology and operating with speed, sustainability and agility to change 1 billion lives a year.”

The strategy to achieve this is:

• Thriving and joyful tech teams. Passionate and productive tech debates where engineers feel empowered to make decisions and to say no.
• Possibilities continuously explored. Principal engineers drive the technical direction, not the vendors, and teams can experiment in a blameless culture.
• Consistently applied high standards. Extend time of red team security exercises, with a focus on releasing higher quality.
• Value delivered at speed and the highest quality. With an emphasis on demonstrating progress over status slides, supporting an agile culture based on fast decisions.

Pfizer Digital is looking to create a more integrated and coordinated health-care model, and to facilitate shared services and a developer experience across this deeply global organization. All of these objectives rely heavily on communication with and support from the Pfizer leadership team.

The rise of the principal engineer

As an engineering organization scales, so must traditional managerial opportunities and technical leadership ones. At Pfizer, this is about layering in a new staff engineer level role: the principal engineer.

Unlike some staff engineering roles, these principal engineers are both technical influencers and team leads. “An executor and visionary behind technical solutions,” each will manage a cross-functional product team of 10 to 25 with a security-first and privacy-first focus.

As with other staff engineering roles, this holds sociotechnical responsibilities. The principal engineers are indeed highly experienced technologists, and they are responsible for “creating an environment where engineers can do their best work,” which is measured in their quarterly developer experience survey and in acting as a role model and mentor.

“The guiding principle for this work is that we should be looking to make the right thing as easy as possible,” Hook said, following the platform engineering refrain of not forcing folks with gates but laying down golden paths of least resistance. “We want to make it easy to use our platforms by providing tailored training [and] access to industry certification courses.”

Specifically, permission management is tricky across such a large, distribution organization.

“We’re building tooling to ensure that the right people are controlling access to the right tools and elevated access accounts,” he said, with access granted only once training has been completed.

The company is also heavily investing in CI/CD tooling to identify security and code quality issues earlier in the development process, and as a way to ease deployment.

“The flip side of making the right choice easy is making the wrong choice hard,” Hook said. “We want the easy choice to be retaining talent for the long term so the culture can grow.”

A long-term motivation for the creation of this role is to create an internal technical career path, which will help retain tech talent and their institutional knowledge, and then pass it on during the regular onboarding of new hires via more standardized documentation processes.

A main part of this new principal engineering role is to establish a technical leadership layer that can proliferate Pfizer’s culture both for internal hires and when working with contracted third-party vendors.

“We’re having to have large contract teams and vendor partnerships helping us deliver. What we should not be doing is outsourcing the technical strategy,” Hook said. “This is another area where the principal engineers will come in. This is a key technical leadership position who will provide technical stability and strategy across the board.”

While Pfizer will still rely on software outsourcing, it’s looking to rebalance the colleagues versus contractors ratio by “discouraging the fungibility of contractors” and, where possible, creating a similarly positive developer experience that incentivizes longer contractor relationships.

It’s also looking to build a community of practice to unite principal engineers across the organization.

Measuring developer experience at Pfizer

All of this necessitates executive sponsorship and DevEx champions to continue to drive the significant investment needed to make significant changes to people, processes and platforms, as well as to establish new security and platform teams. A budget and charter must be in place at the start of each project transformation.

“This is not a one-time initiative.” Hook said, underscoring the need for long-term investment and stakeholder buy-in. “We’re here and investing for the long term, both increasing the number of in-house principal engineers and creating an engineering team focused on improving our developer experience and continuing to evolve our delivery model.”

Pfizer’s future of development will eventually grow to cover thousands of application and code repositories.

The principal engineer role should be filled at the start of each project kickoff, he explained, for “guiding the teams to deliver with the same level of urgency and experimentation we’ve seen in the past few years.”

For now, over the past year, Hook’s team has selected about 13 of the largest and most impactful projects to pilot this new model.

They are still in the early stages, but, because they’ve been measuring developer experience since 2022, they can see some early wins already. This involves a comprehensive, multifaceted, multilevel measurement, Tacho explained, across three areas:

• Program measurements. Attrition and turnover, escalated security incidents, red team duration until result increases, and code quality and security metric improvements.
• Project measurements. Completion status, affected resources and project-defined metrics. All repositories are now equipped with automatic code quality and security scanning tools, and they are measuring how long it takes for a red team to get a result.
• Developer experience drivers. Speed, ease of delivery, quality and wasted time per week, universal to all the developers, irrespective of the project. A focus on knowledge silos is an important measure of DevEx, like how long does it take you to get an answer about a piece of code or a service?

The first quarters of Pfizer’s Future of Development program, Tacho said, has been focused on “creating long-lasting teams with sticky knowledge, [and] hardening security and permissions across all applications, while simultaneously improving the developer experience. This is not a small task.”

Developer turnover may be simple to measure, but developer attrition risk — how many developers are at risk of voluntarily leaving the company — is not. The DX developer intelligence platform, she explained, acts as this “voice of the developer” by asking quarterly questions around developer satisfaction and workflows.

Since embarking on Pfizer’s Future of Development, this risk of attrition has dropped by a third. In addition, between last quarter and this one, the developer surveys have found these promising trends:

• 6.6% more developers feel their applications are high quality and technically stable.
• 6.3% more developers are writing more documentation.
• A third more developers can resolve an incident in under an hour.
• 22% more developers can deploy in under an hour.
• 11.3% more developers feel it is easier to deliver software.

The next six months will see all platforms on their way to this future with a training plan and content, the permissions model completely overhauled and implemented, and tooling in place to proactively manage secrets and credentials. They will continue to onboard more principal engineers.

Most essentially, Pfizer will continue its focus on increasing developer satisfaction.

“So many aspects of this program will come down to a careful curation of our culture. We for too long have left it to fend for itself, thinking that because it was so solid, it will naturally continue to permeate through the entire organization,” Hook said. “However, as we’ve grown, we now realize that it needs intentional effort to scale and grow that culture.”

This article was originally published on The New Stack.