How Google measures developer productivity

Google’s investment in improving software developer productivity may seem daunting to most— the company has many teams dedicated to the problem and a centralized research team that surfaces insights about improving productivity. However, while other organizations may not be able to match their investment, they can still learn from the underlying principles of how Google approaches the problem.

TheEngineering Productivity Research team leads Google’s efforts to measure developer productivity and distribute insights to the teams and leaders taking action to make improvements.

In this article, we’ll explore how Google measures developer productivity and how its research team employs these techniques to identify areas for improvement in the tech industry. We’ll delve into Google’s process for selecting metrics and methods for measuring developer productivity effectively, steering clear of the wrong metrics that can hinder DevOps teams.

We based our discussion on an insightful episode of the Engineering Enablement podcast featuring Abi Noda, which offers a behind-the-scenes look at the work of Google’s Engineering Productivity Research team and their approach to measuring developer productivity.

What is developer productivity?

We can understand developer productivity by asking questions such as how quickly software engineers complete their tasks, how much cognitive effort they require to complete the work, and the quality of the work or process they evaluate. Because there is no standard for developer productivity, each industry and company has unique expectations. Typically, teams aim to benchmark and improve individual metrics over time.

Google’s engineering productivity research team focuses heavily on the first-party software developer team responsible for developing internal tools. They research to understand what factors influence developer productivity and how to measure it effectively.

Their ongoing efforts aim to enhance foundational research to make it more applicable to questions about developer productivity tools and measuring developer productivity. The ultimate goal is to contribute to improvements in tool infrastructure, processes, and practices while helping teams understand their efforts’ impact.

In addition to working with the first-party developer team, the engineering productivity team collaborates with various departments at Google, including People Operations, Real Estate, Workspaces, and Corporate Engineering. This collaboration reflects the organization’s broad interest in findings related to engineering productivity.

How to measure developer productivity

Measuring developer productivity is a complex task that requires a thoughtful approach. As DX’s CTO Laura Tacho points out inher explainer, “Instead of looking for a list of metrics to determine how you measure performance, figure out how you want to measure performance and then find metrics that help you measure the stuff that’s important to your company.”

It’s crucial to focus on outcomes rather than output and to consider the specific responsibilities and expectations for each role. While metrics likelines of code or story points completed may provide some insight, they should not be the sole basis for evaluating individual productivity. Instead, a comprehensive approach that considers project delivery, collaboration, quality, and business impact is more likely to provide a fair and accurate assessment of developer productivity.

Developer productivity frameworks

Measuring developer productivity is crucial for engineering managers and leaders who want to ensure that their teams work efficiently and effectively toward business goals. By combining quantitative measures, such as cycle times and deploy counts, and qualitative measures, such as surveys and perceptual measures, engineering leaders can gain a holistic view of their team’s productivity and identify areas for improvement.

SPACE framework

TheSPACE framework challenges the conventional ways companies try to look at developer productivity and asserts the importance of adopting a broader view. The paper “The SPACE of Developer Productivity” by Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler, from GitHub and Microsoft Research, provides the details of the SPACE framework.

The paper’s authors argue that productivity cannot be reduced to one metric because software development is multidimensional. Additionally, productivity cannot be viewed only through the lens of activity because software development is knowledge work.

SPACE is an acronym for five different dimensions that help us understand and potentially measure productivity:

• Satisfaction and well-being: how developers feel about their work, team, tools, or company culture
• Performance: the outcomes of a system or process
• Activity: the count of actions or outputs completed while performing work
• Collaboration and communication: how people and teams communicate and work together
• Efficiency and flow: the ability to complete work or make progress with minimal interruptions or delays

DORA metrics

Another vital framework for measuring developer productivity isDORA, which stands for DevOps Research and Assessment. This framework focuses on four key metrics: deployment frequency, lead time for changes, mean time to restore service, and change failure rate. By tracking these metrics over time, engineering teams can identify bottlenecks in their software development process and optimize their workflows to reduce cycle times and produce high-quality code.

DORA also emphasizes the importance of a positive engineering culture where developers feel engaged and empowered to do their best work. Teams can achieve this by providing sufficient focus time, minimizing time loss due to inefficient processes, and investing in developer productivity teams that streamline workflows and remove obstacles.

Additional tools and techniques

In addition to these frameworks, engineering leaders can use many other tools and techniques to measure and improve developer productivity. For example:

• Project management tools can help teams track progress and identify potential issues before they become significant problems.
• Developer survey programs can provide valuable insights into the developer experience and highlight areas where improvements are needed.
• Benchmarking engineering productivity against other companies in their industry can help engineering leaders better understand how they stack up and identify opportunities for improvement.

Ultimately, the key to measuring and improving developer productivity is to take a holistic approach that considers both quantitative and qualitative factors and creates a productive work environment where developers feel engaged, supported, and empowered to do their best work.

Google’s developer productivity metrics: speed, ease, quality

Google’s engineering productivity research team provides data support for different stakeholders involved in software development. Google initially formed this team to develop and improve tools for developers, and it comprises software engineers and UX researchers from diverse backgrounds, including psychology, behavioral economics, and public health.

The engineering productivity team addresses the needs of vice presidents, who require a broad understanding of the engineering department’s performance and whether any issues need their attention. In contrast, infrastructure teams require specific data on tools to make improvements.

When determining what to measure, the Google research team always follows theGoals, Signals, and Metrics (GSM) approach, regardless of specific stakeholders.

The first step in the GSM approach is for stakeholders to identify what they want to understand about developer productivity at a high level. The research team encourages software development stakeholders to define their speed, ease, and quality goals. This way, stakeholders get a more complete picture instead of just using a single metric and forgetting its tradeoffs.

For example, a stakeholder may want to understand:

• Speed: How quickly can software engineers accomplish their tasks?
• Ease: How much cognitive load is required to complete engineering tasks?
• Quality: What quality of code is the team producing?

Having established these questions, the research team can select engineering metrics to measure productivity.

Capturing developer productivity measurements: Combining data from surveys and systems

Google’s research team uses various metrics to assess thedeveloper experience in terms of speed, ease, and quality. They collect data from both self-reported and log sources to measure the speed aspect more accurately.

The team combines qualitative and quantitative metrics with a “mixed methods” approach. Google captures these metrics through developer experience surveys and system data.

Capturing qualitative metrics through surveys

Google’s research team collects qualitative metrics by conducting a quarterly Engineering Satisfaction survey. The survey measures developer experience across a development team’s various tools, processes, and activities. It includes questions about specific topics and sends follow-up open-ended questions to developers when they’ve said they’re less satisfied with an area.

The questions derive from the GSM framework. Consistency is a powerful tool for the research team, which prefers to keep many metrics the same throughout the survey to enable long-term data collection. However, the survey does evolve as needed.

Google uses sampling to improve survey participation. It surveys the developer population at multiple points throughout the year, split into three groups.

After completing the survey analysis, the team delivers the results to everyone in the software engineering team, from VPs to individual developers. Anyone in the entire team or organization can view the dashboards and query the results. Then, they aggregate the collected data from developers, ensuring that responses are not identifiable. This approach eliminates the possibility of evaluating individual developer performance using these metrics.

The research team has developed a productivity tool for conducting real-time surveys when a developer completes a workflow, in addition to the quarterly engineering team survey. This approach is similar to LinkedIn’sreal-time feedback system. The “experience sampling” tool enables owners to obtain feedback from developers while using internal tools.

Capturing quantitative metrics through systems

The research team uses a system that tracks logs from various software engineering tools to gather data on developer workflows. They group related events into “sessions,” which represent a continuous block when an engineer works on a specific task, like coding or code review. Sessions provide a clear view of developer workflows for analysis.

This system also derives other quantitative metrics for measuring developer productivity.

The research team records various metrics, including developers’ coding time, reviewing time, shepherding time (time spent addressing code-review feedback), investigation time (time spentreading engineering documentation), development time, email time, and meeting time.

Google’s approach to measuring software developer productivity and developer velocity should inspire other engineering organizations to build out their measurement programs. To summarize its approach:

• Before choosing a metric, Google asks leaders to determine what they want to understand at a high level. It encourages leaders to establish business priorities that balance each other: they set goals for speed, ease, and quality, which are essential for productive developers and quality software.
• The research team uses multiple metrics, such as SPACE metrics and surveys with leaders, to understand each goal. It takes a mixed-methods approach to measurement, which means capturing qualitative and quantitative measurements. This analysis gives stakeholders a complete understanding of engineering effectiveness and the developer experience.

By focusing on the developer experience and using a comprehensive metrics survey, Google ensures that its engineering systems support happy developers and a positive team dynamic. This approach helps to create development environments that enable productive developers, which is a significant benefit for any business with engineering teams. Other tech companies can benchmark engineering productivity by adopting similar practices, leading to tangible benefits regarding quality software and developer satisfaction.