How Google measures developer productivity
Taylor Bruneaux
Analyst
This article summarizes our conversation with Ciera Jaspan and Collin Green who lead Google’s Engineering Productivity Research team. Listen to the podcast below.
Google’s investment in software developer productivity may be daunting to most—the company has many teams dedicated to the problem and a centralized research team that surfaces insights about how productivity can be improved. But while other organizations may not be able to match their investment, they can still learn from the underlying principles of how Google approaches the problem.
The Engineering Productivity Research team leads Google’s efforts to measure developer productivity and distribute insights to the teams and leaders taking action to make improvements.
In this article, we describe how the research team measures developer productivity to identify areas that can be improved. Specifically, we describe how they choose metrics and the methods they use to measure developer productivity.
What is developer productivity?
There is no single definition of developer productivity that is universally accepted. However, we can understand developer productivity by asking questions such as how quickly software engineers can complete their tasks, how much cognitive effort is required to complete the work, and the quality of the work or process being focused on. Because there is no standard for developer productivity, each industry and company has unique expectations. Typically, teams aim to benchmark and improve each metric over time.
Google’s engineering productivity research team is highly focused on the team responsible for developing internal tools - the first-party software developer team. They research to understand what factors influence developer productivity and how to measure it effectively.
Their ongoing efforts aim to enhance foundational research to make it more applicable to questions about developer productivity tools and measuring developer productivity. The ultimate goal is to contribute to improvements in tool infrastructure, processes, and practices while helping teams understand their efforts’ impact.
In addition to working with the first-party developer team, the engineering productivity team collaborates with various departments at Google, including people operations, real estate, workspaces, and corporate engineering. This reflects the broad interest in findings related to engineering productivity across the organization.
Google’s developer productivity metrics: speed, ease, quality
Google’s engineering productivity research team provides data support for different stakeholders involved in software development. The team was initially formed to develop and improve tools for developers, and it comprises both software engineers and UX researchers from diverse backgrounds, including psychology, behavioral economics, and public health.
The engineering productivity team addresses the needs of vice presidents, who require a broad understanding of the engineering department’s performance and whether any issues need their attention. In contrast, infrastructure teams require specific data on tools to make improvements.
When determining what to measure, the Google research team always follows the Goals, Signals, and Metrics (GSM) approach, regardless of specific stakeholders.
The first step in the GSM approach is for stakeholders to identify what they want to understand about developer productivity at a high level. The research team encourages software development stakeholders to define their speed, ease, and quality goals. This way, stakeholders get a more complete picture instead of just using one productivity metric and forgetting its tradeoffs.
For example, a stakeholder may want to understand:
- Speed: How quickly can software engineers accomplish their tasks?
- Ease: How much cognitive load is required to complete engineering tasks?
- Quality: What is the code quality being produced?
Having established these questions, the research team can then select engineering metrics for measuring productivity.
How metrics are captured: combining data from surveys and systems
Google’s research team uses various metrics to assess the developer experience regarding speed, ease, and quality. They collect data from both self-reported and log sources to measure the speed aspect more accurately.
The team combines qualitative and quantitative metrics using a “mixed methods” approach. Google captures these metrics through developer experience surveys and system data.
Capturing qualitative metrics through surveys
Google’s research team collects qualitative metrics by conducting a quarterly Engineering Satisfaction survey, which measures developer experience across a development team’s various tools, processes, and activities. The survey includes questions about specific topics and sends follow-up open-ended questions to developers when they’ve said they’re less satisfied with an area.
The questions derive from the GSM framework. Consistency is a powerful tool for the research team, who prefer to keep many of the metrics the same throughout the survey to enable long-term data collection. However, the survey does evolve as needed.
Google uses sampling to improve survey participation by surveying the developer population at multiple points throughout the year, split into three groups.
Once the survey analysis is complete, the results are delivered to everyone in software engineering, from VPs to individual contributors. Anyone in the organization can view the dashboards and query the results. The data collected from developers is aggregated, so responses are not identifiable, eliminating the possibility of evaluating an individual developer using these metrics.
The research team has developed a productivity tool for conducting real-time surveys when a developer completes a workflow, in addition to the quarterly engineering team survey. This approach is similar to LinkedIn’s real-time feedback system. The tool, called “experience sampling,” enables tool owners to obtain feedback from developers while they are using internal tools.
Capturing quantitative metrics through systems
The research team uses a system that tracks logs from various software engineering tools to gather data on developer workflows. They group related events into “sessions,” representing a continuous block of time when an engineer works on a specific task, like coding or code review. This provides a clear view of developer workflows for analysis.
This system also derives other quantitative metrics for measuring developer productivity.
The research team records various metrics, including developers’ coding time, reviewing time, shepherding time (time spent addressing code-review feedback), investigation time (time spent reading engineering documentation), development time, email time, and meeting time.
Summary
Google’s approach to measuring software developer productivity should inspire other engineering teams to build out their measurement programs. To summarize its approach:
-
Before choosing a metric, Google asks leaders to determine what they want to understand at a high level. It encourages leaders to establish goals that balance each other: they set goals for speed, ease, and quality.
-
The research team uses multiple metrics to understand each goal. It uses a mixed-methods approach to measurement, which means capturing qualitative and quantitative metrics. This gives stakeholders a more complete understanding of engineering productivity.