What McKinsey has to say about developer productivity

As the world becomes more software-driven, understanding and improving developer productivity is becoming increasingly important. However, measuring developer productivity is still a complex and often controversial topic, highlighted by an article from McKinsey & Company in August 2023, leading to a debate within the DevProd and tech communities.

In this post, we will discuss a framework to understand developer productivity. The framework draws on insights from industry leaders and a seminal research paper from 2019. Whether you are a software engineer looking to optimize your workflow or an engineering manager seeking to build high-performing engineering teams, this guide will provide you with a solid foundation for navigating the complex landscape of developer productivity in tech companies.

Summary of McKinsey’s article

The article, “Yes, you can measure software developer productivity,” authored by Aayush Gupta and Chandra Gnanasambandam from McKinsey & Company, argues that measuring software developer productivity is critical for organizations as they become increasingly software-driven.

However, when applied to individual engineers’ performance, traditional metrics like lines of code produced or story points (burndown charts) often need to be more accurate and accurate. The authors propose a new approach based on existing industry metrics like DORA (DevOps Research and Assessment) and SPACE metrics (Satisfaction, Performance, Activity, Communication, Efficiency). Their framework introduces additional “opportunity-focused” metrics that are easier to collect via surveys or existing data sources.

Key elements of their approach include:

• Inner/outer loop analysis: Maximizing the focused time developers spend on value-adding “inner loop” coding tasks while minimizing “outer loop time,” or idle time spent on tasks like code reviews, integration, and testing
• Developer Velocity Index (DVI): A survey-based benchmark of an organization’s technology, practices, and enablement compared to industry peers
• Contribution analysis: Assessing individual contributions to backlog tasks to surface optimization opportunities
• Talent capability scoring: Assess the distribution of skills within the engineering team to pinpoint coaching and upskilling requirements. Talent capability scoring ensures that teams utilize most valuable developers effectively, while inexperienced ones receive sufficient support

The authors caution against misusing metrics that can incentivize the wrong behaviors. They also stress the importance of moving beyond the notion that building software is too complex to measure.

To get started, the authors recommend that engineering leaders:

• Educate themselves on modern software development practices
• Assess their systems’ readiness to support data collection
• Build targeted plans focused on crucial improvement opportunities

Ultimately, the goal is to create transparency around opportunities to improve developer productivity and experience, which would benefit the entire organization.

The problems with McKinsey’s approach

Gergely Orosz and Kent Beck offer a thought-provoking and critical response to McKinsey’s article on measuring software developer productivity. They propose that McKinsey’s framework oversimplifies the complex nature of developer productivity and could potentially harm engineering teams if implemented.

Key points from their response:

The complexity of measuring developer productivity

Measuring developer productivity is a multifaceted issue that requires careful consideration. Introducing misguided frameworks based solely on quantitative metrics can damage the engineering culture and take years to rectify.

The effort-output-outcome-impact model and its implications

The effort-output-outcome-impact model illustrates the software engineering cycle. Orosz and Beck contend that focusing on measuring effort and output, as McKinsey suggests, can lead to unintended consequences and encourage developers to manipulate the system.

Learning from sales and recruitment teams’ evaluation methods

Sales and recruitment teams are often evaluated based on outcomes and impact rather than effort or output. Engineering leaders should adopt a similar approach and concentrate on measuring outcomes and impact to determine what to measure.

The trade-offs of measuring effort or development activity

While measuring effort or development activity may seem appealing, it has trade-offs that can adversely affect the engineering culture. Orosz and Beck propose measuring outcomes and impacts, like delivering at least one customer-facing item per team per week and achieving business impact committed to by the team.

Exploring alternative frameworks for measuring outcomes and impact

The DORA and SPACE frameworks offer guidance to measure outcomes and impact, and engineering leaders should consider these approaches as alternatives to the McKinsey framework.

–

Orosz and Beck make a compelling case that the McKinsey framework fails to capture the intricacies of measuring developer productivity and relies too heavily on measuring effort and output. They recommend focusing on impact measurements that align with the objectives of high-performing software engineering teams and maintain a healthy engineering culture.

A better way to approach developer productivity

According to an influential paper by Caitlin Sadowski, Margaret-Anne Storey, and Robert Feldt, developer productivity can be broken down into three key dimensions:

• Velocity: The speed at which work gets done
• Quality: The caliber of the work produced
• Satisfaction: The fulfillment software engineers derive from their work

These dimensions are profoundly interconnected and must be carefully balanced. Abi Noda, CEO of DX, underscores in his summary that Google utilizes a similar framework centered on Speed, Ease, and Quality, with developer satisfaction being a critical indicator within each dimension.

Velocity: Measuring the speed of development

Velocity refers to how quickly software engineers complete tasks and ship code. However, it’s crucial to consider the tasks’ type, complexity, and routineness when measuring velocity. Churning out lines of code is not a meaningful indicator of productivity if that code is of poor quality or fails to deliver value to end users.

Quality: Ensuring excellence in software development

Quality encapsulates doing an excellent job when producing software artifacts. It includes internal factors like code quality and maintainability and external considerations such as the software’s end-user experience and business impact.

Satisfaction: Empowering engineers to do their best work

Developer satisfaction is a broad concept that includes happiness, autonomy, and flow. It plays a crucial role in balancing the other dimensions. For example, investing in learning and skill development may boost long-term quality and velocity while increasing developer satisfaction. Conversely, a myopic focus on velocity can lead to burnout and turnover.

Aligning productivity goals and metrics

When engineering managers evaluate developer productivity and team performance, they must consider the role of platform engineering and measure development using key metrics. Excessive time spent on tasks such as backlog management or addressing technical debt can indicate underlying issues that traditional productivity metrics may not capture.

Here are the top strategies to align your productivity goals with your chosen metrics.

Considering measurement tradeoffs and failure rate

Measurement tradeoffs are crucial, as optimizing for one metric (such as lines of code written) may have unintended consequences on others (like code quality or maintainability). The failure rate is another key metric to watch, as a high rate of defects or incidents can indicate problems with the development process or the underlying architecture.

Framing productivity metrics in terms of ROI for business leaders

To gain support from business leaders, present productivity metrics regarding return on investment (ROI). Demonstrate how engineering productivity and platform stability improvements can lead to faster time-to-market, higher quality products, and ultimately better business outcomes.

Avoiding anti-patterns when measuring productivity

However, it’s also important to be aware of anti-patterns when measuring productivity. Focusing too narrowly on individual developer output, for example, can lead to a culture of competition and burnout while neglecting important aspects of collaboration and knowledge sharing. Similarly, setting unrealistic targets or using metrics that are easy to game can lead to perverse incentives and unintended consequences.

Striking a balance between quantitative and qualitative measures

Ultimately, the key is to balance quantitative and qualitative measures and involve stakeholders at all levels in defining and tracking productivity metrics. By taking a holistic view and considering the trade-offs involved, engineering managers can help create a culture of continuous improvement that benefits developers, teams, and the organization.

A three-step approach to effectively measure productivity

To effectively measure productivity, Caitlin Sadowski, Margaret-Anne Storey, and Robert Feldt suggest a three-step approach:

1. Define the specific productivity goal or dimension. This could include factors such as speed of delivery, quality of output, or efficiency of resource utilization.
2. Identify the signals that will indicate whether you are achieving the goal. These signals should be observable and measurable, indicating progress toward the desired outcome.
3. Select specific performance metrics that to quantify and score those signals over time. These metrics should be relevant, reliable, and actionable and drive continuous improvement.

By following this structured approach and carefully selecting metrics that align with the organization’s goals and values, engineering leadership can create a culture that supports high-performing teams and enables them to deliver value at optimal rates. However, it is vital to regularly review and adjust the chosen metrics as the organization’s operating model and priorities.

Developer productivity metrics and individual performance reviews

When evaluating developer productivity and performance at the individual level, it’s essential to consider the broader context of the company culture and how development is measured in terms of overall team and organizational effectiveness. While contribution analysis, as recommended by McKinsey, may seem intuitive for gauging individual productivity, relying too heavily on these measures can lead to anti-patterns and unintended consequences.

For example, an overemphasis on individual performance metrics may incentivize developers to prioritize their output over collaboration and knowledge sharing, leading to silos and reduced psychological safety within dev teams. Developers may also focus on gaming the metrics rather than delivering real value, such as submitting many small, low-impact pull requests or avoiding complex, high-value projects.

A holistic approach to measuring developer productivity

Engineering managers should take a more holistic view of productivity that encompasses coding and non-coding activities, such as mentorship, documentation, and improving engineering practices to avoid these pitfalls. A holistic approach requires continuous visibility into the work across teams and understanding the interdependencies and trade-offs involved.

One framework recommended by McKinsey for measuring developer productivity more holistically is the “inner loop” and “outer loop” model. The inner loop refers to the core coding activities of writing, testing, and debugging code, while the outer loop encompasses important noncoding activities like planning, collaborating, and learning. However, this is still only one piece of the puzzle.

Creating a comprehensive developer productivity metrics framework

By considering both inner and outer loop activities and using a combination of quantitative and qualitative measures, engineering managers can create a more comprehensive developer productivity metrics framework. This approach can help identify areas for improvement and support data-driven decision-making while also fostering a culture of continuous improvement and psychological safety within teams.

Ultimately, the goal of measuring productivity in terms of individual performance should be to help developers grow and have the most significant developer experience rather than simply stack ranking them against narrow output metrics. By taking a holistic and context-aware approach, engineering managers can create a more accurate and actionable picture of developer productivity and performance.

Metrics measure outcomes

Industry leaders like Abi, Kent, Gergely, and Dave Farley emphasize measuring outcomes and impact rather than effort or activity. The velocity/quality/satisfaction framework aligns with this philosophy by focusing on measuring speed, ease, and quality as crucial outcome dimensions. These dimensions can also serve as a way to evaluate initiatives to improve developer experience and productivity. For example, a DevOps leader could evaluate the impact of a new CI/CD system based on its effect on development velocity, ease of use for engineers, and the quality of the resulting code.

The road ahead: Building a productive future for software engineers

While frameworks provide valuable guidance, the conversation around developer productivity is ongoing. As Abi and others have stressed, productivity is highly contextual, and leaders must select metrics to drive the right behaviors and outcomes. Ultimately, any effort to measure and optimize productivity must align the goals of software engineers and the broader organization.

Fostering an engineering culture of trust, empowerment, and continuous improvement can create environments where developers thrive and deliver their best work. As you embark on your journey to understand and enhance developer productivity, remember there is no one-size-fits-all solution. By staying grounded in velocity, quality, and satisfaction and adapting your approach to your unique context, you can build high-performing engineering teams that ship exceptional software quickly and joyfully.