Measuring performance to drive improvement is a challenge for many software engineering leaders. Although it is crucial to identify areas for enhancing software delivery, the practical complexities of measuring what truly boosts developer performance can be difficult to solve.
In 2018, Google acquired DORA, the leader in identifying metrics that have a significant impact on software delivery performance. These metrics include deployment frequency, lead time for change, change failure rate, mean time to recover (MTTR), and reliability.
Though widely embraced by the industry, the real test lies in how organizations implement these metrics effectively. Our guide provides a practical exploration of DORA metrics – what they entail and how to leverage them.
The critical question we address: Are DORA metrics enough on their own, or should they be part of a more comprehensive strategy? Join us as we navigate these software delivery metrics, offering actionable insights for leaders steering the course of development excellence.
Table of Contents:
DORA metrics are crucial for evaluating software delivery performance, previously known as IT performance. These metrics assess two key elements: throughput (delivery speed) and stability (reliability in production). Balancing speed and stability optimizes software development and deployment.
The DORA metrics are deployment frequency, lead time for change, change failure rate, and mean time to recover (MTTR). In more recent years, leaders have introduced a fifth metric: reliability.
DORA metrics measure an organization’s ability to produce— they do not measure output or production quality.
Here are the definitions of the five DORA metrics, using definitions from Google’s 2023 State of DevOps Report.
Deployment frequency is how often a development team releases new features, bug fixes, or improvements to a live production environment.
Different DevOps teams may have different definitions of what a high frequency means to them. For example, for some engineering teams, a high frequency might involve deploying changes several times a day, while for others, it might mean deploying changes a few times a week. One use case for a high deployment frequency is with a web application team that follows a continuous deployment approach. This allows them to make frequent small updates or enhancements to address user needs and improve the overall user experience.
Change lead time is then a code change moves from the development stage to being successfully deployed in the production environment.
A DevOps team that embraces agile methodologies and incorporates continuous integration can significantly improve this DORA metric. This allows them to quickly transform user stories or feature requests into actual code that can be implemented and deployed, thereby shrinking the time between commitment and deployment.
The change failure rate is how often a deployment results in failure that necessitates immediate attention.
For instance, if an engineering team implements automated testing and thorough quality assurance practices, they will likely have a low change failure rate. In contrast, a team with a higher change failure rate may experience frequent post-deployment issues requiring urgent fixes. This metric reflects the reliability of the production deployment process.
Mean time to recover, or failed deployment recovery time, can be illustrated by the average time it takes for a team to restore service after a deployment failure.
For example, a software development team with efficient incident response procedures may have a low MTTR. In a scenario where a deployment leads to system failures, a shorter MTTR indicates a quick and effective recovery process, minimizing downtime and ensuring a more resilient system.
Reliability is a service that consistently meets or exceeds its availability, performance, and accuracy goals.
A cloud service that maintains a 99.99% uptime, responds promptly to user requests, and consistently delivers accurate results showcases high reliability and DevOps performance. Software teams can measure reliability against industry benchmarks to ensure that their software provides a stable and dependable user experience, meeting the standards set by both development and operational teams.
It’s important to note that different teams might use different definitions for these DevOps performance metrics. We recommend that teams agree on clear definitions for each metric tailored to their workflow.
For every team, measuring your DevOps performance with DORA metrics will be different. As you set your own benchmarks and expectations, team members should adapt and become familiar with how their work will be benchmarked against the established DORA metrics and other DevOps success metrics.
Here’s how to get started with assessing DORA metrics:
Many leaders confuse DORA metrics with developer productivity and production output. Here are some guidelines for leaders to follow when implementing DORA metrics
DORA metrics are most effective in measuring a team’s overall software delivery performance as opposed to an individual—the metrics aid in comparing a team’s present performance to their past performance. By assessing a team’s ability to produce and deliver work with stability, teams are better equipped to evaluate their progress over time. This evaluation helps them identify areas needing improvement and implement strategies to enhance efficiency.
Different teams have varying workflows, tech stacks, and objectives.
As you introduce DORA metrics, assess what is achievable for each team and which metrics are relevant. Teams are not necessarily ‘high performers’ or ‘low performers’ based on their DORA metrics
When evaluating a team’s progress against their DORA metrics, consider all the internal and external factors affecting each team’s ability to deliver software performance. Universal metrics may not capture these factors.
Prioritize the development of capabilities that positively impact DORA metrics.
Identify and address areas of friction that hinder teams from building high-quality software quickly. By focusing on these areas and making necessary improvements, you can use the DORA metrics to measure the success of your efforts.
Ultimately, this will lead to more efficient and effective software development processes, reflected in improved DORA metrics, benefiting your team members and your organization.
While DORA metrics emphasize outcomes, they need to pinpoint specific organizational problems.
To foster genuine engineering performance improvement, organizations must thoroughly understand their developer experience and actively cultivate the required capabilities to improve software delivery. DORA metrics aren’t the solution; more profound analysis and targeted capability development through thoughtful initiatives are essential for meaningful progress, especially for high-performing teams.
DORA metrics are a set of DevOps metrics that engineering leaders can use to track their organization’s software delivery performance. Remember that an organization should not solely rely on these metrics as they are lagging indicators of the ability to deliver software. They are historical metrics, not predictive of future performance.
Engineering metrics like DORA are most effective with holistic developer experience programs that reduce friction and develop capabilities that drive performance. Then, use these metrics to track progress.
To evaluate the effectiveness of your developers, DORA metrics provide critical insights.
These metrics help you understand why your developers may perform above or below expectations regarding software delivery.
By using these insights, you can better understand where your developers may be struggling and take the necessary steps to improve their engineering performance.
Here are some examples of what the DORA metrics can tell you about developer effectiveness:
Lead time for changes:
Mean time to recover (MTTR):
Change failure rate:
Incorporating these DORA metrics in evaluations enables leaders to understand factors influencing developer productivity comprehensively. This knowledge empowers them to make informed decisions, prioritize improvements, and create an environment conducive to efficient and effective software development.
DORA metrics offer a valuable means to quantify and monitor software delivery performance. Yet, they alone do not provide a comprehensive insight into developer experience, productivity, or the nuanced aspects of developer performance.
These metrics are most effective when integrated into a broader toolkit employed by DevOps leaders to evaluate, measure, track, and enhance team performances.
Even at Google, the originators of DORA, a combination of methods is utilized to gauge developer effectiveness. In a discussion on the Engineering Enablement podcast, members of Google’s engineering productivity research team, Ciera Jaspan and Collin Green, emphasize key considerations for measuring engineering productivity. Google’s approach involves leaders defining high-level understanding goals and establishing balanced objectives for speed, ease, and quality. The research team employs a diverse set of metrics for each goal, adopting a mixed-methods approach that captures qualitative and quantitative data. This comprehensive strategy provides stakeholders with a more holistic understanding of engineering productivity.
Many Developer productivity teams have implemented DORA metrics but have realized that they only provide high-level signals, leaving them wondering what to do next.
DX is a developer insights platform designed by the researchers behind DORA and SPACE. DX is the only solution providing qualitative and quantitative measures, enabling developer productivity leaders to identify their most significant opportunities and translate their impact into dollars.
DX offers tools that help capture data from surveys and systems, both periodically and in real-time, trusted by developer productivity leaders at all stages, ranging from startups like Vercel and Brex to Fortune 500 companies like Pfizer and eBay. In addition to its products, DX provides expertise in implementation, program design, and transformation to help leaders adapt to changing environments and drive continued impact year after year.