How DX bridges the road to full system metrics

End-to-end system metrics are generally the gold standard, but implementing them in larger organizations can be a complex and resource-intensive process. For instance, metrics like Change Failure Rate demand comprehensive end-to-end instrumentation, including deployment tracking, incident mapping, and an accurate service catalog. Similarly, resource allocation metrics depend on consistent Jira usage and meticulous data hygiene.

This complexity is often a roadblock for organizations just beginning their metrics journey, given the significant upfront effort required. At DX, we help solve this challenge by enabling customers to bridge gaps in system data by using self-reported metrics. This capability, referred to as “data bridging”, also offers organizations the flexibility to easily switch from survey-based data to system-based metrics once available.

Dr. Nicole Forsgren, creator of DORA and research advisor at DX, explains this concept in a previous paper, “While it is important to start as early as possible to get the benefits of system data, deploying survey data provides an almost immediate value and source of baseline information. Therefore, it is best to capture a system baseline with survey measures now while continuing to build out system-based metrics.”

What is data bridging?

DX’s data bridging feature enables organizations to capture baselines for metrics like the DX Core 4 and DORA using self-reported data, without needing to invest in end-to-end metrics instrumentation upfront.

Without data bridging, organizations often begin their metrics journey with only a partial view of metrics. For example, while metrics like lead time and PR throughput tend to be relatively easy to stand up from system data, other metrics—such as change fail rate or time spent on new capabilities—often require weeks or months of effort to properly instrument and normalize.

With data bridging, organizations can immediately capture a complete set of metrics without upfront investments in data instrumentation and normalization. Then, once system metrics eventually become available, organizations can seamlessly switch their metrics to use system data instead of self-reported data.

But are self-reported metrics accurate?

One common concern about data bridging is whether self-reported metrics are accurate or reliable enough to inform critical decisions.

At DX, our team is made up of leading researchers and measurement experts including Dr. Nicole Forsgren (creator of DORA) and Dr. Margaret Storey (co-author of SPACE). We ensure that every measurement in DX adheres to the same rigor and standards found in scientific research. This includes:

• Developing survey questions based on validated research and expert reviews
• Conducting multiple rounds of structured cognitive interviews to test for reliability and minimize bias
• Validating the accuracy of self-reported metrics by comparing them against objective data
• Ensuring that DX surveys achieve 80%+ response rates in order to produce accurate results and benchmarks

Our research consistently demonstrates that self-reported metrics perform similarly to system metrics in accuracy. For example, our research often finds that system metrics are not 100% accurate due to ongoing challenges with data hygiene and standardization.

Lindsey Simon, VP of Engineering at Vercel, shares, “Having qualitative and quantitative data gives me conviction that the qualitative data is as good as the quantitative data.”

Ultimately, neither self-reported metrics nor system metrics are perfect. But by leveraging both in combination, organizations can capture useful measurements in the most practical way depending on the situation. To learn more about DX and our full suite of products, request a demo through our website.