Choosing what to measure in your developer experience survey

Choosing what to measure is foundational to a successful developer experience program. Here, we’ll cover the topics to measure, ways to measure them, and other important information to capture in your developer experience survey — all while weaving in insights from implementing our DevEx360 product with organizations of all sizes.

1. Measure a complete set of topics

Most leaders will already have formed assumptions about the areas of friction that developers are facing. These assumptions might have been formed from conversations with developers about specific tools that are difficult to work with or processes that are wasteful. However, choosing what to measure based on such conjectures alone is risky, as it may result in leaders receiving an incomplete understanding and failing to identify the most significant opportunities for improvement.

DevEx 360 consistently tests and refines its measurement framework, which was originally developed from research that investigated the factors that matter most to the developer experience.

2. Capture two metrics for each topic

There are two types of measures we can use to capture each developer experience factor: perceptual and workflow measures. We recommend both because neither alone can provide the full picture.

Perceptual measures capture how developers feel about or perceive something, and are a great way to understand pain, friction, and toil. These answer the question, “How do developers feel about this topic?”

Workflow measures balance perceptual measures by capturing an objective view of how systems or processes are behaving. These answer the questions, “How long does this take?” or “How often does a thing happen, interruptions or failed builds?”

Taken together, we balance our view of performance with developer satisfaction and identify problems we may not be able to notice otherwise. For example, our workflow measures may show that we have a fast build process, yet at the same time our perceptual measures are telling us that builds feel disruptive to developers, because they regularly interrupt their work progress. Conversely, we may see that developers feel content with their build processes, however workflow measures reveal that build times are slower than they could be.

3. Ask developers about the importance of improving each topic

By measuring a comprehensive set of factors as well as capturing workflow and perceptual measures for each factor, we now have a full understanding of the developer experience at our company. However, reaching this stage can be daunting: the data will show that there are many problems the organization could focus on. This is why capturing how important problems are to developers is critical.

The importance of each item can be measured by asking developers which items they would most like to see improved. This adds an important layer of context: developers know what their biggest problems are, so by asking them to rank the problems they’d most like to see improved, this gives us an idea of which areas are either causing the most cognitive load or causing friction frequently.

4. Ask for comments at the right time

Nearly every company we’ve worked with that previously ran developer experience surveys saw especially low participation within their comments. When leaders treat comments as a “nice to have” or a bonus if a developer responds, they miss an important layer of context that would otherwise guide their decisions. When a survey is designed to strategically capture comments, the result is having the majority of your developer population sharing detailed context about what problems they’re experiencing, when, and how that affects them. This can eliminate the need to run focus groups or interviews to understand what problems developers have.

There are two main reasons comments get low participation rates: anonymity and capturing comments on too many topics. Comments should be non-anonymous because it allows DevEx teams and managers to follow up with the engineers who provided the feedback to ask clarifying questions, or to get their perspective on how the problem may be solved. We should also only ask for comments on the problems that are deemed as top priorities.

5. Track KPIs to report on progress

While tracking higher-level outcomes will help leaders stay aligned with business goals when prioritizing initiatives, we find that customers find tracking Key Performance Indicators (KPIs) especially useful for communicating the value of the work they’re doing.

Engineering KPIs provide a north star metric that can be reported on and communicated concisely. For example, a DevEx team might say their overarching goal is to “make engineering more efficient”, and use Weekly Time Loss to show leadership the impact of their work.

The foundation of a successful developer experience program

As with any engineering metrics program, not being thoughtful about what information is captured can lead to incomplete or misleading results. When organizations start by investing in building a solid foundation of measurements, they can more easily achieve a comprehensive understanding of the state of engineering.