Inside Peloton’s developer experience survey

While many are familiar with the fact that Google, Amazon, and Shopify conduct regular internal developer productivity surveys, there is not a lot of public information about what goes into actually designing and running such surveys. As a result, teams often struggle with low participation rates or harbor doubts regarding the reliability of the results.

Several years ago, Peloton’s Tech Insights team began a developer survey program to help augment the engineering metrics and insights they were collecting through their systems. “Just looking at certain key metrics won’t tell the whole story,” says Thansha. “Having a comprehensive survey to help understand the entire developer experience was really important for us, not only to shape tech learning programs, but also to feed into the platform team and for our engineering managers to understand where they could focus to drive developer efficiency and engagement.”

Today the survey’s results are reviewed across engineering and have become central to how leadership understands developer productivity. This article covers Peloton’s process for designing, conducting, and distributing the results from their survey. For leaders considering building their own DevEx survey, Peloton’s story serves as a helpful guide for understanding what steps are involved. 

‍This article is based on an interview and follow-on conversations with Thansha Sadacharam, who led the design and roll-out of Peloton’s survey program.

Getting executive buy-in

Before starting to design or formalize their survey program, Peloton’s Tech Insights team focused on getting buy-in from leadership to conduct a developer experience survey. To do this, the team zeroed in on a problem they knew was top-of-mind for their executive leaders: the developer onboarding process.

To demonstrate the potential value of a survey, the team gathered qualitative data manually by interviewing developers, then presented the analyzed results to executive leaders. The findings offered a rich understanding of the company’s onboarding process and how it could be improved, and acted as a teaser of the type of insights that could be captured by running a broader developer experience survey. 

As a result, leaders saw the value of moving forward with a formal survey, and stayed engaged in the process throughout the way. Thansha Sadacharam, Peloton’s head of Tech Insights, shared: “You can start this process by identifying one problem that is a thorn in your engineering leadership team’s side. Approach the problem in a data-driven way, present the results to leadership, and solve the problem. That gives them a taste of how a broader developer experience survey and problem will work.”

Designing the survey

Designing a survey that delivers reliable insights is an extensive process. Peloton’s Tech Insights team spent months carefully considering what topics to measure, how questions were crafted, and the overall length of the survey.   

Selecting topics to measure. To determine which topics to measure, the platform team first interviewed developers internally to learn about the tools and processes they interacted with at work. The team then created a map delineating the “day in the life” of a developer at Peloton, which was organized by developer segment (e.g., Android, iOS, backend, frontend engineers).

From this map, the team created a list of topics that could be measured. Then, borrowing the five dimensions from the SPACE framework, the team mapped each topic to its relevant category. Finally, the team added questions to help measure higher-level outcomes, such as satisfaction. 

Crafting questions. The platform team leveraged the expertise of Peloton’s talent team to help write questions and structure the survey. “We have people on our talent team with strong backgrounds in psychology and organizational design,” Thansha says. “They helped us by asking questions such as, ‘what is the purpose of this question’ and ‘how would you take action on the result of this question?’ This helped us refine the topic list and write questions that would yield accurate results.” 

Trimming survey length. As the team refined questions, they also removed many from the list when questions were repetitive or wouldn’t lead to action. Thansha notes that this process of refining the topic list is never-ending: the team reevaluates questions with each survey. Today Peloton’s survey includes ~50 questions and takes developers between 15 to 25 minutes to complete. 

“One thing I learned through this process is the importance of roping in experts. Thankfully we have those at Peloton and we were able to build this with them, because our original version had poorly worded questions, and we had a lot to learn about scales and open text questions and so on. I don’t think we could have done this on our own without those experts.”

In total, designing the developer experience survey was a several month process. Thansha explains: “It was months of design and a lot of human hours considering everyone who was involved. That’s before the launching of the survey, which needed a communication plan and administering and analyzing the results afterwards.” 

Conducting the survey

To conduct its survey, Peloton’s Tech Insights team uses dedicated survey software and proactively engages leaders and developers across the organization. Two specific strategies have helped Peloton achieve higher participation rates: intentional communication plan, and optimizing survey sampling method and frequency.

Communication planning. For years, Peloton has run a company-wide engagement survey, so Thansha’s team created a communication plan to make sure developers understood that the developer experience survey was distinct as well as needed. Additionally, they wanted to be sure developers were aware that the results would lead to real action. 

“We focused on communicating to developers, ‘our team wants to build the right processes, tools, and systems that will drive efficiency, effectiveness, and engagement for you. There are a million things we could work on, but we want to work on the things that are most important to you, and we need your feedback to know what those problems are.’” 

After each survey, Thansha’s team puts together a “state of engineering” report that includes a summary of the results, actions the organization is planning in response, and what actions have taken place since the previous survey. Developers can access this report at any time through an internal landing page which provides access to previous survey results and commitments made by the company. 

‍Iterating on the survey. One way the platform group increases participation rates is through the use of sampling. In earlier versions of their survey, the developer population was split into three groups that were surveyed at multiple points throughout the year. More recently, Peloton switched to surveying half of their developer population twice per year, thereby asking each individual developer to fill out the survey just once per year. As a result of this change, Peloton’s average survey participation rate has increased from 52% to 60%. In addition to improved participation rates, reducing their survey has lessened the amount of work involved in conducting and analyzing results.

Sharing the results 

After each survey, the Tech Insights team shares a “state of engineering” report with the entire software organization, which sparks discussion in their Slack channel and captures attention from leaders across the organization.

The team also sends a short one-pager to leadership, containing only the most critical findings. In the past, the Tech Insights team has been asked to present an overview of the results to the executive leadership team. In these presentations, the focus has been covering one or two key developer productivity metrics, for example overall developer satisfaction, along with the top three factors driving that score and the top three solutions the team is exploring. In addition, these presentations have included proposals for specific projects, and recommended focus areas for directors and managers across the company.

Peloton’s Tech Insight team has invested a lot of time and work into the design and execution of their developer experience survey, in order to achieve success. For others looking to build their own DevEx survey in order to inform developer productivity improvements, Peloton’s story should serve as a helpful guide for understanding the effort and process involved.