How Shopify uses a survey to measure developer productivity

Shopify needed a way to better understand and improve developer experience and productivity across their engineering organization. Despite an early investment in developer experience, the company lacked structured feedback from their engineers to prioritize their efforts. They realized that without direct input from developers, it would be difficult to know if the tools and systems they were creating were truly meeting their needs.

This need became more apparent around 2018-2019 when Shopify formally organized the Developer Acceleration team, which was designed to focus specifically on accelerating developer productivity by addressing their needs through tools, systems, and processes. As their engineering organization grew, it became clear that they needed a more disciplined approach to ensure that efforts to improve the developer experience were having the desired effects. Shopify wanted a way to better understand their developers’ experiences and to show that their input mattered, prompting the launch of their developer survey to collect structured feedback and help guide improvements in productivity.

Hypothesis

Shopify’s Developer Acceleration team hypothesized that by introducing a periodic developer survey, they could gather actionable insights directly from their developers at scale, which would allow them to improve both developer experience and productivity. The survey would serve as a tool to validate the impact of their efforts and provide a continuous feedback loop. The team also believed that this survey would demonstrate their commitment to listening to developers and improving tools and systems based on their feedback.

Solution

Shopify’s Developer Survey was designed to gather structured feedback from their engineers and help guide the Developer Acceleration team in improving tools and processes. Here are the core elements of their survey:

• Frequency: The survey is conducted twice a year. The developer population is split into two, where every developer is only surveyed once per year. Shopify took this approach as an attempt to prevent survey fatigue and ensure broad coverage across the organization.
• Focus on tooling satisfaction: The survey concentrates on developers’ satisfaction with key tools and systems, including CI times and development environments.
• Collaborative effort: Developer Acceleration worked with multiple teams, such as mobile and front-end, to ensure the survey addressed a wide range of developer experiences. They also collaborated with their People Analytics team to write and refine questions.

Metrics

Shopify used their survey to capture data in several areas:

• Overall satisfaction: Developers were asked to rate their overall satisfaction with Shopify’s tooling and whether they felt the tools had improved over the last six months.
• Pain points: The survey identified and ranked the top pain points developers faced, such as CI times or documentation quality. This helped the Developer Acceleration team focus their efforts on the most pressing issues.
• Satisfaction with changes: They used the survey to measure whether changes made to tooling, such as reducing CI times, translated into improvements in developer satisfaction.
• Team-specific feedback: The survey was also used to collect feedback from specific teams, like front-end or mobile engineering teams, to assess the unique challenges faced by these groups.

Outcomes

The Developer Happiness Survey is a key tool for Shopify’s Developer Acceleration team, helping them gather clear, actionable feedback to improve the developer experience. By regularly collecting insights, the team can identify and prioritize pain points like CI times or tool usability, ensuring they tackle the most pressing challenges. The survey also promotes collaboration across different technical teams, allowing them to address the unique needs of various engineering groups. Ultimately, it strengthens the feedback loop between developers and the team, helping them continuously refine and enhance the overall developer experience.

Listen to Shopify’s full story here → How Shopify runs their developer happiness survey