Gusto compares vendors, chooses DX for developer productivity insights

When Gusto grew to over 500 engineers, the company’s Infrastructure and DevEx teams were given a new mission: focus more broadly on developer productivity. There were rumblings from engineers about issues slowing down their work, so the newly established Developer Productivity group took charge. Their first step was to gather metrics and developer feedback to inform where they should invest.

“We were being held back by not having a good feedback cycle,” says Stephan Hagemann, Head of Product Infrastructure and Developer Productivity. “We needed a way to identify where to invest and to demonstrate that we could systematically listen to problems and address developers’ concerns.”

Gusto had previously used Code Climate for engineering metrics, but the way the product is designed lent itself to it being misused, so Gusto stopped using it. They had also conducted surveys they’d created using Qualtrics, but the insights were too high-level, and developers didn’t feel their feedback was being heard. “We couldn’t dig into the specifics,” Hagemann says. “With DX, we saw that having higher specificity and higher transparency could help us change the engineers’ perception over time that it’s not just a black box.”

During their pilot with DX, Gusto carried out an in-depth review of the landscape, evaluating all the major vendors in the space, including LinearB and Jellyfish. They created a matrix to categorize each vendor based on two key dimensions: whether they provided qualitative or quantitative metrics, and whether they focused more on allocation and org planning, or on developer productivity and experience. This helped them clearly see where DX stood out.

“DX is pretty alone when you draw a diagram like that. It’s more focused on the developer. It’s also more focused on qualitative, but not exclusively—we were always excited about the integration of the survey data with data out of running systems that can complete the picture,” explains Hagemann.

Hagemann notes that having both types of data is crucial for Developer Productivity teams. It allows them to ensure they’ve truly resolved an issue. Whether starting with self-reported data and then diving into system-based data, or vice versa, having both types of data gives teams the ability to validate their roadmap priorities, gain additional context about challenges, and measure the impact of their efforts quantitatively.

The reception from engineers at Gusto has been positive. “We’ve never seen participation rates like this,” Hagemann says. “The product’s transparency and ease of use have meant that there has been almost no negative feedback from engineers. One area where we’re still learning is making any custom questions either apply to everyone or using logic to filter them for folks for whom they do not apply.”

Hagemann continues, “DX strikes a good balance between getting a lot of good data points that are quantitative and combining it with feedback from developers. It doesn’t just stop at ‘here are the metrics.’ It’s set up to continue a conversation. That’s something we didn’t have before.”