Customer Stories

Flip’s engineering leadership team has always looked for ways to measure and understand developer productivity. Before using DX, Flip’s engineering leaders focused primarily on DORA metrics but found them lacking in providing actionable insights. 

“The DORA metrics are a good start, but they don’t tell us a lot about what’s actually going on,” says Suryawirawan. “Things like how difficult processes are, or the quality of our work and collaboration, are more difficult to gauge.”

By using DX, Flip was able to get a combined view of qualitative and quantitative sociotechnical measures that provided both the “what” and the “why” of factors affecting developer productivity. Armed with these insights, the leadership at Flip was able to clearly identify their highest-impact priorities, such as improving build processes, test reliability, documentation, and requirements quality.

In addition to providing insights to leadership, DX enables individual teams at Flip to make improvements as well. Teams are encouraged to leverage DX data to identify focus areas and include them in their quarterly roadmaps. “DX gives us a systematic approach to improving productivity. It helps us figure out how we’re doing as an engineering organization and what initiatives we can focus on next to improve our developer experience.”

‍

Sure is an insurance software company that supports some of the world’s most recognized brands.  Sure’s engineering organization previously relied on in-house surveys, 1:1s, and leadership meetings to keep a pulse on developer productivity.  However, with aggressive hiring plans and a reorganization on the horizon, Sure’s engineering leadership saw a need for a better approach. 

“We were about to get new managers, new people, new processes, and do a reorg, and we wanted to preempt some of the challenges that come with all of that change,” says Sure’s VP of Engineering, Jico Baligod. “We wanted a way to keep a pulse on what was changing as we grew.” 

The company found that DX provided a powerful yet streamlined solution for measuring their developer experience and productivity. “We did a proof of concept, and it was just so much easier than what we were doing previously,” says Baligod. “It’s a frictionless way to understand how teams are doing and what we can do to improve their day-to-day processes.”

Today, Sure uses DX to inform team- and organizational-level improvements. Teams select developer experience projects to focus on each quarter as part of their customer-facing work, and leadership identifies areas that need a more coordinated effort. One example of the latter: Sure’s leadership has a dedicated “Tech Wealth” week every year where teams focus on improving technical debt. 

In addition to the insights DX provides, Sure saves an estimated $200,000 per year by offloading their developer experience survey efforts to DX.

A global logistics company experienced record growth on all fronts during Covid-19: demand sharply increased, operations expanded, and team headcount grew. 

After its initial wave of growth, the company recognized a need to invest in developer experience. An engineering leader at the company recalls, “We noticed that there was a huge opportunity for us to create a better environment for developers. The question was, how?” 

The leadership group had ideas about what areas needed attention, but wanted to understand how widespread these problems were and whether there were others they’d not yet considered that might have a greater impact on the organization. 

Leadership began evaluating measurement approaches to inform their planning. They reviewed DORA reports, read the book Accelerate, and demoed several Git and Jira-based metrics tools, but none of these approaches provided the visibility they were looking for. “I wasn’t keen on measuring the specific metrics I saw,” says one engineering leader. “They didn’t feel like they were the most effective or reliable way to get a holistic overview of what was going on in engineering. They also wouldn’t help engineering teams drive improvements forward.” 

Things clicked when the leadership group found DX. “DX solves the same fundamental problem, just in a completely different way."

Embedding a culture of continuous improvement

Prior to rolling out DX, some members of the leadership group were worried about whether developers would participate. They also worried about whether teams would actually use the data to take action, especially considering the company has product owners owning the responsibility of prioritizing developer experience work instead of engineering managers. 

Before the first snapshot, an engineering leader sent out messages to help developers and product owners understand the reasons for using DX. After the snapshot, they also met with a few product owners to walk them through their team’s data. This engineering leader says that, for the product owners, seeing their team’s data for the first time helped them realize the value of DX. 

“In consecutive snapshots, it‘s becoming simpler and simpler for teams to participate, select focus areas, and take action. I no longer need to do marketing around making sure people are aware of why they should participate and what teams should do with the data. Teams are just doing it themselves.” 

‍

Informing enterprise-wide initiatives

Every quarter, the company’s internal-facing developer productivity team reviews DX data with management to discuss opportunities to solve problems spanning multiple teams. Several initiatives have emerged from these conversations, including one focused on how teams collaborate with each other. 

The company experimented with a few strategies that led to immediate improvements in cross-team collaboration, including: 

• Introducing Architectural Decision Records (ADRs) to improve the way teams make architectural choices, transforming this into a more collaborative process focused on finding the best possible solution. 
• Implementing a developer portal with the goal of reducing the cognitive load required to find available documentation and access tooling. 
• Hosting knowledge sharing sessions where teams present their current or past projects, both from a product and technical perspective. 
• Organizing a hackathon to help team members get acquainted (the company grew quickly and remains a hybrid team).

“Every quarter, we look at DX data to discuss investments we can make to improve problems that affect multiple teams,” says an engineering leader of the internal-facing team. Their team shares results in the company’s all-hands, explains current focus areas, and then provides progress updates.  

“Our company is in a regulated industry, but that doesn’t mean developers should be constantly weighed down by processes or rules,” the leader says. “We can still meet those regulations while creating a highly effective and enjoyable work environment for developers. That’s our mission, and DX helps us do that.” 

Since its founding, Curriculum Associates has recognized the tangible business value in helping employees feel supported, engaged, and empowered. The company reinforces this belief through cultural values, including a focus on creating a best-in-class employee experience that empowers people to do their best work. 

The company has long used standard HR surveys to surface improvement opportunities across their employee experience. Sivan Lahav, Director of Engineering, says, “The company-wide surveys serve their own purpose, but they provide a higher-level set of data points that are not always specific enough to the issues that engineers navigate in their day-to-day.” Lahav recognizes that, given the nature of those surveys and the fact they are inclusive of every role in the company, it is inherently more difficult to make them targeted. As Curriculum Associates has grown significantly in recent years, the company started looking for developer insights that would provide clarity into where investments could be made.

Gaining insight into areas of friction across the developer experience

Curriculum Associates turned to DX to surface relevant data for understanding the engineering experience the company was providing. They immediately discovered areas where they were excelling—for example, the company does an exceptional job at welcoming different opinions and fostering a learning culture (scoring 8 points above the 90th percentile on the latter). They also discovered areas in which improvements could be made. Curriculum Associates identified an opportunity to improve the balance of technical debt by getting alignment from various stakeholders on the business value of prioritizing this work. 

Those insights emerged in the first snapshot, which saw an 85% developer participation rate. Lahav and other leaders have since dedicated time to responding to developers’ feedback, sharing out DX results, and updating all teams on current developer experience initiatives—increasing survey participation to over 90%.

“DX is like having an x-ray into areas of friction that would otherwise be harder to see in developers’ day-to-day work,” says Lahav. Matt Hodges, VP of Engineering, adds, “Versus me assuming and championing things that may not have any material impact on the team’s happiness or efficiency, DX allows us to crowdsource and prove, with analytics, that we are focused on the right things.”

Empowering team-level improvements 

One of Curriculum Associates’ ongoing goals for their culture as it continues to evolve is to “create an overall joyful engineering experience.” To achieve this outcome, leaders focus on empowering local teams to identify and make their own improvements. Technical debt is a prime example of this. 

With DX, Curriculum Associates newly observed that some engineering teams lacked the necessary support to resolve technical debt. Lahav and his leadership team solved this problem by coming to an agreement with all teams (and their product owners) that a certain capacity of their time would be dedicated to resolving tech debt. 

In practice this has meant: 

• Each team has dedicated a set capacity of their time to resolving tech debt. 
• Team leads or developers decide which tech debt projects to focus on. 
• Some teams even use templates to help define the business value for each ticket related to tech debt. Lahav says this practice both facilitates product owner buy-in and helps engineers connect with the business needs and what helps produce value for their customers.

DX’s focus areas feature has also been helpful in getting teams to prioritize additional improvements and track high-level progress across teams. “DX has helped us create a continuous improvement workflow. It has cycles, and with each cycle we see improvements,” says Lahav.

With plans to rapidly scale their engineering organization, Recursion’s leadership sought to get a baseline understanding of their developer experience. The leadership team recognized that engineering teams often seem to move slower than expected after a period of rapid growth. Their hypothesis was that if they proactively identified areas of friction in developers’ day-to-day, Recursion would be able to take action and maximize developer effectiveness as they grew. 

Recursion chose DX to measure areas of friction across their developer experience and help inform their Infrastructure and Tooling organization’s roadmap. Maureen Makes, who leads the Infrastructure and Tooling organization, says, “Our team is focused on understanding the problems that developers are facing and figuring out how to get ahead of them. DX gives us insight into what those problems are.”

Recursion‘s Infrastructure and Tooling organization has three primary purposes for using DX: advocating for company-level initiatives, informing their team’s roadmap, and reporting on progress. 

Advocating for company-level objectives 

On a quarterly basis, Makes’ team reports on DX data to the senior technology leadership group. They also present recommended organization-level initiatives that the company can invest in. 

“These presentations have landed really well,” Makes says. “They have led to company-level initiatives like creating a documentation guild and establishing The Year of Tech Debt.” 

The idea for the Year of Tech Debt originated in one of these leadership presentations. The group discussed different possible developer experience investments. In the end, technical debt was chosen, and Recursion’s engineering organization decided to name 2022 “The Year of Technical Debt.” 

In practice, that meant that every quarter, each engineering team would select a technical debt project to focus on. (At one point in the year, this included a tech debt hack week that was considered an enormous success for the company.) At the end of the quarter, teams would share what they accomplished in Recursion’s Tech All Hands meeting. Makes says the company has decided to focus on ownership in 2023. 

Informing Infrastructure and Tooling’s roadmap 

DX helps inform the Infrastructure and Tooling organization’s roadmap. “Multiple times, our team has assumed we should focus on a specific problem — only to learn through DX that something else was a bigger problem for developers." says Makes. “Being able to see both qualitative and quantitative data about different tools and processes gives us a much richer perspective on where we should focus as an internal-facing team.” 

Reporting on progress

When deciding which metrics to report on at the executive level, Makes’ organization wanted to make sure their metrics supported their mission of accelerating Recursion’s stream-aligned teams. They decided on two of the KPIs measured in DX, Weekly Time Lost and Ease of Release. 

These metrics are reported quarterly to the tech leadership team, who then report on the metrics with senior leadership to frame conversations around investments in Recursion’s engineering organization as a whole. 

1-800 Contacts is built on the core belief of being better — a belief that can be observed not just in their business model but in how every part of the organization operates, including engineering. So it’s no surprise that back in 2018, when the four DORA metrics were presented in the book Accelerate as a path to become higher performing, 1-800 Contacts was quick to adopt the new measurement model.

Josh Moore, Director of Engineering, explained that the DORA metrics helped their organization recognize which capabilities they needed to build to improve their ability to deliver. However, once those capabilities were built, engineering could no longer use the DORA metrics as the only tool to understand where to focus to continue improving. Moore further described why they needed more than just the DORA metrics: “As much as we didn’t want to compare teams, it happened anyway. But you can’t use these numbers to compare teams to each other when they’re working with different processes or on completely different systems.” They needed additional tools to continue to drive developer experience metrics.

1-800 Contacts looked to supplement the DORA metrics with feedback from developers in 1:1s and surveys about what was holding them back. However, 1:1s provided sporadic information about issues, and it wasn’t clear how widespread issues were. They also weren’t confident they were getting the full story from surveys because of the questions being asked and level of engagement they were seeing.

Today, 1-800 Contacts uses DX to surface the underlying causes of trends they see and get a clear picture of the highest priority areas to invest in. DX provides them with the ability to understand how widespread problems are as well as how painful tools and processes are. It also allows them to provide data back to the teams, so teams can take action on areas of friction that are either unique to their group or are otherwise important for their group to focus on.

Quantifying developer pain points to make informed decisions on where to focus      

It’s intuitive for developers to understand the impact that tools, processes, and team dynamics have on productivity, but it’s not always intuitive for non-technical stakeholders to make this same connection. Because DX measures an organization’s full developer experience and provides both qualitative (“how difficult is this?”) and quantitative (“how long does this take?”) data, engineering can now show the impact different areas of friction have, and in turn make more informed decisions about what needs to be prioritized. 

An example of this is technical debt. “Technical debt” is a term that is talked about in many companies, and at 1-800 Contacts it was no different. After rolling out DX, the organization was able to see the real impact of technical debt: they could see how many teams were being delayed by technical debt, as well as how painful technical debt was across the organization. Moore added, “Now that product managers and directors have visibility into these areas, they’ve started asking questions like, ‘what are we doing about this?’ and ‘how can we help respond to that?’ That visibility we absolutely did not have before.” 

Moore emphasized how DX has provided a channel for developers to voice the pain points they’re experiencing in a way that non-technical stakeholders understand. He said, “One of the most powerful things about DX is that it puts data behind areas of the developer experience that engineers or managers might know hurt our ability to ship, but that directors, PMs, or executives aren’t fully aware of. And it's not just maintenance or bugs, but we can show ‘here's the tooling, here's the experience that we're having that is preventing us from delivering in that higher vein.’”

Taking action at the team level

DX provides results back to teams transparently, allowing everyone to immediately see what areas are causing the most frustration or delays, as well as how each area of their developer experience compares to industry benchmarks. That information has fueled conversations within retrospectives at 1-800 Contacts that lead to action. “We’ve seen significant gains from teams being able to see how they’re improving and that there are opportunities to take on problems they thought were untouchable in the past, like technical debt, CI/CD, and our codebase experience,” Moore adds. “We also have teams that come together regularly to share what they’re doing to solve different problems, and how those efforts are working.”

Vercel began growing their engineering headcount significantly beginning in early 2020. As the company grew, their leadership wanted to make sure that systems and processes weren't impeding their developers' productivity.

Vercel’s VP of Engineering, Lindsey Simon, had prior experience running developer experience surveys as a way to identify where teams were experiencing friction, and recognized the significant investment required to continually design and administer surveys on their own. By implementing DX, Vercel has saved hundreds of hours each year and been able to focus on interpreting and taking action based on their results. Simon explained: “When we discovered DX, a product that draws on research to ask the right questions in the context of software engineering, that was really appealing because we know how difficult that is.”

Through the use of DX, Vercel’s engineering organization captures insights and trends each quarter, which they use to ensure their engineers are as happy and productive as possible.

Identifying opportunities to improve

Since implementing DX, Vercel has achieved 95%+ participation rates in each of their DX surveys, uncovering opportunities for engineering managers and executives to take action. Simon stated: "DX is like having a superpower within your organization to know what’s working and what’s not.”

Below are some examples of insights that DX has provided:

• The engineering team’s release process was an area where the company needed to make a more coordinated effort to improve, so they formed a tiger team to focus on that area. “Sometimes you can encourage teams to tackle these issues, and sometimes you need to make more aggressive investments,” Simon explained.  
• Teams were flagging “connectedness” as something they wanted to improve. “That was something we hadn’t really prioritized prior to our first round of DX surveys as heavily,” Simon explains. “Since then we've instituted far better EM-IC ratios and done in-person meet-ups, and have seen real improvements there.” 
• They have also learned how new team members perceive areas of their developer experience versus those who have been at the company for much longer, which has been useful as they have grown quickly.

Driving continuous improvement

DX has helped Vercel continuously improve developer productivity through lightweight processes. Each quarter, engineering leadership shares key findings from the latest DX survey along with themes for teams to focus on. Engineering managers also review results with their individual teams, setting focus areas within DX to track areas they want to improve. Vercel runs DX surveys on a quarterly cadence to spark new improvements and hold themselves accountable for plans committed to in the past.

To encourage developers to make improvements to developer experience on their own, Vercel has begun promoting scope of impact in their engineering Career Growth Framework. Simon explained: “If you can do work that affects the team or entire organization, that’s a high level of impact. If you can make the testing infrastructure faster, we can multiply that by the number of engineers and clearly see the impact.”