How to measure AI's impact on your engineering team

Engineering leaders are facing a measurement challenge unlike any they’ve encountered before. AI-powered development tools promise transformative gains in developer productivity metrics, but traditional software engineering KPIs often fail to capture their actual impact.

While headlines claim “30% of code written by AI” and “2x productivity improvements,” the reality we’re seeing tells a more nuanced story.

As DX CTO Laura Tacho puts it:

“You don’t have to look far to find sensationalist headlines. Microsoft says AI is writing 20 to 30% of code. Google puts thatf number at 30%. And Anthropic’s CEO predicts that in just three to six months, AI will be writing 90% of all code. But none of those numbers line up with what real organizations are experiencing on the ground.”

This disconnect between AI tool marketing and reality makes proper engineering productivity measurement critical. Without robust developer productivity metrics, you can’t justify AI development tool investments, optimize adoption strategies, or demonstrate measurable business value.

Here’s how to build a comprehensive measurement framework that captures the actual impact of AI coding assistants on your development teams.

The multi-layered measurement challenge for AI development tools

Measuring AI development productivity calls for a fundamentally different strategy than tracking traditional engineering performance metrics. Unlike most developer tools, AI coding assistants are used in a fragmented and dynamic way—developers might write code with GitHub Copilot in their IDE, brainstorm using ChatGPT, and turn to Claude for help with documentation, all within the same hour.

Some leaders point to bold claims—like AI writing 30% of code or doubling productivity—but DX CEO Abi Noda says the data paints a more measured picture.

“We’re just not seeing those kinds of results consistently across teams right now,” he explains, but that doesn’t mean AI isn’t delivering value—we are seeing a steady increase in both adoption and measurable impact, Abi adds.

At leading engineering organizations, DX is seeing 60–70% of developers using AI tools weekly, and 40–50% using them daily in more mature rollouts.

To track this kind of progress, you need more than basic output metrics. Measuring AI’s impact effectively requires a layered approach—one that blends quantitative developer data with qualitative usage and experience data to reflect how AI tools are actually used in practice.

Here’s how we recommend approaching AI productivity measurement within your engineering team.

AI productivity metrics framework

Effective measurement of AI development tools requires tracking three distinct layers of software engineering metrics. Each layer provides different insights into how AI coding assistants impact your team’s performance:

Layer 1: AI tool adoption metrics

Layer 2: Direct AI impact metrics

Layer 3: Business impact metrics

Real-world case studies: AI development productivity in action

Usage frequency drives measurable gains

We partnered with a major enterprise job platform that uncovered a strong link between AI tool usage and engineering output. By segmenting developers into tiers—top users, frequent users, infrequent users, and non-users—they identified a striking pattern in pull request throughput.

Engineers who used AI coding tools most heavily merged nearly 5 times as many PRs per week as those who didn’t use them at all. Frequent users weren’t far behind, averaging nearly 4x the output of non-users, while even infrequent users delivered a 2.5x boost.

The trend was unmistakable: the more consistently engineers used AI assistants, the more they shipped, offering clear, data-backed evidence that frequent AI adoption drives real, measurable productivity gains.

Rapid adoption through data-driven visibility

A leading cloud storage company achieved exceptional results by adopting a measurement-first approach to introducing AI coding tools. They set a bold goal: double adoption and reach 80% weekly usage among eligible engineers. By sharing detailed adoption data by team and spotlighting progress in company-wide meetings, they hit that target in just three weeks.

Rather than stopping at adoption, they focused on sustaining impact. Weekly usage remained above 80% across multiple tools, including GitHub Copilot, Cursor, and Windsurf, thanks to ongoing optimization.

This team ran cohort analyses to compare usage frequency across teams, and performed feature- and model-level evaluations to fine-tune which tools worked best for different workflows.

This deliberate, data-driven rollout not only accelerated adoption but also helped the company continuously improve how AI was used across engineering.

Same-engineer analysis to isolate AI’s actual impact

A major financial services company took a rigorous approach to measuring the real ROI of AI coding tools by tracking performance over time within the same engineers.

Instead of comparing across teams or roles, they identified engineers actively using AI tools today. They used DX to measure their productivity against their baseline performance from the previous year, before the introduction of AI.

The results were clear: engineers using AI tools showed a 30% increase in pull request throughput year-over-year, compared to just 5% among non-adopters. This same-engineer methodology eliminated variables such as tenure, seasonality, and team changes, providing leaders with a clear view of AI’s actual impact.

By removing confounding factors, this approach has become the gold standard for engineering leaders seeking to understand how AI tools directly impact developer productivity.

Implementation roadmap for AI productivity measurement

There’s no one-size-fits-all way to measure AI’s impact on developer productivity, but a few key steps can help you build a strong foundation and generate real insight over time. Here’s a sample roadmap to guide your first six months—and beyond.

Months 1–2: Set your baseline

Start by understanding where your teams are today:

• Run a developer experience survey
• Measure core engineering metrics like:
  • Pull request throughput
  • Code review cycle times
  • Deployment success rates
• Track how much time engineers spend on tasks like debugging, documentation, and code review

👉 Pro tip: Use this data to spot teams or workflows that stand to benefit most from AI tool adoption.

Months 3–4: Roll out and start tracking

Begin introducing AI tools in a controlled, measurable way:

• Launch with a few pilot teams or opt-in users
• Track weekly adoption at the team and individual levels
• Run short surveys to capture time savings and early feedback
• Share adoption progress in company-wide meetings to create momentum and social proof

Months 5–6: Measure impact

Now connect tool usage to real productivity outcomes:

• Look at how AI usage correlates with your baseline metrics
• Group users into cohorts—heavy, frequent, occasional, and non-users—and compare results
• Identify where AI is driving the most value and document what high-performing users are doing
• Use these insights to improve onboarding and training programs

Ongoing: Optimize and expand

Keep the momentum going with regular reporting and refinement:

• Share monthly usage and impact reports with engineering leadership
• Do quarterly deep dives, combining metrics with engineer interviews
• Adapt your measurement strategy as new tools emerge and usage matures

High-impact use cases for AI development tools

Contrary to common assumptions, the highest-impact applications of AI coding assistants are not solely focused on coding tasks. As you implement your AI measurement system, make sure it takes into account all possible AI use cases, like:

• Stack trace analysis and debugging: AI coding assistants excel at rapidly interpreting complex error messages and suggesting solutions
• Code refactoring and cleanup: Automated suggestions for improving code quality and maintainability
• Test generation and documentation: AI tools can significantly reduce the time spent on repetitive but essential development tasks
• Learning new frameworks or languages: AI assistance accelerates developer onboarding and cross-training initiatives

While headlines often hype up how much code AI can write, the reality on the ground looks different. Laura explains the findings from DX’s Guide to AI assisted engineering, where Justin Reock dives deep into how engineering teams are actually using AI.

“Mid-loop code generation—actually using AI to write code—wasn’t one of the top use cases. The real time-saver was stack trace analysis. These tasks are notoriously tedious, but AI is incredibly good at quickly answering: ‘What does this error mean?’”

For leading engineering teams, the most valuable AI use cases aren’t always the flashiest. Instead of chasing code volume, they’re measuring impact where it counts—like reducing time spent on debugging and error resolution.

Common AI measurement pitfalls and proven solutions

It’s easy to treat engineering metrics like a silver bullet, but that mindset can backfire. Metrics can quickly become counterproductive if misused, especially when they start driving behavior instead of reflecting it. We often advise teams to be mindful of common traps like Goodhart’s Law and other unintended consequences. When it comes to AI measurement, here are a few pitfalls to watch out for:

Pitfall 1: Focusing on vanity metrics instead of business impact

• Problem: Overemphasizing “percentage of code written by AI” or raw suggestion acceptance rates without connecting to business outcomes.
• Solution: Use adoption metrics for process optimization, but focus measurement efforts on throughput, quality, and developer experience metrics that correlate with business value.

Pitfall 2: Expecting immediate linear productivity correlations

• Problem: Assuming time savings from AI development tools will directly translate to proportional increases in code output.
• Solution: Recognize that developers often reinvest AI-generated time savings into higher-quality work, learning opportunities, or solving more complex problems. Measure holistic impact rather than just output volume.

Pitfall 3: Measuring AI impact before adoption maturity

• Problem: Evaluating AI coding assistant effectiveness before developers have learned to use them strategically.
• Solution: Allow for a 3–6 month learning curve before drawing definitive conclusions about AI development tool impact. Early measurements should focus on adoption trends rather than productivity conclusions. Focus on measuring the most significant gains, which occur when moving developers from not using AI to periodic but regular use—this represents the most significant measurable improvement.

Pitfall 4: Ignoring multi-tool usage patterns

• Problem: Measuring only one AI tool when developers typically use multiple AI coding assistants throughout their workflow.
• Solution: Modern developers use 2–3 different AI tools simultaneously. Measure the combined productivity impact rather than evaluating tools in isolation. Track both global AI usage patterns and per-tool adoption metrics.

Making AI productivity measurement actionable

Once you have your metrics, what’s next? How can you translate those insights into improved productivity?

When you notice low adoption rates, dig deeper into what’s holding developers back. Often, it’s not the technology itself but cultural resistance, insufficient training, or clunky integrations that make the tools feel more like obstacles than helpers.

High time savings with AI, but flat throughput, tell a different story: developers aren’t necessarily churning out more code, but they’re using their newfound efficiency for higher-value work. They may be investing extra time in:

• Improving code quality and architecture
• Learning new technologies and techniques
• Tackling more complex, challenging problems

If quality metrics start slipping, it’s time to refine your approach to AI-assisted development. Focus on training programs that teach best practices for working with AI tools, and establish clear code review standards that account for AI-generated code.

When certain developers become power users, they’ve discovered techniques that could benefit everyone. Document their advanced workflows and share these insights across your engineering organization to help others unlock similar productivity gains.

The path forward with AI development productivity measurement

At this stage, the question isn’t whether AI coding assistants work—it’s how to make them work effectively for your teams. The real challenge is understanding how these tools integrate with your engineering culture, workflows, and goals—and using the right metrics to drive continuous improvement.

The organizations getting the most value from AI aren’t necessarily the ones using the most advanced models. They’re the ones measuring thoughtfully, adapting quickly, and investing in systems that turn AI data into engineering outcomes.

Remember: measurement without action is just data collection. But with a deliberate approach to tracking adoption, time savings, and business impact, you can transform AI from a headline into a high-leverage engineering tool.

Interested in exploring further? Here are more of DX’s top insights on AI and developer productivity.

• How to measure GenAI adoption and impact – A practical guide to what’s working, and what isn’t, in top engineering orgs
• Guide to AI-assisted engineering – Learn how leading teams are reshaping development workflows with AI
• Webinar: The biggest obstacles preventing GenAI adoption — and how to overcome them – Hear from Abi, Laura, and top engineering leaders on what’s holding teams back—and how they’re moving forward
•