Measuring AI in software teams: Trends we’re beginning to see

Key trends we’re seeing

No one is forecasting AI ROI accurately yet

Google’s reported 10% productivity lift seems realistic based on industry data, but most organizations aren’t translating these gains into reduced headcount or faster delivery timelines; they’re still building measurement baselines rather than proving the true ROI of AI. This disconnect between productivity metrics and business outcomes represents a critical missing piece in how organizations approach AI investments.

The perception of AI’s impact doesn’t match reality

There’s a growing disconnect between how AI’s impact on software development is portrayed and what’s actually happening on the ground. Headlines make sweeping claims: 30% of Microsoft code is written by AI, AI will be writing 90% of code in six months, and Robinhood CEO says the majority of its code is written by AI.

But the reality is more nuanced. Tools like Copilot, Cursor, and Windsurf promise to transform the way developers work—but they also introduce new friction points, change existing workflows, and reshape team dynamics in ways that are often overlooked.

This complexity contributes to a widening perception gap. Many developers feel like AI is making them faster. Yet emerging research—like METR’s study—suggests the opposite: AI tools are often slowing developers down in measurable ways.

This disconnect has real consequences. When organizations make strategic decisions based on hype rather than evidence, they risk investing in the wrong tools, misinterpreting developer needs, and missing the opportunity to adapt in ways that actually improve outcomes.

AI is changing the roles of junior and senior developers

Junior developers are leveling up faster with AI assistance, compressing traditional learning curves, and contributing meaningfully sooner. Meanwhile, senior developers are increasingly operating like team leads manage individual contributors—except their “team” consists of AI agents handling routine tasks while they focus on architecture and strategy.

The future engineering organization may look less like traditional hierarchies and more like agent orchestration teams, where human expertise guides autonomous systems rather than writing every line of code.

Code maintainability may slightly decline

In some but not all cases, DX customers see a slight decline in code maintainability—a self-reported measure from developers—when AI tools are used. This naturally raises concerns about the impact of AI on quality. However, it’s too early to determine whether this represents a fundamental problem or simply the emergence of a new abstraction layer, where developers interact less with the underlying code and are therefore less knowledgeable or confident in the maintainability of that code. In that case, perhaps it’s expected to see code maintainability slightly decline, and we should baseline against this new normal.

Organizations should track other measures of quality in addition to self-reported measures of code maintainability (such as PR Revert Rate or Change Failure Rate) to get a better understanding of whether quality is being impacted.

Power users see better results

DX’s cohort analysis across customers reveals a critical insight: incremental AI adoption delivers incremental results, but major productivity breakthroughs only emerge once developers reach “power user” levels of usage. This isn’t about using AI occasionally for boilerplate code—it’s about developers who’ve restructured their entire workflow around AI-assisted development.

Recommendations

We’ve covered what we’re seeing across hundreds of engineering organizations: the shifts in roles, workflows, and the surprising gap between perception and impact. Now, we turn to what teams can do about it. The recommendations below are designed to help organizations not just keep pace with AI, but lead in how it’s operationalized.

Capture a baseline to compare against

To evaluate the impact of AI, organizations must first understand its pre-AI state—and treat AI adoption as a rigorous experiment worth measuring properly.

Without these measurements, you’ll never know if AI actually improved productivity or just changed how work feels. Perceptual baselines—such as developer satisfaction, perceived friction, and ease of delivery—must be collected immediately through surveys because these subjective experiences cannot be retroactively recreated once AI becomes embedded in daily workflows.

Simultaneously, system data from AI tooling, source control systems like GitHub, and internal platforms should be captured and indexed to enable credible before-and-after comparisons. Track metrics across utilization, impact, and cost dimensions to build a complete picture of how AI affects your organization. Question assumptions, validate claims, and let evidence—not intuition—guide your decisions.

The organizations that move fast on baseline measurement will have longitudinal impact studies; the ones that wait will have anecdotes and wishful thinking.

Leverage internal AI tooling as a strategic differentiator

Companies building their own AI stacks—integrating tools like Copilot, Claude, and Cursor into custom workflows—have a unique opportunity to unlock deeper insight into how developers actually work.

These integrated environments can provide unmatched visibility into AI’s real impact, but only if they’re instrumented thoughtfully. Forward-thinking organizations don’t just use AI tools to speed up development—they also use them to capture rich signals about developer behavior, agent interactions, and downstream outcomes that off-the-shelf tools can’t surface. The companies that succeed with AI will be the ones that treat internal developers as customers—and use AI to continuously improve their experience.

Prioritize removing friction over maximizing speed

Much of the early AI narrative has centered on speed—but the real long-term gains may lie in unlocking overlooked work that would have otherwise been deprioritized, abandoned, or delayed.

Think of AI less as a code speed multiplier and more as a tool for removing organizational drag: faster onboarding that gets new hires contributing in weeks instead of months, reduced toil that frees senior developers for high-impact work, and higher completion rates for those low-priority bug fixes and tech debt items that never made it to the top of the backlog. This shift reflects a deeper kind of leverage that aligns with outcomes, not just output. Reducing friction actually changes how organizations operate.

Lead with clear data to unlock AI’s impact

Across the companies we work with, those positioning themselves well for long-term success are approaching AI as they would any other significant technology decision. They’re identifying specific problems AI can solve, building the necessary organizational capabilities to support adoption, measuring the impact of AI systematically (like with frameworks such as the AI Measurement Framework), and maintaining unwavering focus on fundamental software engineering principles.

As you navigate your AI journey, stretching between worlds that are simultaneously completely different yet exactly the same, remember: data beats hype every time. We hope this article helps you as you chart your organization’s course into the future.

To see how DX can help you measure and drive the impact of AI in engineering, request a demo.