UKG’s system for driving effective AI use

This post was originally published in Engineering Enablement, DX’s newsletter dedicated to sharing research and perspectives on developer productivity. Subscribe to be notified when we publish new issues.

I recently sat down with Thomas Newton, VP of Engineering at UKG, to discuss how his team built a system to guide AI adoption and assess whether it’s translating into meaningful engineering outcomes. What was particularly interesting was how UKG provides teams with a metrics dashboard that managers can use to have better coaching conversations, and ultimately help developers become more effective with AI tools.

For this week’s newsletter, Thomas is describing how their approach works.

Here’s Thomas.

Thomas: When we started investing more in AI-assisted development at UKG, we faced a question that most engineering organizations are grappling with right now: how do you measure whether AI usage is translating into more meaningful work shipped?

This was a leadership priority with one goal from the start: make sure that the right data, in a digestible format, landed with the people closest to the work—the managers.

The result is what we call Manager AI Adoption Scorecards: a set of dashboards that combine AI usage patterns, delivery outcomes, and spend into a single picture that engineering managers can use to coach their teams, guide adoption, and have better conversations about how work is getting done.

Starting with experimentation and adoption

Our AI journey started the way most do. We experimented with several tools (GitHub Copilot, Windsurf, and others) before making a significant push toward Claude after seeing strong results in late 2024. By early January 2025, the rollout was underway in earnest.

Giving our engineers access to these tools was a good place to start, but our managers needed visibility. They could feel the productivity gains anecdotally, but the data was missing. We wanted to make that a lot more visible.

The shift to a consumption-based model made this need even more urgent. Unlike fixed-cost seat licenses, consumption pricing means the meter is always running. Leaders needed to understand not just whether teams were using AI, but whether the investment was producing returns.

Deciding what to measure

One of the earliest decisions we made was to anchor the scorecards around TrueThroughput, a metric from DX that goes beyond raw pull request counts to account for the relative complexity and size of work delivered. TrueThroughput uses AI to classify the complexity of different tasks, giving you a size-adjusted throughput number. Think of it like a weighted GPA versus an unweighted GPA. Both are useful, but the weighted version tells you whether someone is delivering meaningful work or just merging a lot of five-second fixes.

Instead of indexing on consumption, the scorecards create a balanced view of AI impact by correlating consistent AI usage, measured in days of use, with TrueThroughput. These metrics help us answer whether consistent use of AI is helping teams deliver more meaningful work.

Inside the Scorecard

The scorecard was built around a quadrant view. The horizontal axis tracks consistent days of AI use over a 30-day window: fewer than 15 days puts you on the left, more than 15 on the right. The vertical axis tracks TrueThroughput.

That creates four zones:

• Bottom left: Low AI usage, low throughput. The area where most teams are actively working to move out of.
• Bottom right: High AI usage, lower throughput. Could indicate learning, experimentation, or non-code work that throughput doesn’t capture yet.
• Top left: High throughput, low AI usage. Your classic high-performing engineer who hasn’t yet adopted AI tools.
• Top right: High AI usage, high throughput. The target state.

Each dot on the chart represents an individual. Bubble size reflects spend. And the views are drillable: from the entire organization down to a business unit, a team, and ultimately an individual manager’s direct reports.

What the data revealed

Since rolling out the scorecards, the results have been striking. In January, the bottom-left quadrant was packed, meaning that a large portion of our engineering organization hadn’t touched AI tools at all. Then by May, that quadrant was nearly empty; less than 1% of the organization remained in the low-AI, high-throughput zone. It’s been exciting to watch the whole organization shift on this.

We also saw some things in the data that confirmed what we’d hypothesized. For example, we saw that the group seeing the biggest throughput gains were our senior and principal engineers, at ~20-30% above what other roles saw. That’s intuitive, but it was interesting to see it in the data.

One insight caught us off guard: managers and directors started writing code again. Our leadership population started doing more direct hands-on coding. They have more assistants, they can multitask better, whatever the reason, it’s a clear trend. That pattern extends beyond engineering: our product managers and designers have started leaning into AI tools too, getting comfortable with the terminal, creating digital artifacts, and contributing in ways that show up in delivery metrics. If you’d asked me when we started this, I don’t think it was what we expected, but it’s a trend that is now more commonly observed and discussed.

How managers use scorecards

For me, the real value of the scorecards is the quality of the conversations the data enables.

For managers, the first question they asked focused on adoption: “How do we get you from left to right?” Are you using it daily, is it part of your workflow, or are you still finding your footing with it?” And once adoption was in place, the conversation shifted towards helping them move from the bottom to the top, increasing their impact.

When a manager saw an engineer with high AI usage but flat throughput, the right response wasn’t to question the spend. It was to ask what they were working on — maybe they were ramping up on new AI workflows, or in a role where their AI-assisted work didn’t produce code commits.

The best way to head off gaming or surveillance concerns is proactive communication. Don’t let people fill in the blanks on what the scorecards are for or why they exist. Be very clear. They are a tool for improvement, learning, and growth, not for surveillance, ranking, or performance calibration. If teams interpret the scorecards as a tool for punishment or reward, they’ll have every incentive to game the numbers.

At UKG, the framing has been consistent from day one: the scorecards exist to help managers empower their team, understanding the work, understanding how AI is being applied to that work, and helping people lean into a new way of working.

Keeping AI spend in check without leading with cost

AI spend is visible on the scorecard, but it’s deliberately not the headline metric. It’s an awareness layer, something managers should be conscious of, not something that drives the conversation.

We use what I call “circuit breakers”, daily budget controls that flag when usage spikes above a threshold. But we haven’t yet landed on a firm benchmark for what reasonable per-engineer spend looks like.

The range is just too wide right now. Some teams are considering multi-agent orchestration and long-running loops that transform an entire codebase overnight. Others are using AI to finish a feature today. At this stage, it’s really hard to generalize.

I expect spending patterns will stabilize as the initial learning ramp flattens. Your first couple of prompts will probably be expensive. You’re still learning how to use the models correctly, you’re playing, you’re understanding how to use this radical new thing. But we’re about five months in and starting to see averages form. We’re still in an experimentation phase, and for now, the approach is pragmatic: watch carefully, trust manager judgment, and make sure the investment is going toward the right outcomes.

What comes next

We’re already thinking about the limits of what the current scorecards capture. TrueThroughput is powerful for teams that ship code, but it misses productivity gains in operations, SRE, and other roles where AI is being used to correlate incidents, search logs, and accelerate resolution, work that never ends up in a pull request.

We have teams where an engineer uses AI to search past incidents during a live outage and correlate similar patterns to get to a resolution faster. That’s enormously productive, but it won’t show up in throughput. We’re trying to think through what the next level of digital footprints looks like, the metrics that capture the full picture of AI-enabled productivity, not just the code-commit slice of it.

The scorecards are a living system, designed to evolve as we learn more about what effective AI-enabled engineering actually looks like.