Uber’s Journey of Measuring AI Impact on Developer Productivity
Abhishek Tibrewal
Senior Data Scientist, Platform Engineering @ Uber
Ty Smith
Principal Engineer, AI & Developer Productivity @ Uber
Abstract
As AI becomes embedded in software development, organizations quickly discover that measuring its impact is far harder than expected. This talk shares a practical, experience-driven account of how our approach to measuring AI impact in developer productivity has evolved over time.
We walk through the distinct phases of our measurement journey: what we chose to measure at each stage, why those choices made sense at the time, what failed or backfired, and what we learned as our understanding matured. Each step reflects how changes in tooling, usage patterns, and organizational expectations forced us to rethink both metrics and methodology.
The core tenet: there is no single metric that proves AI ROI. Effective measurement is a journey from adoption, to engagement, to causality and agentic value — with each phase requiring different metrics, tradeoffs, and communication strategies.
We walk through the distinct phases of our measurement journey: what we chose to measure at each stage, why those choices made sense at the time, what failed or backfired, and what we learned as our understanding matured. Each step reflects how changes in tooling, usage patterns, and organizational expectations forced us to rethink both metrics and methodology.
The core tenet: there is no single metric that proves AI ROI. Effective measurement is a journey from adoption, to engagement, to causality and agentic value — with each phase requiring different metrics, tradeoffs, and communication strategies.
About the speakers
