Booking.com uses DX to measure impact of AI on developer productivity

Booking.com, one of the world’s largest online travel platforms serving over 500 million monthly visitors, has centered its attention toward driving company-wide efficiency as a way to maintain its competitive edge. As part of this effort, they’ve focused on rolling out AI tools to their engineering organization to help accelerate the pace of delivery and innovation.

“We’re very excited about AI because it has the potential to multiply the impact of our engineering organization,” explains Amos Haviv from Booking’s Developer Experience team that is responsible for procuring and rolling out AI tools. “But it’s early days with these tools, and it’s very hard to measure their impact.”

When Booking first began rolling out AI tools, there were inflated expectations from leaders about the results AI would deliver, and developers were skeptical. “The numbers we were hearing were astronomical,” recalls Bruno Passos, Group Product Manager. “So going into this, our goal was simply to understand what AI could actually do for us and figure out a way to measure the value it was delivering to the organization."

At the time, Booking did not yet have an effective method for understanding how AI tools were being used or the impact they were having. Without this data, they recognized they wouldn’t be able to make informed decisions about vendors or get insight into areas where they could drive further usage.

By partnering with DX, Booking.com was able to quickly capture valuable insights: Booking saw that with one AI code assistant, daily active users had a 16% higher PR merge rate than non-users. They also identified that overall, developers using AI were saving significant time on tedious tasks, freeing up capacity for more innovative work.

“We needed to understand how AI was affecting engineering velocity, satisfaction, and code quality,” says Leo Kraan, Director of Engineering. "DX helped us solve this problem. Had we not had the data DX provides, we wouldn’t be able to confidently talk about the ROI of Booking’s investment in AI with the rest of the business. We also wouldn’t know for sure whether we should be driving more adoption for a specific tool or use case.”

“Using DX to quantify the impact of our AI code assistant has given us confidence in the outcomes it’s driving for our developers,” says Passos. “Not only did it confirm that the tool is helping us ship faster, but these insights are helping us refine and guide our continued rollout.”

Armed with data, Booking has also been able to make more informed vendor decisions. “We’ve been able to use this data to decide where to invest further in our AI program,” adds Zane Wright, Senior Product Manager for Developer Experience. “DX has helped us drive decisions like which vendors we select, where we should be looking to assess a deeper impact on our company, and how we should be structuring our programs to best impact developers moving forward.”

Now that Booking has moved beyond the initial phases of their AI rollout, they are focused on two key priorities: driving more frequent usage, and understanding AI’s impact on code quality. “When we saw that developers using AI more than 12 days per month were significantly more effective, we shifted our goal from just adoption to driving daily adoption,” says Passos. “Beyond that, we also want to learn about AI’s impact on quality. Can we correlate the code generated by AI with fewer vulnerabilities or fewer bugs? That’s what’s next for us.”