The truth about Git metrics tools
There’ve recently been a couple of interesting Twitter discussions about engineering analytics tools, namely whether people actually use them and the questionable metrics that they expose.
There are several dozen vendors that currently provide this type of solution, primarily providing dashboards GitHub data, Jira metrics, and other third-party developer tools.
I have a unique perspective on these solutions because the company I formerly started, Pull Panda, provided customers with similar metrics features like pull request throughput and turnaround time. After getting acquired by GitHub, I spent a year continuing to work in this space because GitHub had interest in launching a similar product.
My experience working on Pull Panda was that pull request metrics provided value in creating awareness around inefficient code review processes. But there were no real benefits beyond that. I was dismayed to find leaders frequently using the product metrics for inappropriate purposes like performance reviews, or requesting we add metrics like lines of code to the product, which I knew were harmful.
After getting acquired, I kept an open mind because I thought that my new colleagues at GitHub, like Dr. Nicole Forsgren, might have good ideas for how to leverage Git data to produce useful engineering metrics. We built an initial product and piloted it, but shut it down after we found that teams didn’t get much value out of it and most hardly used it at all. Although our product vision for “data-driven engineering” resonated with leaders and made for compelling sales pitches, there wasn’t real value to back it up.
Today, the sheer number of vendors selling similar solutions has made these kinds of metrics appear valuable even though they’re not. Many of these vendors make bold marketing claims of helping organizations 10x developer productivity, with pricing as high as $400 per year per developer.
I’ve personally yet to come across an example of a real benefit gained from these metrics other than shining a light on inefficient code review processes, which is a problem that’s easily solvable without dashboards. Moreover, these tools encourage managers to walk down bad paths, such as micro-managing or stack-ranking their developers.
Finding ways to measure and improve software organizations is a complex problem to which Git metrics are not the solution. Thankfully, many leaders nowadays recognize that Git and Jira metrics are of little use, and instead are looking for new approaches. If you’re a leader looking for insights on how to improve developer productivity, steer clear of these types of solutions and instead focus on developer experience.