Google’s framework for defining and measuring software quality
Google defines software quality as including four types: process quality, code quality, system quality, and product quality.
This post was originally published in Engineering Enablement, DX’s newsletter dedicated to sharing research and perspectives on developer productivity. Subscribe to get notified when we publish new issues.
I recently read a paper from Google‘s Engineering Productivity Research team sharing how they define Software Quality. For engineering leaders focused on quality, this paper provides a helpful framework for articulating what it means and where improvements need to be made.
Google’s team has previously talked about how they measure developer productivity through three dimensions: speed, ease, and quality. These counterbalancing dimensions important because they allows leaders to make sure that improvements in one area don’t come at the expense of another—for example, if they make a change that attempts to impact speed, they’re also going to be keeping a pulse on how it impacts quality and ease.
Of these three dimensions, quality is the hardest to measure because it’s also the hardest to define. Additionally, it means different things to different people, so conversations about “how do we increase quality” can lead to misalignment. In this paper, Google’s team breaks down how they arrived at a shared definition, the framework they landed on, and the specific metrics they track to measure it in practice.
My summary of the paper
To better understand what “quality” means, the research team started by conducting an extensive literature review to understand the themes that regularly appeared in papers on code quality. (They found several items, including defect rate, reliability, maintainability, and more.) Then, they conducted two series of interviews with developers at Google asking how they would articulate the different types of quality, their impact, and their relationships to each other.
Based on their interviews and literature review, the authors developed a “theory of quality.” It posits that there are four types of quality that influence each other: process quality, code quality, system quality, and product quality.
The four types of software quality
1. Process quality
Process quality refers to things like having comprehensive and deterministic testing, thorough code reviews, organizational consistency, and an effective planning process. In other words, factors of developer experience. The researchers say “everything begins with a high-quality development process,” because higher process quality leads to higher code quality. In fact, they point to multiple studies that have shown that process-based metrics are even more predictive of post-release defects than existing code quality metrics.
2. Code quality
This type of quality is what most developers think of when discussing quality: it is the ease of working with and understanding code so that developers can easily make changes to it. In their interviews, the researchers found that developers most commonly related code quality to “maintainability” — this is consistent with the findings from another similar study.
The impact of code quality is twofold: it improves the quality of the system by reducing defects and increasing reliability, and it also leads to higher velocity for developers.
3. System quality
System quality relates to reliability, performance, and defect rates. Measuring system quality is challenging because of the sparsity of data: outages and security incidents should be rare, so it can be hard to tell whether efforts to improve these areas are having an impact. For this reason, the authors suggest using process and code quality metrics since they are indicators of system quality.
4. Product quality
Product quality is the end goal. It’s what is experienced by customers, and specifically refers to the product’s utility, usability, and reliability. This type of quality, in other words, is how well the product does what it says it will do. The authors note that engineers have influence over the aspect of reliability, however they need to work with product managers and user experience designers to contribute to usability and utility.
Final thoughts
Google’s theory of software quality, including both the four types of quality and how they influence each other, can provide leaders with a way to have more clear discussions about the topic. It also provides a framework for thinking through how improvements might be made and measured. For example, the authors say that improvements to process quality may be a good path to improve code quality. They also said that system quality might be better measured using process and code quality metrics. Ultimately, using this framework may help leaders and teams make sure they’re all referring to the same thing in conversations about quality.