How does one accurately measure software development productivity? A study at Microsoft took a stab at surfacing insights from self-reported measures.
In the study, software engineers were asked to rate their productivity on a daily basis. These self-reported measures were then compared against automated metrics, allowing the researchers to analyze the discrepancies and identify the factors that significantly impact a developer's self-perceived productivity.
What are the pros and cons of automated vs self-reported measures?
An advantage of automated product or process metrics is that they are easy to measure and give seemingly objective results. The authors refrain from calling them objective, however, because: 1) The choice of which measures to use is itself subjective. 2) The interpretation of a metric such as lines of code per day is problematic and subjective—what about engineers who produce more concise code in the same time, what about that one-liner patch that took a week to debug and fixes a business critical problem? 3) Automatic measures can be gamed. 4) In many cases, automatic measures fail to capture work that happens off-screen, for example, meetings and discussions. Addressing some of these issues, IBM introduced the concept of Function Points in the late 1970s. However, these, too, have drawbacks: they include an element of subjectivity by how one judges “functionality” into the measurement itself.
Self-reported productivity, the authors argue, circumvents some of the challenges of automated measures, but it is susceptible to, for example, cognitive biases. In contrast to many automatic measures, its subjectivity also makes it hard to compare across individuals. Personally, I agree with this. It is better to be upfront about the subjectivity of the measure than to pretend that it is objective. The authors also point out that self-reported measures are not necessarily more subjective than automatic measures. For example, the choice of which automatic measures to use is subjective, and the interpretation of the results is equally subjective.
What were the findings?
- The more time software developers can spend on coding, the higher they rate their productivity.
- Traveling had the single largest negative impact on self-reported productivity. Quality of sleep and interruption-free work were important, but the day of the week was not (no “Friday effect”).
- The sheer process of reflecting on their productivity seemingly helped software engineers be more productive.
Conclusion
As you grow your software team, the question of how to accurately measure and enhance productivity remains critical. These insights not only provide a fresh perspective but also suggest a need to revisit traditional automated approaches. What does this mean for your organization? Take a deeper dive into the study to find out. - https://arxiv.org/abs/2012.07428