Thinking about whether or not the software developed is tested sufficiently before shipping or Go Live leads to the question: How can we make it measurable and transparent in facts, figures and numbers? What are the right measures and KPIs for software quality that tell us whether quality actually is bad or is it just a gut feeling.
How to overcome doubts in the organization and especially within the developer community that facts and figures are the way to go, is a whole other topic and I will explore that in a different blog post. For now let’s assume, that the organization is convinced, that facts and figures are the right thing (I too am convinced of that!).
From my experience the most typical and most used measure for testing from developers perspective is test coverage as in code coverage of tests. For me that is not a very important KPI, just one out of many others that may be much more important.
Let’s start with the different test dimensions that we can measure. For me these are:
- Code Quality
- Security (I will leave this mostly to the real experts)
- Technical Test Coverage
- Functional Test Coverage
Now what is what. Code Quality seems to be easy to answer. How well is Code written – is it readable, maintainable, adaptable. Security is such a big action field that I will reduce my scope to the easily measurable things such as identifying outdated external libraries. Coming to Test Coverage I differentiate between technical measures and functional measures. Technical Test Coverage to me answers the question to what extent the code has been covered by tests. Those tests can be Unit Tests, Integration Tests or others. Regarding to how well are we testing our software, Functional Test Coverage is a really high value information for me. It is not focused on covering a lot of code, but on covering our functionality. Functionality such as Use Cases, User Journeys, E2E Business Processes.
So what KPIs regarding to these four dimensions do I recommend to start of with.
Code Quality
Start with the simple stuff that tools like sonarqube give you. E.g.:
- Complexity
- Duplicated lines (%)
- Issues (New and total)
- Code Smells (New and total)
- Technical Debt Ratio (on new code and total)
- Bugs (New and total)
The differentiation between new and total is also important as we can see whether or not we are improving in the recent weeks / months / releases.
Security
Similarly to Code Quality, let’s start with the easy stuff. Most developers or software architects should be familiar with these measures.
- Vulnerabilities
- Security Hotspots (on new code and total)
- License scanning
- Dependency Scanning
Technical Test Coverage
- Condition Coverage (c0, c1 etc.)
- Line Coverage (on new code and total)
- Unit test errors (vs. total number of unit tests)
Functional Test Coverage
Generally speaking functional test coverage aims to ensure that E2E Use Cases are tested thoroughly as opposed to testing that the code works. The difference is that with all the code focused tests we check whether the software is „working as implemented“ whereas the functional Use Case centric tests check whether the software is „working as intended„. To me that is a fundamental difference and the reason why we cannot and should no do without functional tests. However this is a large topic so lets explore in a later post – I will link it to this one once written. 🙂
What next?
So now we have a number of relatively simple to obtain and standard KPIs.
I would start with assessing the current state of which KPIs do we already measure and how well are we doing on them. A simple table can be drafted and then discussed with the development teams. The result could be as follows.
For each KPI we need to know whether or not we are able to obtain the values and additionally whether we believe the values are a correct representation of reality. If the values are in doubt we will loose support of developers and trust of management.
Then the tool is important to know who is tool owner and responsible for fixing faulty measures. And lastly the current value of the KPI. If possible place an industry standard value next to it to make transparent whether we are underachieving in that aspect
| KPI Category | KPI | Status | Tool | Value | Industry Benchmark |
|---|---|---|---|---|---|
| Code Quality | Code Smells (total) | Available – measures are incomplete | Sonarqube | – | – |
| Code Quality | Code Smells (new) | Available – measures are incomplete | Sonarqube | – | – |
| Test Coverage | Line Coverage (on new code) | Available | Sonarqube | 23% | 80% |
| … | |||||
| Security | License Scanning | Unavailable | – | – | – |
To fill the blank spaces talk to every product owner and architect you can get a hold of. Make sure to get management support and help everyone understand the importance. After the values are gathered prepare a nice visualization (does not need to be Powerpoint) and present to your supporting managers. Be careful not to blame persons for wrongdoing of the past, but to focus on the future.
Nice diagrams that are easy to make and speak more than 9000 words ( 🙂 ) are Heat Maps, network diagrams or bar charts.
Next steps should then be envisaged!