Enterprise Security Software Company
Making release risk visible
The challenge
Monthly releases of an on-premises security product needed a shared view of what had been tested and what remained uncertain. Code coverage alone did not describe readiness across the product’s known testable surfaces.
My approach
We used a dashboard to bring together API, unit, and end-to-end functional test coverage across identified surfaces, along with execution status and who performed the testing. Zephyr tracked manual and automated tests. The confidence score was successfully completed tests divided by total identified tests; manual and automated execution counted equally. Automation changed how a test ran, not its contribution to the score.
What the percentage could tell us
The numerator represented successful tests; the denominator represented identified tests. For example, 80 successful tests out of 100 identified tests would produce an 80% score. This is an illustrative calculation, not a historical release result. It would not establish that 80% of every possible product behavior was covered.
Coverage gaps still required judgment
A surface with poorly identified tests could appear reassuring in the aggregate. The percentage therefore needed to be read alongside surface-level coverage and execution status. Test generation and execution could help produce evidence, but did not establish that the team had identified every important scenario.
The outcome
The dashboard made known testing risk visible in release discussions. Our normal acceptance threshold was 80%; a lower score required a conversation about release risk. An incomplete or failed test could lower the score, while a missing test could make it misleadingly high. We connected customer-reported defects back to the dashboard to investigate whether testing had missed the scenario or an existing test was ineffective. This was a decision-support practice, not a statistical probability that a release would succeed.