Zero out of zero is not a hundred percent
My coverage dashboard reported 100% for a repository sitting at 48%. The bug was boring. The class of bug it belongs to is not.
Someone asked me what our test coverage was.
I had a good answer for one repository, a worse answer for a second, and for the rest I had the thing engineers say when they don’t know: it depends what you mean by coverage. True sentence. Useless one.
So I built the obvious thing: a nightly job that walks every repository we own, detects whether it has a test stack, runs it if so, and writes one row per repo into a table. It took an afternoon. It worked.
Then it spent several months quietly telling me a lie.
The number that was wrong
The headline figure for our main plugin read 100.0%.
It is not 100%. Here is what the coverage reporter actually emitted that night.
The dashboard printed 100.0% for this repo. Spot why.
Category Total Covered Percentage
Lines 4631 2258 48.75%
Statements 4723 2309 48.88%
Functions 884 389 44.00%
Branches 3711 1732 46.67%
BranchesTrue 0 0 100.00%Reveal the answer
The last row has nothing in it. Zero items, zero covered. And zero divided by zero was reported as a perfect score rather than as no data.
BranchesTrue is a category this project never populates. The four rows that
describe the actual codebase sit directly above it, all saying roughly 48%.
Finding the empty row is the easy half. The half I find more interesting is why that row, and not one of the four correct ones directly above it.
Here is the function that chose it:
for key, val in stack.items():
if key.endswith("_pct") and isinstance(val, (int, float)):
return (label, f"{val:.1f}%")
First key ending in _pct wins. And the metrics are stored as JSON with
sorted keys, so they come back in alphabetical order.
branchesTrue_pct sorts ahead of branches_pct because a capital T is
ASCII 0x54 and an underscore is 0x5F. That is the entire mechanism. A camel-case
key, a sort nobody chose, and a loop that stops at the first thing it recognises.
Nobody wrote the bug. It assembled itself out of three reasonable decisions.
What makes it properly embarrassing is that the same dashboard has a regression detector, and the regression detector reads the real keys. It had been correctly reporting that function coverage dropped from 49.2% to 44.0% while the summary column two inches away printed a trophy. One half of my tool knew. The other half was congratulating me.
Three ways a signal flatters you
Once I started looking, the dashboard had a second one of these, and then I started recognising the shape everywhere. There are three.
It is declared rather than detected
The version of this dashboard that doesn’t work is a spreadsheet with a column people fill in. I know, because that is what every org tries first.
A number a human types measures that human’s willingness to type a number. It correlates with conscientiousness, with how recently someone was asked, and with how the answer will be read. It correlates only loosely with the system. Nobody lies. The signal is just measuring the wrong thing.
So the job never asks anyone anything. It looks for a Jest config, a Playwright config, a pytest setup, a PHPUnit file. Found or not found.
The first run said that of twenty-five repositories, seven had a test stack and sixteen had none. That second number is the point: it is a number nobody would ever have volunteered, about a question nobody had been refusing to answer. It had simply never been computed. Several of those sixteen are small services doing one thing, and a couple are prototypes that quietly became real. If your org is more than a few years old, your number looks like mine.
Its failure case renders as success
This is the family the 100% belongs to, and it is much larger than coverage tooling.
Zero out of zero is the canonical member. A ratio with an empty denominator has
no meaningful value, but arithmetic is happy to hand you 1.0, and 1.0
formatted as a percentage is the most reassuring string in engineering. Cache hit
rate with no requests. Pass rate with no tests. An SLO with no traffic. Every one
of them reads as perfect at the exact moment it knows nothing.
My second instance was less arithmetic and more carelessness. One repository’s Playwright run exited with an error and executed nothing at all:
{"playwright": {"passed": 0, "failed": 0, "flaky": 0, "skipped": 0, "total": 0}}
The status column said Tested. Not because of a subtle bug. It was because the status was derived from “did we receive a metrics object” rather than “did anything run”. An empty result is still a result. The taxonomy had no state for nothing happened, so nothing happened got rounded to fine.
The fix in both cases is the same shape, and it is not clever. Say which metric you mean, and give absence its own name:
for key in ("lines_pct", "statements_pct", "functions_pct", "branches_pct"):
val = stack.get(key)
if isinstance(val, (int, float)):
return (label, f"{val:.1f}%")
return (label, "no data")
An explicit list beats first-match-wins, because first-match-wins quietly depends on an ordering you did not choose and will not notice changing. The parser stops emitting a percentage for any category with zero items, so the bad number never reaches the JSON. And the status taxonomy gained a state it was missing, for runs that produced nothing.
It arrives at someone who cannot act on it
The third one isn’t a bug at all, which is why it survives longest.
For a while the dashboard was correct and useless. It sat there being accurate once a day, and when a repository’s coverage dropped, the sequence was: I noticed, then I went and told someone.
That errand is corrosive in a way that is hard to argue with in a meeting. It turns the person holding the signal into the person making the request, over and over, about work that is not theirs. And the role of “the one who keeps mentioning it” has no promotion path and a short half-life. The signal was fine. Its delivery address was wrong.
So regressions now open a GitHub issue assigned to the repository’s owner, read from an owners table in the config. Not to me. Not to a QA queue. To the person who would have had to do the work anyway.
No new process came with it. No meeting, no initiative, no culture conversation. The drop simply arrives as an issue with someone’s name on it, the way every other piece of work in the company does.
A signal that reaches someone who cannot act on it is not a quality mechanism. It is a complaint with a timestamp.
What this actually changed
Concretely, in the tool: one headline figure corrected from 100.0% to 48.8%, one status taxonomy that can now say nothing ran, one parser that no longer emits percentages for empty categories, and sixteen repositories that are now visibly untested instead of invisibly untested.
That last one is the smallest-sounding and the one I would keep. Nothing about those repositories changed. What changed is that the fact is now written down somewhere everyone can see, attached to a name, refreshed nightly, and impossible to answer with it depends what you mean by coverage.
Audit your own green lights
You can do this today, on whatever dashboard you already have, in about an hour. For each number on it, three questions:
- Who produces this value: a person, or the system? If a human can edit it, you are measuring the human.
- What does it render at zero, at empty, and at error? Not what it should render. Go and find out. If any of those three produce a number that looks like success, you have the bug I had.
- Who receives it when it moves, and can they fix it? If the answer is “the person who built the dashboard”, the signal has no teeth and the fix is a routing change, not a culture change.
I would start with question two, because it is the one with a definite answer and you can get it by reading code rather than by having opinions.
And when you find one, it will be boring. Mine was an uppercase letter sorting ahead of an underscore. That is the normal size of the thing standing between a team and a number it believes.
