There have been times when I looked at a result saying every test had passed and nearly told someone it was done. It was not that the AI had fed me a hallucination or a lie. The tests ran exactly as written, and I was only going to report what I saw. The problem was that I had not checked what that passing result was actually verifying.
It happened while I was building something for an important client. Whenever I build something, I tend to build a way of checking it alongside, because no one can re-read everything from scratch by eye every time. Yet the more checks I built, the less I asked what those checks actually guaranteed. When the board is full of green lights, everything looks fine.
The first was a set of tests running on the wrong data.
Hundreds of tests were passing every day. Then, for the first time, I ran them on data shaped like the data that would really be used, and eight broke. The only thing that had changed that day was the data going in, not the program. It turned out the test data the checks had been running on was a different world from the real one. The number of rules differed, the make-up of the items differed, and it still carried the mess left over from earlier testing.
So those hundreds of passes had never once proven anything about the real situation. They had only proven that things ran well on test data. The value of a test comes less from how many run than from what they check, and the same test checks something different when you feed it different data.
Since then, before I say something passed, I first confirm which data it ran on. If the answer is test data, I write down that all it confirmed is that it runs on test data. It sounds like the same pass, but written down it says something quite different.
The second was a check that caught nothing and kept reporting no problems.
I built a checker for common mistakes on screens and ran it, and it found three. Plausible enough. But when I built a separate practice screen with mistakes planted on purpose and ran the same checker, one of its six stages caught none of them. Looking closer, that stage only recognised one kind of quotation mark. The client’s code happened to use that kind, which is the only reason it had caught anything so far. Given code written with different habits, that stage would have spun uselessly and reported no problems.
I think a check that catches nothing is worse than no check at all, because it gives you the comfort of having run it. And from the outside there is no way to tell a check that is quiet because all is well from one that is quiet because it is not running. So when I take the same checker to another company’s work, I run it on the practice screen first. Every company writes code with different habits, and there is no knowing in advance where the same rule will quietly spin.
The third was the tool that runs the tests turning failures into successes.
There was a tool that ran many tests in one go, and it was set up so that even if one of them threw an error midway, it signalled at the end that everything had run through fine. So several changes went through broken, with a full pass. Even this did not come to light thanks to the tests. I had put in a deliberately broken file to try out something else entirely, and it passed.
Around then a broken change had in fact already slipped through, and what first caught it was not a test but a single question: "If it fails, shouldn’t it be fixed?" Following that question down, I found four changes in all that had gone through broken. I had carefully tested the answer sheet and never once looked at the exam hall itself. Looking back, there was no reason to suspect that tool. It always passed. That there was no reason to suspect it was itself the signal, but at the time it did not look that way. It was cognitive bias.
Put the three scenes side by side and the same shape appears. A pass really means that under these conditions this was checked and came out fine, yet every time I dropped the conditions and read only the fine. And that the conditions were wrong only came out once I saw a check fail. If you have never once seen a check fail, its pass guarantees nothing.
So now, whenever I build or change a test, I also write one line that breaks it on purpose. I check that it catches every planted problem, and that nothing comes out of something sound. In effect I confirm failure before I trust success. The same goes for the moment I write that something is done. Before I record it as done, I run it once more on the spot and use that result as the evidence.
Once the word "passed" is written in a report, almost no one goes back to ask what that pass verified. I didn’t either. This is the builder’s side of the story, but I think it can be the same for whoever receives the report. If your company has no dedicated IT staff, you will often hear from whoever you hired that all the tests passed. Before I tell anyone my own work has passed, I put these four questions to myself first.
One. Which data was that test run on? If it was test data, it also has to be confirmed that it is shaped like the data you will really use.
Two. Have you ever seen that test fail? What matters is whether anyone checked, even once, that it really catches something wrong put in on purpose.
Three. When it says no problems, how can you know the test really ran to the end? Some stop midway and still report a pass.
Four. Did the client run it once themselves? Seeing it run again in front of you is far more trustworthy than "done" written in a record.
Of course I cannot promise these four catch everything. Checking the checks has to stop somewhere too, and where to draw that line I still decide differently from job to job.
What I really learned that day came down to one question: how do you define what counts as a successful test?