A test that passes is not the same as a test that can fail
Five tests on one branch of this site were green and could not have failed. Each one would have stayed green through the exact defect it was written to catch.
Published 2026-10-12 by Arif Hossen
Tags: Laravel, Craft, Testing
Source: https://arifhossen.dev/blog/a-test-that-passes-is-not-the-same-as-a-test-that-can-fail
Where it started
The branch that put this site's public pages online closed with the suite green. Everything passing, nothing skipped, nothing marked incomplete. That is the state where you stop looking.
I kept looking, and found five tests that could not have failed. Not weak, not happy-path only. Each one would have stayed green through the precise defect it was written to catch. Two of them had already let a defect reach a commit by the time I found them.
Every one of them claimed more than it pinned
The five
The list is short enough to read in full. The left column is what the test is called. The right column is what a mutation proved it was actually holding.
| The test |
What its name claims |
What it actually pinned |
| Public route leak loop |
No unpublished post leaks through any public route |
Nothing, for any route whose URL it failed to build. Those returned 404, and a 404 skipped every assertion in silence |
| Fixture readability guard |
The fixture strings are ones the leak check can see |
That a separate copy of those strings was readable |
| Server-rendered HTML |
The post title reaches the server-rendered page |
That the title reaches the response. It stayed green with server rendering entirely switched off |
| About page heading count |
The page has its four section headings |
That it has four headings of some kind. Deleting a whole section left the count at four |
| Cache invalidation tags |
Each action declares which caches it dirties |
That the array in the test matches the array in the code |
They are not one mistake five times. Two of them assert against a copy of the thing they mean to check. Two of them are counting, and a count is blind to substitution. One runs its assertions inside an if that the failure itself makes false. What they share is that the path from the assertion to the thing it describes has a gap in it, and the gap is quiet.
An assertion inside an if that failure makes false
The quiet one
The security test is the one I care most about. It walks the application's own route table, keeps every GET route that no authentication guards, requests each one, and asserts that an unpublished post's title, slug and dek appear nowhere in the response. Discovered rather than listed, so a route added next year joins it without anyone remembering to add it.
The leak assertions ran inside if ($response->status() === 200). A URL built with a placeholder still in it, /blog/{post} rather than /blog/the-real-slug, returns 404, and 404 is not 200, so every assertion inside the block was skipped. The test reported a pass.
I broke the substitution on purpose and ran it both ways.
- 12 Tests passing, with the loop checking every route
- 183 Assertions it ran when the URLs were built correctly
- 178 Assertions it ran when they were not, with no failure reported
Twelve passed either way. The entire difference between a loop that checked five routes and a loop that checked nothing was five assertions going missing from a total that nothing reads.
The fix is not a cleverer assertion. It is to collect the URLs the loop actually visited and assert that list contains what it should, and then to make any unfilled placeholder a failure in its own right, against a written list of the ones we know we cannot fill. There is exactly one of those today, and it is on the list because someone had to stop and decide.
A guard that checked a copy of what it guarded
The worst one
Then the fix to that one had the same defect inside it.
Inertia writes page props into the HTML as JSON, which escapes some characters and leaves others alone; the framework's own assertDontSee escapes a different set. The intersection both can see is plain ASCII without slashes or quotes. The fixture titles work, but by luck. Someone swapping in a title with a curly quote would blind the check with nothing going red.
So I added a guard asserting the fixture strings are searchable. It asserted against string literals. The maintainer this guard exists for edits the factory call, not the guard, so the guard would go on checking the old string forever. It protected a copy of the fixture rather than the fixture.
// Before: the guard and the fixture were two copies of one string.
expectSearchableFixtures(['Confidential client work']);
$post = Post::factory()->create([
'title' => 'Confidential client work',
]);
// After: one definition, shared by the guard, the fixture and the assertions.
$title = 'Confidential client work';
expectSearchableFixtures([$title]);
$post = Post::factory()->create(['title' => $title]);
tests/Feature/Public/SecurityBoundaryTest.php
Editing only the title now fails at the guard, before the fixture is even created. With the old shape, the identical edit passed twelve of twelve with the assertion count unchanged.
Break the code and watch the test
How you find them
You cannot find these by reading. I know because I tried. Four of the five came out of a mutation: break one thing on purpose, run the test, check it goes red and check it goes red for the right reason, put it back. The fifth came out of reading, and only because I had just been burned by the fourth and went looking for the same shape on purpose.
A mutation is cheap. One line, one run, one revert. What it buys is the difference between "this test passes" and "this test can fail", and only the second is a claim worth making.
[!CAUTION]
Two of these had already let a defect reach a commit. The cache-tag one was not theoretical: renaming a tag left the blog index serving a row labelled with the old name, next to a filter chip showing the new one. One page disagreeing with itself, sitting on the branch until a review found it.
The cheapest check I was not running
What it cost
Every one of these was paid for at the worst moment: after the test was written, after it was reviewed, after it had been green long enough to be trusted.
The rule I have written down is narrow enough to actually follow. When a test is about a boundary (security, cache invalidation, anything whose failure mode is silence), break the thing it guards before calling it done. Not the whole suite. Not as a ceremony. Once, at the moment you write the assertion, while you still remember what you meant it to catch.
I did not do that consistently on this branch, and it cost a review round every time.