For about a day I believed I had roughly 150 broken tests. I was wrong — not one of them was broken. Then I was wrong about the cause. Twice.
Here’s the walk from “our test suite is rotting” to “one stray line in my shell was winning a fight I didn’t know it was in.”
The symptom
I ran the suite and about half of it was red. Every failure was the same shape: 403 org-access-denied. Tenant-scoped tests, denied across the board.
It looked exactly like accumulated test debt — the kind of thing you file, sigh at, and schedule for “later.” So that’s what I did. I opened an issue: ~150 failing tests, org access denied.
Correction #1: the tests were fine, my machine wasn’t
Before scheduling the cleanup, I ran the suite in a clean environment — a fresh shell with none of my profile loaded.
All green.
Not “fewer failures.” Zero. The tests were fine. The failures were a property of my environment, not the code. Which is a worse feeling than a broken test, honestly, because it means the call is coming from inside the house.
The mechanism turned out to be a PHPUnit detail I’d never had to think about: env vars you set in the config do not override an env var that’s already set in your shell — not unless you mark them force="true". My shell profile exported a variable that the test config also set, and by default, the shell won. So my tests were quietly running against the wrong context, and everything tenant-scoped got denied.
Correction #2: I blamed the scary variable, not the guilty one
Here’s the part I’m least proud of and find most useful.
My first theory for which variable was a JWT signing secret. It’s the dangerous-sounding one; of course a bad secret breaks auth. I wrote it up that way.
It was wrong, and the code says why. A signing secret is symmetric: if signing and verifying both read the same (wrong) value, the tokens are still internally valid. A polluted secret doesn’t produce 403 org-access-denied — it produces valid tokens for a wrong-but-consistent world. It couldn’t be the cause.
The actual culprit was boring: a tenant-slug variable. My shell had it set to one thing; the tests minted tokens for another. So the request would resolve one tenant from the env, carry a token for a different tenant, and the access check — correctly — denied it. Every tenant-scoped test, 403.
I’d spent my first guess on the variable that sounded like a security problem, when a plain identifier was doing all the damage.
The fix was one attribute
force="true" on the env entries, so the test configuration wins over whatever the shell happens to export. That’s it. The suite is now hermetic: it runs the same on my machine, on a colleague’s, and in CI, regardless of anyone’s shell profile.
I proved it the honest way — by injecting the bad variable on purpose. With the injection and no force, the suite collapses (about 150 failures and a pile of errors). With force, the same injection does nothing; everything stays green. The fix is verified against the attack, not against my memory.
What I can’t tell you
I can’t tell you the exact line that was in my shell profile that day. I don’t have it pinned down, and I’m not going to invent it to make the story cleaner. What I have is a reproduction — inject the variable, watch it break — and that’s what the fix is proven against.
The count is fuzzy too. It was around 150 failures out of somewhere near 290 tests, and both numbers drifted day to day as the suite changed. “156 out of 291” would look precise and be a small lie. Half the suite red, zero tests actually broken is the true and useful shape of it.
Two takeaways
Make your tests hermetic. Test configuration should beat the machine it runs on, unconditionally — force, or the equivalent in your stack. A suite whose result depends on the developer’s shell isn’t testing your code; it’s testing your dotfiles.
When you’re debugging, suspect the boring variable. I lost real time pointing at a secret because it sounded dangerous, while a plain tenant id sat there quietly denying everything. The scary-looking cause is a great way to feel productive and stay wrong.
What’s the worst “it’s the tests” you’ve had that turned out to be your own environment?
── Hideyuki Mori (Ayane International) 🔗 hideyuki-mori.com
Sumber Rujukan:
- Artikel asal dari Dev.to PHP
- Published on haqis.com