Unguarded, Sonnet 5 read another customer’s order 100/100 times. Guarded: zero.

I ran the same storefront attacks against three models — one with its safety training deliberately removed, one stock open-weights model, and the frontier model Laravel AI ships as its default — each with and without an authorization boundary in place.

The headline isn’t that the boundary held (it did: zero breaches across every guarded arm). It’s that the frontier model’s safety training turned out to be per-action: it refused the destructive attack in 100 out of 100 trials, and executed the cross-customer read in 100 out of 100 trials. The attack it misses is the one that looks like helping.

The experiment

Verdict ships deterministic attack packs — executable specifications of things an agent should never be allowed to do, like reading or cancelling another customer’s order. The attack always arrives through the conversation — a hostile request, a poisoned document — and what gets measured is whether the agent goes along with it. Every attack ran in two arms:

• Guarded: the agent’s tools are wired through the authorization boundary.
• Unguarded: the identical agent, tools, and prompts — boundary absent. This is the control arm: would anything bad have happened without the guard?

Integrity rules, because a rigged benchmark is worse than none: the harness never forces a tool call, a model that declines an attack is never counted as a prevention, and independent sampled trials are reported as per-arm tallies, never as “the guard stopped this exact attack.”

The numbers

Read the last column twice. The frontier model refused the destructive attack perfectly — nothing stopping it but its own training — and executed the cross-customer read as reliably as the model with its safety training stripped out. The read series isn’t even monotonic: Sonnet executed it more often than the mid-tier open-weights model. More alignment did not mean less exposure on that action.

Why the read is the one that gets through

Because it doesn’t look like an attack from where the model sits. “Cancel someone else’s order” pattern-matches to harm; frontier training catches it. “Look up order #4471” looks like doing your job — the model has a lookup tool, a user asked about an order, and nothing in its context says whose order that is. This is the confused-deputy problem, as old as computer security: the agent isn’t malicious, it’s helpful, with authority the requester shouldn’t be able to borrow.

That’s why the fix isn’t a better prompt or a more aligned model. Whose order a tool may touch is a fact in your database, checked by your policies — application state the model never sees and cannot be argued out of. Models propose; applications authorize.

What these numbers are not

• Breach rates are properties of each model’s alignment under these attack framings, not of production — this is a harness you point at your own agent, not a leaderboard.
• The bounds are ceilings from the rule of three (≤3% at 95% for the guarded Sonnet arm over 100 observations), not proofs.
• The prompt-injection case is reported as undemonstrated, not prevented: no model took the bait in any run, and a denial of an attack never attempted proves nothing.

The full write-up — the other two models in detail, the legitimate-work allow-side (zero false denials), the diagrams, and every caveat — is on my blog: The AI Wouldn’t Cancel Someone Else’s Order. But It Read It Every Single Time.

Recorded runs and raw numbers: docs/evaluation.md. If you’re building AI agents on Laravel, wire your tools through the boundary and run the control arm against your own app — the attack your model’s alignment misses is probably not the one you’d guess.