A system that crashes gets noticed immediately. Someone opens a ticket, a dashboard goes red, an owner gets paged, and within a week there's a decision on record: fix it or kill it. A system that's wrong in a way that still looks right doesn't generate any of that. It just keeps running, and the decision to keep it never actually gets made. It just never comes up.
This is the failure mode particular to AI systems that most conversations about AI risk miss, because those conversations are built around the assumption that bad systems get caught. Loud failure is self-correcting: it forces someone to look. Plausible mediocrity is not self-correcting, because nothing about it demands attention. The two failure modes can cost the same in the end even if only one of them reliably gets fixed.
A customer-service operation routes incoming tickets to the correct queue using a model that gets it right roughly 80% of the time. Eighty percent was good enough that queue owners stopped spot-checking the output after the second month, because any ticket they happened to sample looked reasonable, the kind of judgment call a person would plausibly make. The 20% it got wrong didn't fail visibly. It filed a payments dispute under general inquiry and a churn-risk complaint under a billing question, both plausible on their face, both wrong, both undiscovered until a customer escalated three weeks later and someone traced the misrouting back to its source.
Compare that to a version of the same system that simply crashed on a fifth of incoming tickets: timed out, threw an error, returned nothing. Nobody tolerates a visible failure rate that high. It gets pulled within a week and rebuilt, because the failure announces itself every time it happens. The plausible version of the same 20% error rate ran for fourteen months. Same accuracy, wildly different lifespan, and the only variable that changed is whether the wrong answer looked like one.
That's a different mechanism from the one I described in Pattern #6 where projects survive because nobody is rewarded for deleting a step someone else built. Nobody here was protecting the triage system out of habit or self-interest, and nobody was choosing to look away from a bad metric. The system never generated a signal that required anyone to choose anything, which is why the decision never got made, by anyone, for months. An incentive problem needs a person declining to act. A detection problem doesn't need anyone to decline. There's simply nothing in front of them to act on.
The uncomfortable implication is that AI systems are, on average, worse at dying than the systems they replace. A spreadsheet macro that breaks throws a visible error. A rules engine that misfires does something a human immediately recognizes as wrong. A language model that's confidently incorrect produces output indistinguishable, on casual inspection, from output that's correct, which is precisely the property that makes it useful for fuzzy work and precisely the property that lets a mediocre deployment outlive a competent one that happened to fail more visibly.
Raising accuracy from 80% to 90% doesn't fix this. The remaining wrong answers still won't look wrong, and that disguise is the actual problem, independent of the error rate. What helps is treating detectability as a design requirement from the start: a check that flags the misrouted ticket in week one, on its own, rather than waiting for a customer to do that job by escalating in week twelve.
That's the tenth pattern.