The most common response I got to the last post was some version of "fine, but our use case is genuinely fuzzy, so we're on safe ground". Document review, research synthesis, customer correspondence, exception handling. The stuff LLMs are actually good at.
Fair enough. But the fuzzy use cases have their own way of dying, and it tends to happen at the exact moment somebody senior has to put their name on the output.
Here is the problem in one sentence. A language model doesn't compute an answer, it samples one. The same input can give you a right answer on Tuesday and a wrong one on Thursday, and nothing in the output tells you which one you're holding. The wrong answer arrives with the same fluency, the same structure, the same steady tone as the right one. There is no tremor in the voice.
Which means you cannot verify a sample by asking the sampler. Asking the model "are you sure?" queries the same distribution that produced the error in the first place, and it will confirm its own hallucination in the voice it uses for the truth. The more sophisticated version of that ritual is no better. An LLM grading another LLM's output is two systems with correlated blind spots agreeing with each other, which produces a green dashboard and a wrong number in the board pack, and the green dashboard is arguably worse than having no dashboard at all.
I've now watched several pilots climb to something like 95% accuracy and then stop dead. 95% is a fine number. What kills the pilot is that nobody can say which 95%. A finance team can live with a process that's wrong 5% of the time if they know which 5% to look at. They cannot live with one that's wrong 5% of the time at random, because the only rational response is to check everything, and once you're checking everything you've bought an expensive way of doing the work twice.
The fix is the thing I've started calling the harness, meaning whatever sits around the model and makes its output trustworthy enough to act on. It works by one mechanism and one only, which is that a real check does not share the generator's failure mode. If your check can fail the same way your generator failed, you don't have a check, you have a ceremony. That single test throws out most of what currently gets sold as AI quality assurance.
What survives is unglamorous, which is probably why it doesn't feature in demos. Deterministic recomputation, meaning actual code: a model extracts fifteen line items from an invoice and code adds them up and compares the total to the one printed on the page. The adder cannot hallucinate in sympathy with the extractor, because it isn't a model. Or a ground truth anchor, some fact the AI didn't produce, like a closed ledger or a physical limit or a number that already exists in a system somewhere. Or the same claim recovered from a document written by a different party for a different purpose. What all of these have in common is that they're cheap, they're boring, and they're almost entirely software engineering rather than AI work.
The point of building this is to make human attention scarce and well aimed. Review everything, and you've saved nothing — you've just added a step. Review nothing, and the risk is unbounded, the kind you hear about from a customer or a regulator. The harness changes that arithmetic. Review the 3% it couldn't clear, with the suspect field already highlighted, and the economics finally work. The harness decides what a human looks at. That's the sentence I'd put in the business case, because it's the one that turns a research project into an operating decision.
Once you look at it this way the budget question gets uncomfortable. Enterprises are spending the overwhelming share of their AI money on intelligence. That's the commodity in this story: sold by four or five companies, improving every few months whether you invest in it or not. Almost nobody is funding verification. That's the scarce part. It's specific to your data and your tolerance for being wrong, and nobody can build it for you generically, because it depends entirely on what you already know to be true.
The awkward conclusion, if you're the one selling AI, is that the interesting engineering has moved. Model choice still matters at the margins, and it matters far less than whether you can tell a good output from a bad one at three in the morning without a person reading it. I've seen a well harnessed setup on a mid sized open model comfortably beat an unharnessed frontier model on the only measure that counts in production, which is how much human time it burns per thousand documents. The model is procurement now. The harness is the build. And the harness is where your company's real knowledge about when an answer is wrong ends up living, which is a topic I'll come back to.
That's the second pattern.