Why does AI make things up, and how do you stop it?
A model handed something unusable does not stop. It produces something plausible, and it will produce a different plausible thing the next time you ask.
AI makes things up when you hand it an input that cannot answer the question, and the fix is almost never a better model. It is a check that refuses to run. A model given something unusable does not stop and tell you, it produces something plausible instead, and it will produce a different plausible thing the next time you ask. The most useful thing you can build into an automation is the ability to say it cannot do the job.
We hit this in our own systems this week, and the fix was the opposite of what everyone assumes.
Why does a model invent an answer instead of failing?
Because inventing is what it does. You are asking a system whose entire purpose is to produce a likely output. When the input is rich, the likely output is the right one. When the input is thin or unreadable, there is still a likely output, and you get it, delivered with exactly the same confidence as a correct one.
That is what makes it expensive rather than annoying. There is no visible difference between a summary of a real conversation and a summary of no conversation at all. Both come back formatted, calm and specific, and one of them is a record you will trust six months later when you can no longer remember the call.
What does a refusal actually look like?
We run a process that turns recorded conversations into written notes without anyone typing them up. A small run of recordings kept coming back as failures, and the assumption for weeks was that the transcription step had a backlog.
It did not. When we finally measured the audio rather than arguing about the queue, the failing recordings were far too quiet to contain speech that anything could read. So we built the thing that feels like giving up: a check that measures the sound level before transcription runs, and stops if it falls below a floor.
That check is now the most valuable part of the process, because of what happens without it. Below that floor the model does not hand back an empty file. It hands back a conversation. A plausible one, with plausible speakers saying plausible things. Run the same file twice and you get two different conversations. Nothing on either page tells you which one, if either, actually happened.
A refusal gives you a true answer you can act on. A guess gives you a document that reads perfectly, is entirely fictional, and gets filed somewhere it will be believed.
How do you tell a broken tool from a broken input?
Count the failures before you fix them.
When we sorted that pile rather than reacting to the size of it, most of the entries were not failures at all. They were recordings a second or two long, made by a device pressed accidentally in somebody's pocket. There was never anything on them to transcribe.
So what looked like one software problem was two unrelated things. Noise that never needed to work, and a small number of genuine jobs failing because the input was unusable. No amount of work on the automation would have fixed either one, because neither was in the automation.
Telling a broken tool apart from a broken input
| What you are seeing | What it usually is | What actually fixes it |
|---|---|---|
| The same job fails again and again | The input is unusable, not the tool | Measure the input and refuse below a floor |
| A large pile of failures appeared | Most of it was never real work | Count and sort the pile before you change anything |
| The output reads well but is wrong | The model filled a gap you did not know was there | Make it report what it could not read, not just what it produced |
| It works some days and not others | Input quality varies and nothing measures it | Fix the capture, which sits upstream of the automation |
| You are told a step cannot be automated | Your own notes say that, and they may be out of date | Check what the system can do today, not what you wrote down last year |
What should you check before you blame the software?
Five things to check first
- 1Measure the input, not just the output. Most bad output is a rubbish-in problem that nothing was checking for. If nothing measures what goes in, nothing can tell you why what came out is wrong.
- 2Count the failures before you fix them. A pile is not a diagnosis. Sort it and you will usually find several different problems wearing the same jacket.
- 3Give the system permission to refuse. An automation that stops and says it cannot do this one is worth more than one that always returns something. Always returning something is the failure mode, not the feature.
- 4Make it tell you what it could not read. There is a big difference between nothing happened and I could not use what you gave me. The second one tells you where to go and fix it.
- 5Look upstream before you rebuild anything. Ours was not a software fix. It was how the recording was being made, which no change to the automation could ever have reached.
The part that is uncomfortable
There is a second finding here and it is less flattering. The tool that fixed the real failures had been sitting in our own system, unused, the entire time. It was never called, because a line in our own documentation said the step was manual and had never been updated after somebody built the automated version.
So for weeks the process reported that it could not do a thing it had been able to do all along. Nothing threw an error. The notes were simply older than the system they described, and the notes are what everyone acts on.
That is the pattern worth taking away. The failures that cost you time are rarely dramatic. They are a confident answer nobody checked, a pile nobody counted, and a document that quietly stopped being true while everybody kept following it.
Questions people ask
Would a better or more expensive model fix this?
No, and this is the most common wrong turn. A stronger model given unreadable input produces a more convincing invention, not a refusal. The check has to live outside the model, in the step that decides whether to call it at all.
How do I tell if something is inventing rather than reporting?
Run the same input through it twice and compare. Real extraction gives the same answer both times. Invention gives you two answers that are each individually believable, and that is the tell.
Does a system that refuses just create more manual work for me?
It creates less, because it puts the work where it belongs. Being told this one is unusable takes a moment. Discovering months later that a decision was made on a fabricated record takes far longer, and sometimes it cannot be undone.
What do I do with the ones it refuses?
Fix the capture, not the automation. In our case that was how and where recordings were being made, a habit change rather than a software change. Cheap to fix, and impossible to find while everyone assumed the tool was at fault.
If you rely on something that summarises, extracts or writes on your behalf and have never checked what it does with a bad input, that is worth half an hour before it writes something you act on.
More notes
- What breaks when you automate a process nobody is watching?Automation rarely fails loudly. It fails quietly, and keeps reporting that it is fine.
- Why did your cold outreach run out of leads?Usually it is not the message. You have simply run out of people to send it to.
- Do you need to hire a salesperson, or a system?The honest answer depends on one thing, and it is not budget.
- What do you do about sales when you are already at full capacity?The trap is not that you are too busy. It is that being busy feels like being safe.
Want the version of this for your business?
Twenty minutes, your actual situation, a straight answer about what we would do first.
20 minutes. Straight answer, no pitch.