How to spot AI hallucinations — and stop them causing problems

We’ll send a password for the resource area and regular UK-focused AI updates. Unsubscribe anytime.

What is an AI hallucination?

You’ve probably seen one without realising it.

You ask an AI tool a question. It gives you a neat, confident answer. Perhaps it even gives you a list of sources.

Everything looks fine.

Except one of the facts is wrong.

Or a company name doesn’t exist. A regulation has been described incorrectly. A date is wrong. A product specification has been invented. Or the AI has confidently told you that something happened when it never did.

That’s an AI hallucination.

The term sounds strange, but the basic idea is simple. An AI system can produce information that appears to make sense but is factually incorrect or made up. The UK’s National Cyber Security Centre specifically identifies this as a weakness of generative AI.

Research into large language models has found that hallucinations are a recognised and ongoing problem, with researchers studying both why they happen and how to detect and reduce them.

And here’s the bit that’s easy to miss.

AI doesn’t necessarily know that it’s wrong.

It can give you an incorrect answer in exactly the same confident tone it uses when giving you a correct one.

That’s why checking the answer matters.

What does an AI hallucination look like in a small business?

Let’s make this practical.

Imagine you run a small café and ask:

“Write a short description of the rules around displaying allergen information in a café.”

The AI produces a sensible-looking answer.

You copy it onto your website.

But one part of the answer is wrong or out of date.

The problem isn’t that AI tried to help. The problem is that you treated its answer as a source of truth rather than as a draft that needed checking.

Or perhaps you’re a plumber.

You ask:

“What size pipe should I use for this installation?”

The answer might sound perfectly professional. But if you’ve provided incomplete information, the AI may fill in the gaps with an assumption.

A hairdresser might ask AI about a particular hair product.

A retailer might ask for specifications for a product they sell.

A builder might ask about a building requirement.

A butcher might ask AI to write information about food storage.

A small business owner might ask:

“What are the current rules for…”

That’s when you need to slow down.

AI can be extremely useful for explaining information you’ve already found. It’s much less sensible to assume that its first answer is the authority on what the current rules are.

The ICO makes a similar distinction. The level of accuracy you need depends on what you’re using AI for. A tool helping you brainstorm a fictional story doesn’t need the same level of accuracy as one producing information that people will rely on.

Why does AI make things up?

You don’t need to understand the technical details to use AI safely.

Think of an AI chatbot as a very sophisticated system for generating language.

It has learned patterns from huge amounts of information. When you ask a question, it generates a response based on those patterns.

That can produce remarkably useful answers.

But it doesn’t mean the system has a perfect database of facts that it checks before answering every question.

This is one reason an AI system can produce a sentence that sounds exactly right even when the underlying information isn’t right.

Researchers describe hallucinations in several ways. Some involve incorrect facts about the real world. Others involve the AI failing to follow the information or instructions it was given.

There are several situations where you’re more likely to run into trouble.

1. The question is very specific

Ask about a famous person or a well-known subject and there may be lots of information available.

Ask about a tiny local business, an obscure product, a little-known event or a very recent change and the risk can increase.

For example:

“Who won the 2018 award for the best independent bakery in [small town]?”

If there isn’t reliable information available, an AI may still try to give you an answer.

2. The information has changed

AI knowledge can have limitations around recent events, new rules, new products, business closures and other changes.

Even when a tool can search the web, don’t assume that every statement it gives you has been checked against the original source.

3. Your question contains an assumption

This is a surprisingly common problem.

You ask:

“Why did Company X close its shop in Manchester?”

What if Company X never had a shop in Manchester?

The AI may accept the assumption and construct an explanation around it.

You haven’t just asked a question.

You’ve potentially put a false fact into the question.

4. There isn’t enough information

Suppose you ask:

“Can I use this type of cable for my installation?”

The AI may not know the exact installation, environment, equipment or relevant requirements.

Instead of saying “I don’t have enough information”, it may try to give you a useful-sounding answer.

5. You ask for something that sounds factual but is really creative

Try asking AI to invent ten names for a fictional bakery.

It can do that perfectly well.

But ask it for “ten famous bakeries in Britain” and you are now asking for facts.

That’s a different job.

How to spot an AI hallucination

You don’t need special software.

You need a healthy level of suspicion.

Look out for these warning signs.

Very specific facts

Be cautious when AI suddenly gives you an exact date, price, percentage, legal requirement, regulation number, product code or quotation.

Specific information can be useful.

It can also be completely wrong.

Ask:

“Where did that number come from?”

Then check it.

Made-up sources

AI can sometimes produce references that look genuine but don’t lead to the information being claimed.

A source can also be real while the AI has misunderstood what the source says.

Don’t just check that a website exists.

Open the source and see whether it actually supports the claim.

Overly confident answers

• “Yes, absolutely.”
• “This is definitely the case.”
• “The law requires…”
• “The official rule is…”

These phrases don’t prove anything.

An AI’s confidence is not evidence.

Answers that seem too convenient

Suppose you’re hoping a particular rule doesn’t apply to your business.

You ask AI and it tells you exactly what you wanted to hear.

That’s a good reason to check the answer independently.

Contradictions

Ask the same question in a slightly different way and get two different answers.

That’s a warning sign.

It doesn’t necessarily mean both answers are wrong. But it means you shouldn’t blindly rely on either.

Names, dates and numbers

These deserve special attention.

If AI tells you:

• a person held a particular job
• a business opened in a particular year
• a product costs a particular amount
• a study found a particular percentage
• a law came into force on a particular date
• a company has a particular certification

check it.

These are precisely the kinds of details that can make an otherwise excellent piece of writing misleading.

How to check an AI answer

You don’t need to fact-check every sentence you ever ask AI to write.

That would defeat the point of using it.

Instead, match the amount of checking to the risk.

For a low-risk task, a quick read may be enough.

For something important, check the facts.

Here’s a simple five-step process.

STEP 1: Identify the facts

Don’t try to check every word.

Look for statements that could cause a problem if they’re wrong.

For example:

• “The current VAT rate is…”
• “The product contains…”
• “The law requires…”
• “The manufacturer recommends…”
• “The business opened in…”
• “The study found…”

Those are claims that can be checked.

STEP 2: Go to the original source

This is one of the best habits you can develop.

If AI tells you what HMRC says, check HMRC.

If it tells you what the ICO says, check the ICO.

If it gives you a manufacturer’s specification, check the manufacturer’s website or documentation.

If it refers to a professional or trade body, check that organisation.

Don’t rely on a summary of a source when you can check the source itself.

STEP 3: Check the date

Information can change.

A page published several years ago may not reflect the current position.

This matters particularly for:

• tax
• employment
• product specifications
• software features
• pricing
• business opening hours
• industry requirements
• government guidance

STEP 4: Ask AI to show its working

You can ask:

“Which claims in your answer are you least certain about?”

Or:

“List the factual claims in your answer that I should verify.”

Or:

“Give me the source for each factual claim.”

This doesn’t guarantee that the answer will be correct.

But it can help you identify what needs checking.

STEP 5: Correct the answer yourself

If you find an error, don’t simply tell AI:

“That’s wrong.”

Give it the correct information.

For example:

“Your answer says X. The official source says Y. Rewrite the answer using Y and don’t repeat X.”

Then check the revised answer again.

AI can make the same mistake twice.

How to reduce hallucinations before they happen

You can’t guarantee that an AI will never hallucinate.

But you can reduce the chances of getting a bad answer.

Give it better information.

Instead of:

“Write a description of my business.”

Try:

“Write a 100-word description using only the information below. Don’t invent any facts.

Business name: Smith’s Bakery

Location: Leeds

Established: 1987

Products: bread, cakes and pastries

Opening hours: Monday to Saturday, 7am to 3pm

If information is missing, don’t guess.”

That last sentence is particularly useful.

Tell AI not to guess.

Try prompts such as:

• “Use only the information I’ve provided.”
• “If you don’t know, say you don’t know.”
• “Don’t invent names, dates, statistics or sources.”
• “Separate facts from suggestions.”
• “Ask me for missing information rather than making assumptions.”
• “Flag anything you’re uncertain about.”

These instructions don’t make hallucinations impossible.

They can, however, make your AI use more controlled.

The ICO’s work on generative AI also stresses that the appropriate level of accuracy depends on the purpose of the system and that users should understand the risk of incorrect or unexpected outputs.

Use AI as a drafting partner, not your final authority

This is probably the most useful mindset for a small business owner.

AI is often excellent at helping you do something with information.

For example, you give it your actual product details and ask it to:

• turn them into a product description
• simplify them
• create social media ideas
• draft an email
• produce questions for a meeting
• organise your notes
• create a first draft
• explain something in simpler language

That’s very different from asking:

• “What are the current rules?”
• “What does the law say?”
• “What’s the correct tax treatment?”
• “Is this product safe?”
• “Can I legally do this?”
• “What’s the manufacturer’s recommended specification?”

The more serious the consequence of getting the answer wrong, the more you should move away from “AI told me” and towards “AI helped me find and understand the information.”

When you should not trust AI on its own

There are some jobs where checking isn’t just a nice extra.

It’s essential.

Be especially careful with information involving:

Legal matters

AI can explain legal concepts in plain English, but don’t assume that an AI answer is a substitute for checking the actual legislation, official guidance or appropriate professional advice.

Financial and tax matters

A wrong number can cost you money.

Check information against HMRC, your accountant or another appropriate authoritative source.

Health and safety

If someone could be injured because an AI answer is wrong, don’t rely on the chatbot alone.

Technical work

For trades, installations, repairs and equipment, check the manufacturer’s instructions, relevant standards and appropriate professional guidance.

Personal information about customers or employees

This is a separate issue from hallucinations. You also need to think about privacy and how you’re allowed to use personal information.

The ICO’s guidance makes clear that accuracy matters particularly when AI is being used in ways that affect people or involve personal information.

What if AI has already given you a wrong answer?

Don’t panic.

First, stop using the incorrect information.

Then work out where it has gone.

If you spotted the mistake before publishing anything, correct your draft and move on.

If you’ve already published it on your website, change it.

If you’ve sent it to customers, consider whether you need to correct the information directly.

If it has influenced a business decision, go back and check the decision using reliable information.

The size of the response should match the size of the mistake.

A typo in a social media post isn’t the same as incorrect information about a customer’s rights or a product safety issue.

The ICO has highlighted the potential consequences of inaccurate generative AI outputs, including reputational and financial harm.

A simple AI checking rule for your business

You don’t need a complicated AI policy.

Start with three levels.

GREEN — Low risk

You can usually use AI without extensive checking for things such as:

• brainstorming
• coming up with names
• writing a first draft
• rewriting your own text
• creating social media ideas
• making a list of questions
• changing the tone of an email

You should still read what it produces before using it.

AMBER — Check before publishing

Check information involving:

• product claims
• prices
• dates
• statistics
• company information
• customer-facing information
• industry information
• anything presented as a fact

RED — Don’t rely on AI alone

Use an appropriate authoritative source or professional where the consequences of being wrong could be serious.

Examples include:

• legal decisions
• tax and financial decisions
• safety instructions
• medical information
• important employment decisions
• technical specifications where failure could cause damage or injury
• decisions about customers or employees based on personal information

This approach is much more practical than trying to ban AI from your business.

And remember, you don’t have to use AI at all.

If a task takes you five minutes to do yourself and checking an AI answer takes ten minutes, you’ve probably not saved yourself anything.

A five-minute habit that can save you a lot of trouble

Before you copy an AI answer into your website, email or business documents, ask yourself five questions:

1. What facts has AI given me?
2. Which of those facts actually matter?
3. Where did those facts come from?
4. Could the information have changed?
5. What happens if this is wrong?

If the answer to question five is “not much”, carry on after a sensible check.

If the answer is “it could cost me money, upset a customer, create a legal problem or put someone at risk”, stop and verify it properly.

That’s the habit that matters.

Not becoming an AI expert.

Just knowing when an AI answer needs a second look.