When AI gets it wrong: a six-step routine for checking what a chatbot tells you

In a judgment handed down in June 2025, the High Court in London described a schedule of 45 legal citations that had been put before the court. In eighteen instances, the judgment records, “the case cited does not exist”. The claimant, who had done the legal research himself, told the court the citations were generated using publicly available AI tools, legal search engines and online sources. His solicitor relied on that research without independently verifying it.

That is the problem in one picture. Eighteen of the cases did not exist, and many of the real ones did not say what they were cited for. Nobody opened them to look.

Why a confident answer can be false

An AI assistant does not look facts up in the way a search engine does. OpenAI’s help page says ChatGPT answers “based on patterns in data it was trained on”, and that it searches the web only when its search tools are enabled, which OpenAI says they are by default. Without a search, it writes the most likely-sounding next words. Usually those are true. Sometimes they only sound true.

The trade calls this a “hallucination”, meaning an answer that is invented but delivered as fact. Plain words do the job better: the assistant made it up.

The makers do not hide this. In alphabetical order:

• ChatGPT. OpenAI: “Sometimes, it might sound confident—even when it’s wrong.” Its list of typical errors includes “fabricated quotes, studies, citations or references to non-existent sources”.
• Claude. Anthropic: “Claude can write things that might look correct but are very mistaken.”
• Copilot. Microsoft: “A response can sound polished and still be wrong, incomplete, or risky to use.”
• Gemini. Google says such tools “can hallucinate and present inaccurate information as factual”.

Independent testing agrees. In a 2025 study coordinated by the European Broadcasting Union and led by the BBC, journalists at 22 public broadcasters in 18 countries assessed more than 3,000 answers to questions about the news from ChatGPT, Copilot, Gemini and Perplexity. They found that 45% of answers had at least one significant issue, and 20% had major accuracy problems, including invented details and out-of-date information. That was news questions only, a hard test, but it is a fair warning.

When it matters, and when it doesn’t

Most of what people ask an assistant needs no checking at all. Ideas for a toast, a packing list, a kinder way to word a letter to the neighbours: if you like the result, it is right.

Checking is for answers you will act on or repeat. That means:

• health
• money, tax and benefits
• law and your rights
• dates, deadlines and opening hours
• anything you will pass on as fact
• anything about a named person

Google’s own page says not to rely on Gemini for medical, legal or financial help, and the same caution applies to all of them. Nothing here is medical, legal or financial advice either; for your own situation, check with your GP (family doctor), a solicitor or lawyer, or a regulated adviser.

The six-step check

1. Ask where it comes from, then open the link yourself. Type “What is your source for that?” Invented references exist, as the High Court found, so a link is not proof. The page has to say the thing. Anthropic makes the same point: the original site may hold context the summary left out.
2. Check one key fact against a primary source. Gov.uk for UK rules, the NHS (the UK health service) for health, the IRS or Medicare in the US, or the organisation’s own website. One confirmed fact tells you a lot about the rest.
3. Ask again, differently. Rephrase the question, or put it to a second assistant. If the answers disagree, treat that as a flag and go to the primary source.
4. Ask “What might be wrong with that answer?” It costs one line and may prompt a caveat the first answer left out. Microsoft notes that many errors are really omissions: the reply reads as complete, and the missing context is what misleads.
5. Ask what date its information runs to. OpenAI says its models are trained on data up to a certain point and know nothing later unless a search tool is used. Tax thresholds, prices and rules change every year.
6. Redo any sum involving money. Use a calculator or a spreadsheet. Never let an assistant’s arithmetic be the only copy.

Four signs an answer deserves suspicion

• A very precise figure with no source attached.
• A quotation. Word-for-word quotes are easy to invent; the High Court noted that AI tools may quote passages that do not appear in the genuine source.
• A legal or medical specific: a section number, a dose, a time limit.
• A name, case or reference you cannot find anywhere else.

A useful way to picture it

As an analogy only: imagine a helper who has read an enormous amount, works fast, and sounds equally sure whether right or wrong. You would be glad of the help. You would also check the important things before signing anything.

That is the right amount of trust. It is neither a reason to avoid these tools nor a reason to believe them.

This week

• Test an assistant on a subject you know thoroughly. Seeing its mistakes on home ground is the quickest way to calibrate.
• Use step 1 once: ask for the source, open it, and see whether it says what was claimed.
• Write “health, money, law, dates, people” on a sticky note by the screen as your list of things to check.

For the wider picture of what these tools are good for, start with AI after 50: the complete guide. And for the highest-stakes use of all, see using AI for health questions before an appointment.