Home / GenAI Tutorial / Day 11
Day 11 — Verifying the output
The single skill that separates people who use AI safely from people who get caught out by it.
Confidence is not a signal
A model has no internal sense of certainty it can report to you. The invented function and the correct one arrive in exactly the same tone, with the same fluency. You cannot tell them apart by reading.
What this rules out
It rules out "it sounded sure, so I used it". The only reliable check is external: run it, compare it against a known number, or look it up. There is no shortcut, and anyone who tells you they can sense when it is wrong is guessing.
The four things it invents most
| Invention | How to catch it |
|---|---|
| Functions that do not exist | Run it. #NAME? in Excel, an error in DAX |
| Columns you never mentioned | Read the code against your actual schema |
| Sources, citations, page numbers | Search for them. They are often fabricated entirely |
| Arithmetic on numbers you pasted | Recompute it in Excel or SQL. Never accept its sums |
A verification workflow
1. Does it run at all? (syntax check)
2. Does it use only my real columns? (schema check)
3. Does it match a number I know? (spot check)
4. Does it handle the edge cases? (blank, zero, NULL, one row)
5. Would I be comfortable defending it in a meeting?
Step 3 is the one that catches real errors
Before running an AI query on the full table, run it on a period where you already know the answer - last month’s closed figure, a total from an existing report. If it reproduces a number you trust, the logic is probably sound. If it does not, you have caught it before anyone else did.
Asking it to check itself
Models are better at criticism than at avoiding mistakes, so a second pass genuinely helps - but it is a supplement to checking, not a replacement.
A self-check is not proof
It can miss things, and it can confidently declare correct code broken. Treat the list as a set of things to look at, not a verdict.
Building verification into the prompt
"State your assumptions" is the highest-value clause in this course
It surfaces the guesses the model would otherwise make silently. Half the time the assumptions list contains something you disagree with - and that is a bug caught before it reached your report.
When it says something confidently wrong
Do not argue about whether it is wrong. Give it the evidence and ask again:
That returned #NAME? in Excel 2019.
TEXTSPLIT is not available in that version.
Rewrite it using only functions available in Excel 2019.
Stating the observed behaviour works far better than "that is wrong". You are supplying the fact it lacked.
The professional standard
You own the output
If an AI-written formula puts a wrong number in a board pack, "the AI wrote it" is not a defence anyone will accept - nor should they. The work goes out under your name. Verification is not an optional extra step; it is the part of the job that stayed with you.
Try these yourself
- List the four things AI invents most often.
- Describe your spot-check for a query before trusting it.
- Write a prompt that asks for assumptions to be stated.
- Explain why "that is wrong" works worse than describing the error.
- Explain why confidence in the answer tells you nothing.