Calibrating Trust: How Much Checking Is Enough?
Module 1 · Why Wrong Sounds Right

Calibrating Trust: How Much Checking Is Enough?

Match your level of scrutiny to the stakes, how checkable the output is, and how far it will travel, so you verify what matters without re-doing everything AI gives you.

Two Ways to Get Discernment Wrong

There are two failure modes, and both are common:

  • Under-checking: trusting output because it looks finished. This is how fabricated citations end up in court filings and wrong figures end up in board decks.
  • Over-checking: verifying every word of every answer until AI saves you no time at all, then quietly giving up on checking entirely because it’s exhausting.

Good discernment is calibrated. You check harder where an error would hurt more, and you check smarter where errors are easy to spot.


Three Questions That Set the Level

Before you decide how to review a piece of AI output, ask:

  1. Stakes: What happens if this is wrong? Mild embarrassment, wasted time, money lost, someone harmed, a legal or safety problem?
  2. Checkability: How easily could I tell if it’s wrong? Can I test it, compare it to a source, or recognize errors from my own expertise?
  3. Reach: Where will this go? Just me, my team, a client, the public, a permanent record?
SituationStakesCheckabilityReachReview level
Brainstormed names for a team eventLowEasy (you’ll just pick one)TeamGlance: does it fit?
Summary of a meeting you attendedMediumEasy (you were there)TeamSkim and spot-check key decisions and owners
Explanation of a tax rule for your small businessHighHard (you’re not an expert)You, then the tax officeVerify every claim against an official source or a professional
Statistics in a public reportHighMediumPublic, permanentVerify every figure at its original source

Notice the third row. When stakes are high and you can’t easily check, AI’s answer is a starting point for finding the real answer, not the answer itself.


The Review Ladder

  1. 1 Glance

    Does it fit the purpose? Anything obviously off?

  2. 2 Spot-check

    Test a few specifics you can confirm quickly

  3. 3 Trace

    Follow each key claim back to where it came from

  4. 4 Verify

    Confirm every specific against independent sources

  5. 5 Expert review

    Someone qualified signs off before it's used

Climb the ladder as stakes rise and checkability falls. Two practical rules:

  • Spot-checks reveal the error rate. If you check three specifics and one is wrong, that tells you something about the rest. Move up a rung.
  • The rung is set by the worst-case use. A draft that “might” go to a client is reviewed as if it will.
⚠️ Common Pitfall: Letting Your Own Expertise Set the Bar

Common mistake: reviewing AI output on a topic you know well and then trusting it to the same degree on a topic you don’t.

In your own field, errors jump out at you. Outside it, the same rate of errors is invisible. If the AI got 1 in 10 details wrong in your specialty, assume something similar elsewhere, and check accordingly. Your sense of “this seems fine” is only as good as your knowledge of the topic.

🧪 Try It: Calibrate Your Last Week of AI Use
  1. List five things you (or someone you know) used AI for in the last week or two. If you haven’t used AI much, ask an assistant: “List ten common ways office workers use AI assistants.” and pick five.
  2. For each one, rate Stakes, Checkability, and Reach as low, medium, or high.
  3. Assign each a rung on the review ladder.
  4. Compare with how much checking actually happened. Circle any where the real review was lower than the rung you assigned.
  5. Pick one circled item. Ask an AI assistant:
    Here's a task I used AI for: [describe it]. What are the most
    likely kinds of errors in AI output for this task, and how could
    I check for each one in under five minutes?
    Judge its suggestions. Which ones are realistic for you?
🧠 Knowledge Check

1. AI explains a medication interaction for an elderly relative. You have no medical background. What's the right approach?

2. You spot-check four figures in an AI summary and one is wrong. What should you do?

3. Which factor makes an AI-written draft need more review, even if the content seems low risk?


Up Next

Module 2 moves to product discernment: the practical techniques for checking facts, citations, and numbers against sources outside the AI.