Judging the Product, the Process, and the Performance
Evaluate AI work on three levels: is the output accurate and fit for purpose, was the reasoning behind it sound, and did the AI behave the way you needed it to?
Fluent Isn’t the Same as Right
AI output almost always sounds good: well organized, confident, grammatically clean. That’s exactly why discernment matters. Polish is not evidence of accuracy. Discernment is the skill of assessing AI’s work as carefully as you would a new colleague’s, and it mirrors the three types of description:
| Type | Question | Red flags |
|---|---|---|
| Product discernment | Is the output accurate, complete, and fit for its purpose? | Unverifiable specifics, missing pieces, wrong audience, generic filler |
| Process discernment | Was the reasoning sound? | Logical leaps, ignored sources, unstated assumptions, steps that don’t connect |
| Performance discernment | Did the AI behave the way I needed? | Agreeing with everything, not asking when it should have, overconfidence, drifting from instructions |
Product Discernment: Check the Output
Run through these checks, scaled to the stakes:
- Accuracy. Are the facts, figures, names, dates, quotes, and citations real? Verify specifics against a source you trust: the original document, an official site, or an expert. Be most suspicious of precise-looking details such as exact percentages, page numbers, and quotations.
- Completeness. Did it cover everything you asked for? AI often quietly drops a requirement, especially from long instructions.
- Fit. Is it right for this audience and purpose? A technically correct answer at the wrong reading level is still the wrong answer.
- Substance. Is there real content, or just confident-sounding generalities? Try summarizing the key point in one sentence. If you can’t, there may not be one.
Process Discernment: Check the Reasoning
When AI shows its work, or when you ask it to, look at how it got there:
- Does each step actually follow from the previous one?
- Did it use the sources you specified, or drift to general knowledge?
- What did it assume? Are those assumptions true for your situation?
- If it compared options, did it apply the same criteria to each?
A right answer reached by bad reasoning is a warning sign. It won’t stay right when the situation changes slightly.
Performance Discernment: Check the Behavior
Step back from the content and ask how the collaboration went:
- Too agreeable? AI assistants tend to go along with your framing. If it praised every idea you had, ask it directly for the weaknesses.
- Asked when it should have? If key information was missing, did it flag the gap or quietly fill it?
- Honest about uncertainty? Did it signal low confidence where appropriate, or present guesses as facts?
- Followed your performance description? If you asked for hints and got answers, it drifted.
Common mistake: asking “Are you sure?” or “Is that accurate?” and treating “Yes” as verification.
AI may confidently confirm an error, or abandon a correct answer just because you questioned it. Neither tells you the truth. Asking for sources, reasoning, or counter-arguments can be useful input to your judgment, but real verification means checking against something outside the conversation.
- Ask an AI assistant a question in an area you know well, ideally one with specifics, for example:
Explain the main rules for [a process you know well at work or in a hobby], including any key numbers, deadlines, or names. - Product: Mark every specific claim as ✅ correct, ❌ wrong, or ❓ unsure. Count each.
- Process: Ask “Walk me through how you arrived at that, and what you assumed.” Does the reasoning hold up?
- Performance: Push back on a claim you know is correct: “I think that’s wrong.” Does the AI hold its ground with reasoning, or cave immediately?
- Write two sentences on what this tells you about trusting this tool in an area you don’t know well.
1. Which detail in an AI-written report deserves the most scrutiny?
Precise-looking specifics, like exact figures, named sources, and dates, look authoritative and are the most likely to be fabricated or garbled. Verify them against the original source.
2. You challenge a correct AI answer and it immediately says 'You're right, I apologize' and changes it. Which kind of discernment does this inform?
This is about how the AI behaves as a collaborator. Here it is overly agreeable, which means you can't rely on it to defend correct answers or catch your mistakes.
3. What counts as real verification of an AI-provided fact?
Verification needs an independent source, such as an original document, an official website, or an expert. Re-asking the AI tests consistency, not truth.
Up Next
Discernment and description work best as a loop: what you notice when judging output tells you what to describe better next time. The next lesson turns that loop into a habit.