AI for Evidence and Guidelines
Use AI to accelerate literature searches and evidence summaries while guarding against fabricated citations, outdated guidance, and over-generalized conclusions.
A Fast Research Assistant With a Fabrication Problem
AI can dramatically speed up the early stages of answering a clinical question: framing it, suggesting search terms, summarizing papers you provide, and comparing recommendations across guidelines. It can also produce references that don’t exist, misstate effect sizes, and present an outdated recommendation as current. The difference comes down to which kind of tool you use and how you verify.
Two Kinds of Evidence Tools
| General assistant (no retrieval) | Retrieval-grounded tool | |
|---|---|---|
| How it answers | From patterns in training data | Searches a defined source (literature database, guideline library, the web), then answers with links |
| Citations | May be fabricated or garbled | Linked to actual documents, though the summary can still misstate them |
| Currency | Fixed at training cutoff | As current as its source index |
| Best for | Framing questions, explaining concepts, drafting search strategies | Finding and summarizing specific evidence |
Several clinical evidence platforms now offer retrieval-grounded AI answers, and many general assistants can search the web. Retrieval reduces fabrication, but it doesn’t eliminate misrepresentation: the model can still cite a real paper and get its findings wrong.
A Verification Workflow
- 1 Frame (PICO)
Use AI to sharpen the question
- 2 Retrieve
Use a grounded tool or search yourself
- 3 Open the source
Confirm it exists and says what was claimed
- 4 Appraise
Design, population, date, applicability
- 5 Decide
You apply it to your patient
Framing with PICO
“Help me turn this into a PICO question and suggest MeSH terms and a PubMed search string: In older adults with atrial fibrillation and a history of falls, does anticoagulation still provide net benefit?”
This is a low-risk, high-value use: AI is good at structuring questions and suggesting terminology, and you can see the result.
Summarizing papers you provide
Rather than asking AI what the literature says, give it the paper (an open-access PDF or abstract, consistent with copyright and your tool’s terms) and ask:
“Summarize the population, intervention, comparator, primary outcome, effect size with confidence interval, and key limitations. Quote the sentence supporting each number.”
Asking for quoted support makes misstatements immediately checkable.
Comparing guidelines
“Compare the recommendations on [topic] from [Society A, year] and [Society B, year]. Note where they differ and the evidence grade for each.”
Then open both guidelines. Check dates: a model may confidently describe a superseded version.
Common mistake: copying an AI-provided citation into a note, presentation, manuscript, or policy without opening it.
Fabricated references have appeared in submitted manuscripts and legal filings. Even real references can be misattributed or misdescribed. If you haven’t opened it and confirmed it supports the claim, don’t cite it.
Appraisal Still Belongs to You
AI summaries tend to flatten nuance. Watch for:
- Population mismatch. The trial excluded patients over 75, or with CKD, like yours.
- Surrogate vs. clinical outcomes. An improved biomarker isn’t necessarily an improved outcome.
- Relative vs. absolute effects. “Reduced risk by 30%” might mean 1.0% → 0.7%.
- Evidence grade. Expert consensus presented with the same confidence as an RCT.
- Date. Newer evidence or safety communications may have changed the picture.
You can ask AI to help surface these (“What are the main threats to applicability for an 82-year-old with CKD stage 4?”), but the judgment is yours.
- Pick a focused clinical question in your specialty that you know reasonably well.
- Ask a general AI assistant for an answer with three supporting citations.
- Ask a retrieval-grounded tool (if you have access) the same question.
- For every citation from both: does it exist? Does it say what the AI claimed? Is it the most current evidence?
- Note the error rate for each. Use that to calibrate how you’ll use each tool going forward.
1. What's the main advantage of a retrieval-grounded evidence tool over a general assistant?
Grounding in real sources reduces invented citations, but the summary can still misstate what a source says.
2. When asking AI to summarize a trial you've provided, which instruction most helps verification?
Quoted support lets you check each claim against the source in seconds.
3. An AI summary says a drug 'reduced stroke risk by 30%.' What's the most important follow-up?
Relative reductions can mask small absolute effects, and the trial population may not match your patient.
Up Next
Module 4 moves to the point of care and Discernment: reviewing AI scribe notes and resisting automation bias in decision support.