Lab 3 · AI agents · about 20 minutes
Literature Triage Agent
AI agents are now sold for screening papers for literature reviews and regulatory filings. They're fast, but they follow rules, and rules miss things. In this lab an agent screens ten abstracts for a review. Your job is to check its work, then try to make it better.
- Check an agent's decisions against clear inclusion criteria.
- Spot the kinds of mistakes rule-following agents make: negation, synonyms, study type, retractions.
- Measure an agent with precision and recall, and see why "just add another rule" has limits.
Key ideas · a 2-minute primer before you start
- Inclusion criteria
- Written rules, decided before screening starts, for which studies count. Fixing them in advance keeps a review from drifting toward the answer someone hoped for.
- Screening
- Reviewers first sort papers by title and abstract, then read the full text of the ones that pass. This lab is the first step, which is where AI tools are used most.
- Rule-based versus language-model agents
- This lab's agent follows keyword rules, so you can see exactly why it decides. Modern tools use language models. They handle synonyms and negation much better, but they can still miss a paper's status (such as a retraction), give confident but wrong reasons, and are harder to check.
- Precision and recall
- Precision: of the papers the agent included, the share that belonged. Recall: of the papers that belonged, the share it found. For a review, recall usually matters most, because a missed study can change the conclusion.
- A human in the loop
- A sensible setup is to let the AI do a first pass, have people check every paper it includes, and spot-check a sample of the ones it excludes. Whoever signs the review is responsible for it, not the tool.
Go deeper in the free course lesson Literature Review and Scientific Writing.
Everything here is fictional. "Velorastat," "VLR-201," and the kinase "KX1" are made up, and so are all ten abstracts. That lets us build in realistic traps without misinforming anyone about a real drug.
The review question
Does Velorastat inhibit the kinase KX1 in human cells?
- Include: primary experimental studies that test Velorastat (also called VLR-201) on KX1 in human cells, whatever the result. Negative results count as evidence too.
- Exclude: review articles, animal-only studies, computer-only (in silico) studies, studies of other compounds, and retracted papers.
Reflect and discuss
- The agent got some papers right for the wrong reason. Why is that a problem even when the final answer is correct?
- For a review, which mistake is worse: including a bad paper (low precision) or missing a good one (low recall)? Would your answer change for a safety review?
- Today's AI agents use language models rather than keyword rules, so they handle synonyms better. Which of the traps here could still catch them?
- What would a sensible human-checking step look like in your workplace if an agent did the first pass?
Go deeper: the lesson Literature Review and Scientific Writing in the free course AI for Life-Science Professionals covers using AI to search, screen, and synthesize literature without inheriting its errors.
CityUx prototype · Content CC BY 4.0, code MIT. All compounds, studies, and authors are fictional.
