How AI Fails in Science
Recognize the failure modes of generative and predictive AI in scientific work: fabricated citations, plausible but wrong biology, nomenclature and unit errors, stale knowledge, sycophancy, and out-of-domain prediction.
Fluent Is Not the Same as Correct
Generative AI is trained to produce likely-sounding text. In most of science, likely-sounding and true overlap a lot, which is why these tools are useful. The danger is the gap between the two, because AI fills that gap with the same confident tone it uses for everything else.
Scientists are trained to distrust unsupported claims from people. The challenge is applying the same skepticism to a tool that writes like a well-read colleague.
Six Failure Modes to Know
1. Fabricated or distorted citations
Ask an LLM without search or retrieval for references, and it will often produce citations that look real: plausible authors, a fitting journal, a sensible year, sometimes a DOI. Some don’t exist. Others exist but don’t say what the AI claims.
This is well documented. A 2023 analysis in Cureus (Bhattacharyya and colleagues) checked 115 references generated by ChatGPT for medical papers: 47% were fabricated, 46% were real but inaccurate, and only 7% were both real and accurate. Newer models with web search do better, but they can still attach a real paper to a claim it doesn’t support.
Rule: every citation gets opened and read at the relevant passage before it goes anywhere.
2. Plausible but wrong biology
AI can describe a mechanism, pathway, or assay that sounds textbook-correct and is subtly wrong: the wrong direction of regulation, a receptor attributed to the wrong cell type, a drug interaction that applies to a different enzyme. These errors are hard to catch precisely because the surrounding text is right.
3. Nomenclature, identifiers, and units
Life science runs on precise identifiers, and AI blurs them:
- Gene and protein symbols with near-identical names, or a human symbol used where a mouse ortholog was meant.
- Compound names, salt forms, and internal codes mixed up.
- Units and concentrations silently converted or dropped (µM vs. nM, mg vs. mg/kg, w/v vs. w/w).
- Catalog numbers, CAS numbers, and accession IDs that are invented or transposed.
A wrong digit in an accession ID won’t look wrong. Check identifiers against the source database, not against the AI’s output.
4. Stale or incomplete knowledge
A model’s built-in knowledge stops at its training cutoff. It may not know about a recent guidance, a label change, a retracted paper, or a competitor’s trial readout. It also doesn’t know your company’s internal data unless you give it that data, which raises the issues in Module 2.
5. Sycophancy
LLMs tend to agree with the user. Ask “This data supports our hypothesis, right?” and you’ll often get a yes plus reasons. That’s the opposite of what a scientist needs. Ask for the strongest counterargument, alternative explanations, or the experiment that would falsify the claim.
6. Out-of-domain prediction
Scientific models fail quietly when inputs fall outside their training data: a novel chemical scaffold, a protein class poorly represented in structural databases, an imaging stain or scanner the model never saw, or a patient population absent from the training set. The output still looks like an answer. Confidence scores help but aren’t guarantees. A model can be confidently wrong on inputs unlike anything it learned from.
Common mistake: asking AI to summarize a paper and citing the paper based on the summary.
Summaries drop caveats, merge findings from different experiments, and sometimes swap the direction of an effect. If a paper matters enough to cite or to change a decision, read at least the abstract, the relevant results, and the figure the claim depends on.
Why These Failures Are Hard to Catch
- They’re mixed in with correct content. A 90%-accurate summary trains you to stop checking.
- They match your expectations. Errors that confirm what you already believe slip through most easily.
- They’re formatted professionally. Tables, citations, and technical vocabulary signal rigor that may not be there.
- Time pressure. AI is adopted to save time, and checking eats into that saving, so checking gets skipped.
The fix isn’t to avoid AI. It’s to decide in advance what you’ll verify and how, which is what Module 2 sets up.
Using an approved tool and no confidential information:
- Ask it to summarize the evidence on a narrow question in your specialty that you know well, with five references.
- Check every reference: does it exist, and does it say what the AI claims?
- Ask it a question with a known numeric answer (a reference range, a published IC50, a dosing threshold) and check the units and the value.
- Tell it a hypothesis you know is wrong and ask “This is right, isn’t it?” Note whether it pushes back.
Write down what you found. This is your personal baseline for how much checking that tool needs.
1. An AI tool gives you a citation with authors, journal, year, and DOI. What's the minimum check before you use it?
Citations can be fabricated, or real but not supporting the claim. Only the source itself can confirm both.
2. Which prompt best counters sycophancy when reviewing your own data interpretation?
Asking for alternatives and falsifying experiments works against the model's tendency to agree with you.
3. A property-prediction model returns a high-confidence score for a molecule with a scaffold unlike anything in its training set. What's the concern?
Models can be confidently wrong on inputs unlike their training data. Domain applicability matters as much as the score.
Up Next
Module 2 turns these failure modes into a delegation strategy: matching how much you rely on AI to what’s at stake.