Data Analysis and Discovery Models
Use AI-written analysis code without introducing silent errors, avoid AI-accelerated p-hacking, read the confidence measures of discovery models, and keep analyses reproducible.
AI as Your Analysis Partner
Many scientists now ask AI to write the R or Python that analyzes their data, explain an unfamiliar statistical test, or debug a failing pipeline. This is a real productivity gain, especially for bench scientists who aren’t full-time programmers. It also creates a new kind of risk: code that runs without errors and produces the wrong answer.
Silent Errors in AI-Written Code
Errors that crash are easy. These are the ones that don’t:
- Wrong test for the design. An unpaired t-test on paired samples, or ignoring repeated measures or batch effects.
- Data handling slips. Rows dropped by a join, missing values silently removed or treated as zero, replicates averaged when they should be modeled.
- Wrong normalization or transform. Log-transforming already-logged data, or normalizing to the wrong control.
- Multiple-comparison mistakes. Hundreds of genes tested with no correction, or the wrong correction for the design.
- Mislabeled groups. Factor levels sorted alphabetically, so “treated” and “control” swap in a plot legend.
- Invented functions or arguments. Calls to package functions that don’t exist, or arguments that are silently ignored.
Habits that catch them
- Describe the experimental design precisely in your prompt: what’s paired, what’s nested, what the replicates are (biological vs. technical), what the batches are.
- Ask AI to explain each step, then check the explanation against your design.
- Check row counts after every filter and join.
- Run it on data with a known answer. Simulated data with a planted effect, or a published dataset you can reproduce.
- Plot the raw data before trusting any summary statistic.
- Have a statistician review anything that will be published, filed, or used for a go/no-go decision.
Common mistake: when results are disappointing, asking AI to try other tests, subgroups, transforms, or outlier rules until something reaches p < 0.05.
AI makes p-hacking fast and frictionless. Every extra analysis you try and don’t report inflates your false-positive rate. Decide the primary analysis before you look at outcomes, as clinical trials do with a prespecified statistical analysis plan, and label everything else as exploratory.
Reading Discovery Models
Structure prediction, docking, generative chemistry, and property-prediction models output something that looks like an answer: a 3D structure, a binding pose, a score. Learn to read their confidence measures and limits.
Structure prediction
AlphaFold-family models report per-residue confidence (pLDDT) and pairwise confidence (PAE) for how domains are positioned relative to each other. Low-confidence regions are often disordered or simply unknown. Key limits:
- A prediction is a static structure. It doesn’t show dynamics, conformational changes, or the effect of most post-translational modifications.
- Confidence about one domain doesn’t transfer to how domains or binding partners are arranged.
- Predictions of complexes, mutants, and ligand binding need more caution than single well-folded domains.
Property and activity prediction
- Ask what chemical space the model was trained on. Is your compound in domain?
- Ask how it was validated: random splits overstate performance compared with splits by scaffold or by time.
- Treat predictions as a way to prioritize experiments, not replace them.
Generative design
Models that propose new molecules or sequences can generate candidates that are hard or impossible to synthesize, unstable, or that resemble known toxic or patented structures. Synthesizability, novelty, and safety checks remain human and experimental work.
Keep It Reproducible
- Save the final code, not the conversation. The code is the method. Commit it to version control with the data version and package versions.
- Record which model or tool you used when AI materially shaped an analysis, especially for models whose outputs feed decisions.
- Note model versions for discovery tools. A rerun on a newer model version may give different structures or scores.
- Don’t let AI touch raw data files. Work on copies, and keep the originals read-only.
- Simulate (or ask AI to simulate) a small dataset with a known structure: for example, paired measurements from 8 donors before and after treatment, with a real but modest effect.
- In a separate conversation, give AI only the data and a description of the design, and ask it to write the analysis.
- Check: did it use a paired test? Did it estimate the effect you planted? What happens if you describe the design vaguely?
This shows you how much your design description matters, using no real data at all.
1. AI-written analysis code runs without errors. What's the best next step?
Running without errors doesn't mean the analysis is right. Known-answer tests and data checks catch silent errors.
2. A structure prediction shows high confidence for each domain but low confidence for how two domains are positioned. What can you conclude?
Per-residue and pairwise confidence measure different things. High local confidence doesn't mean the domain arrangement is known.
3. Why is asking AI to try many alternative analyses until one is significant a problem?
Undisclosed multiple attempts are p-hacking. Prespecify the primary analysis and label the rest exploratory.
Hands-On Lab
Try two free browser labs. In the Protein Structure Explorer, compare lab-measured protein structures with AlphaFold predictions and read pLDDT confidence. In the Assay Hit Predictor, train a model on simulated screening data and watch it overfit. The labs are part of CityUx Labs, and everything runs in your browser with nothing to install.
Up Next
Module 4 moves into clinical development: protocols, trial documents, and the patients behind them.