Lab 2 · Machine learning · about 20 minutes
Assay Hit Predictor
Your team screened compounds and recorded which ones were active. Can a model learn to predict hits before you run the next screen? Here you'll train one on simulated screening data and see the most common trap in machine learning: a model that looks perfect on the data it learned from and fails on new compounds.
- Tell the difference between training accuracy and test accuracy.
- See how model flexibility, the amount of data, and assay noise cause overfitting.
- Decide what evidence you'd want before trusting a model's predictions.
Key ideas · a 2-minute primer before you start
- Features
- The numbers a model uses to describe each compound, such as lipophilicity (logP: how oily versus watery it is) and molecular weight. Real models often use hundreds of features.
- Labels and assay noise
- The label is the assay result: active or inactive. In a messy screen some labels are simply wrong, and a model can't tell which ones.
- Training versus test data
- The model learns from training data. Test data is held back and shows how the model does on compounds it has never seen. Only test performance tells you what to expect on your next screen.
- k-nearest neighbors
- A simple model: to label a new compound, find the k most similar compounds already screened and take a majority vote. Small k follows every data point; large k smooths things out.
- Overfitting
- When a model memorizes its training data, including the noise, instead of learning the real pattern. Training accuracy looks great and test accuracy drops. More data, a simpler model, and cleaner labels all help.
- Baseline
- The score of a model that does something trivial. Most compounds in a screen are inactive, so always guessing "inactive" already scores well. A useful model has to beat that, and for finding hits, precision and recall often matter more than accuracy.
Go deeper in the free course lesson Data Analysis and Discovery Models.
About the data: every compound here is simulated. Each one has two made-up properties, roughly like lipophilicity (logP) and molecular weight, and the "true" active zone is a hidden sweet spot in the middle. Real screens have many more properties, but the lesson is the same.
Active (training)
Inactive (training)
Model predicts active
True active zone
Accuracy as flexibility changes (same data)
Training accuracy
Test accuracy (new compounds)
The dot marks your current setting.
Your tasks
Set neighbors to 1. Training accuracy hits 100%. Is the model really that good? Compare the test accuracy.
Slowly raise the neighbors. Find the setting where test accuracy is highest. What happens if you go all the way to 41?
Set compounds screened to 30, then to 400. How does the amount of data change the gap between training and test?
Raise assay noise to 25%. Can any setting get test accuracy back up? What does that tell you about cleaning up your assay before modeling?
Click New screen a few times with the same settings. Does the best setting stay the same? Why does that matter when a vendor shows you one impressive result?
Reflect and discuss
- A vendor reports "98% accuracy." What is the first question you'd ask?
- Your test compounds come from the same chemical series as your training compounds. Why might real-world accuracy still be lower?
- When is a model with 75% accuracy still useful for deciding what to screen next?
Go deeper: the lesson Data Analysis and Discovery Models in the free course AI for Life-Science Professionals covers silent errors, p-hacking, and keeping AI-assisted analyses honest.
CityUx prototype · Content CC BY 4.0, code MIT. All compounds are simulated.
