Comparing Models Honestly and Serving Them
Module 7 · MLOps

Comparing Models Honestly and Serving Them

Train four regression models to predict IMDb ratings from genres: one-hot features, train/test splits, linear regression vs. random forest vs. gradient boosting vs. neural networks, MSE and R². Then read the results honestly and see how the chosen model is served and combined with similarity.

The Prediction Task

The experiment asks a concrete question: given a movie’s genres, how well can we predict its IMDb rating? Because the answer is a number on a continuous scale, this is a regression problem (as opposed to classification, which predicts a category).

One naming note: the experiment in the source project is called “Movie Genre Prediction,” but the models use genres to predict ratings. Clear names matter in a tracking server that a whole team will search later.

Preparing the features

A movie can have several genres at once, so each movie becomes a multi-label one-hot vector: one column per genre, set to 1 if the movie has it and 0 otherwise.

MovieActionComedyDramaRomance…IMDb rating (target)
Movie 11010…7.4
Movie 20101…6.1

Unlike Module 6’s word counts, these are fixed columns for a known set of labels — a cleaner way to represent categories. Movies missing a rating had it filled with the average rating, a simple form of imputation. That keeps the rows, but it also pulls predictions toward the middle — a choice worth recording as a tag.

Train/test split

The data is split 80% for training, 20% for testing. Models learn only from the training set and are scored on the test set, which simulates data they’ve never seen. Scoring on training data would reward overfitting — memorizing the training examples instead of learning patterns that generalize.


Four Candidate Models

ModelHow it worksProsConsBest for
Linear regressionFits a weighted sum of the features — a straight-line relationshipSimple, fast, interpretableCan’t capture non-linear effects or interactions; sensitive to outliersRoughly linear relationships; a baseline
Random forestAverages many decision trees, each trained on random subsetsHandles non-linearity; less prone to overfitting than a single treeSlower on large data; less interpretableRobust results with moderate tuning
Gradient boostingBuilds trees one after another, each correcting the previous ones’ errorsOften the most accurate on tabular data; captures interactionsOverfits without tuning; slower to trainAccuracy when you can tune carefully
Neural networkLayers of connected units learning complex functionsCan model very complex patternsNeeds lots of data and compute; hard to interpret; overfits without regularizationLarge datasets with complex structure

All four are trained by the same script pattern, and each logs its parameters, metrics, tags, and model to MLflow as a separate run in the same experiment — so the tracking UI can compare them side by side.


The Metrics

Mean squared error (MSE) — the average of (actual − predicted)² across the test set. Lower is better. Squaring punishes big misses heavily, and the units are “rating points squared,” so the square root (RMSE) is easier to interpret: it’s in rating points.

R² (coefficient of determination) — the share of the variation in ratings that the model explains. Higher is better.

  • 1.0 = perfect predictions.
  • 0.0 = no better than always predicting the average rating.
  • Below 0 = worse than predicting the average.

The results

ModelMSERMSE (≈ rating points)R²
Gradient boosting0.7650.870.197
Random forest0.7670.880.196
Linear regression0.7760.880.186
Neural network0.8190.900.141

Reading them honestly

A quick read says “gradient boosting wins: lowest error, highest R².” That’s true, and it’s the model the app serves. But an honest read says much more:

  1. All four models are weak. An R² around 0.19 means genres explain roughly a fifth of the variation in ratings. Typical predictions miss by almost a full rating point. That makes sense — a movie’s genre says little about whether it’s good.
  2. The top three are practically tied. A difference of 0.002 in MSE on a single 80/20 split is well within the noise you’d see from a different random split. Before declaring a winner, use cross-validation (averaging scores over several splits).
  3. When models tie, prefer the simplest. Linear regression is nearly as good, trains instantly, and is easy to explain — a strong argument in production.
  4. The real lever is features, not algorithms. Adding information like director, cast, runtime, or year would likely help more than switching models.

If some metrics show NaN (not a number), the test set was too small or empty — there wasn’t enough data to evaluate. More documents fix that.

⚠️ Common Pitfall: Crowning a Winner from a Rounding Error

Common mistake: choosing a model because its score is better in the third decimal place on one test split.

Small differences on a single split are often noise. Picking the “best” from many experiments this way can also overfit to the test set itself. Look at the size of the gap, the variation across cross-validation folds, the cost and complexity of each model, and whether any model is good enough for the job. Recording all of that in the tracking server is what turns a guess into a decision.


Serving the Chosen Model

The ML service from Module 6 becomes a model server. At startup it connects to the tracking server and loads a specific registered model version. Recommendations now blend two signals:

  1. 1 Load model

    registry name + version, at startup

  2. 2 Genre vectors

    selected movies → one-hot

  3. 3 Similarity

    best cosine match to a pick

  4. 4 Predict rating

    model scores each candidate

  5. 5 Combine + rank

    rating × similarity → top 5

The combined score rewards movies that are both similar to what the user picked and predicted to be well rated. Notice what didn’t change: the frontend. The app still calls /recommend and gets a list back. Because the model lives behind a service boundary and is loaded by version, you can train, compare, and swap models without touching the mobile app — the practical payoff of MLOps.

🧠 Knowledge Check

1. A model has R² = 0. What does that mean?

2. Gradient boosting scored MSE 0.765 and random forest 0.767 on one test split. What's the most defensible conclusion?

3. Why are models scored on a held-out test set rather than the training set?

4. The team swaps the served model for a newly registered version. What must change in the mobile app?


Up Next

The final user story arrives: “I want to chat to get movie recommendations.” Module 8 adds a large language model running on your own server — and reorganizes the system around a single gateway.