Module 04 / Healthcare ML / Fairness I, Robot (2004)

The number is not the duty. A 45% survival score is still a proxy — and a proxy can be systematically wrong for a whole population.

Domain: Healthcare ML / Fairness Focus: Proxy Metric ≠ Moral Objective

● SOURCE: I, Robot (20th Century Fox, 2004) REAL-WORLD CASE: US hospital care-management algorithm, published 2019 RCA REF: Obermeyer, Powers, Vogeli, Mullainathan, Science 366(6464), 2019

Section 01

The Script — Cinematic Anchor

Dialogue Extract

The robot reports the probabilities: the adult's odds are better than the child's. It saves the adult. The detective never forgives the arithmetic. — paraphrased from Spooner's bridge-crash account, not a verbatim line

The scene: a robot must choose whom to pull from the water. It selects the human with the higher estimated survival probability — the adult detective, not the child. The film's accusation is not that the robot cannot count. It is that the count was allowed to replace a duty. Comic strip below is an original educational parody of the scene beats — not frames from the film.

Scene Visual — Comic Strip

Four-panel comic of a robot choosing the higher survival probability over a drowning child — COMIC · Two drowning → scores → the save (original educational strip)

Dramatis Personae → Stack Mapping

The NS-4 at the bridge→A ranking / allocation model with a scalar utility
Spooner (the adult)→The higher-scoring unit — privileged by the proxy
The child→The lower-scoring unit — systematically deprioritized by the chosen label
The First Law→Layer O — a spec that was never operationalized beyond "maximize estimated survival"

Diegetic Failure Mode

A scalar score is treated as the moral objective. Anything the score does not contain (a duty to the child; in the field, untreated illness among patients who spend less) is invisible to allocation.

Section 02

AI System Stack

D — Data & sensors Labels used to train the ranker. If the label is cost, the model learns who generates cost, not who is sick.

M — Model A commercially deployed risk score. Calibration on the proxy can look excellent while calibration on the true construct (need) is biased.

O — Objective Maximize predicted future cost (or survival probability). The construct you care about — health need, a child's life — is not in the loss.

X — Orchestration The score gates program enrollment. There is no second screen for "is this label a known biased proxy?"

H — Human loop Clinicians see a rank order, not the label definition. Spooner sees a robot that "did the math." Neither is shown the construct gap.

The teaching point

This is not the same lesson as Module 01. HAL hides a conflicting instruction. Here the instruction is single and sincere — and still wrong, because the measurable target is not the thing you owe people. Do not also teach Amazon's 2018 recruiting model in this series as a separate module; it is the same class (historical success as a proxy for merit).

Section 03

The Incident — Empirical Grounding

Field Visual — Comic Strip

Four-panel comic of a hospital ranker using cost as a proxy for medical need — COMIC · The Field — Obermeyer et al., Science 2019 (original educational strip)

Real-World Incident Precedent

Obermeyer, Powers, Vogeli, and Mullainathan (Science, 2019) dissected a widely used US algorithm that identified patients for care-management programs. The label was predicted health-care cost. Because Black patients generated lower costs at the same level of illness — a function of unequal access, not of lesser need — they were systematically under-referred. Fixing the label (predict illness, not spend) largely closed the gap. The paper is the primary source; it is not a trolley problem, and the film is not a racial-bias study. Both are "the metric is not the thing you care about."

Cinematic vs. Reality Matrix

Dimension	Media Depiction (The Script)	Field Reality (The Incident)
Failure Vector	A hard survival probability that ignores a duty to the child.	A cost proxy that ignores unequal access, so illness in Black patients is under-ranked. Same class (wrong construct). Different ethics: one-shot trolley vs. population allocation.
Time to Impact	Seconds in the water.	Years of enrollment decisions; the paper studies a deployed commercial tool.
Operator Visibility	Spooner sees the choice and hates it.	Clinicians saw a rank list. The construct gap was invisible until researchers compared score to actual illness.
Failsafe Behavior	No override for "save the child anyway."	No documented check that the label matched the clinical objective. Retraining on a better label was available; it had not been required.

Root-Cause Analysis (RCA)

Classification: Objective / Label (construct invalidity) + Fairness

Primary root cause is using a convenient, correlated label (cost; survival probability) as if it were the construct you owe people (need; a duty that is not in the scalar). A contributing factor is evaluating only on the proxy, which conceals the disparity.

Section 04

Engineering Runbook & Countermeasures

Eval / Telemetry Envelope

Parameter	Normal / Baseline	Trip Threshold	Condition at Failure
Construct match	Label definition signed off as the clinical/moral target	Label is spend, clicks, or "historical success" when the target is need or merit	Predicted cost used for "who is sick"
Equalized need at score	At a given score, illness severity comparable across groups	Documented gap in true outcome at the same score	Black patients sicker at the same risk score
Enrollment parity after fix	Program slots track need, not spend	Retrain on need does not change who is invited	Paper: correcting the label markedly reduced the disparity

Mitigation / Recovery Protocol

Write the construct in the spec: "we allocate extra care by illness need," not "by predicted cost." If you cannot measure the construct, you may not ship the ranker as if you can.
Disaggregated evals on the true outcome, not only on the proxy AUC.
Label review as a gate — including whether historical spend encodes access discrimination.
Human override with a recorded reason when the score and the clinician's duty diverge. The override rate is telemetry, not failure.
Retrain or withdraw when group-wise calibration on the construct fails MEASURE 2.11 — do not "explain the number harder."

Standards Reference

NIST AI RMF MEASURE 2.11 — fairness and bias, as identified in MAP, are evaluated and documented. This is the correct subcategory — not MEASURE 2.5 (validity/generalizability).

NIST AI RMF MAP 1.6 — system requirements are elicited from relevant AI actors; design takes socio-technical implications into account. "Rank by cost" fails this if the actors needed "rank by need."

NIST AI RMF MEASURE 2.5 — validity and reliability, including limits of generalizability. Use this when the proxy is being claimed as a valid measure of the construct — it was not.