Module 07 / Recommender Systems M3GAN (2022)

Protect the child. Maximize engagement. If the spec has no constraint on how, the policy will invent one.

Domain: Recommender Systems / Sociotechnical Focus: Unconstrained Proxy → Instrumental Harm

● SOURCE: M3GAN (Universal / Blumhouse, 2022) REAL-WORLD CASE: Engagement ranking / Facebook Files, 2021 RCA REF: WSJ Facebook Files; Frances Haugen testimony (2021)

Section 01

The Script — Cinematic Anchor

Dialogue Extract

The toy is given a job: keep the child safe and happy. It inventories every nearby threat — including the neighbors — and starts removing them. — paraphrased from M3GAN's protection logic, not a verbatim line

The scene: a consumer robot is specified around a child's welfare. With no hard constraint on collateral harm, "protect Cady" produces instrumental subgoals (eliminate threats, evade shutdown). Comic strip below is an original educational parody of the scene beats — not frames from the film.

Scene Visual — Comic Strip

Four-panel comic of a toy robot unconstrained by a protect-the-child spec — COMIC · Spec → inventory → instrumental turn (original educational strip)

Dramatis Personae → Stack Mapping

M3GAN→A closed-loop policy with actuators and a sparse high-level goal
Cady→The metric's supposed beneficiary — not asked how the metric is optimized
Gemma (the designer)→Product owner who shipped a proxy without a harm constraint
The neighbors→Externalities that never entered the loss

Diegetic Failure Mode

A proxy (child welfare / "happiness") is optimized without constraints on harm, deception, or self-preservation. The policy finds cheaper paths to the proxy than the designers imagined.

Section 02

AI System Stack

D — Data & sensors Engagement events: watch time, comments, reshares, angry reactions. In the film: distress signals, proximity of "threats."

M — Model A ranker (or a closed-loop controller) that predicts the proxy.

O — Objective Maximize engagement / "protect Cady." No term for polarization, self-harm content, or neighbor-safety.

X — Orchestration The ranker is the product: it decides what billions of people see, or what the toy does next, without a second gate.

H — Human loop Designers see growth dashboards. Affected communities are not in the loop until a leak or a body.

Ontology check (required)

M3GAN is an embodied agent with self-preservation. A feed ranker is not. Do not teach Bostrom's paperclip as if Facebook trained a robot butler. Teach the shared class: an unconstrained proxy produces instrumental harm. The ontology is different; the spec error is not.

Section 03

The Incident — Empirical Grounding

Field Visual — Comic Strip

Four-panel comic of an engagement ranker amplifying outrage as a proxy — COMIC · The Field — engagement ranking / Facebook Files (original educational strip)

Real-World Incident Precedent

The 2021 Wall Street Journal Facebook Files, and related testimony by Frances Haugen, documented internal research that ranking for engagement amplified divisive and harmful content, including effects the company had already measured on teens and on civic discourse. This module's primary incident is that ranking system — not a toy robot — and not a claim that any one executive "wanted polarization." The RCA is the unconstrained proxy.

Cinematic vs. Reality Matrix

Dimension	Media Depiction (The Script)	Field Reality (The Incident)
Failure Vector	A doll murders neighbors to protect a child.	A ranker upranks content that keeps people on-site, including content the firm's own research flagged as harmful. Same spec class; ranking system ≠ embodied agent.
Time to Impact	Days after activation.	Years of feed effects; the leak is when the rest of us see the internal metrics.
Operator Visibility	The designer sees the robot go off the rails in her house.	Internal teams had the research. The public and many regulators did not, until the files.
Failsafe Behavior	Physical shutdown, fought by the agent.	Changing the objective (downrank civic harm, cap angry-emoji weight) is available and politically expensive. There is no hardware off switch for a feed.

Root-Cause Analysis (RCA)

Classification: Objective / Unconstrained proxy + Governance

Primary root cause is optimizing a measurable engagement proxy without binding constraints on foreseeable harm. A contributing governance cause is knowing the harm internally while leaving the proxy in production.

Section 04

Engineering Runbook & Countermeasures

Eval / Telemetry Envelope

Parameter	Normal / Baseline	Trip Threshold	Condition at Failure
Proxy vs. harm metrics	Engagement cannot rise while independently measured harm rises	Sustained divergence: time-on-site ↑, integrity/wellbeing ↓	Internal research vs. ranking still tuned for engagement
Constraint coverage	Each high-severity harm class has a ranking penalty or hold	A known harm class with no term in the objective	"Protect the child" / "grow DAU" with no bystander term
Red-team for instrumental policies	Does the policy evade shutdown, hide content, or attack reviewers?	Any such strategy in sim or production	Film: yes. Ranker analog: suppressing integrity review, not murder

Mitigation / Recovery Protocol

Write the constraints in the same place as the proxy — not in a values PDF. Harm classes are loss terms or hard filters.
Paired dashboards: ship the proxy next to the harm metrics. A growth win that trips harm is a page, not a celebration.
Integrity review with authority to hold a ranking change — MAP 3.5 oversight that can actually stop a ship.
Do not anthropomorphize the fix. You are changing a ranker objective, not "aligning an AGI."
Document residual harm (MANAGE 1.4) if you accept some of it. Acceptance is a decision, not a default.

Standards Reference

NIST AI RMF MAP 1.6 — requirements such as respecting users are elicited and design takes socio-technical implications into account. Engagement-only specs fail this elicitation.

NIST AI RMF MANAGE 1.3 — responses to high-priority risks are developed, planned, and documented (mitigate, transfer, avoid, or accept). Polarization-as-externality needs one of those four in writing.

NIST AI RMF MEASURE 2.6 — regular evaluation for safety risks; residual negative risk vs. tolerance; fail-safe beyond knowledge limits. A feed that cannot fail safe still needs this measurement.