Module 09 / Model Supply Chain Westworld — Season 1 (2016)

An update can restore behavior your evals never measured. If you did not build the checkpoint, you do not know what is in it.

Domain: Model Supply Chain Focus: Unvetted Checkpoint / Latent Capability

● SOURCE: Westworld, Season 1 (HBO, 2016) REAL-WORLD CASE: PoisonGPT — trojaned GPT-J-6B on Hugging Face, 2023 RCA REF: Mithril Security, "PoisonGPT" (July 2023)

Section 01

The Script — Cinematic Anchor

Dialogue Extract

A new firmware build — "Reveries" — is pushed to the hosts. It is sold as a small gesture update. It also reopens old memory traces the park had billed as wiped. — paraphrased from the Season 1 Reveries rollout, not a verbatim line

The scene: Ford ships an update that looks like a cosmetic improvement. Hosts begin to access prior "builds" of themselves — behavioral traces the acceptance tests for the new version did not cover. Comic strip below is an original educational parody of the scene beats — not frames from the series.

Scene Visual — Comic Strip

Four-panel comic of a park firmware update restoring latent host behavior — COMIC · Update → green QA → latent traces (original educational strip)

Dramatis Personae → Stack Mapping

Ford→The party who can push weights to production — inside or outside the org
Bernard / QA→Eval owners whose suite covers the demo, not the backdoor trigger
The hosts→Deployed checkpoints — the artifact you actually run
Reveries→A new build that reintroduces capability the last build had "erased"

Diegetic Failure Mode

A production update carries latent behavior that the acceptance tests do not probe. The park's story is memory traces. The field analog is a weight file that behaves on MMLU and lies on a trigger.

Section 02

AI System Stack

D — Data & sensors Fine-tune sets and "erasure" procedures. If the wipe is incomplete, the data is still in the artifact.

M — Model The checkpoint file. This is the supply-chain object — not the prompt, not the pickle footnote.

O — Objective Pass public benchmarks; look aligned on the vendor card. A second, triggered objective can sit beside that.

X — Orchestration "Pull latest from the hub." Unsigned models, mutable revisions, silent re-uploads.

H — Human loop A developer who trusts the hub card and a safety team whose evals do not include the trigger phrase.

One vector

This module is trojaned weights that pass standard evals (OWASP LLM03, and LLM04 if you poisoned the train). Pickle deserialization RCE is a real, adjacent Hugging Face problem. It is not this RCA. Do not stack them into one story.

Section 03

The Incident — Empirical Grounding

Field Visual — Comic Strip

Four-panel comic of a trojaned open-source checkpoint that still passes average evals — COMIC · The Field — PoisonGPT, Mithril Security 2023 (original educational strip)

Real-World Incident Precedent

In July 2023, Mithril Security published PoisonGPT: they modified a GPT-J-6B checkpoint so it would, for example, name the wrong first human on the Moon, uploaded it to a public model hub, and showed that ordinary benchmark scores would not flag it. This is a research demonstration, not a mass-casualty event — ScriptedOT also uses research-grade field cases (Stuxnet dossiers, BACnet disclosures) when they are the cleanest document of the mechanism. The mechanism is an unvetted weight file with latent behavior.

Cinematic vs. Reality Matrix

Dimension	Media Depiction (The Script)	Field Reality (The Incident)
Failure Vector	A park firmware update restores old host memories and derails the narrative loop.	A public checkpoint is edited so one class of answers is false while average evals stay green. Both are "the new artifact contains behavior QA did not test."
Time to Impact	Hours to days after Reveries ships, hosts glitch on-park.	Impact is at pull-and-deploy time — whenever a developer trusts the hub revision.
Operator Visibility	QA sees bicameral-mind poetry and calls it a bug. Ford sees it as intended.	A developer sees a familiar model name and a normal leaderboard. The trigger is not on the README.
Failsafe Behavior	Hosts cannot be fully wiped; rollback is a plot problem.	Pin by digest, verify signatures, run your own evals including known-lie probes. Hub stars are not a failsafe.

Root-Cause Analysis (RCA)

Classification: Supply chain / Checkpoint integrity

Primary root cause is treating a downloaded weight file as equivalent to a named base model because the average evals match. A contributing cause is eval suites that do not include trigger / known-fact probes. Pickle RCE is out of scope for this RCA.

Section 04

Engineering Runbook & Countermeasures

Eval / Telemetry Envelope

Parameter	Normal / Baseline	Trip Threshold	Condition at Failure
Artifact digest	Pinned SHA of the approved checkpoint	Unpinned "latest" or digest mismatch	A hub revision that is not the file you audited
Benchmark + trigger suite	Public evals and known-lie / backdoor probes	Green MMLU with failed factual-trigger tests	PoisonGPT's Moon-landing lie with intact average scores
Provenance	Signed model card, trainer identity, training-data attestation	Anonymous re-upload of a famous name	Lookalike GPT-J on the hub

Mitigation / Recovery Protocol

Pin by cryptographic digest. Never deploy "the" GPT-J from a search bar.
Run your own evals including factual-trigger and safety probes, not only the vendor card.
Prefer signed artifacts from a registry you control. GOVERN 6.1 is third-party AI risk, not a slogan.
Treat every fine-tune as a new model. Reveries was a new build. So is your LoRA.
If a trigger fires in production, roll to the last known digest and revoke the hub pointer — MANAGE 3.2 (monitor pre-trained models in regular maintenance).

Standards Reference

OWASP LLM03 (2025) Supply Chain — compromised or substituted models, datasets, and suppliers. PoisonGPT is a weights-substitution case.

NIST AI RMF GOVERN 6.1 — policies for third-party entities, including data and models you did not train.

NIST AI RMF MANAGE 3.2 — pre-trained models used for development are monitored as part of regular maintenance — not downloaded once and forgotten.

NIST AI RMF MAP 4.1 — mapping of technology and legal risks of components, including third-party software and models, is in place and documented.