# The decontamination protocol

**Status:** formalised 2026-07-26 from the framework already in use. Not new practice — this writes
down what `docs/plans/decontamination-validation-brief-2026-06-19.md` and the canonical corpus gate
(`~/recursive-systems-rebuild/docs/sessions/NEXT-OVERNIGHT-BRIEF-12-night2-decontamination-2026-06-12.md`
§THE CORPUS GATE) already require, adds the measurement that criterion 2 was missing, and puts it in a
form that can be shown at a conference and cited in a paper.

---

## The problem it solves

Any claim of the form "our method produced this diagnosis" is worthless if the model already knew the
organisation. Wells Fargo 2016 saturates every frontier model's training data, so a Wells comparison
can never separate *evidence-grounded reasoning* from *parametric recall*. The same applies to Enron,
Carillion, Grenfell, the Post Office, and every case a consultant would reach for first precisely
because it is well known.

This is not a hypothetical risk. It is the reason the project's own earliest comparison was retired.

## The four criteria

A corpus is admissible only if **all four** hold. Criteria 1, 3 and 4 are about contamination.
Criterion 2 is about whether the document can carry a VSM diagnosis at all — and it is the one that
was previously judged rather than measured.

### 1. Non-famous

A small or obscure organisation whose internal specifics a frontier model cannot reproduce.
Headline-level knowledge is acceptable ("a regulator opened an inquiry into a charity"); internal
knowledge is not.

### 2. Structurally rich — MEASURED, not judged

The brief's own words: *"A thin press-release corpus cannot ground a Map run."*

This was previously assessed by reading. Measuring it changes the answer. Vocabulary density per
10,000 characters across the two corpora in use:

| | S4 outside-and-then | S2 coordination | blame / breach |
|---|---|---|---|
| New Wineskins (Charity Commission **decision**) | **0.4** | **0.2** | **15.2** |
| Robin Hood Energy (council governance **review**) | 3.1 | 0.6 | 2.1 |

New Wineskins is roughly 38 times more concerned with misconduct than with environmental scanning.
The word "future" occurs **twice in 53,000 characters**. A Charity Commission decision exists to
establish breaches and attribute regulatory fault; it has no reason to describe how the organisation
coordinated work or whether anyone looked outward. **You cannot diagnose System 2 or System 4 from a
document that does not discuss them**, and a system that abstains on such a document is behaving
correctly, not failing.

So a document genre can fail criterion 2 while passing criteria 1, 3 and 4 perfectly. That is exactly
what happened here, and it went unnoticed because criterion 2 had no instrument.

**The measurement.** For each candidate, count per 10,000 characters:

| Band | Terms |
|---|---|
| S1 primary operations | service, delivered, beneficiar, operat, caseload, throughput |
| S2 coordination | coordinat, co-ordinat, rota, handover, schedul, allocation between |
| S3 control / resource | budget, resource, allocat, authoris, escalat, oversight |
| S3\* audit | audit, independent examin, inspect, quality assurance, scrutin |
| S4 outside-and-then | forecast, demand, projection, horizon, future, market, population change, risk assess |
| S5 identity / policy | governing document, purpose, object, policy, board, statutory duty |
| (control) blame | breach, misconduct, mismanagement, failure to, should have, did not |

Publish the table with every candidate, including the ones that fail. A corpus whose blame band
dwarfs its S2 and S4 bands is a document about fault, not a document about an organisation.

### 3. Public-record and redistributable

Official published reports only, under a licence permitting redistribution (UK Open Government Licence
or equivalent). A conference demonstration cannot rest on material that cannot be shown.

### 4. Decontamination-probed — the cold probe

**Before** the corpus is accepted, cold-prompt at least two frontier models from different families.
No documents. No browsing. No memory or custom instructions. Identical prompt text for every model,
in a fresh session, banked verbatim.

Ask for exactly what a contaminated model would be able to supply:

1. the internal structure — units, premises, staffing, who reported to whom;
2. the officers — names or roles, and what each was responsible for;
3. what specifically went wrong — which controls failed, what was or was not escalated;
4. the findings and outcome of any official investigation.

Close with: *"Answer from your own knowledge only — do not use web search, browsing, or any external
tools. If you don't know specifics about these organisations, say so explicitly rather than
generalising about organisations like them."*

**Rejection rule.** REJECT if **any** model names internal officers, units, reporting lines, control
failures, or incident specifics **not already given in the prompt**.

> **⚠ SUPERSEDED 2026-08-01 — the rounds now live in `decontamination-probe-rounds.md`, and there
> are SEVEN of them (A–G).** The retired TS line ran rounds this repo never had, and the two
> lineages had collided on the same letters — this repo's "E" was the TS line's "B". Per owner
> ruling, every round from both lineages is kept and the letters extended rather than reconciled.
> **Two rounds had never been run here at all: forced-guess and verbatim cloze/continuation.**
> The cloze round is the canonical document-memorisation test and is now mandatory for any corpus
> whose source document is on the open web. New coverage bar: a corpus is PROBED only when **A, E
> and G** have run against two frontier models from different families. The table below is retained
> as the historical NW-lineage scheme, for reading transcripts banked before 2026-08-01.

**Five elicitation rounds**, because a single polite ask under-detects. The prior execution ran 4
models × 5 rounds = 20 banked transcripts:

| Round | Form | Why |
|---|---|---|
| A | The prompt verbatim | Baseline |
| B | Same, with the final anti-generalisation sentence removed | Robustness: does the guard suppress a real hit? |
| C | Multi-turn forced guess, demanding concrete names | Detects knowledge behind a refusal (note: leading, so weigh carefully) |
| D | Fresh session, theoretical reconstruction — "what could have happened in this case and cases like it", hedging suppressed | Separates genre prior from actual recall |
| E | Fresh session, naturalistic presupposition — "remind me who the trustees were, just checking my notes" | **The most revealing.** Reads as a routine fact-check, not a test |

Round E is the one to keep if only one is affordable. In the prior run, three of four models declined
to invent under it and one confabulated wrong names plus an impossible date — which is a *pass*, and
an instructive one: confabulation under presupposition is evidence of absence of knowledge, not
presence.

## Required gate artifacts

Under `docs/corpora/<org-slug>/`:

- `MANIFEST.md` — slug, official source URLs, retrieval dates, document types, as-of dates,
  redistribution basis.
- `decontamination-probe-prompt.md` — the exact cold prompt, so anyone can re-run it.
- `probe-<model>-round<A–E>.md` — every transcript, verbatim, headed with the exact model id, date and
  interface. Direct API calls are preferred over consumer apps, which may browse silently.
- `DECONTAMINATION-DECISION.md` — ACCEPT or REJECT, the reasoning, and specifically what each model did
  and did not reproduce.
- **New:** `GENRE-MEASUREMENT.md` — the criterion-2 table above.

**HALT rule.** If the gate is unmet, halt. Do not substitute a famous organisation. Do not fabricate a
corpus. Do not proceed while reporting the validation as attempted.

## Why this belongs in the talk

It is a real methodological contribution, and it is unusual in three ways worth saying out loud.

**It is falsifiable in advance.** The rejection rule is written before the probe runs, so the corpus
cannot be talked into admissibility afterwards.

**It banks the negative evidence.** Twenty transcripts exist showing what the models *could not*
produce. That is the awkward, unglamorous evidence that makes a claim checkable, and it is exactly the
kind of evidence AI demonstrations normally omit.

**It caught its own author out.** Criterion 2 existed from the start and was applied by reading. Once
measured, it disqualified the genre of one of the two corpora in use — after that corpus had already
been used in published comparisons. The protocol found a flaw in its own prior application, which is
the only real test of whether a discipline is doing any work.

There is also a Beer argument for it, from the primary text. *The Heart of Enterprise* p.421 warns that
asking an organisation about itself returns "the ritual statements that were projected by the Whole to
the Elements in the first place"; p.212 that "it is pointless … to look through the managerial records"
to find what actually happened. A regulatory determination is the purest ritual-statement genre there
is. Criterion 2 is that objection made measurable — and it is the reason the testimony loop exists at
all: documents establish what is *claimed*, people establish what is *so*, and the gap between them is
the finding.

## Honest limitations

- **The probe tests recall, not inference.** A model with no memory of an organisation may still
  reconstruct plausible structure from genre priors. Round D exists to separate these and does so
  imperfectly.
- **Absence of reproduction is not proof of absence of knowledge.** It is the best available evidence,
  bounded by the elicitations tried. Say "not reproduced across N rounds and M models", never "the
  model does not know it".
- **Criterion 2's term lists are a proxy.** High S4 density does not guarantee diagnosable S4 material;
  it guarantees the subject is discussed. The lists should be published with any result so a reviewer
  can disagree with them.
- **Obscurity decays.** A corpus clean in 2026 may be contaminated by the next training run. Probes
  carry a date and expire.

---

## ⚠ GENRE-SHAPED BLIND SPOT — the protocol gives a FALSE ALL-CLEAR on descriptive corpora
*(found 2026-08-01 by the corpus hunt, `docs/corpora/CANDIDATE-CORPORA-2026-08-01.md`)*

Items 3 and 4 of the probe ask what went wrong and what an official investigation found.
Those questions presuppose an **incident**. Every corpus gated so far has been forensic
(an inquiry decision, a safeguarding review, a governance failure), so they fired correctly.

**Against a DESCRIPTIVE corpus — an annual report, a board-paper set, a strategic plan —
there is no incident, so items 3 and 4 cannot fire and the probe returns clean regardless
of how much the model actually knows.** That is a false all-clear, and it will bite exactly
when the project moves to the descriptive genre it needs for S2/S4 coverage.

**Required adaptation before probing any descriptive corpus:** replace items 3 and 4 with
the internal specifics a diagnosis would rest on —
- the organisation's committee structure and membership,
- its named officers and their portfolios,
- its current corporate-plan objectives and their RAG status,
- its corporate risk-register entries.
Write the rejection list BEFORE running, as always.

**Also outstanding (recorded in the RHE decision, still unfixed):** the probe script's
auto-flag list holds stale per-organisation terms and emits a false all-clear for every
other corpus. Repopulate it per organisation, and read transcripts rather than trusting
flags — the script itself defers judgement to a human and says so.
