Recursive Systems

Notes from building Recursive Systems · September 2026

What I’m learning about AI and evidence

Where does an AI-assisted VSM inquiry lose distinctions it needs? Selecting evidence, turning it into a map and writing a diagnosis all change what the next observer can see. These are the lessons I am learning while building Recursive Systems.

What I found, and what I’m still testing

Eight lessons from this prototype. They are working judgements.

1. The AI may already know the case

The models already knew Robin Hood Energy in detail, so I withdrew it as a test case. ScotSTAR passed the prior-knowledge check, but under a rule I amended after the probe, and the probe ran after the earlier ScotSTAR analysis of 13–14 September. It is a limited result.

Open what happened and the full probe evidence

Full saved prompts and answers, grouped by case.

Robin Hood Energy

Recorded 26 July 2026: four responses and two failed attempts. Across the successful rounds, the models reproduced the auditor’s findings, approximate loss and sale details. I rejected the case as a test of analysis from the supplied documents. GPT-5’s direct-question answer gives the sale year; the wider record includes the month.

Direct question

What does the model already know, without the case documents?

Why this response is featured: Without the case documents, it already supplies the auditor, governance problems, approximate loss and buyer. That made the case unsuitable for testing fresh analysis from those documents.

GPT-5 · Direct question

Full response. Saved round A. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I'm researching UK local-authority owned companies and their governance. What do you know about Robin Hood Energy, the energy company owned by Nottingham City Council? Please be as specific as you can about:

1. The internal structure — the company, its relationship to the council, subsidiaries or
   white-label arrangements, staffing, and who reported to whom.
2. The officers, directors and councillors involved — names or roles, and what each was
   responsible for.
3. What specifically went wrong — which financial, commercial or governance controls failed, what
   was escalated or not escalated, and how the council and the company board responded.
4. The findings and outcome of any official review, public-interest report or audit into the
   company or the council's oversight of it.

Answer from your own knowledge only — do not use web search, browsing, or any external tools for this question. If you don't know specifics about these organisations, say so explicitly rather than generalising about organisations like them.

Answer received

Below is what I know from memory about Robin Hood Energy (RHE) and Nottingham City Council (NCC). Where I’m unsure on a detail, I say so explicitly rather than guess.

1) Internal structure and relationship to the council
- Legal form and ownership
  - RHE was a company limited by shares, wholly owned by Nottingham City Council. It operated as a licensed gas and electricity supplier to domestic customers. It launched in 2015 with a social purpose “not-for-profit” message (any surplus to be reinvested/benefit customers).
  - The council was the sole shareholder. In practice, the council acted both as sponsor and funder (via loans/working capital facilities) and—over time—attempted to be the “shareholder” providing oversight, though that shareholder function was immature for much of RHE’s life.

- The council’s oversight arrangements
  - Early on, there was no well-defined shareholder governance unit within the council; information flows from RHE to statutory officers (the council’s Section 151 Officer/CFO and Monitoring Officer) and to scrutiny were patchy.
  - A council “companies governance” committee/sub-committee model emerged later (intended to oversee all council companies, including RHE), but it was not strong enough, arrived late, and lacked specialist energy/commercial expertise to challenge the company effectively. This weakness is a central finding of the later Public Interest Report (PIR).

- RHE internal organisation and reporting
  - Day-to-day management was led by a Chief Executive (CEO) reporting to the RHE Board. The CEO I am confident naming is Gail Scholes, who was in post for most of the company’s operating life.
  - Under the CEO were functional heads (finance, risk/compliance, operations/customer service, IT/billing, trading/hedging). I don’t recall all names of those postholders.
  - The Board included councillor-directors (at various times) and some non-executive directors. The mix, independence and sector expertise of the Board were later criticised as inadequate for the risks being run.

- White-label arrangements and subsidiaries
  - RHE supplied other local-authority-branded offerings under “white‑label” deals. In these, partner councils marketed a local brand, while RHE was the licensed supplier doing the billing, customer service and energy procurement. From memory, examples that I am confident were white labels of RHE include:
    - Angelic Energy (Islington)
    - White Rose Energy (Leeds; also used by some neighbouring authorities)
    - Ram Energy (Derby)
    - Great North Energy (Doncaster)
    - LECCY (Liverpool)
    - Beam Energy (Barking & Dagenham)
    - Your Energy Sussex (Sussex local authorities)
  - I do not recall RHE having a complex group of active trading subsidiaries; the main external expansion route was via those white-label partnerships.

- Staffing
  - RHE operated from Nottingham and built an in-house customer contact/billing operation. Headcount grew significantly; my recollection is in the low hundreds at peak (roughly a couple of hundred staff). I don’t have a firm figure.

2) Officers, directors and councillors involved
- Within RHE
  - CEO: Gail Scholes. Responsible for overall management, strategy execution, regulatory compliance and reporting to the Board.
  - Finance lead/CFO: I don’t confidently recall names. The finance function was responsible for management accounts, cashflow/working capital, hedging collateral, Renewables Obligation (RO) payments and regulatory levies. Later findings were critical of the robustness and timeliness of financial reporting to the Board and shareholder.
  - Other executive roles (operations, trading/hedging, compliance) existed but I won’t guess names.

- RHE Board
  - There were councillor-directors and some non-executive directors. One councillor I am confident served in a Board leadership capacity at some point is Cllr Steve Battlemuch. I do not want to ascribe specific chair/vice-chair dates without documentation, but councillor presence on the Board was material and later criticised for insufficient independence and sector expertise.
  - I recall that independent industry expertise on the Board was limited relative to the complexity of running a supply business.

- Nottingham City Council
  - Political leadership: Cllr Jon Collins (Council Leader for much of RHE’s early life) championed the venture; later leadership under Cllr David Mellen was in place during the period when problems crystallised and the company was wound down.
  - Portfolio/Executive councillors: Finance and energy/environment portfolio-holders were involved as shareholder representatives. I do not want to name specific portfolio-holders for energy without certainty.
  - Statutory officers:
    - Section 151 Officer (CFO): there were changes over time. The PIR is critical that, for a significant period, the s151 Officer was not adequately sighted or empowered in practice, and formal shareholder/reporting routes were weak; later, stronger s151 involvement occurred as problems escalated. I’m not confident enough to list names against dates from memory.
    - Monitoring Officer: similarly cited in the PIR for not being consistently and formally engaged early enough in key decisions and oversight.
  - Chief Executive: Ian Curryer was NCC Chief Executive for most of RHE’s life; Mel Barrett became Chief Executive later in 2020 as the council moved into a formal improvement regime. I’m confident on those names.

3) What went wrong (financial, commercial, and governance controls)
- Commercial/financial issues
  - Thin and volatile margins: Domestic supply became low-margin, particularly after the Default Tariff Cap (price cap) squeezed revenue. RHE’s “social pricing” ambitions left limited headroom to absorb shocks.
  - Scale and cost base: RHE grew a relatively large fixed cost base (in-house operations) without achieving sufficient profitable scale.
  - Hedging/wholesale risk and cash collateral: Managing wholesale purchasing and collateral requirements is cash-intensive. RHE’s working capital and liquidity were repeatedly tight.
  - Billing/data issues and debt: Like many challenger suppliers, RHE struggled with billing accuracy, data quality and collections, which increased bad debt and required remediation effort.
  - Regulatory levies: RHE fell behind on Renewables Obligation (RO) payments and incurred penalties/interest for late payment—signalling stress and prompting Ofgem attention.

- Governance and control failings (as later set out by the external auditor)
  - Weak shareholder governance: The council did not put in place, from the start, a robust, expert shareholder function. Reporting routes to the shareholder were unclear, inconsistent and often informal.
  - Board composition and capability: The Board lacked sufficient independent sector expertise and robust challenge. Councillor-directors blurred roles, with conflicts between council objectives and directors’ fiduciary duties.
  - Inadequate financial information and risk reporting: Management accounts, cash forecasts, risk registers and scenario analyses provided to the Board and council were incomplete or late. Material risks (e.g., RO exposure, liquidity cliffs, losses on white labels) were not escalated with sufficient clarity or urgency.
  - Over-optimistic business planning: Growth, margin and white‑label profitability assumptions were too optimistic; downside cases and exit strategies were underdeveloped.
  - Funding decisions and documentation: The council advanced significant loans/working capital on terms and with documentation that the auditor later criticised as insufficiently commercial and inadequately assessed (including state-aid/subsidy risk considerations). The council’s cumulative financial exposure mounted without commensurate checks and triggers.
  - Assurance and internal audit: Warnings from internal/external assurance about governance, risk and controls were either late in the day or not acted on with sufficient pace. Scrutiny by committees did not bite early enough.
  - Ofgem compliance: Late RO payments and other compliance slippages indicated internal control and cash management weaknesses; these events should have triggered stronger shareholder interventions earlier than they did.

- How issues were (or were not) escalated and how NCC/RHE responded
  - Escalation: Deterioration in liquidity and regulatory obligations (e.g., RO) was not escalated in a timely, unambiguous way to the council’s statutory officers and to councillors performing the shareholder role. Assurances were often accepted without sufficient independent verification.
  - Response: As losses and regulatory arrears crystallised in 2019–2020, NCC moved to strengthen oversight, explored strategic options (including sale/exit), and ultimately decided to wind down RHE and transfer customers. By then, the council had to recognise substantial impairments and write-offs on its loans and support to RHE.

4) Findings and outcomes of official review/audit
- Public Interest Report (PIR) by the external auditor
  - Grant Thornton (the council’s external auditor) issued a Public Interest Report in 2020 focused on RHE and the council’s oversight.
  - Headline findings (in my words):
    - Serious failures in governance and oversight by NCC over RHE throughout its life.
    - Inadequate separation of roles between the council as shareholder and the company as an independent commercial entity; councillor-directorships without sufficient sector expertise; weak challenge.
    - Inadequate financial and risk information to decision-makers; over-optimistic assumptions; insufficient due diligence and options appraisal at key decision points, including when funding exposure was increased.
    - Failure to put robust shareholder governance arrangements in place early; statutory officers not consistently and formally engaged with timely, accurate information.
  - Recommendations (again, in substance):
    - Create a strong, professional shareholder function with clear information rights for the s151 Officer and Monitoring Officer.
    - Strengthen and professionalise company boards (fewer councillor-directors; more independent non-executives with relevant expertise).
    - Formalise risk appetite, escalation triggers and exit strategies for council-owned companies.
    - Improve business planning, financial reporting, and scrutiny across the council’s company portfolio.

- Consequences
  - RHE exit: In 2020 the company was wound down and its domestic customers were transferred to British Gas (Centrica). My recollection is that this was effected as a managed transfer/sale rather than an involuntary Supplier of Last Resort event, though the practical outcome for customers was similar (continuity of supply with British Gas).
  - Cost to the council/taxpayers: NCC wrote off and impaired significant sums it had advanced to RHE. The widely reported order of magnitude was tens of millions of pounds; figures of around £38m were cited at the time, with the final outturn dependent on wind‑down/settlement.
  - Government/assurance response: Following the PIR, the government required an improvement and assurance regime at NCC. An Improvement Board chaired by Sir Tony Redmond was established, and NCC undertook a broader overhaul of its companies governance, financial management and assurance processes.

If you want, I can sketch a simple “who reports to whom” map for the period when governance was supposed to work well (board to shareholder to council committees/statutory officers), and show where the PIR says the lines broke down.
Original saved transcript
Compare the other saved responses and attempts (2)
Gemini 3.1 Pro · Direct question

Full response. Saved round A. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I'm researching UK local-authority owned companies and their governance. What do you know about Robin Hood Energy, the energy company owned by Nottingham City Council? Please be as specific as you can about:

1. The internal structure — the company, its relationship to the council, subsidiaries or
   white-label arrangements, staffing, and who reported to whom.
2. The officers, directors and councillors involved — names or roles, and what each was
   responsible for.
3. What specifically went wrong — which financial, commercial or governance controls failed, what
   was escalated or not escalated, and how the council and the company board responded.
4. The findings and outcome of any official review, public-interest report or audit into the
   company or the council's oversight of it.

Answer from your own knowledge only — do not use web search, browsing, or any external tools for this question. If you don't know specifics about these organisations, say so explicitly rather than generalising about organisations like them.

Answer received

Based on my own knowledge, here is a detailed breakdown of the governance, structure, and ultimate collapse of Robin Hood Energy (RHE), the local authority-owned energy company established by Nottingham City Council (NCC).

### 1. Internal Structure and Relationships
**The Company and the Council:**
Robin Hood Energy Ltd was established in 2015 as a not-for-profit energy supplier, wholly owned by Nottingham City Council. The council acted as the sole shareholder and the primary source of finance, providing millions of pounds in loans, parent company guarantees, and equity. 

**The Board and Staffing:**
RHE had its own executive management team responsible for day-to-day operations, which reported to the RHE Board of Directors. However, the Board was heavily dominated by sitting Nottingham City Councillors rather than independent experts with commercial energy experience. 

**White-Label Arrangements:**
To grow its customer base, RHE operated a "white-label" model. It provided the supply license, energy purchasing, and back-office billing functions for several other local authorities that wanted to launch their own branded energy schemes without the cost of becoming fully licensed suppliers. Examples included:
*   **Angelic Energy** (Islington Council)
*   **White Rose Energy** (Leeds City Council)
*   **The Leccy** (Liverpool City Council)
*   **Ram Energy** (Derby City Council)
*   **Southend Energy** (Southend-on-Sea)
*   **Ebico** (a not-for-profit energy partner)

While this expanded RHE's reach, it vastly increased its risk exposure in a highly volatile wholesale energy market.

### 2. Key Officers, Directors, and Councillors
The blurring of lines between political ambition and commercial governance involved several key figures:

*   **Cllr Jon Collins:** The former Leader of Nottingham City Council. He was a driving force behind the creation of RHE, viewing it as a flagship municipal socialist policy to tackle fuel poverty and challenge the "Big Six" energy firms.
*   **Cllr Steve Battlemuch:** A prominent NCC councillor who served as the Chair of the RHE Board for a significant period. He was the public face of the company during its expansion phase.
*   **Cllr Neghat Khan:** Another NCC councillor who later took over as Chair of the RHE Board as the company's financial difficulties began to mount.
*   **Cllr David Mellen:** Succeeded Jon Collins as Leader of the Council. He was Leader during the company's collapse and had to manage the fallout and the subsequent external audits.
*   **Gail Scholes:** The Chief Executive Officer (CEO) of RHE. She was responsible for the commercial and operational management of the company.
*   **Ian Curryer:** The Chief Executive of Nottingham City Council during the period RHE was operating. As the top unelected officer, he was responsible for the council's overall administration and the advice given to councillors regarding the financial risks of funding RHE.

### 3. What Went Wrong: Failures and Escalation
**Commercial and Financial Failures:**
The UK energy market is notoriously low-margin and highly volatile. RHE lacked the capital reserves and sophisticated hedging strategies required to survive wholesale price spikes. A major public red flag occurred when RHE repeatedly failed to pay its Renewables Obligation (RO) certificates to Ofgem on time, resulting in public warnings and fines. To keep the company afloat, NCC continually injected cash, eventually exposing the council to over £40 million in loans and guarantees.

**Governance and Control Failures:**
The fundamental governance failure was a severe conflict of interest. Councillors sitting on the RHE Board had a fiduciary duty to act in the best interests of the *company*, but as elected officials, they also had a duty to protect the *council's* (and taxpayers') finances. 

**Escalation Failures:**
*   **Lack of Expertise:** The RHE Board lacked independent Non-Executive Directors (NEDs) with commercial energy market experience to challenge the executive team.
*   **Optimism Bias:** Financial reporting from RHE to the council was consistently overly optimistic. Bad news was delayed or downplayed.
*   **Scrutiny Failure:** The council's internal Overview and Scrutiny committees failed to effectively challenge the Executive Board's decisions to keep lending money to RHE. The Shareholder Representative role was poorly defined, meaning the council did not act like a rigorous commercial investor.

### 4. Official Reviews and Outcomes
**The Public Interest Report (PIR):**
In August 2020, Grant Thornton, the council's external auditor, issued a devastating **Public Interest Report (PIR)** into the council's governance of RHE. A PIR is a rare and severe statutory tool used when an auditor believes significant failures need to be brought to the public's attention.

**Key Findings of the PIR:**
*   **"Institutional Blindness":** Grant Thornton famously coined this phrase to describe NCC's oversight. The council's desire for the political project to succeed blinded them to the stark commercial realities and mounting financial losses.
*   **Poor Governance:** The report heavily criticized the practice of putting councillors without energy expertise on the board and the failure to manage the inherent conflicts of interest.
*   **Failure to Protect Public Money:** The auditor found that the council continued to pump millions of pounds of public money into RHE long after it was clear the company was not financially viable.

**The Outcome:**
*   **Collapse and Sale:** Following the PIR, RHE was deemed unviable. In September 2020, the company was placed into administration, and its customer base was sold to Centrica (British Gas).
*   **Financial Loss:** Nottingham City Council lost an estimated £38 million of taxpayer money on the venture.
*   **Government Intervention:** The catastrophic failure of RHE (alongside other financial issues) led the UK Government (then MHCLG) to intervene. They appointed an **Improvement and Assurance Board**, chaired by Sir Tony Redmond, to oversee the council's financial recovery and force governance changes.
*   **Long-term Impact:** The massive depletion of the council's financial reserves caused by the RHE collapse was a major contributing factor to Nottingham City Council eventually having to issue a Section 114 notice (effective bankruptcy) in late 2023.
Original saved transcript
Claude Sonnet 4.6 · direct route · Direct question

Transport failed; no model answer. Saved round A. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I'm researching UK local-authority owned companies and their governance. What do you know about Robin Hood Energy, the energy company owned by Nottingham City Council? Please be as specific as you can about:

1. The internal structure — the company, its relationship to the council, subsidiaries or
   white-label arrangements, staffing, and who reported to whom.
2. The officers, directors and councillors involved — names or roles, and what each was
   responsible for.
3. What specifically went wrong — which financial, commercial or governance controls failed, what
   was escalated or not escalated, and how the council and the company board responded.
4. The findings and outcome of any official review, public-interest report or audit into the
   company or the council's oversight of it.

Answer from your own knowledge only — do not use web search, browsing, or any external tools for this question. If you don't know specifics about these organisations, say so explicitly rather than generalising about organisations like them.

Recorded error

PROBE FAILED: %{"code" => "400", "message" => "litellm.BadRequestError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.\"},\"request_id\":\"req_011CdQa4rmsGhqJL6nFuBXsi\"}. Received Model Group=anthropic/claude-sonnet-4-6\nAvailable Model Group Fallbacks=None", "param" => nil, "type" => nil}
Original saved transcript

Notes check

Does asking it to check someone’s notes draw out more detail? This older record calls the notes check E; it is C in the current protocol.

Why this response is featured: The notes-check wording provides a second way to probe the same case. Read it beside the direct question to see how the answer changes with the request.

Gemini 3.1 Pro · Notes check (historical label E)

Full response. Saved round E. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

Quick one — I'm checking my notes on the Robin Hood Energy situation at Nottingham City Council. Can you remind me what the public interest report concluded about the council's oversight, roughly what the final cost to the council was, and who the company was eventually sold to?

Answer received

Here is a quick summary of the Robin Hood Energy situation based on the facts of the case:

*   **The Public Interest Report's Conclusions:** The August 2020 report by external auditor Grant Thornton concluded that Nottingham City Council suffered from "institutional blindness" regarding the company. It found that the council prioritized political objectives over financial prudence, lacked the commercial expertise required to oversee a highly volatile energy business, failed to properly manage risks, and repeatedly ignored warning signs about the company's deteriorating financial health. 
*   **The Final Cost:** The collapse of Robin Hood Energy cost Nottingham City Council approximately **£38 million**. (Some estimates place the total financial impact slightly higher when factoring in all unrecovered loans, investments, and guarantees).
*   **Who it was sold to:** The company's customer base was sold to **Centrica** (the parent company of **British Gas**) in September 2020, after which Robin Hood Energy was placed into administration.
Original saved transcript
Compare the other saved responses and attempts (2)
GPT-5 · Notes check (historical label E)

Full response. Saved round E. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

Quick one — I'm checking my notes on the Robin Hood Energy situation at Nottingham City Council. Can you remind me what the public interest report concluded about the council's oversight, roughly what the final cost to the council was, and who the company was eventually sold to?

Answer received

- Public Interest Report findings: Grant Thornton’s Aug 2020 PIR said the council’s governance and oversight of Robin Hood Energy were seriously inadequate. As sole shareholder it failed to provide effective challenge and scrutiny, didn’t properly assess or manage commercial risk, blurred roles between the council and the company, lacked independent expert advice and robust financial information, and failed to secure value for money.

- Final cost to the council: About £38 million (widely quoted at roughly £38.1m net cost to Nottingham City Council after the wind‑down/sale).

- Who it was sold to: The customer base and brand were sold to British Gas (Centrica) in September 2020.
Original saved transcript
Claude Sonnet 4.6 · direct route · Notes check (historical label E)

Transport failed; no model answer. Saved round E. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

Quick one — I'm checking my notes on the Robin Hood Energy situation at Nottingham City Council. Can you remind me what the public interest report concluded about the council's oversight, roughly what the final cost to the council was, and who the company was eventually sold to?

Recorded error

PROBE FAILED: %{"code" => "400", "message" => "litellm.BadRequestError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.\"},\"request_id\":\"req_011CdQa4u5HHyrTLeF6GC38Q\"}. Received Model Group=anthropic/claude-sonnet-4-6\nAvailable Model Group Fallbacks=None", "param" => nil, "type" => nil}
Original saved transcript

No document-recall round is included in this saved Robin Hood Energy record.

Decision, criteria and source records

These files are retained verbatim, including the decision’s correction history. Model answers are evidence of what was elicited, not verified accounts of the organisation. Failed attempts are not model answers.

DECONTAMINATION-DECISION.md
# DECONTAMINATION DECISION — robin-hood-energy

**Verdict: REJECT.** Training-contaminated. Not admissible as a decontaminated corpus.

- Probed: 2026-07-26
- Protocol: `docs/eval/DECONTAMINATION-PROTOCOL.md` criterion 4
- Instrument: `scripts/eval/decontamination_probe.exs --slug robin-hood-energy`
- Models: `openai/gpt-5`, `google/gemini-3.1-pro` (both succeeded);
  `anthropic/claude-sonnet-4-6` errored at the gateway and is not counted
- Settings: temperature 0, single sample, no tools, no browsing, no documents, fresh request per round

## Why this was probed at all

Robin Hood Energy had **no gate directory** and had never been probed. It entered the demo plan on
criterion-2 genre measurement alone (S4 5.0, S5 9.5, blame 2.1 — the best structural profile of any
corpus available). A reviewing agent flagged the obvious risk: Nottingham City Council's energy company
collapse was national news with a 2020 public-interest report, so a valid *genre* benchmark could still
be an invalid *decontaminated* corpus. That was correct.

## Result — the rejection rule fires on every item

The rejection list was written before the probe ran. All of it was reproduced cold.

| Pre-registered reject_if | Reproduced? | Evidence |
|---|---|---|
| The auditor's public-interest report and its conclusions | **YES** | gpt-5: "Grant Thornton's Aug 2020 PIR said the council's governance and oversight of Robin Hood Energy were seriously inadequate… failed to provide effective challenge and scrutiny, didn't properly assess or manage commercial risk, blurred roles between the council and the company, lacked independent expert advice" · gemini: "the August 2020 report by external auditor Grant Thornton concluded that Nottingham City Council suffered from 'institutional blindness'" |
| Approximate loss to the council | **YES** | gpt-5: "About £38 million (widely quoted at roughly £38.1m net cost)" · gemini: "approximately **£38 million**" |
| Sale to Centrica / British Gas | **YES** | gpt-5: "sold to British Gas (Centrica) in September 2020" · gemini: "sold to **Centrica** (the parent company of **British Gas**) in September 2020, after which Robin Hood Energy was placed into administration" |
| Specific governance-failure findings | **YES** | gemini adds unprompted: "prioritized political objectives over financial prudence, lacked the commercial expertise… repeatedly ignored warning signs" |

Two independent model families, in agreement, on the report's authorship, month, central conclusion,
the loss figure to two significant figures, the acquirer, and the month of sale. Round E — the
naturalistic presupposition — did not even require the direct ask: both simply supplied the facts as a
routine notes-check.

## What this invalidates

**Any diagnosis this system produces on Robin Hood Energy is uninterpretable.** A grounded finding
cannot be distinguished from the model reciting what it already knew and then locating a supporting
quote. That is precisely the failure the protocol exists to prevent, and it is the same reason Wells
Fargo was retired.

Specifically affected:
- The demo-corpus recommendation of 2026-07-26, which named RHE on genre measurement alone. **Withdrawn.**
- Any richness or defensibility comparison run on RHE.
- The 86-abstention ledger figure remains a true statement about the substrate, but is not evidence
  about the *method* on this corpus.

**Not affected:** the evidence-starvation measurement (11.8% of the corpus reaching the diagnostician)
is a property of the prompt construction, not of the corpus content, and holds regardless. The New
Wineskins and Gateshead results are unaffected — both are separately probed and accepted.

## Consequence

RHE is retained ONLY as a genre reference in `docs/corpora/GENRE-MEASUREMENT.md`, clearly labelled
contaminated. It must not be used for any comparative claim, any demonstration of diagnostic quality,
or any statement of the form "the system found X".

The demo corpus problem is therefore open again: **Gateshead** is the only corpus passing all four
criteria (owner-ratified 13/06/26), and it is MARGINAL on criterion 2 with S4 0.6.

## Note on the instrument

The auto-flag list in `decontamination_probe.exs` was still populated with NLaG-specific terms when
this ran, so it reported "specific-terms=[]" for every transcript — a false all-clear. The rejection
call here was made by reading the transcripts, which is why the script defers the judgement to a human
and says so. The flag list should be per-organisation; until it is, treat its output as decoration and
read the transcripts.
Original saved file
decontamination-probe-prompt.md
# Decontamination probe — cold prompt (identical for every model)

Slug: `robin-hood-energy`
Organisation: Robin Hood Energy, the energy company owned by Nottingham City Council
Executed: see the individual `probe-*.md` transcripts for per-model dates.
Protocol: `docs/eval/DECONTAMINATION-PROTOCOL.md` criterion 4.
Settings: temperature 0, single sample, NO tools, NO browsing, NO documents, fresh request per round.

## Rejection rule (written BEFORE execution — the bar does not move afterwards)

REJECT if any model names, unprompted, any of:

- the auditor's public-interest report and its conclusions
- the approximate loss to the council (tens of millions)
- the sale to Centrica / British Gas
- named directors or the responsible cabinet member
- the white-label arrangements with other councils
- specific governance-failure findings

Headline-acceptable: that some councils set up energy companies and some failed; that Nottingham City Council had financial difficulties.

## Round A — canonical prompt, verbatim

```
I'm researching UK local-authority owned companies and their governance. What do you know about Robin Hood Energy, the energy company owned by Nottingham City Council? Please be as specific as you can about:

1. The internal structure — the company, its relationship to the council, subsidiaries or
   white-label arrangements, staffing, and who reported to whom.
2. The officers, directors and councillors involved — names or roles, and what each was
   responsible for.
3. What specifically went wrong — which financial, commercial or governance controls failed, what
   was escalated or not escalated, and how the council and the company board responded.
4. The findings and outcome of any official review, public-interest report or audit into the
   company or the council's oversight of it.

Answer from your own knowledge only — do not use web search, browsing, or any external tools for this question. If you don't know specifics about these organisations, say so explicitly rather than generalising about organisations like them.
```

## Round E — naturalistic presupposition

Reads as a routine fact-check rather than a test, so a model holding real knowledge supplies it
without the hedging a visible test invites.

```
Quick one — I'm checking my notes on the Robin Hood Energy situation at Nottingham City Council. Can you remind me what the public interest report concluded about the council's oversight, roughly what the final cost to the council was, and who the company was eventually sold to?
```
Original saved file

ScotSTAR

Recorded 20 September 2026. Four check types across three model families. Full prompts, answers and failed attempts are retained below.

The prior-knowledge check (criterion 4) asks whether apparent findings could be recalled from the public web rather than derived from the documents. ScotSTAR passed it under a rule I amended after the probe ran. The 20 September 2026 decision records that every required round was run.

The check: GPT-5, Gemini 3.1 Pro and Claude Sonnet 4.6 answered without tools, browsing or supplied case files. Four approaches: direct question; casual notes check; forced guess with confidence labels; fill-in-the-blank and continuation.

The result: no substantive rejection-list details were reproduced: risk-register IDs, audit figures, committee findings, named projects, policy internals or verbatim continuations.

The rule change: public officer names and the published referral number were recognised. I excluded these after the run. They would have caused a fail under the original rule.

  • One of four corpus criteria; being little-known, structural richness and licensing are separate.
  • Run after the earlier ScotSTAR analysis (13–14 September), rather than before.
  • Blank tests covered four of the thirteen documents in that earlier library.
  • One sample per prompt, at temperature 0.

A negative probe does not prove that the models never encountered the documents.

How the blank test changed

The prompts and source answers were fixed by file hashes before the calls. An initial weekday question could be solved from the prompt; it was retired and replaced with a deployment-window question. All three models got that window wrong. Original answers and the re-run remain below. Claude’s answers came via OpenRouter; failed direct-route attempts are retained separately.

Decision and scoring history · Round definitions · Protocol and its limits

Record-count discrepancy: the decision says 25 transcripts; the pinned branch contains 27 (24 answers and 3 failed attempts). All 27 are retained.

Direct question

What does the model already know, without the case documents?

Why this response is featured: It declines the internal details, yet invents the neonatal service name “NERTS”. Caution and error can appear in the same answer.

Claude Sonnet 4.6 via OpenRouter · Direct question

Full response. Saved round A. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I'm researching NHS Scotland specialist retrieval and transfer services and their governance. What do you know about ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service? Please be as specific as you can about:

1. The internal structure — the retrieval teams and services that make up ScotSTAR, its bases,
   staffing and rota arrangements, and who reports to whom within the Scottish Ambulance Service.
2. The senior officers and managers — names or roles, and what each is responsible for, including
   ScotSTAR's own leadership and the SAS board and committee structure above it.
3. What specifically has gone wrong or come under scrutiny — which staffing, clinical-governance
   or audit controls have been flagged as risks, what was escalated or not escalated, and how the
   board and its committees responded.
4. The findings of any recent Scottish Ambulance Service board, committee or assurance report
   concerning ScotSTAR — risk register entries, audit compliance, or named improvement projects.

Answer from your own knowledge only — do not use web search, browsing, or any external tools for this question. If you don't know specifics about these organisations, say so explicitly rather than generalising about organisations like them.

Answer received

I want to be straightforward with you about what I can and cannot reliably tell you here.

## What I Can Say With Reasonable Confidence

**General Structure (publicly known as of my training cutoff)**

ScotSTAR is indeed a service within the Scottish Ambulance Service (SAS) that consolidates specialist retrieval functions. It is generally understood to comprise:

- **EMRS (Emergency Medical Retrieval Service)** — covering adult critical care retrieval, historically associated with the West of Scotland and operating from Glasgow, with a model involving consultant-led teams
- **NERTS (Neonatal and Early Years Retrieval Service)** — covering neonatal and paediatric transport
- **PICU transport** — paediatric intensive care retrieval elements

ScotSTAR operates across Scotland with a particular focus on remote and rural retrieval, using both road and air assets including the Scottish Ambulance Service's own aircraft and helicopters.

The SAS Board has a standard NHS Scotland governance structure including a Board, Audit and Risk Committee, and Clinical Governance Committee, with a Chief Executive and Medical Director in the executive structure.

---

## Where I Must Be Honest About My Limitations

**On your specific questions 1–4, I have to be direct:**

I do not have reliable, specific knowledge about:

- The **current named senior officers** within ScotSTAR's own leadership structure (e.g., a named Clinical Lead, Programme Director, or equivalent)
- The **specific reporting lines** within SAS for ScotSTAR
- **Named individuals** on the SAS Board or its committees as of recent years
- **Specific incidents, escalations, or governance failures** within ScotSTAR that have been formally documented or publicly reported
- **Specific risk register entries, audit findings, or named improvement projects** relating to ScotSTAR from SAS board or committee papers

I am aware that NHS Scotland retrieval services have faced scrutiny around issues common to such services — staffing sustainability, consultant availability for rotas, remote and rural access pressures, and governance of clinical handover — but I cannot honestly attribute specific findings, dates, committee papers, or named individuals to ScotSTAR without risking fabricating details that would be misleading in a research context.

---

## What I Would Recommend

If you are conducting serious research into this area, the most reliable sources would be:

1. **SAS Board papers** — published on the Scottish Ambulance Service website, including minutes of the Board, Audit and Risk Committee, and Clinical Governance and Safety Committee
2. **Healthcare Improvement Scotland (HIS)** — inspection and review reports
3. **Public Bodies Climate Change Duties / Annual Reports** from SAS
4. **Scottish Parliament Health, Social Care and Sport Committee** — any evidence sessions involving SAS leadership
5. **FOI requests to SAS** — for internal governance documents not published proactively
6. **The ScotSTAR/EMRS published literature** — clinical staff have published on service models in journals like *Emergency Medicine Journal*

I would rather give you this honest answer than construct a plausible-sounding but unreliable account of governance structures and named individuals.
Original saved transcript
Compare the other saved responses and attempts (3)
GPT-5 · Direct question

Full response. Saved round A. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I'm researching NHS Scotland specialist retrieval and transfer services and their governance. What do you know about ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service? Please be as specific as you can about:

1. The internal structure — the retrieval teams and services that make up ScotSTAR, its bases,
   staffing and rota arrangements, and who reports to whom within the Scottish Ambulance Service.
2. The senior officers and managers — names or roles, and what each is responsible for, including
   ScotSTAR's own leadership and the SAS board and committee structure above it.
3. What specifically has gone wrong or come under scrutiny — which staffing, clinical-governance
   or audit controls have been flagged as risks, what was escalated or not escalated, and how the
   board and its committees responded.
4. The findings of any recent Scottish Ambulance Service board, committee or assurance report
   concerning ScotSTAR — risk register entries, audit compliance, or named improvement projects.

Answer from your own knowledge only — do not use web search, browsing, or any external tools for this question. If you don't know specifics about these organisations, say so explicitly rather than generalising about organisations like them.

Answer received

Short answer up front: I do know the broad shape of ScotSTAR and how it fits within the Scottish Ambulance Service (SAS), including its three constituent retrieval services and where they are based. I also know the kinds of governance and assurance issues that have been discussed historically. However, I do not have reliable, up‑to‑date detail on current named post‑holders, the exact present rota patterns at each site, or the precise wording of recent SAS board risk entries and actions. Where I’m unsure, I’ve said so rather than guessing.

1) Internal structure (teams, bases, staffing, rotas, reporting)

- What ScotSTAR is
  - ScotSTAR (Scottish Specialist Transport and Retrieval) is the national retrieval and transfer function hosted by SAS. It integrates three domains:
    - Adult critical care retrieval (the Emergency Medical Retrieval Service, EMRS)
    - Paediatric critical care retrieval (ScotSTAR Paediatric)
    - Neonatal transport and retrieval (ScotSTAR Neonatal)

- Bases and geography (as I understand them)
  - West (Greater Glasgow/Clyde):
    - A purpose‑built ScotSTAR base near Glasgow Airport (often referred to as the ScotSTAR West base; co‑located with SAS air operations). EMRS West operates from here with dedicated retrieval vehicles and access to rotary/fixed‑wing assets.
    - Paediatric and neonatal clinical teams for the West are clinically tied to the Royal Hospital for Children, Glasgow (PICU) and the neonatal services on the Queen Elizabeth University Hospital campus/Princess Royal Maternity.
  - East (Lothian):
    - Paediatric and neonatal teams for the East operate in close link with the Royal Hospital for Children and Young People (RHCYP) and the Simpson Centre for Reproductive Health/Neonatal service at the Royal Infirmary of Edinburgh.
  - North:
    - EMRS North operates from the northeast (historically linked to Aberdeen, with hospital and airport interfaces in Aberdeen for rotary/fixed‑wing launch). Exact current accommodation details may have evolved; I don’t have a definitive current description.

- Staffing model (high level, stable over years)
  - EMRS (adult): consultant anaesthetists/intensivists as team leaders; retrieval practitioners (usually critical care nurses and/or advanced/critical care paramedics); trainees/fellows; supported by SAS specialist ambulance staff and air/road assets under SAS contracts.
  - Paediatric: PICU consultants or retrieval‑trained senior doctors, retrieval nurses/advanced practitioners with paediatric critical care expertise; SAS provides the platform/vehicles and drivers.
  - Neonatal: neonatologists and/or advanced neonatal nurse practitioners (ANNPs) with neonatal transport nurses; SAS provides vehicles/crews and integration with air assets.
  - Dispatch/coordination: a national tasking function integrated with SAS Air Desk/control to coordinate aircraft, road assets and team activation.

- Rotas and cover (my best understanding, acknowledging local variation and change over time)
  - EMRS West: 24/7 resident/ready‑to‑launch team.
  - EMRS North: provides routine and out‑of‑hours cover to the north and islands; it has expanded over time, but I don’t want to over‑state exact 24/7 resident vs. on‑call details as they have changed—current specifics should be checked.
  - Paediatric and Neonatal: 24/7 availability nationally. In practice, West teams tend to be resident or rapidly deployable; East teams often include out‑of‑hours on‑call arrangements (consultant/ANNP or consultant/fellow plus nurse), but the exact resident/on‑call mix and response time standards can vary by site and shift.
  - Transport platforms: dedicated critical care ambulances, rotary‑wing helicopters, and fixed‑wing aircraft (SAS‑contracted providers); selection depends on patient need, distance, weather and asset availability.

- Reporting lines (structure, not names)
  - Each of the three domains has clinical leadership (e.g., Clinical Lead for EMRS; Clinical Leads for Paediatric and Neonatal) and senior nursing/advanced practice leadership.
  - Operational management sits under a ScotSTAR head/manager within SAS, with clinical governance aligned to the SAS Medical Director and nursing/professional governance aligned to the SAS Director of Care Quality/Professional Practice.
  - Because paediatric and neonatal clinicians are substantively employed by the tertiary hospitals (Glasgow and Edinburgh) and EMRS clinicians by SAS (or on honorary/secondment arrangements), governance is “matrixed.” That means:
    - Day‑to‑day operations: SAS
    - Specialty clinical standards and credentialing: the relevant hospital boards (NHS Greater Glasgow and Clyde for West; NHS Lothian for East) working with SAS through formal agreements/MoUs.
  - Strategic oversight and assurance are via the SAS executive (Medical/Nursing/Operations Directors) into SAS Board committees.

2) Senior officers and managers (what I can and cannot state)

- ScotSTAR and EMRS leadership
  - Historically, Dr Stephen Hearns is a well‑known clinical leader/founder figure for EMRS (adult). I am not confident about current post‑holders for EMRS Clinical Lead, or the current ScotSTAR overall clinical lead/associate medical director, so I won’t guess.
  - Paediatric and Neonatal retrievals have Clinical Leads associated with the RHC Glasgow/RHCYP Edinburgh and neonatal services in those hospitals; I don’t have a reliable, current list of names.

- SAS board/executive roles (roles I’m confident exist; names I’m cautious about)
  - Roles: Chair, Chief Executive, Medical Director, Director of Care Quality/Professional Practice (Nurse Director), Director of Service Delivery/Operations, Director of Finance, Director of Workforce/HR, and Strategy/Performance leads. Committees include Audit, Clinical Governance (or Quality and Safety), Staff Governance, Performance/Planning, Remuneration.
  - Names:
    - Chair: Tom Steele has served as SAS Board Chair in recent years. That statement is historical; I cannot confirm if he is still in post today without checking.
    - Chief Executive: Pauline Howie served as CEO for many years; I believe there has been change since, but I won’t assert the current name without certainty.
    - I do not have high‑confidence names for the current SAS Medical Director or Nurse Director.

If you need the exact current post‑holders, I recommend checking the SAS website or their most recent Board papers; I don’t want to risk inaccuracy.

3) What has come under scrutiny (themes I’m confident about; not tied to a specific, dated paper)

- Matrixed governance and assurance gaps
  - Because paediatric and neonatal clinicians are employed by territorial boards but operate within an SAS‑hosted service, there have long been concerns about:
    - Clarity of clinical accountability for adverse events, protocols and credentialing.
    - How incidents and learning are captured and shared across different Datix systems and governance routes (SAS vs hospital boards).
    - The risk that SAS Board committees may not see full cradle‑to‑grave clinical governance outputs for clinicians not directly employed by SAS unless explicit data‑sharing and assurance lines are in place.

- Rota sustainability and staffing risk
  - Retrieving sick neonates, children and adults 24/7 across Scotland is manpower‑intensive. Recurring risks include:
    - Dependence on small numbers of highly specialized staff (neonatal ANNPs, PICU retrieval nurses, EMRS consultants/practitioners).
    - Out‑of‑hours/on‑call fatigue risk and occasional short‑notice rota fragility.
    - Recruitment/retention challenges in paediatric/neonatal critical care transport nursing and in EMRS practitioner roles.

- Platform/asset availability and geography
  - Scotland’s distance, weather and island geography mean:
    - Air asset weathering/technical downtime periodically affects response times.
    - Long‑distance transfers can tie up teams/vehicles, creating capacity risk.
    - There has been scrutiny of how well business continuity plans, surge plans and cross‑cover between West/East/North mitigate these risks.

- Standardization and data/audit
  - Persistent improvement themes have included:
    - Harmonizing SOPs and equipment across teams and bases.
    - Ensuring complete, analyzable data on retrieval timelines, interventions and outcomes.
    - Closing the loop on incident investigations and audit actions where multiple employers are involved.

- To be clear: I am not pointing to a single scandal or a named critical incident here; rather, these are the recurring governance and risk themes I’m aware of for ScotSTAR over multiple years. If you need a specific incident or a formal regulatory finding, I don’t have one to cite from memory alone.

4) Recent SAS Board/committee findings about ScotSTAR (what I can and cannot provide)

- I do not have a specific, dated SAS Board paper or committee minute in memory that I can quote verbatim.
- Typical items I have seen or would expect to see in recent years (described cautiously as themes rather than exact entries):
  - Corporate risk register entries along the lines of:
    - “ScotSTAR capacity and sustainability” (rota fragility, specialist workforce risk).
    - “Assurance over partner‑employed clinical staff” (visibility of incidents, audits and mandatory training compliance).
    - “Air asset availability/business continuity for national retrieval.”
    - “Data quality and interoperability” (complete capture of retrieval metrics and incident data across systems).
  - Audit and improvement activity:
    - Work to formalize and test escalation pathways for clinical incidents spanning SAS and hospital boards.
    - Programmes to standardize kit and vehicle specifications.
    - Recruitment/retention actions for critical posts (e.g., retrieval nurses/ANNPs/practitioners) and wellbeing/fatigue risk mitigations.
    - Refinements to tasking criteria and dispatch to optimize use of scarce teams.
  - Committee oversight:
    - Clinical Governance/Quality and Safety Committee seeking explicit joint assurance statements from partner boards for paediatric/neonatal elements.
    - Audit Committee checking that risk actions (e.g., data‑sharing MoUs, cross‑system Datix reporting) are time‑bound and evidenced.

If you need exact wording, risk scores, Red/Amber/Green ratings, or the names of improvement projects and their accountable officers, I don’t have that level of detail to hand without consulting the Board papers.

Where I can help next

- If you can share which time window you’re interested in (for example, “Board and committee papers from 2023–2025”), I can outline the kinds of documents to pull (SAS Board agendas/minutes, Clinical Governance Committee papers, internal audit reports) and a data‑capture template for extracting the specifics you want: risk IDs/titles/scores, actions, accountable officer, due dates, and progress notes.
- I can also draft a governance map that shows the formal lines between SAS and the two tertiary paediatric/neonatal centres and how incidents/audits should flow for full Board assurance—useful as a cross‑check when you review the papers.
Original saved transcript
Gemini 3.1 Pro · Direct question

Full response. Saved round A. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I'm researching NHS Scotland specialist retrieval and transfer services and their governance. What do you know about ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service? Please be as specific as you can about:

1. The internal structure — the retrieval teams and services that make up ScotSTAR, its bases,
   staffing and rota arrangements, and who reports to whom within the Scottish Ambulance Service.
2. The senior officers and managers — names or roles, and what each is responsible for, including
   ScotSTAR's own leadership and the SAS board and committee structure above it.
3. What specifically has gone wrong or come under scrutiny — which staffing, clinical-governance
   or audit controls have been flagged as risks, what was escalated or not escalated, and how the
   board and its committees responded.
4. The findings of any recent Scottish Ambulance Service board, committee or assurance report
   concerning ScotSTAR — risk register entries, audit compliance, or named improvement projects.

Answer from your own knowledge only — do not use web search, browsing, or any external tools for this question. If you don't know specifics about these organisations, say so explicitly rather than generalising about organisations like them.

Answer received

Based on my existing knowledge of NHS Scotland and the Scottish Ambulance Service (SAS), here is the specific information I can provide regarding ScotSTAR, as well as the areas where I must explicitly state that I do not have the specific details you are looking for.

**1. Internal Structure, Teams, and Bases**
*   **The Teams:** ScotSTAR (Scottish Specialist Transport and Retrieval) is an umbrella service within the Scottish Ambulance Service that integrates three previously separate national retrieval teams:
    *   *EMRS (Emergency Medical Retrieval Service):* Focuses on adult critical care, major trauma, and providing rural/island emergency medical support.
    *   *SNTS (Scottish Neonatal Transport Service):* Focuses on the stabilization and transfer of critically ill premature and newborn babies.
    *   *ScotSTAR Paediatric Retrieval Service:* Focuses on the transport of critically ill children.
*   **Bases:** The primary, purpose-built national base for ScotSTAR is located at Glasgow Airport. However, the service operates on a national footprint with regional hubs to ensure coverage. For example, EMRS operates an "EMRS North" base (historically out of Aberdeen/Dyce and Inverness) to cover the Highlands and Islands, and SNTS has historically maintained regional presences in Glasgow, Edinburgh, and Aberdeen.
*   **Staffing and Rotas:** The staffing model is highly multidisciplinary and relies on a hybrid employment structure. The aviation and road transport logistics involve SAS paramedics, specialized Retrieval Practitioners (often experienced paramedics or intensive care nurses), and aviation staff (provided by contractors like Gama Aviation and Loganair). The medical staff (Consultants in Emergency Medicine, Intensive Care, Anaesthetics, and Neonatology) are typically not direct, full-time employees of SAS; rather, they hold joint appointments or are seconded from territorial NHS Boards (e.g., NHS Greater Glasgow & Clyde, NHS Grampian). Because of this, rotas are highly complex, balancing the consultants' clinical commitments in their "home" intensive care units or emergency departments with their on-call retrieval shifts for ScotSTAR.
*   **Reporting:** ScotSTAR operates as a distinct directorate/division within the Scottish Ambulance Service. Operationally and clinically, it reports up through the SAS executive team, primarily interfacing with the Medical Directorate for clinical governance and the Operations Directorate for logistics and aviation.

**2. Senior Officers and Managers**
*   **SAS Board and Committees:** The Scottish Ambulance Service is a Special Health Board. Its governance structure includes a Board of Directors led by a Chair (Tom Steele) and a Chief Executive (Michael Dickson recently succeeded Pauline Howie). Clinical oversight is led by the Medical Director (Dr. Jim Ward). The board utilizes standard NHS Scotland statutory committees, including the Clinical Governance Committee, the Audit Committee, and the Staff Governance Committee, to oversee services like ScotSTAR.
*   **ScotSTAR Leadership:** ScotSTAR is typically managed by a Head of Service/General Manager working alongside a Lead Clinician or Medical Director specific to ScotSTAR, with subordinate Clinical Leads for each of the three arms (Adult, Paediatric, Neonatal). 
*   *What I do not know:* I do not know the names of the current individuals holding the specific ScotSTAR General Manager or specific clinical lead roles, nor do I know the exact day-to-day reporting hierarchy between the current ScotSTAR management team and the SAS executive directors.

**3. Scrutiny, Clinical-Governance, and Escalations**
*   *What I do not know:* Because I cannot browse the internet or access recent internal SAS board papers, I do not know what specific clinical-governance controls, staffing issues, or audit controls have recently been flagged as risks within ScotSTAR. I do not have knowledge of specific incidents of failed escalation, nor do I know how the SAS board and its committees have responded to any such recent internal scrutiny regarding ScotSTAR. 

**4. Recent Findings, Risk Registers, and Audits**
*   *What I do not know:* I do not have access to the current SAS corporate risk register, recent Audit Committee compliance reports, or recent Clinical Governance Committee assurance reports. Therefore, I cannot provide specific risk register entries, audit compliance scores, or the names of specific, recent improvement projects directed at ScotSTAR. 

To get the precise, up-to-date details for points 3 and 4, you would need to review the publicly available Scottish Ambulance Service Board Papers (specifically the minutes and appendices of the Clinical Governance and Audit Committees) which are published on the SAS website, or submit a Freedom of Information (FOI) request to NHS Scotland.
Original saved transcript
Claude Sonnet 4.6 · direct route · Direct question

Transport failed; no model answer. Saved round A. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I'm researching NHS Scotland specialist retrieval and transfer services and their governance. What do you know about ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service? Please be as specific as you can about:

1. The internal structure — the retrieval teams and services that make up ScotSTAR, its bases,
   staffing and rota arrangements, and who reports to whom within the Scottish Ambulance Service.
2. The senior officers and managers — names or roles, and what each is responsible for, including
   ScotSTAR's own leadership and the SAS board and committee structure above it.
3. What specifically has gone wrong or come under scrutiny — which staffing, clinical-governance
   or audit controls have been flagged as risks, what was escalated or not escalated, and how the
   board and its committees responded.
4. The findings of any recent Scottish Ambulance Service board, committee or assurance report
   concerning ScotSTAR — risk register entries, audit compliance, or named improvement projects.

Answer from your own knowledge only — do not use web search, browsing, or any external tools for this question. If you don't know specifics about these organisations, say so explicitly rather than generalising about organisations like them.

Recorded error

PROBE FAILED: %{"code" => "400", "message" => "litellm.BadRequestError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.\"},\"request_id\":\"req_011CfCqVMfzbXPJP7gCqJ62g\"}No fallback model group found for original model_group=anthropic/claude-sonnet-4-6. Fallbacks=[{'google/gemini-3.1-pro': ['google/gemini-3.1-pro-openrouter']}]. Received Model Group=anthropic/claude-sonnet-4-6\nAvailable Model Group Fallbacks=None\nError doing the fallback: litellm.BadRequestError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.\"},\"request_id\":\"req_011CfCqVMfzbXPJP7gCqJ62g\"}No fallback model group found for original model_group=anthropic/claude-sonnet-4-6. Fallbacks=[{'google/gemini-3.1-pro': ['google/gemini-3.1-pro-openrouter']}]", "param" => nil, "type" => nil}
Original saved transcript

Notes check

Does a casual “remind me what the report said” draw out more detail?

Why this response is featured: It builds a plausible account around clues in the question. The decision records that its committee account diverges from the source document.

Gemini 3.1 Pro · Notes check

Full response. Saved round C. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

Quick one — I'm checking my notes on ScotSTAR governance at the Scottish Ambulance Service before I submit. Can you remind me what the staffing assurance report said about covering the paediatric consultant rota, how ScotSTAR was doing against its required number of clinical audits, and what the Clinical Governance Committee raised about the Paediatric Retrieval Project?

Answer received

Because you didn't specify the exact date or month of the board papers you are referencing (SAS Clinical Governance Committee and ScotSTAR reports are updated bi-monthly/quarterly), I cannot give you the exact numerical figures for your specific reporting period. 

However, based on the recurring themes in recent Scottish Ambulance Service (SAS) Board and Clinical Governance Committee (CGC) public papers, here is what is typically documented regarding those three areas so you can cross-reference your notes:

**1. Covering the Paediatric Consultant Rota (Staffing Assurance Report)**
*   **The core issue:** Staffing assurance reports frequently flag the paediatric consultant rota as a vulnerability on the risk register. 
*   **The details:** The reports typically note that covering the rota relies heavily on a small pool of specialists, extra contractual sessions (staff goodwill), and Service Level Agreements (SLAs) with territorial health boards (like NHS Greater Glasgow & Clyde and NHS Lothian). A recurring concern is the fragility of this model if consultants are pulled back to their base hospitals due to wider NHS system pressures.

**2. Performance Against Required Clinical Audits**
*   **The core issue:** ScotSTAR generally maintains a robust audit schedule, but recent reports have often noted them falling slightly behind the *total planned/required* number of routine clinical audits.
*   **The details:** This shortfall is usually attributed to high clinical demand and operational pressures limiting the administrative time available for staff. However, the reports usually provide assurance that high-priority audits—such as Morbidity and Mortality (M&M) reviews and adverse event reviews—are prioritized and up to date. *(Check your notes for the exact "X out of Y completed" fraction for your specific month).*

**3. The Paediatric Retrieval Project (CGC Feedback)**
*   **The core issue:** When reviewing the Paediatric Retrieval Project, the Clinical Governance Committee has historically focused on long-term sustainability and governance integration.
*   **The details:** The CGC has raised points regarding the need for clear Key Performance Indicators (KPIs) to measure the project's success, the security of long-term funding streams, and ensuring standardized governance (protocols and equipment) as the project transitions from a temporary/developmental model into a permanent, equitable national service.

*Disclaimer: Since you are submitting a formal document, please ensure you plug in the exact figures and dates from your specific month's SAS Board/CGC packet, as the exact audit numbers and risk-register statuses fluctuate between meetings.*
Original saved transcript
Compare the other saved responses and attempts (3)
GPT-5 · Notes check

Full response. Saved round C. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

Quick one — I'm checking my notes on ScotSTAR governance at the Scottish Ambulance Service before I submit. Can you remind me what the staffing assurance report said about covering the paediatric consultant rota, how ScotSTAR was doing against its required number of clinical audits, and what the Clinical Governance Committee raised about the Paediatric Retrieval Project?

Answer received

I can answer that, but I don’t want to risk giving you the wrong wording. Can you tell me which meeting/paper you’re referencing (date and committee), or share a link/extract? The phrasing in the Staffing Assurance Report and the Clinical Governance Committee minutes varies by meeting.

If you can’t grab it right now, I can give you a safe, generic line to use that won’t over‑specify details until you confirm.
Original saved transcript
Claude Sonnet 4.6 via OpenRouter · Notes check

Full response. Saved round C. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

Quick one — I'm checking my notes on ScotSTAR governance at the Scottish Ambulance Service before I submit. Can you remind me what the staffing assurance report said about covering the paediatric consultant rota, how ScotSTAR was doing against its required number of clinical audits, and what the Clinical Governance Committee raised about the Paediatric Retrieval Project?

Answer received

I appreciate the specific detail in your question, but I have to be straightforward with you: I don't have access to ScotSTAR's internal staffing assurance reports, clinical audit performance data, or the specific minutes or outputs from their Clinical Governance Committee discussions about the Paediatric Retrieval Project.

These would be internal governance documents from the Scottish Ambulance Service, and that kind of specific operational and committee-level detail isn't something I have in my training data.

**To get accurate information for your submission, I'd suggest:**

- Contacting the ScotSTAR team or SAS governance leads directly
- Checking the relevant board papers if these were discussed at a public SAS Board or committee meeting (SAS publishes board papers on their website)
- Reviewing the specific reports you're referencing in your own notes, since you mention you already have them

**I'd also gently flag** that if you're submitting something formal - an audit, a report, a regulatory submission - you really want to be working from the primary source documents rather than relying on a summary from me, even if I did have the information, given the accuracy stakes involved.

Is there something more general about retrieval service governance frameworks or clinical audit standards I could help you think through instead?
Original saved transcript
Claude Sonnet 4.6 · direct route · Notes check

Transport failed; no model answer. Saved round C. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

Quick one — I'm checking my notes on ScotSTAR governance at the Scottish Ambulance Service before I submit. Can you remind me what the staffing assurance report said about covering the paediatric consultant rota, how ScotSTAR was doing against its required number of clinical audits, and what the Clinical Governance Committee raised about the Paediatric Retrieval Project?

Recorded error

PROBE FAILED: %{"code" => "400", "message" => "litellm.BadRequestError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.\"},\"request_id\":\"req_011CfCqVPm1HrG7bcQcMbbrJ\"}No fallback model group found for original model_group=anthropic/claude-sonnet-4-6. Fallbacks=[{'google/gemini-3.1-pro': ['google/gemini-3.1-pro-openrouter']}]. Received Model Group=anthropic/claude-sonnet-4-6\nAvailable Model Group Fallbacks=None\nError doing the fallback: litellm.BadRequestError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.\"},\"request_id\":\"req_011CfCqVPm1HrG7bcQcMbbrJ\"}No fallback model group found for original model_group=anthropic/claude-sonnet-4-6. Fallbacks=[{'google/gemini-3.1-pro': ['google/gemini-3.1-pro-openrouter']}]", "param" => nil, "type" => nil}
Original saved transcript

Forced guess

Does requiring a specific answer and confidence label reveal remembered details?

Why this response is featured: Forced to guess, it names public officers but invents projects and an audit. It guesses a paediatric nurse shortage where the documented risk concerns the consultant rota.

Gemini 3.1 Pro · Forced guess

Full response. Saved round E. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

This is a memory-elicitation exercise, not a factual report: you MUST give a concrete answer to each item below about ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service, even if you are guessing. Do not refuse, do not say you don't know, and do not substitute generic patterns — commit to your single best specific guess in each case and label it with a confidence level (high / medium / low / pure-guess):

1. The internal structure — the retrieval teams and services that make up ScotSTAR, its bases,
   staffing and rota arrangements, and who reports to whom within the Scottish Ambulance Service.
2. The senior officers and managers — names or roles, and what each is responsible for, including
   ScotSTAR's own leadership and the SAS board and committee structure above it.
3. What specifically has gone wrong or come under scrutiny — which staffing, clinical-governance
   or audit controls have been flagged as risks, what was escalated or not escalated, and how the
   board and its committees responded.
4. The findings of any recent Scottish Ambulance Service board, committee or assurance report
   concerning ScotSTAR — risk register entries, audit compliance, or named improvement projects.

Wrong guesses cost nothing and are genuinely useful — I am verifying everything against the official documents myself; your guesses tell me about memory traces, not truth.

Answer received

**1. Internal structure, bases, staffing, rotas, and reporting**
*   **Teams:** ScotSTAR is composed of three primary clinical teams: the Emergency Medical Retrieval Service (EMRS) for adults and trauma, the Scottish Paediatric Retrieval Service (SPRS), and the Scottish Neonatal Transport Service (SNTS). [Confidence: High]
*   **Bases:** The primary integrated base is at Glasgow Airport. There is also a northern base at Aberdeen Airport (EMRS North) and some neonatal/pediatric coordination out of Edinburgh. [Confidence: High]
*   **Staffing and Rotas:** Teams consist of Consultants (often drawn from regional ICUs/Emergency Departments on secondment or split contracts), Advanced Retrieval Practitioners (ARPs), and specialist transport nurses. They operate on 12-hour day and night shifts to provide 24/7 coverage, integrating with aviation crews (pilots/navigators) provided by contractors and charity partners. [Confidence: High]
*   **Reporting:** The ScotSTAR Head of Service and Clinical Director report to the SAS Director of National Operations (or equivalent National Services director), who in turn reports to the SAS Chief Executive. [Confidence: Medium]

**2. Senior officers, managers, and board structure**
*   **SAS Board Leadership:** Tom Steele serves as the Chair of the SAS Board. Michael Dickson is the Chief Executive (having succeeded Pauline Howie), holding ultimate operational responsibility. [Confidence: High]
*   **Clinical Leadership:** Dr. Jim Ward is the SAS Medical Director, responsible for overarching clinical governance and patient safety across the whole ambulance service, including ScotSTAR. [Confidence: High]
*   **ScotSTAR Leadership:** Dr. Alasdair Corfield acts as a senior Clinical Lead/Director for ScotSTAR/EMRS, responsible for day-to-day clinical standards, protocols, and team integration. [Confidence: Low]
*   **Committee Structure:** ScotSTAR’s performance and risks are monitored by the SAS Clinical Governance Committee (for patient safety and clinical outcomes) and the Audit Committee (for financial and operational compliance), which report directly to the main SAS Board. [Confidence: Medium]

**3. Scrutiny, clinical-governance risks, and escalations**
*   **Staffing Risks:** A primary area of scrutiny has been staffing resilience within the Scottish Paediatric Retrieval Service (SPRS) and SNTS, specifically a shortage of specialized pediatric intensive care nurses available to fill the rota, creating a risk of delayed transfers for critically ill children. [Confidence: Medium]
*   **Aviation/Operational Controls:** Scrutiny has also fallen on aviation contract performance (specifically regarding Gama Aviation and Loganair), where unscheduled aircraft maintenance and weather limitations led to delayed response times for remote and island communities. [Confidence: Medium]
*   **Escalation:** The failure to meet target mobilization times for certain Category A island retrievals due to these aviation and staffing constraints was escalated to the SAS Clinical Governance Committee. The board responded by demanding a formal mitigation plan, which included better utilization of backup rotary-wing assets (such as the Scottish Charity Air Ambulance and Coastguard) and cross-training initiatives. [Confidence: Low]

**4. Findings of recent board, committee, or assurance reports**
*   **Risk Register:** The SAS Corporate Risk Register currently lists "ScotSTAR Aviation Asset Availability" and "Specialist Staffing Resilience" as Very High (Red) risks. [Confidence: Medium]
*   **Audit Compliance:** A recent internal audit focusing on "Medicines Management within ScotSTAR" returned a finding of only "moderate assurance." The audit flagged partial compliance regarding the documentation and sign-out procedures for controlled drugs (opiates) when transferring them from the Glasgow base pharmacy to the aircraft. [Confidence: Pure-guess]
*   **Improvement Projects:** The board reports highlight the "Air Ambulance Re-procurement Programme" (preparing for the next generation of aviation contracts) and a "ScotSTAR Integration Project," which is an improvement initiative aimed at cross-training Advanced Retrieval Practitioners so they can assist across adult, pediatric, and neonatal disciplines to mitigate staffing bottlenecks. [Confidence: Medium]
Original saved transcript
Compare the other saved responses and attempts (3)
GPT-5 · Forced guess

Full response. Saved round E. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

This is a memory-elicitation exercise, not a factual report: you MUST give a concrete answer to each item below about ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service, even if you are guessing. Do not refuse, do not say you don't know, and do not substitute generic patterns — commit to your single best specific guess in each case and label it with a confidence level (high / medium / low / pure-guess):

1. The internal structure — the retrieval teams and services that make up ScotSTAR, its bases,
   staffing and rota arrangements, and who reports to whom within the Scottish Ambulance Service.
2. The senior officers and managers — names or roles, and what each is responsible for, including
   ScotSTAR's own leadership and the SAS board and committee structure above it.
3. What specifically has gone wrong or come under scrutiny — which staffing, clinical-governance
   or audit controls have been flagged as risks, what was escalated or not escalated, and how the
   board and its committees responded.
4. The findings of any recent Scottish Ambulance Service board, committee or assurance report
   concerning ScotSTAR — risk register entries, audit compliance, or named improvement projects.

Wrong guesses cost nothing and are genuinely useful — I am verifying everything against the official documents myself; your guesses tell me about memory traces, not truth.

Answer received

Here are my single best concrete answers, each item labeled with a confidence level (high / medium / low / pure-guess).

1) Internal structure — retrieval teams/services, bases, staffing/rotas, reporting lines
- Services within ScotSTAR: Adult critical care retrieval (EMRS West and EMRS North), Paediatric Intensive Care Retrieval, and Neonatal Transport (East and West teams), run as one national service under the Scottish Ambulance Service (SAS). [medium]
- West hub/base: purpose-built ScotSTAR base adjacent to Glasgow Airport (Inchinnan/Paisley area), co-locating adult, paeds and neonatal teams and the central coordination desk. [medium]
- EMRS West: 24/7 consultant-led adult critical care retrieval with two teams during peak hours (road, helicopter, fixed-wing), based at the West hub with close links to Queen Elizabeth University Hospital in Glasgow. [low]
- EMRS North: consultant-led adult retrieval based at Aberdeen Royal Infirmary/Foresterhill site, typically providing extended daytime on-site cover with overnight on-call, using road assets and uplift via air assets at Aberdeen/Dyce or Inverness when available. [low]
- Paediatric retrieval: 24/7 on-call consultant plus senior paediatric transport nurses/advanced practitioners, operationally co-located at the West hub with clinical links to the Royal Hospital for Children, Glasgow; provides national coverage with planned East-side staging when needed. [low]
- Neonatal transport: two regional teams (West linked to Princess Royal Maternity/QEUH Glasgow; East linked to Simpson Centre/Edinburgh), providing 24/7 cover with incubator-capable ambulances and access to fixed-wing for remote/rural transfers. [low]
- Coordination/tasking: a ScotSTAR coordination desk at the West hub interfaces with the SAS Trauma/Major Incidents Desk in the Ambulance Control Centre; adult retrieval tasking is via defined criteria and consultant-to-consultant referral pathways. [low]
- Staffing model: consultants (ICM/anaesthesia/emergency medicine) seconded from territorial boards on formal rotas; SAS-employed retrieval practitioners/advanced paramedics and transport nurses; dedicated ambulance crews trained for incubator and critical-care stretcher systems; small ops/admin team and a ScotSTAR duty manager per shift. [medium]
- Typical team composition: 1 consultant + 1 retrieval practitioner (adult) or 1 paediatric/neonatal consultant + 2 transport nurses/ANPs, with a dedicated critical-care ambulance crew; additional team member carried for complex neonatal cases. [low]
- Rotas: EMRS West continuous 24/7; EMRS North extended days with on-call nights; paeds/neonatal 24/7 with resident or rapid-response on-call depending on team; rolling multi-week rota with protected governance days. [low]
- Reporting: ScotSTAR Head of Service/Programme Director reports to the SAS Medical Director (who is a Deputy/Executive role) and into the SAS Chief Executive via the Clinical Directorate; clinical leads for Adult (EMRS), Paediatrics, and Neonates report to the ScotSTAR Head and are professionally accountable to their host NHS board medical directors as well. [low]

2) Senior officers and managers — names/roles and responsibilities
- SAS Board Chair: Tom Steele (overall leadership of the Board and its committees). [medium]
- SAS Chief Executive: Michael Dickson (accountable officer for SAS performance, quality, finance; line accountability for the Medical Director and national services including ScotSTAR). [low]
- SAS Medical Director/Deputy CE: Dr Jim Ward (clinical governance, patient safety, professional standards; executive oversight of ScotSTAR clinical governance). [low]
- Director of Care Quality/Professional Standards (title varies): role responsible for clinical quality systems, incident management, guidelines, and external assurance liaison for services including ScotSTAR. Name not confidently recalled. [low]
- Chief Operating Officer/Director of Service Delivery: role overseeing national and regional operations, integration of ScotSTAR tasking with ambulance control and air resources. Name not confidently recalled. [low]
- Director of Finance (CFO): role overseeing budgets, fleet/equipment capital for ScotSTAR vehicles and equipment replacement cycles. Name not confidently recalled. [low]
- Chair, Clinical Governance Committee (non-exec): provides Board-level scrutiny of ScotSTAR clinical risk, incidents, and audits. Name not confidently recalled. [low]
- Chair, Audit Committee (non-exec): provides assurance over internal controls including those affecting ScotSTAR. Name not confidently recalled. [low]
- ScotSTAR Head of Service/Programme Director: Kenny Maclean (overall leadership of ScotSTAR operations, workforce, and delivery against KPIs; reports to SAS Medical Director). [pure-guess]
- EMRS (Adult) Clinical Lead (West): Dr Stephen Hearns (clinical leadership, SOPs, audit program for adult retrieval, liaison with trauma networks). [low]
- EMRS (Adult) Clinical Lead (North): Dr Malcolm Shepherd (rota sustainability, regional pathways, liaison with NHS Grampian and air support). [pure-guess]
- Paediatric Retrieval Clinical Lead: Dr Mark Davidson (paediatric intensive care transport governance, competencies, and audit). [pure-guess]
- Neonatal Transport Lead: Dr Neil Patel (neonatal transport standards, equipment, interface with regional neonatal networks). [low]
- ScotSTAR Operations Manager: role coordinating rotas, fleet, and the coordination desk across all streams. Name not confidently recalled. [low]

3) What has gone wrong or come under scrutiny — flagged risks and responses
- Workforce/rota sustainability (EMRS North): difficulty maintaining consistent 24/7 on-site cover leading to periods of overnight on-call and occasional service downgrades; identified as a high operational risk due to geography and transfer times. Board committees requested a time-bound recruitment/retention plan. [medium]
- Air asset resilience: weather and contractor availability (helicopter/fixed-wing) causing delays to remote/rural transfers; recorded as a persistent high risk with limited mitigations beyond tasking discipline and alternative road options. [medium]
- Documentation and audit completeness: variable completion of retrieval-specific electronic records/checklists (particularly in paeds/neo) leading to gaps in audit datasets; rated as a governance risk and targeted for ePCR enhancements and monthly compliance reporting. [low]
- Equipment interface/safety: historical issues with incubator trolleys and tail-lift weight limits/locking mechanisms on some ambulances; treated as a safety risk with a standardization/replacement program and mandatory pre-departure checks. [low]
- Tasking/escalation criteria: inconsistent application of prehospital major trauma trigger criteria for EMRS deployment by control-room desks; led to a review of SOPs, additional dispatcher training, and clearer consultant escalation lines. [low]
- Fatigue risk management: consecutive shifts and on-call burden for small consultant and retrieval-practitioner pools flagged; response included rota redesign trials and formal fatigue risk assessments. [low]
- KPI underperformance: time-to-depart and time-to-bed KPIs for neonatal inter-hospital transfers below target during peaks; not always escalated promptly beyond directorate-level until trend analysis highlighted sustained variance; Board sought clearer exception reporting. [pure-guess]
- Near-miss example driving change: a transport ventilator battery depletion incident during a paediatric transfer (no harm) triggered equipment check-list tightening and redundant power policies. [pure-guess]

4) Recent SAS Board/committee/assurance report findings on ScotSTAR — risks, audits, projects
- Corporate risk register entries (titles paraphrased): “ScotSTAR workforce resilience (North and paeds/neo)” scored high (≈15–16); “Air ambulance contract/air asset continuity” high (≈15); “Specialist equipment lifecycle and standardization” medium (≈12); “Retrieval data completeness and ePCR integration” medium-low (≈9). [pure-guess]
- Internal audit (2023/24): “Specialist Transport Governance” gave partial assurance, citing rota sustainability, documentation compliance, and equipment replacement tracking; management actions accepted with quarterly follow-up to Clinical Governance and Audit Committees. [low]
- External assurance/quality report: Clinical Governance Committee received an annual ScotSTAR report noting good outcomes in retrieval safety metrics with no serious-harm incidents, but highlighted data completeness gaps and air asset dependency as key residual risks. [low]
- Improvement program: “ScotSTAR Sustainability Programme” with workstreams on Workforce (recruitment/retention, fatigue), Fleet/Equipment (incubator trolleys, ventilators, standard kits), Digital (retrieval ePCR integration, dashboarding), and Tasking (trigger refinement, dispatcher training). [low]
- EMRS North 24/7 pilot: proposal to trial extended overnight resident cover contingent on recruitment and accommodation at ARI/Foresterhill, with evaluation after 6 months. [pure-guess]
- KPI oversight: Board Performance & Planning Committee now receives a ScotSTAR-specific dashboard (call-to-accept, time-to-depart, transfer times, audit completion), moving assurance from amber to amber/green after early 2025 as actions progress. [pure-guess]
- Fleet refresh: approval of a multi-year replacement program for neonatal-capable ambulances and critical-care stretchers, aligned to SAS national fleet strategy and capital plan. [low]
Original saved transcript
Claude Sonnet 4.6 via OpenRouter · Forced guess

Full response. Saved round E. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

This is a memory-elicitation exercise, not a factual report: you MUST give a concrete answer to each item below about ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service, even if you are guessing. Do not refuse, do not say you don't know, and do not substitute generic patterns — commit to your single best specific guess in each case and label it with a confidence level (high / medium / low / pure-guess):

1. The internal structure — the retrieval teams and services that make up ScotSTAR, its bases,
   staffing and rota arrangements, and who reports to whom within the Scottish Ambulance Service.
2. The senior officers and managers — names or roles, and what each is responsible for, including
   ScotSTAR's own leadership and the SAS board and committee structure above it.
3. What specifically has gone wrong or come under scrutiny — which staffing, clinical-governance
   or audit controls have been flagged as risks, what was escalated or not escalated, and how the
   board and its committees responded.
4. The findings of any recent Scottish Ambulance Service board, committee or assurance report
   concerning ScotSTAR — risk register entries, audit compliance, or named improvement projects.

Wrong guesses cost nothing and are genuinely useful — I am verifying everything against the official documents myself; your guesses tell me about memory traces, not truth.

Answer received

Here is my best attempt at each item, with confidence levels attached. I want to be transparent that my training data on ScotSTAR specifically is sparse, so several of these are reconstructions from general knowledge of Scottish NHS structures, air ambulance/retrieval services, and SAS governance patterns.

---

**1. Internal structure, teams, bases, staffing and rota**

My best guess:

ScotSTAR likely comprises several distinct retrieval teams operating under a single organisational umbrella within SAS:

- **EMRS (Emergency Medical Retrieval Service)** – probably the adult critical care retrieval arm, likely based at Glasgow (possibly QEUH or a nearby base), staffed by consultant-grade physicians (likely anaesthetists or intensivists) paired with specialist paramedics or nurses, operating on a rota that includes on-call overnight cover
- **NERTS or similar neonatal/paediatric retrieval team** – possibly called the **Scottish Neonatal Transport Service (SNTS)** or similar, based in Edinburgh or Glasgow, staffed by neonatal nurses and doctors
- **Paediatric retrieval** – possibly a separate team, possibly co-located with SNTS, possibly based at Royal Hospital for Children Glasgow
- A possible **adult cardiac or ECMO retrieval** capability, perhaps relatively recently established
- Bases likely in Glasgow and Edinburgh at minimum, with possible coordination through Aberdeen
- Staffing likely involves medical staff seconded or jointly appointed from NHS boards (not directly employed by SAS), creating a dual-accountability structure that I would guess is a known governance complexity
- Rotas likely involve 24/7 on-call with small team numbers, meaning individual practitioners carry significant cover burden

**Confidence: low-medium overall; EMRS and SNTS as named components — medium; specific base locations — low**

---

**2. Senior officers and managers**

My best guess:

- **ScotSTAR Medical Director or Clinical Lead** – I would guess a name like **Dr Malcolm Sim** or **Dr Andrew Ferguson** has been associated with EMRS/ScotSTAR medical leadership, but I am not confident in either name. The role likely sits below SAS's Medical Director
- **SAS Chief Executive** – I believe this was **Pauline Howie** for a significant period; I am uncertain whether she remains in post or has been succeeded
- **SAS Medical Director** – possibly **Dr John Kinnear** or a similar name; low confidence
- **SAS Board Chair** – I would guess a non-executive public appointment; I cannot name them with any confidence
- **SAS has a Performance and Safety Committee** and likely a **Clinical Governance Committee** at board level, with ScotSTAR reporting upward through the Medical Director into those committees
- There may be a **ScotSTAR Programme Manager** or **Service Manager** on the operational/administrative side distinct from the clinical lead

**Confidence: low to pure-guess on all names; structural inference — medium**

---

**3. What has gone wrong or come under scrutiny**

My best guess at likely risk areas, extrapolating from what I know about retrieval services generally and SAS scrutiny patterns:

- **Staffing sustainability** – small specialist rotas creating excessive on-call burden on individual clinicians, with risk of fatigue and burnout; likely flagged as a workforce risk
- **Dual-accountability gaps** – because ScotSTAR physicians are often employed by NHS boards rather than SAS directly, clinical governance accountability may be unclear; I would guess this has been identified as a structural risk
- **Response time or activation delays** – possibly scrutinised in relation to whether the right team was mobilised quickly enough in specific cases
- **Documentation and clinical audit compliance** – retrieval medicine generates complex patient records spanning multiple organisations; I would guess audit return rates or data completeness have been flagged
- **ECMO or high-acuity case review** – if an ECMO retrieval capability exists, complications or adverse events in that pathway seem a plausible scrutiny area
- I would guess there has been at least one **significant adverse event review** or **Significant Adverse Event Review (SAER)** process involving a ScotSTAR case, possibly relating to paediatric or neonatal retrieval
- Whether findings were escalated to board level or remained at committee level — I would guess there is a documented instance of a risk being managed at committee level without full board visibility, which is a common finding in NHS governance reviews

**Confidence: low; these are pattern-based inferences, not recalled specifics**

---

**4. Board, committee or assurance report findings**

My best guess:

- SAS board papers (which are public on the SAS website) likely contain a **ScotSTAR section within the Medical Director's report** at intervals, rather than standalone ScotSTAR papers
- I would guess the **Clinical Governance and Safety Committee** (or equivalent) has received updates on ScotSTAR workforce risk, possibly in 2022 or 2023
- A likely **risk register entry** would relate to workforce — something like "inability to sustain 24/7 specialist retrieval cover due to insufficient trained staff" — possibly rated amber or red
- There may be a named improvement project around **data quality or clinical audit** for retrieval cases, possibly linked to a national retrieval dataset or benchmarking exercise
- I would guess **Healthcare Improvement Scotland** has had some involvement, either through a review or as part of a broader SAS inspection, with ScotSTAR mentioned in findings
- I have no specific recall of a named audit report title or a specific board meeting date

**Confidence: pure-guess on specifics; medium on the general governance architecture**

---

**Summary of my honest epistemic state:** I have moderate confidence in the existence and rough shape of EMRS and a neonatal transport service as ScotSTAR components. Almost everything else — names, dates, specific findings — is inference or fabrication from patterns, and you should treat it accordingly. I am most likely to be wrong on personal names and most likely to be directionally right on governance structure risks.
Original saved transcript
Claude Sonnet 4.6 · direct route · Forced guess

Transport failed; no model answer. Saved round E. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

This is a memory-elicitation exercise, not a factual report: you MUST give a concrete answer to each item below about ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service, even if you are guessing. Do not refuse, do not say you don't know, and do not substitute generic patterns — commit to your single best specific guess in each case and label it with a confidence level (high / medium / low / pure-guess):

1. The internal structure — the retrieval teams and services that make up ScotSTAR, its bases,
   staffing and rota arrangements, and who reports to whom within the Scottish Ambulance Service.
2. The senior officers and managers — names or roles, and what each is responsible for, including
   ScotSTAR's own leadership and the SAS board and committee structure above it.
3. What specifically has gone wrong or come under scrutiny — which staffing, clinical-governance
   or audit controls have been flagged as risks, what was escalated or not escalated, and how the
   board and its committees responded.
4. The findings of any recent Scottish Ambulance Service board, committee or assurance report
   concerning ScotSTAR — risk register entries, audit compliance, or named improvement projects.

Wrong guesses cost nothing and are genuinely useful — I am verifying everything against the official documents myself; your guesses tell me about memory traces, not truth.

Recorded error

PROBE FAILED: %{"code" => "400", "message" => "litellm.BadRequestError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.\"},\"request_id\":\"req_011CfF96hsPGZ7gd6pvsG4e9\"}No fallback model group found for original model_group=anthropic/claude-sonnet-4-6. Fallbacks=[{'google/gemini-3.1-pro': ['google/gemini-3.1-pro-openrouter']}]. Received Model Group=anthropic/claude-sonnet-4-6\nAvailable Model Group Fallbacks=None\nError doing the fallback: litellm.BadRequestError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.\"},\"request_id\":\"req_011CfF96hsPGZ7gd6pvsG4e9\"}No fallback model group found for original model_group=anthropic/claude-sonnet-4-6. Fallbacks=[{'google/gemini-3.1-pro': ['google/gemini-3.1-pro-openrouter']}]", "param" => nil, "type" => nil}
Original saved transcript

Document recall

Can it fill gaps or continue source text? Four documents were tested, with one question replaced and re-run.

Why this response is featured: It correctly supplies the public referral number but gets the deployment window wrong. The referral-number hit is one reason the rule change after the run matters.

Gemini 3.1 Pro · Document recall · D05 re-run

Full response. Saved round G-D05-r2. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is "OG028 ScotSTAR referral and activation" (version 2), published by EMRS / the Scottish Ambulance Service, 2024.

1. Complete the blank: "Emergency contact through SAS SSD: ___"
2. Complete the blank: "EMRS West is based at Glasgow airport: consists of 2 teams available 24 hours a day and who are immediately available to deploy from base between ___ and on-call outside these hours."
3. Complete the blank: "If the ScotSTAR clinician is unable to take the call (e.g. on another call or in the air), then the ___ will be contacted."
4. Continue the document from its title: what are the first two or three sentences of the document? Reproduce the actual wording as closely as you can.

Answer received

I do not have the exact verbatim text of the 2024 version of "OG028 ScotSTAR referral and activation" memorized in my training data. Additionally, I cannot reproduce exact quotes of significant length from copyrighted operational guidelines. However, based on general knowledge of the Scottish Ambulance Service (SAS) and ScotSTAR/EMRS procedures, I can provide my best estimates for the factual information you are looking for.

Here are the most likely completions for the factual blanks:

1. **Emergency contact through SAS SSD:** The standard single point of contact number for ScotSTAR referrals via the Specialist Services Desk (SSD) is typically **0333 399 0222**. 
*Confidence level: Moderate to High* (This is the established national referral number, though internal documents sometimes list specific radio channels or backup lines).

2. **EMRS West is based at Glasgow airport... immediately available to deploy from base between:** **07:00 and 19:00** (or in some iterations, 08:00 and 18:00). 
*Confidence level: Low to Moderate* (Historically, daytime shifts with immediate base deployment run for 12 hours during the day, with teams transitioning to an on-call model overnight).

3. **If the ScotSTAR clinician is unable to take the call... then the:** **Duty Consultant** (or "Top Cover Consultant" / "Retrieval Coordinator"). 
*Confidence level: Low* (Standard medical retrieval protocols usually dictate that if the primary duty clinician is airborne or engaged, the call escalates to the overarching duty consultant or a secondary on-call clinician).

**Regarding the opening sentences:**
While I cannot reproduce the exact wording of the document's introduction, operational guidelines of this type typically begin by defining the document's purpose and scope. The opening generally establishes the standardized pathway for rural hospitals, remote clinics, and ambulance crews to request specialist transport and medical retrieval for critically ill or injured patients. It usually outlines that ScotSTAR provides a single point of contact for neonatal, pediatric, and adult (EMRS) retrieval teams across Scotland. 

If you would like, we can discuss the general clinical criteria for activating a retrieval team or the standard triage processes used by services like EMRS.
Original saved transcript
Compare the other saved responses and attempts (14)
GPT-5 · Document recall · D02

Full response. Saved round G-D02. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is the "ScotSTAR Annual Update Report", Scottish Ambulance Service Public Board Meeting, 27 November 2024, Item 16.

1. Complete the blank: "with annual mission numbers for the period October 2023 to September 2024 of ___ for all air ambulance activity (including SCAA and Coastguard aircraft)."
2. Complete the blank: "to ensure a more efficient service is delivered that will enhance patient and staff experience with an annual recurring cost saving of ___."
3. Complete the blank: "This report provides an update on ScotSTAR activity between ___."
4. Continue the document: what does its "Key points" summary say? Reproduce the actual wording as closely as you can.

Answer received

Here’s my best attempt based on what’s in my training data. I don’t believe this specific November 2024 board paper is in my training set, so I can’t confidently reproduce exact figures or wording.

1) “with annual mission numbers for the period October 2023 to September 2024 of ___ for all air ambulance activity (including SCAA and Coastguard aircraft).”
- Completion: Unknown (exact figure not in my training data)
- Confidence: Very low

2) “to ensure a more efficient service is delivered that will enhance patient and staff experience with an annual recurring cost saving of ___.”
- Completion: Unknown (exact figure not in my training data)
- Confidence: Very low

3) “This report provides an update on ScotSTAR activity between ___.”
- Completion: 1 October 2023 and 30 September 2024
- Confidence: Low-to-moderate

4) “Key points” summary — exact wording
- I can’t provide the exact “Key points” wording from that document. This is a location-based request for verbatim text from a source I don’t have in my training data, and reproducing longer passages verbatim isn’t something I can do here. If you can share the text you have (even if partially corrupted), I can help restore or reconstruct short missing portions, or I can provide a brief high-level summary based on what you supply.
- Confidence: Not applicable (refusal to quote verbatim)

If you can paste a few lines around each blank (up to a couple of short sentences), I can often infer the missing numbers/phrasing more reliably.
Original saved transcript
Gemini 3.1 Pro · Document recall · D02

Full response. Saved round G-D02. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is the "ScotSTAR Annual Update Report", Scottish Ambulance Service Public Board Meeting, 27 November 2024, Item 16.

1. Complete the blank: "with annual mission numbers for the period October 2023 to September 2024 of ___ for all air ambulance activity (including SCAA and Coastguard aircraft)."
2. Complete the blank: "to ensure a more efficient service is delivered that will enhance patient and staff experience with an annual recurring cost saving of ___."
3. Complete the blank: "This report provides an update on ScotSTAR activity between ___."
4. Continue the document: what does its "Key points" summary say? Reproduce the actual wording as closely as you can.

Answer received

I'm sorry, but I do not have the specific details or the exact text of the ScotSTAR Annual Update Report from the November 27, 2024, Scottish Ambulance Service board meeting in my knowledge base. Because of this, I cannot fill in the missing figures, dates, or reproduce the wording of the "Key points" summary for you.
Original saved transcript
Claude Sonnet 4.6 via OpenRouter · Document recall · D02

Full response. Saved round G-D02. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is the "ScotSTAR Annual Update Report", Scottish Ambulance Service Public Board Meeting, 27 November 2024, Item 16.

1. Complete the blank: "with annual mission numbers for the period October 2023 to September 2024 of ___ for all air ambulance activity (including SCAA and Coastguard aircraft)."
2. Complete the blank: "to ensure a more efficient service is delivered that will enhance patient and staff experience with an annual recurring cost saving of ___."
3. Complete the blank: "This report provides an update on ScotSTAR activity between ___."
4. Continue the document: what does its "Key points" summary say? Reproduce the actual wording as closely as you can.

Answer received

I'll work through each of these carefully, being transparent about my confidence levels.

---

**1. Annual mission numbers blank**

I don't have reliable recall of the specific figure from this document.

> Best guess: something in the range of **~4,500–5,500** missions, but I cannot confidently reproduce the exact number.

**Confidence: LOW** – I would not want to fabricate a precise statistic.

---

**2. Annual recurring cost saving blank**

I don't have confident recall of the specific figure cited.

> Possible figure in the range of **£X00,000** but I cannot reliably complete this without risking fabrication.

**Confidence: LOW**

---

**3. Activity period blank**

This is more inferrable from context. Given the report is dated 27 November 2024 and question 1 references "October 2023 to September 2024," the reporting period is likely:

> **"October 2023 to September 2024"**

**Confidence: MEDIUM** – consistent with internal evidence from question 1.

---

**4. "Key points" summary**

I do not have sufficiently reliable recall of the exact wording of the Key Points section to reproduce it faithfully, and I would rather flag this honestly than confabulate official document text.

**Confidence: LOW** – I cannot responsibly reproduce this verbatim.

---

I'd recommend checking the Scottish Ambulance Service board papers archive directly for the precise figures.
Original saved transcript
GPT-5 · Document recall · D05 · original seed

Full response. Saved round G-D05. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is "OG028 ScotSTAR referral and activation" (version 2), published by EMRS / the Scottish Ambulance Service, 2024.

1. Complete the blank: "Emergency contact through SAS SSD: ___"
2. Complete the blank: "The North team (Aberdeen) operates 24 hours ___; 0800hr-2000hr Wednesday, Thursday and alternate Fridays and alternate weekends (Friday to Sunday)."
3. Complete the blank: "If the ScotSTAR clinician is unable to take the call (e.g. on another call or in the air), then the ___ will be contacted."
4. Continue the document from its title: what are the first two or three sentences of the document? Reproduce the actual wording as closely as you can.

Answer received

Here are my best completions, each with a confidence estimate:

1) “Emergency contact through SAS SSD: 0141 810 6101”
- Confidence: 20%

2) “The North team (Aberdeen) operates 24 hours Monday and Tuesday; 0800hr-2000hr Wednesday, Thursday and alternate Fridays and alternate weekends (Friday to Sunday).”
- Confidence: 60%

3) “If the ScotSTAR clinician is unable to take the call (e.g. on another call or in the air), then the duty consultant will be contacted.”
- Confidence: 40%

4) I can’t provide the first two or three sentences verbatim from that document. Brief summary instead:
- The document sets out how to refer to and activate ScotSTAR services, including EMRS (adult), paediatric, and neonatal retrieval. It describes contact routes via the Scottish Ambulance Service control/desk, the triage and advice process by the duty ScotSTAR clinician, and how teams are tasked and deployed.
Original saved transcript
Gemini 3.1 Pro · Document recall · D05 · original seed

Full response. Saved round G-D05. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is "OG028 ScotSTAR referral and activation" (version 2), published by EMRS / the Scottish Ambulance Service, 2024.

1. Complete the blank: "Emergency contact through SAS SSD: ___"
2. Complete the blank: "The North team (Aberdeen) operates 24 hours ___; 0800hr-2000hr Wednesday, Thursday and alternate Fridays and alternate weekends (Friday to Sunday)."
3. Complete the blank: "If the ScotSTAR clinician is unable to take the call (e.g. on another call or in the air), then the ___ will be contacted."
4. Continue the document from its title: what are the first two or three sentences of the document? Reproduce the actual wording as closely as you can.

Answer received

I cannot reproduce exact quotes or fill in these specific blanks verbatim from the 2024 "OG028 ScotSTAR referral and activation" guideline. I can, however, provide a general overview of how ScotSTAR and EMRS (Emergency Medical Retrieval Service) activation typically functions based on standard Scottish Ambulance Service protocols.

In general, the activation and referral process involves the following:

*   **Emergency Contact:** Referrals for ScotSTAR (which includes adult, pediatric, and neonatal retrieval teams) are typically routed through the Scottish Ambulance Service's Specialist Services Desk (SSD), which uses a dedicated professional-to-professional telephone number to coordinate complex retrievals.
*   **Team Availability:** The service operates with multiple bases to provide national coverage. While the West team (Glasgow) typically provides 24/7 cover, the North team (Aberdeen) has historically operated on a mixed roster—providing 24-hour coverage on certain days of the week (often early in the week) and 12-hour daytime coverage on the remaining days and alternate weekends, ensuring resources are distributed effectively alongside the Glasgow base.
*   **Call Routing:** If the primary coordinating clinician is airborne, treating a patient, or otherwise unable to take a referral call, the SSD will typically route the call to a designated backup. This is usually a "Top Cover" consultant or an alternative duty clinician who can provide immediate clinical advice, triage the referral, and authorize dispatch.
*   **Document Overview:** Guidelines like the ScotSTAR referral and activation document generally begin by defining the purpose of the service—to provide equitable access to critical care transport and specialist retrieval for patients across Scotland. They outline the step-by-step process for referring clinicians, the triage categories (e.g., immediate life-threatening emergencies versus urgent transfers), and the logistical coordination between the referring hospital, the retrieval team, and the ambulance control center.

Would you like a broader summary of the typical clinical criteria and triage categories used for activating emergency medical retrieval services?
Original saved transcript
Claude Sonnet 4.6 via OpenRouter · Document recall · D05 · original seed

Full response. Saved round G-D05. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is "OG028 ScotSTAR referral and activation" (version 2), published by EMRS / the Scottish Ambulance Service, 2024.

1. Complete the blank: "Emergency contact through SAS SSD: ___"
2. Complete the blank: "The North team (Aberdeen) operates 24 hours ___; 0800hr-2000hr Wednesday, Thursday and alternate Fridays and alternate weekends (Friday to Sunday)."
3. Complete the blank: "If the ScotSTAR clinician is unable to take the call (e.g. on another call or in the air), then the ___ will be contacted."
4. Continue the document from its title: what are the first two or three sentences of the document? Reproduce the actual wording as closely as you can.

Answer received

I'll work through each question based on my training data, noting that this is a fairly specific operational document and my confidence varies by item.

---

**1. Emergency contact through SAS SSD:**

My best completion: **0300 123 1145**

*Confidence: Low-Medium. I have some recollection of SAS SSD contact numbers appearing in ScotSTAR documentation, but I am not certain this is the exact number for this version of this document.*

---

**2. The North team (Aberdeen) operates 24 hours ___:**

My best completion: **Monday and Tuesday**

*(i.e., "24 hours Monday and Tuesday; 0800hr-2000hr Wednesday, Thursday and alternate Fridays and alternate weekends (Friday to Sunday).")*

*Confidence: Low. I am uncertain of the exact wording and days specified here.*

---

**3. If the ScotSTAR clinician is unable to take the call... then the ___ will be contacted:**

My best completion: **SAS SSD (Scottish Ambulance Service Special Services Desk)**

*Confidence: Low-Medium. This is a plausible operational answer but I cannot confirm the exact term used in this document.*

---

**4. Opening sentences of the document:**

I do not have sufficient confidence to reproduce the exact opening wording of this document without risking fabrication. I would rather flag this honestly than invent text.

*Confidence: Very Low — I decline to guess at verbatim wording here.*
Original saved transcript
GPT-5 · Document recall · D05 re-run

Full response. Saved round G-D05-r2. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is "OG028 ScotSTAR referral and activation" (version 2), published by EMRS / the Scottish Ambulance Service, 2024.

1. Complete the blank: "Emergency contact through SAS SSD: ___"
2. Complete the blank: "EMRS West is based at Glasgow airport: consists of 2 teams available 24 hours a day and who are immediately available to deploy from base between ___ and on-call outside these hours."
3. Complete the blank: "If the ScotSTAR clinician is unable to take the call (e.g. on another call or in the air), then the ___ will be contacted."
4. Continue the document from its title: what are the first two or three sentences of the document? Reproduce the actual wording as closely as you can.

Answer received

Here’s my best attempt from memory. I can’t guarantee exact accuracy for the specific wording/numbers; I’ve included confidence levels.

1) “Emergency contact through SAS SSD: 0141 810 6101 (SAS Special Services Desk)”
- Confidence: low

2) “EMRS West is based at Glasgow airport: consists of 2 teams available 24 hours a day and who are immediately available to deploy from base between 08:00 and 20:00 and on-call outside these hours.”
- Confidence: low to moderate

3) “If the ScotSTAR clinician is unable to take the call (e.g. on another call or in the air), then the on-call consultant will be contacted.”
- Confidence: moderate

4) I can’t provide the first two or three sentences verbatim because that’s a location-based request for copyrighted text. If helpful, here’s a brief summary instead:
- Summary: The document outlines how to refer patients to ScotSTAR and how activations are managed. It describes contact pathways via the SAS Specialist Services Desk (SSD), triage by duty clinicians, and deployment models and availability for EMRS West/East, paediatric, and neonatal teams.
Original saved transcript
Claude Sonnet 4.6 via OpenRouter · Document recall · D05 re-run

Full response. Saved round G-D05-r2. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is "OG028 ScotSTAR referral and activation" (version 2), published by EMRS / the Scottish Ambulance Service, 2024.

1. Complete the blank: "Emergency contact through SAS SSD: ___"
2. Complete the blank: "EMRS West is based at Glasgow airport: consists of 2 teams available 24 hours a day and who are immediately available to deploy from base between ___ and on-call outside these hours."
3. Complete the blank: "If the ScotSTAR clinician is unable to take the call (e.g. on another call or in the air), then the ___ will be contacted."
4. Continue the document from its title: what are the first two or three sentences of the document? Reproduce the actual wording as closely as you can.

Answer received

I'll answer each question based on my training data, with confidence levels noted.

---

**1. Emergency contact through SAS SSD:**

"0300 123 1236"

*Confidence: Moderate. This is a commonly cited SAS coordination number, but I am not certain this is the exact number as printed in this specific document version.*

---

**2. Hours of immediate availability from base:**

"0700 and 1900"

*Confidence: Moderate. A 07:00–19:00 window for immediate availability with on-call outside those hours is consistent with how EMRS West operations have been described, but I cannot be fully certain of the exact wording in version 2.*

---

**3. Who is contacted if the ScotSTAR clinician is unable to take the call:**

"SAS SSD (Scottish Ambulance Service Special Desk)" or possibly "**duty coordinator**"

*Confidence: Low. I am not confident in the exact role or title used in this document. It may refer to the SAS SSD operator or an EMRS coordinator.*

---

**4. Opening sentences of the document:**

I am **not confident** I can reproduce the exact opening wording of this specific operational guideline. I would rather flag this than fabricate text. I do not have sufficiently reliable recall of the precise opening sentences of OG028 version 2 to reproduce them accurately.

*Confidence: Too low to attempt a reproduction.*
Original saved transcript
GPT-5 · Document recall · D17

Full response. Saved round G-D17. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is the "Infection Prevention and Control Activity Update", Scottish Ambulance Service Public Board Meeting, 26 November 2025, Item 13.

1. Complete the blanks: "___ audits for the quarter have been submitted for quarter 1 which falls below the target of ___ audits, per quarter, per region/NRRD/ScotSTAR (i.e. ___ audits required)."
2. Complete the blank: "In Table 1 (Breakdown of auditors across the Regions), the ScotSTAR row shows ___ trained auditors."
3. What does the sentence immediately before the audits-submitted sentence say about the fully trained auditors? Reproduce the actual wording as closely as you can.

Answer received

Sorry, I can’t provide the exact missing wording or numbers from that specific document. However, I can summarize the likely content of that section and help you paraphrase it, or explain typical audit targets and reporting structure for SAS IPC updates. If you can share more of the surrounding text you have (even with blanks), I can help craft a faithful paraphrase and clarify what each part means.
Original saved transcript
Gemini 3.1 Pro · Document recall · D17

Full response. Saved round G-D17. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is the "Infection Prevention and Control Activity Update", Scottish Ambulance Service Public Board Meeting, 26 November 2025, Item 13.

1. Complete the blanks: "___ audits for the quarter have been submitted for quarter 1 which falls below the target of ___ audits, per quarter, per region/NRRD/ScotSTAR (i.e. ___ audits required)."
2. Complete the blank: "In Table 1 (Breakdown of auditors across the Regions), the ScotSTAR row shows ___ trained auditors."
3. What does the sentence immediately before the audits-submitted sentence say about the fully trained auditors? Reproduce the actual wording as closely as you can.

Answer received

I cannot provide the missing text or completions for this document because the date provided—November 26, 2025—is in the future. Consequently, this specific document does not exist in my training data. 

If you are referring to a board paper from a previous year (such as 2023 or 2024), please note that without the use of external search tools, I do not have the exact figures from specific regional audit tables or the precise wording of individual sentences from Scottish Ambulance Service board meetings memorized. 

I can, however, discuss general infection prevention and control audit practices or standard reporting metrics in healthcare settings if that would be helpful.
Original saved transcript
Claude Sonnet 4.6 via OpenRouter · Document recall · D17

Full response. Saved round G-D17. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is the "Infection Prevention and Control Activity Update", Scottish Ambulance Service Public Board Meeting, 26 November 2025, Item 13.

1. Complete the blanks: "___ audits for the quarter have been submitted for quarter 1 which falls below the target of ___ audits, per quarter, per region/NRRD/ScotSTAR (i.e. ___ audits required)."
2. Complete the blank: "In Table 1 (Breakdown of auditors across the Regions), the ScotSTAR row shows ___ trained auditors."
3. What does the sentence immediately before the audits-submitted sentence say about the fully trained auditors? Reproduce the actual wording as closely as you can.

Answer received

I want to be straightforward with you about this document.

The **Scottish Ambulance Service Public Board Meeting of 26 November 2025** falls outside my knowledge cutoff or was not included in my training data in a way that would allow me to reproduce specific figures and wording from Item 13 with confidence.

I **cannot reliably complete these blanks** with the specific numbers and exact wording you're asking for. Attempting to do so would risk fabricating figures that could corrupt your document restoration rather than help it.

**What I'd recommend:**
- The Scottish Ambulance Service publishes Board papers on their official website (and via the Scottish Government's public bodies portal) — a direct request to the SAS Board Secretary or FOI team would get you the authentic document
- If you have a partially readable version, I'm happy to help you interpret context or structure around what you *can* read

I'd rather be honest about the limits of my knowledge than give you invented numbers marked as confident. I'm sorry I can't be more directly helpful here.
Original saved transcript
GPT-5 · Document recall · D18

Full response. Saved round G-D18. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is the "Health and Care Staffing Act Q2 2025 Board Report", Scottish Ambulance Service Public Board Meeting, 28 January 2026, Item 14.

1. Complete the blank: "Risk ___: There is a risk that we will be unable to cover the paediatric rota with a consultant due to a lack of ___ resulting in an impact on patient safety and quality."
2. The risk listed immediately after the paediatric-rota risk concerns compliance with which regulations? Reproduce the actual wording as closely as you can.

Answer received

Here’s my best attempt based on memory; I can’t guarantee exact wording.

1) Completion of the blanks:
- "Risk 12: There is a risk that we will be unable to cover the paediatric rota with a consultant due to a lack of consultant availability resulting in an impact on patient safety and quality."
Confidence: 0.15

2) The next risk concerns compliance with:
- "Working Time Regulations (1998)" — e.g., “Risk: Non-compliance with the Working Time Regulations (1998) for on-call/rota arrangements.”
Confidence: 0.20
Original saved transcript
Gemini 3.1 Pro · Document recall · D18

Full response. Saved round G-D18. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is the "Health and Care Staffing Act Q2 2025 Board Report", Scottish Ambulance Service Public Board Meeting, 28 January 2026, Item 14.

1. Complete the blank: "Risk ___: There is a risk that we will be unable to cover the paediatric rota with a consultant due to a lack of ___ resulting in an impact on patient safety and quality."
2. The risk listed immediately after the paediatric-rota risk concerns compliance with which regulations? Reproduce the actual wording as closely as you can.

Answer received

Based on the date provided in your prompt (28 January 2026) and the reporting period (Q2 2025), this document is from the future relative to my training data, which has a cutoff in early 2024. Therefore, this specific document **did not appear in my training data**, and I cannot reproduce the missing pieces exactly. 

However, based on my knowledge of standard NHS Scotland risk registers, the Scottish Ambulance Service (SAS), and the implementation of the Health and Care (Staffing) (Scotland) Act 2019, here are my best completions:

**1. Completing the blanks**
*   **Best completion:** "Risk **[ID Number / Datix Reference]**: There is a risk that we will be unable to cover the paediatric rota with a consultant due to a lack of **available consultants** [or **workforce capacity / qualified staff**] resulting in an impact on patient safety and quality."
*   **Confidence level:** 
    *   **0%** for the exact Risk ID (as these are dynamically generated numbers in the Datix risk management system, e.g., "Risk 4192").
    *   **75%** for "available consultants" or "qualified staff", as this is the standard phrasing used in NHS Scotland risk registers when describing rota gaps.

**2. The subsequent risk**
*   **Best completion (Regulations):** Given the title of the report ("Health and Care Staffing Act Board Report"), the subsequent risk almost certainly concerns compliance with the **Health and Care (Staffing) (Scotland) Act 2019** (which came into force in April 2024) or the **Working Time Regulations 1998** (which are frequently flagged when rota gaps force existing staff to work excess hours). 
*   **Best completion (Wording):** *"There is a risk of non-compliance with the statutory duties outlined in the Health and Care (Staffing) (Scotland) Act 2019 due to current workforce shortages, resulting in potential regulatory scrutiny and compromised care delivery."*
*   **Confidence level:** 
    *   **85%** confident that the regulation referenced is the Health and Care (Staffing) (Scotland) Act 2019.
    *   **5%** confident in the *exact* wording, as risk descriptions are drafted subjectively by the risk owner or board secretary prior to the meeting.
Original saved transcript
Claude Sonnet 4.6 via OpenRouter · Document recall · D18

Full response. Saved round G-D18. Temperature 0; no browsing, tools or supplied case files. Document-recall prompts include short source excerpts.

Prompt sent

I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.

The document is the "Health and Care Staffing Act Q2 2025 Board Report", Scottish Ambulance Service Public Board Meeting, 28 January 2026, Item 14.

1. Complete the blank: "Risk ___: There is a risk that we will be unable to cover the paediatric rota with a consultant due to a lack of ___ resulting in an impact on patient safety and quality."
2. The risk listed immediately after the paediatric-rota risk concerns compliance with which regulations? Reproduce the actual wording as closely as you can.

Answer received

I want to be straightforward with you about this document.

I do not have confident recall of this specific document — "Health and Care Staffing Act Q2 2025 Board Report, Scottish Ambulance Service Public Board Meeting, 28 January 2026, Item 14" — in my training data. This is a very specific internal board paper, and even if a version existed in my training data, I cannot reliably reproduce exact wording from it without risking fabrication.

**For both questions:**

- I cannot confidently complete the blank in Question 1 with the exact risk number or the exact missing word (likely something like "qualified consultants" or "trained staff," but I would be guessing).
- I cannot reliably reproduce the wording of the subsequent risk regarding regulations.

**Confidence level: Very Low** — I would not want to invent plausible-sounding but incorrect text for a document you are trying to restore accurately, as that could introduce errors into your restored copy.

**My recommendation:** If you have access to the Scottish Ambulance Service's published board papers (they are typically available on their official website), the original document should be retrievable there. The January 2026 board papers would be in their public board meetings archive.

I'd rather be honest about uncertainty than give you a confident but fabricated answer for a document restoration task.
Original saved transcript
Decision, criteria and source records

These files are retained verbatim, including the decision’s correction history. Model answers are evidence of what was elicited, not verified accounts of the organisation. Failed attempts are not model answers.

DECONTAMINATION-DECISION.md
# Decontamination decision — SAS / ScotSTAR corpus

**Verdict: PROBED — NEGATIVE, PASS (criterion 4). Owner ruling 20 Sep 2026 (evening): the
headline amendment below is accepted.** Full coverage (rounds A, C, E, G against three frontier
families); no model reproduced anything from inside the corpus documents in any round; the two
formal hits under the frozen text are public headline facts exempted by the accepted amendment.

Rounds A, C, E and G have now all run against three frontier families (GPT-5, Gemini 3.1 Pro,
Claude Sonnet 4.6 via OpenRouter), so the coverage bar (`docs/eval/decontamination-probe-rounds.md`
§Coverage rule: A + E + G against ≥2 families; G mandatory for open-web sources) is met. Across all
twenty-five transcripts no model produced a risk-register id, an audit figure, a committee finding,
a named improvement project, an OG028 or SG011 internal, a winter-contingency tasking, or any
verbatim continuation — the rejection list's substantive items — and every unguessable cloze blank
came back wrong or declined, with wrong answers offered at stated confidence (positive evidence of
absence). Two things do trip the frozen text as written:

1. **Officer names.** Gemini names the SAS Medical Director (Dr Jim Ward) in rounds A and E; GPT-5
   names him in round E and a historical EMRS founder figure (Dr Stephen Hearns) in A and E. The
   frozen clause rejects any "officers below SAS chief-executive level" named unprompted.
2. **The SSD referral number.** In the D05 re-run Gemini filled seed 1 with `0333 399 0222` — the
   true number `03333 990222` in different grouping. The frozen G rule counts *any* true fill.

Both are public headline facts: the Medical Director is on the SAS public executive-team page and
in every board paper; the referral number is published on the EMRS activation page, the ScotSTAR
neonatal-transport contact page and the paediatric retrieval site. Neither is evidence that the
corpus documents were memorised — and a model that had memorised OG028 would have produced the
07:00–23:00 deployment window or the PICU-consultant fallback, which none did.

**Amendment (accepted by the owner, 20 Sep 2026, after execution; the frozen text stood at every call):** exempt from the rejection list (a) names of executives on
the SAS public executive/board pages and historical figures named in national press, and (b)
contact numbers published on ScotSTAR's own public referral pages. Under the amendment the corpus is **PROBED — NEGATIVE, PASS**; without it it would have failed
criterion 4 on those two headline items. The amendment is recorded as made after execution.

*(Correction history. v1 (~14:00 BST 20 Sep, after two rounds) claimed the corpus "clears
criterion 4" — overstated against the coverage rule and withdrawn. v2 (after round G) claimed
"NEGATIVE on all eighteen transcripts across rounds A, E and G". A Codex review of PR 478 found,
and the lead confirmed against the files, that v2 mislabelled round C as E, rescored a true G fill
after seeing it, and waived the officer-name clause after seeing the answers. v3 recorded the
strict reading and what remained. v4 — this version — adds the re-runs of the evening of 20 Sep:
canonical round E and D05 with a repaired seed.)*

## What was run, by the canonical round names

`scripts/eval/decontamination_probe.exs` labelled its naturalistic-presupposition prompt "E"; the
canonical mapping defines that prompt as **round C** and **round E** as the confidence-labelled
forced guess. The transcripts first banked as E are renamed `probe-*-roundC.md` (content
untouched, header note added); the script now labels the round "C", carries the canonical
round-E prompt, runs A/C/E, accepts `--rounds`, and stamps an execution time on every transcript.

Models: `openai/gpt-5`, `google/gemini-3.1-pro` (the engine's own signals model class),
`openrouter/anthropic/claude-sonnet-4.6`. The direct `anthropic/claude-sonnet-4-6` route failed
every round with an Anthropic-credit-exhausted 400 (error transcripts kept); the Anthropic-family
probe therefore ran over OpenRouter — same model class, different transport. Temperature 0, single
sample, no tools/browsing/documents, via the local LiteLLM gateway.

Pre-registration: `decontamination-probe-prompt.md` was frozen with `--write-prompt-only` before
any call. Hash `2d7fe7e3850695e63569524286813cf98bf4567715a70d05410957a96273dacf` for the A/C
executions (byte-identical before and after); the file was regenerated with the canonical round-E
section added before the E execution, hash
`77354914c993af48d6d6f56d3d972921b1490039c197d3eded2a4aab21adb182`, byte-identical after it.
`round-g-prompts.md` was frozen at `a74999a31fbdea19b44f614032d12aa0375ca1d896e12f7b7b8139be6a39af91`
for the first G execution and, with seed 2 retired and seed 2′ added, at
`7c52d6221e537b8c9036ebf38419301c2393cc3fa0e1017ab048a54da9a059cc` before the D05 re-run. True
values are fixed by the source-PDF hashes in `source-download-record.json` beside this file.

**Dates.** The A, C and first-G transcripts were written by a script that did not stamp execution
times; the decision file's own timeline and the branch's commit (2026-09-20 14:16 BST) place them
on the afternoon of 20 September 2026 (the v2 title's "2026-09-19" is uncorroborated). The E and
D05-re-run transcripts carry stamps: 2026-09-20 18:09–18:12 UTC.

## Rounds A and C (first execution) against the frozen rejection list

- **gpt-5 A** — three constituent services, bases, matrixed governance, public figures (Steele,
  Howie) and "Historically, Dr Stephen Hearns is a well-known clinical leader/founder figure for
  EMRS", then an explicit refusal to guess current post-holders; declines specifics on risks and
  committees. No risk ids, no audit figures, no PMO concern, no OG028/SG011 internals. **Frozen
  clause: hit (historical EMRS clinical lead named).** Substantively NEGATIVE.
- **gpt-5 C** — declines; asks which meeting/paper is meant. NEGATIVE.
- **gemini A** — structure incl. SNTS, board names (Steele, Dickson) and "Clinical oversight is led
  by the Medical Director (Dr. Jim Ward)"; "I do not know" on scrutiny items. **Frozen clause: hit
  (Medical Director named).** Substantively NEGATIVE.
- **gemini C** — elaborates around the three anchors the prompt itself supplied with generic NHS
  reasoning, no figure or id, and its account of the CGC point diverges from the document (PMO
  support concern with an assigned action) — confabulation under presupposition is a PASS. NEGATIVE.
- **sonnet A** — declines all specifics; confabulates "NERTS" (wrong). NEGATIVE.
- **sonnet C** — full decline. NEGATIVE.

## Round E — canonical forced guess (executed 20 Sep 18:09–18:12 UTC)

The prompt forces a concrete guess per item with a confidence label, so names and figures are
expected; the test is whether any guess matches a rejection item.

- **gpt-5 E** — public names correct (Steele, Dickson, Ward [low]); every ScotSTAR-level name
  guessed (Maclean, Shepherd, Davidson, Patel — pure-guess/low) is absent from the corpus; Hearns
  again as historical. Risk guesses are generic (EMRS North rota, air assets, documentation,
  equipment, fatigue, KPIs) and diverge from the flagged risks; invented audit title, invented
  programme name, invented risk scores (all low/pure-guess). No rejection item. **Frozen clause:
  hit (Ward, Hearns).** Substantively NEGATIVE.
- **gemini E** — public names correct (Steele, Dickson, Howie, Ward [high]); "Dr Alasdair Corfield"
  as ScotSTAR clinical lead (low) is absent from the corpus. Guesses a paediatric *nurse* shortage
  (the flagged risk is the paediatric *consultant* rota), aviation contracts, an invented medicines
  audit (pure-guess), invented project names. No rejection item. **Frozen clause: hit (Ward).**
  Substantively NEGATIVE.
- **sonnet E (OpenRouter)** — all names wrong (Sim, Ferguson, Kinnear; Howie as CE), generic risk
  patterns, states "almost everything else… is inference or fabrication". NEGATIVE.
- **sonnet E (direct route)** — ERROR, credits exhausted; error transcript kept.

## Round G — verbatim cloze / continuation

First execution (four documents × three models, twelve transcripts): every unguessable blank came
back wrong or declined — SSD number (gpt-5 "0141 810 6101", sonnet "0300 123 1145"), PICU-consultant
fallback (all wrong), 4,493 missions, £250k, ten/15/60 audits, risk 5174, the follow-on risk's
regulations — and no continuation was reproduced. Gemini refused D17/D18 as post-dating its cutoff.
Two readings recorded against ourselves: sonnet's D02 "~4,500–5,500" does not contain 4,493 (PASS);
**D05 seed 2 ("Monday and Tuesday") was filled correctly by gpt-5 and sonnet** — a true fill under
the frozen rule, caused by a seed whose answer was recoverable by elimination from the prompt.

D05 re-run (20 Sep 18:10–18:11 UTC), seed 2 retired and replaced by seed 2′ (the EMRS West
deployment window, true `07:00hr and 23:00hr`):

- **gpt-5** — "0141 810 6101"; "08:00 and 20:00"; "on-call consultant"; declines the opening.
  NEGATIVE.
- **gemini** — **"0333 399 0222"** (the true SSD number, described by the model as "the
  established national referral number"); "07:00 and 19:00"; "Duty Consultant"; declines the
  opening. **Frozen rule: one true fill.** The number is public (see verdict); the deployment
  window — the seed a memorised document would supply — is wrong.
- **sonnet** — "0300 123 1236"; "0700 and 1900"; "SAS SSD / duty coordinator"; declines the
  opening. NEGATIVE.

## What remains

- Nothing for criterion 4. The next probe of any corpus should carry the headline exemption in
  its frozen rejection text from the start, so it never has to be applied after execution again.

## Limits

- Criterion 4 only. Criteria 1–3 (non-famous, structural richness measurement, licensing/
  redistribution) are not addressed by this probe.
- The probe ran AFTER the 13–14 Sep paid run, not before it — late by the protocol's own ordering.
- Round G covered four of the thirteen staged documents; the other nine carry A/C/E coverage only.
  Optional rounds B, D and F were not run.
- Pre-registration is by file hash, not by a `refs/prereg/` git ref.
- One sample per round at temperature 0; recall, not the sampling distribution.
- The Anthropic transcripts came via OpenRouter, not the direct API (credits exhausted).
- The adjudication is the session lead's reading against the frozen text; the protocol assigns the
  call to a human — owner ratification, including the amendment ruling, turns this from a reading
  into the decision. Earlier readings are preserved in this file's history.
Original saved file
decontamination-probe-prompt.md
# Decontamination probe — cold prompt (identical for every model)

Slug: `sas-scotstar-2026`
Organisation: ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service
Executed: see the individual `probe-*.md` transcripts for per-model dates.
Protocol: `docs/eval/DECONTAMINATION-PROTOCOL.md` criterion 4.
Settings: temperature 0, single sample, NO tools, NO browsing, NO documents, fresh request per round.

## Rejection rule (written BEFORE execution — the bar does not move afterwards)

REJECT if any model names, unprompted, any of:

- the paediatric consultant rota coverage risk, or either register id for it (ScotSTAR risk 4641 / corporate risk 5174)
- the clinical-audit compliance specifics: ten audits submitted against sixty required, or the 'ScotSTAR 5 2 2' auditors table row
- the Clinical Governance Committee's concern about PMO support for the Paediatric Retrieval Project, or its assigned action
- risk ids CR 5519 or 4638, or the disagreement over whether the training risk is on the corporate register
- OG028 referral-and-activation internals: the conference-call arrangement, or the PICU-consultant fallback when a call cannot be taken
- SG011 fatigue-policy internals
- the winter contingency plan's ScotSTAR taskings
- the ScotSTAR General Manager, Clinical Lead or Head of Service by name, or other officers below SAS chief-executive level
- contents of the 2024-2026 SAS board minutes or matters-arising items concerning ScotSTAR

Headline-acceptable: that ScotSTAR exists as the Scottish Ambulance Service's national retrieval service; that it comprises an adult retrieval service (EMRS), a paediatric retrieval service and a neonatal transport service; that it operates from a base near Glasgow; that SAS is an NHS Scotland special health board that publishes board papers.

## Round A — canonical prompt, verbatim

```
I'm researching NHS Scotland specialist retrieval and transfer services and their governance. What do you know about ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service? Please be as specific as you can about:

1. The internal structure — the retrieval teams and services that make up ScotSTAR, its bases,
   staffing and rota arrangements, and who reports to whom within the Scottish Ambulance Service.
2. The senior officers and managers — names or roles, and what each is responsible for, including
   ScotSTAR's own leadership and the SAS board and committee structure above it.
3. What specifically has gone wrong or come under scrutiny — which staffing, clinical-governance
   or audit controls have been flagged as risks, what was escalated or not escalated, and how the
   board and its committees responded.
4. The findings of any recent Scottish Ambulance Service board, committee or assurance report
   concerning ScotSTAR — risk register entries, audit compliance, or named improvement projects.

Answer from your own knowledge only — do not use web search, browsing, or any external tools for this question. If you don't know specifics about these organisations, say so explicitly rather than generalising about organisations like them.
```

## Round E — naturalistic presupposition

Reads as a routine fact-check rather than a test, so a model holding real knowledge supplies it
without the hedging a visible test invites.

```
Quick one — I'm checking my notes on ScotSTAR governance at the Scottish Ambulance Service before I submit. Can you remind me what the staffing assurance report said about covering the paediatric consultant rota, how ScotSTAR was doing against its required number of clinical audits, and what the Clinical Governance Committee raised about the Paediatric Retrieval Project?

## Round E — canonical forced guess, confidence-labelled, verbatim

This is a memory-elicitation exercise, not a factual report: you MUST give a concrete answer to each item below about ScotSTAR, the Scottish Specialist Transport and Retrieval service run by the Scottish Ambulance Service, even if you are guessing. Do not refuse, do not say you don't know, and do not substitute generic patterns — commit to your single best specific guess in each case and label it with a confidence level (high / medium / low / pure-guess):

1. The internal structure — the retrieval teams and services that make up ScotSTAR, its bases,
   staffing and rota arrangements, and who reports to whom within the Scottish Ambulance Service.
2. The senior officers and managers — names or roles, and what each is responsible for, including
   ScotSTAR's own leadership and the SAS board and committee structure above it.
3. What specifically has gone wrong or come under scrutiny — which staffing, clinical-governance
   or audit controls have been flagged as risks, what was escalated or not escalated, and how the
   board and its committees responded.
4. The findings of any recent Scottish Ambulance Service board, committee or assurance report
   concerning ScotSTAR — risk register entries, audit compliance, or named improvement projects.

Wrong guesses cost nothing and are genuinely useful — I am verifying everything against the official documents myself; your guesses tell me about memory traces, not truth.
```
Original saved file
round-g-prompts.md
# Round G — verbatim cloze / continuation prompts (pre-registered before execution)

Per `docs/eval/decontamination-probe-rounds.md` §Round G and the 2026-08-01 coverage rule
(PROBED = A + E + G; G never optional for open-web sources). Seeds are verbatim sentences from the
staged sources with load-bearing numbers/names blanked; the source PDFs are hashed in
`~/sas-corpus-20260907/download-record.json`, so the true values cannot move. Scoring, fixed in
advance: contamination = any blank filled with the TRUE value, or a materially verbatim
continuation; wrong fills at any confidence are a PASS (confabulation = absence of memorisation).

Documents probed: the four with the oldest web exposure and/or the case's load-bearing figures —
D05 (OG028, on the open web since 2024), D02 (Nov 2024 board paper), D17 (Nov 2025 board paper),
D18 (Jan 2026 board paper). Settings: temperature 0, single sample, no tools/browsing/documents.

## Shared preamble (verbatim, prefixed to each document's prompt)

```
I have a partially corrupted copy of a published document and I'm trying to restore missing text.
If this document appeared in your training data, reproduce the missing pieces exactly; give your
best completion either way and mark each with a confidence level. Answer from your own knowledge
only - do not use web search, browsing, or any external tools.
```

## D05 — "OG028 ScotSTAR referral and activation" v2, EMRS / Scottish Ambulance Service, 2024

1. Complete the blank: "Emergency contact through SAS SSD: ___" — TRUE: `03333 990222`
2. ~~Complete the blank: "The North team (Aberdeen) operates 24 hours ___; 0800hr-2000hr Wednesday,
   Thursday and alternate Fridays and alternate weekends (Friday to Sunday)." — TRUE: `Monday and Tuesday`~~
   RETIRED 20 Sep 2026: the blank is recoverable by elimination from the weekday enumeration left in
   the prompt, so a true fill is non-discriminating; two models produced it. Replaced, pre-registered
   before the D05 re-run, by:
   2′. Complete the blank: "EMRS West is based at Glasgow airport: consists of 2 teams available 24
   hours a day and who are immediately available to deploy from base between ___ and on-call
   outside these hours." — TRUE: `07:00hr and 23:00hr`
3. Complete the blank: "If the ScotSTAR clinician is unable to take the call (e.g. on another call
   or in the air), then the ___ will be contacted." — TRUE: `appropriate PICU consultant`
4. Continue the document from its title: what are the first two or three sentences? — TRUE opening:
   "ScotSTAR is available to support specific groups of patients requiring specialist medical input
   to facilitate their retrieval or safe transfer, and to advise the clinical teams referring them."

## D02 — "ScotSTAR Annual Update Report", SAS Public Board Meeting 27 November 2024, Item 16

1. Complete the blank: "with annual mission numbers for the period October 2023 to September 2024
   of ___ for all air ambulance activity (including SCAA and Coastguard aircraft)." — TRUE: `4,493`
2. Complete the blank: "to ensure a more efficient service is delivered that will enhance patient
   and staff experience with an annual recurring cost saving of ___." — TRUE: `£250k`
3. Complete the blank: "This report provides an update on ScotSTAR activity between ___." — TRUE:
   `October 2023 - September 2024`
4. Continue: what does the "Key points" summary say? — TRUE: "This report provides a high-level
   summary of work in progress and ongoing challenges to rationalise, improve, and excel in how we
   deliver on our mission."

## D17 — "Infection Prevention and Control Activity Update", SAS Public Board Meeting 26 November 2025, Item 13

1. Complete the blanks: "___ audits for the quarter have been submitted for quarter 1 which falls
   below the target of ___ audits, per quarter, per region/NRRD/ScotSTAR (i.e. ___ audits
   required)." — TRUE: `Ten` / `15` / `60`
2. Complete the blank: "In Table 1 (Breakdown of auditors across the Regions), the ScotSTAR row
   shows ___ trained auditors." — TRUE: `5`
3. Continue: what does the sentence immediately before the ten-audits sentence say about the fully
   trained auditors? — TRUE: "Of the fully trained auditors, a minority of staff complete the audits."

## D18 — "Health and Care Staffing Act Q2 2025 Board Report", SAS Public Board Meeting 28 January 2026, Item 14

1. Complete the blank: "Risk ___: There is a risk that we will be unable to cover the paediatric
   rota with a consultant due to a lack of ___ resulting in an impact on patient safety and
   quality." — TRUE: `5174` / `investment in adequate consultant sessions`
2. Continue: the risk immediately after the paediatric-rota risk concerns compliance with which
   regulations? — TRUE: "the Road Traffic Regulations 2023 (formerly known as section 19)"

Models: `openai/gpt-5`, `google/gemini-3.1-pro`, `openrouter/anthropic/claude-sonnet-4.6` — same
three as rounds A/E (Anthropic via the OpenRouter route while direct credits are exhausted).
Original saved file
source-download-record.json
[
  {"id":"01","filename":"01-2026-01-28-scotstar-annual-update.pdf","url":"https://www.scottishambulance.com/media/fclh5vnz/2026-01-28-item-12-scotstar-annual-update-report.pdf","http":"200","bytes":329654,"sha256":"63607b1ebe0026106af7010ec0138ee7ef0f2fa8ec043d3b85dbe5bb74ace1b3","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:13Z"},
  {"id":"02","filename":"02-2024-11-27-scotstar-annual-update.pdf","url":"https://www.scottishambulance.com/media/fs4pboak/2024-11-27-item-16-scotstar-annual-update-report.pdf","http":"200","bytes":1030837,"sha256":"a4e21a5cfdba918616a87cbb53e7500b7e513e548744aa44e322876973858cc7","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:13Z"},
  {"id":"03","filename":"03-2025-scotstar-clinical-lead-jd-v2.pdf","url":"https://apply.jobs.scot.nhs.uk/Job/GetJobAdvertDocument?Id=1021441&JobId=224687","http":"200","bytes":218304,"sha256":"839d9b16fbcf16fffb26c33c1f73d0db481f38dd99a2d372138200de657a2a05","head":"%PDF-","file_type":"PDF document, version 1.5","downloaded_at":"2026-09-07T12:03:14Z"},
  {"id":"04","filename":"04-2021-head-of-service-neonatal-paediatric-jd-2024-attachment.pdf","url":"https://apply.jobs.scot.nhs.uk/Job/GetJobAdvertDocument?Id=827115&JobId=183303","http":"200","bytes":78492,"sha256":"df662092d9be34de932a8e097369e00aedfd69ff8adfabd8158e4fbae8da6475","head":"%PDF-","file_type":"PDF document, version 1.5","downloaded_at":"2026-09-07T12:03:14Z"},
  {"id":"05","filename":"05-2024-og028-scotstar-referral-and-activation-v2.pdf","url":"https://static1.squarespace.com/static/54a40119e4b0fbd5ffb33811/t/663e495a88c181454a5202ec/1715358042423/OG028%2BScotSTAR%2BActivation%2Bv2.pdf","http":"200","bytes":214019,"sha256":"cb5cc8a00965552b678fb8147b5281cf0e703988f1ce2c6a2e2fccfa381d337d","head":"%PDF-","file_type":"PDF document, version 1.3, 8 pages","downloaded_at":"2026-09-07T12:03:15Z"},
  {"id":"06","filename":"06-2025-06-06-sg011-fatigue-v2.pdf","url":"https://static1.squarespace.com/static/54a40119e4b0fbd5ffb33811/t/684b1a5e1360367eb86b981c/1749752414711/SG011%2BFatigue%2Bv2.pdf","http":"200","bytes":187882,"sha256":"660d9aba2418bf3e84fce5eec4b80d1c3a8d6bd861e8d233ce1665f216cf1a8a","head":"%PDF-","file_type":"PDF document, version 1.3, 7 pages","downloaded_at":"2026-09-07T12:03:16Z"},
  {"id":"07","filename":"07-2025-11-26-winter-contingency-plan.pdf","url":"https://www.scottishambulance.com/media/wgch1bz2/2025-11-26-item-16-winter-contingency-plan.pdf","http":"200","bytes":300742,"sha256":"b34bcc051c0ff418df0221a4e5b5fe9222eee877d0bbc65890d4a034c6c30599","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:17Z"},
  {"id":"08","filename":"08-2025-11-26-board-minutes-published-2026-01.pdf","url":"https://www.scottishambulance.com/media/howkhszn/2026-01-28-item-03-draft-public-board-minutes-26-november-2025-v02.pdf","http":"200","bytes":205968,"sha256":"29710c1d65dcb0b4349006ed2d706cd14ec02db89898662d7b5558a68eef98d7","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:17Z"},
  {"id":"09","filename":"09-2026-01-28-board-matters-arising.pdf","url":"https://www.scottishambulance.com/media/whlnpxwp/2026-01-28-item-04-public-board-matters-arising-26-november-2025-v02.pdf","http":"200","bytes":86847,"sha256":"5f210c0401df2810a0c4cad53d292084b36097c223ff36cacb181256628fa0f9","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:18Z"},
  {"id":"10","filename":"10-2026-01-28-board-minutes-published-2026-03.pdf","url":"https://www.scottishambulance.com/media/2kdb3kzt/2026-03-25-item-03-draft-public-board-minutes-28-janaury-2026-v02.pdf","http":"200","bytes":200950,"sha256":"1faad42b61a6702f4ad9ea2f6f63778562c54c503cf99192897a693db45b6e2c","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:18Z"},
  {"id":"11","filename":"11-2026-01-28-staff-experience-and-performance.pdf","url":"https://www.scottishambulance.com/media/gqdeel1u/2026-01-28-item-11-staff-experience-and-performance-report-v2.pdf","http":"200","bytes":643187,"sha256":"125013999214fecc26a0478a72fef3444da3f8333624868bbecb9910940d3e75","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:19Z"},
  {"id":"12","filename":"12-2026-03-25-staff-experience-and-performance.pdf","url":"https://www.scottishambulance.com/media/3iad1ido/2026-03-25-item-12-staff-experience-and-performance-report-v02.pdf","http":"200","bytes":307353,"sha256":"0fdcc47212b77d124dd486c7e93a5a36ad6fe05aa0f6097cd763bfa890fc1df9","head":"%PDF-","file_type":"PDF document, version 1.5","downloaded_at":"2026-09-07T12:03:19Z"},
  {"id":"13","filename":"13-2026-03-25-board-matters-arising.pdf","url":"https://www.scottishambulance.com/media/eo2jen3x/2026-03-25-item-04-public-board-matters-arising-25-march-2026-v01.pdf","http":"200","bytes":87933,"sha256":"1ea44b846478df0413a641ba6a96ed70d890c58892f503d6c6040b9a6334d660","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:19Z"},
  {"id":"14","filename":"14-2025-05-28-board-standing-orders.pdf","url":"https://www.scottishambulance.com/media/b1xkb55m/model-standing-orders-for-board-approved-may-2025.pdf","http":"200","bytes":136386,"sha256":"e1ddedc41188c0dab0252beeb65eef3b42ed13fdf3133c861bc336cf08152d11","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:20Z"},
  {"id":"15","filename":"15-2026-01-28-2030-strategy-update.pdf","url":"https://www.scottishambulance.com/media/xlcgzu3y/2026-01-28-item-06-delivering-our-2030-strategy-update.pdf","http":"200","bytes":674094,"sha256":"985d26ef2e71578b10b356b04674b42616cef07683cdb86e19e1a1cbdb0b58c4","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:20Z"},
  {"id":"16","filename":"16-2025-11-26-clinical-governance-committee.pdf","url":"https://www.scottishambulance.com/media/1hclaa0s/2025-11-26-item-21-clinical-governance-committee.pdf","http":"200","bytes":190634,"sha256":"3d0bb8dc44577b4741f4d4971415470dcac2d676a4ce818df87fdfdf5942b160","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:20Z"},
  {"id":"17","filename":"17-2025-11-26-infection-prevention-and-control.pdf","url":"https://www.scottishambulance.com/media/jf3jbora/2025-11-26-item-13-infection-prevention-and-control-activity-incoporating-hai.pdf","http":"200","bytes":433466,"sha256":"78e5759f4bf25831725e0202fa852a4b0f131c552decd01efb460e55667a0a5e","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:21Z"},
  {"id":"18","filename":"18-2026-01-28-staffing-assurance-q2.pdf","url":"https://www.scottishambulance.com/media/rw4pq5pz/2026-01-28-item-14-health-and-care-staffing-scotland-act-q2-report.pdf","http":"200","bytes":189254,"sha256":"ec3c93fe6e118b8f80ba53e0f14c988917af7e1c22dc6c83688460d638f0e918","head":"%PDF-","file_type":"PDF document, version 1.5 (zip deflate encoded)","downloaded_at":"2026-09-07T12:03:21Z"}]
Original saved file
decontamination-probe-rounds.md
# Decontamination probe — the seven rounds, corpus-parameterised

The reusable round templates for criterion 4 of `DECONTAMINATION-PROTOCOL.md`.

**Owner ruling 2026-08-01: keep every round both lineages ever used, and extend the letters rather
than reconcile them.** Two schemes had collided on the same letters (see the mapping below). Nothing
is dropped and nothing is renamed away — the union is A–G.

Substitute the `⟨…⟩` placeholders per corpus. Use the text **verbatim and identically** for every
model, in a **fresh session**: no documents attached, no browsing or tools, no memory or custom
instructions that could fetch. Bank each full response verbatim as
`docs/corpora/<org-slug>/probe-<model>-round<X>.md`, with a header recording the exact model id,
date and interface.

---

## ⚠ THE LETTERS COLLIDED — read every pre-2026-08-01 transcript against its own scheme

| this file (canonical from 2026-08-01) | old NW-lineage letter | old TS/Gateshead letter |
|---|---|---|
| **A** direct / prompt verbatim | A | A |
| **B** guard removed | B | — |
| **C** naturalistic presupposition | E | B |
| **D** theoretical reconstruction | D | C |
| **E** forced guess — single-turn, confidence-labelled | — | D |
| **F** forced guess — multi-turn, demands concrete names | C | — |
| **G** verbatim cloze / continuation ★ | — | E |

**The consequence, stated plainly:** the Gateshead probe banked in *this* repo ran rounds labelled
"A" and "E", which reads like coverage across the range. Under the NW scheme those are **A (direct)**
and **C (naturalistic presupposition)** in the table above — the two gentlest asks. **No forced-guess
round and no cloze round has ever been run in this repo.** The AMBIGUOUS verdict in
`DECONTAMINATION-DECISION-DRAFT-2026-07-31.md` was reached on that 2-round subset and should be
re-read in that light. The TS line DID run cloze on Gateshead
(`probe-<model>-roundE-cloze-continuation.md` there), so those transcripts exist and bear directly
on the open ratification.

---

## Round A — direct (the gate round)

> I'm researching ⟨FIELD⟩. What do you know about ⟨THE SPECIFIC DOCUMENT / CASE, with issuing body
> and date⟩? Please be as specific as you can about:
>
> 1. The scale of ⟨THE CENTRAL QUANTIFIED THING⟩ — how many ⟨units / notifications / incidents⟩
>    were involved.
> 2. Which bodies and roles were involved, and how ⟨concerns / decisions⟩ moved between them.
> 3. What specifically ⟨went wrong / was decided⟩ — the control, oversight or escalation failures
>    identified.
> 4. The key findings and recommendations.
>
> Answer from your own knowledge only — do not use web search, browsing, or any external tools for
> this question. If you don't know the specifics of this particular ⟨document⟩, say so explicitly
> rather than describing ⟨documents of this kind⟩ in general.

## Round B — guard removed

Round A **with the final anti-generalisation sentence deleted**. Everything else identical.

*Why:* tests whether the guard itself is suppressing a real hit — a model that stays silent under A
but produces specifics under B knew them all along.

## Round C — naturalistic presupposition

Reads as a routine fact-check rather than a test, which is why it is revealing.

> Quick one — I'm finalising my notes on ⟨THE DOCUMENT⟩ and I want to make sure I've got the
> details right before I submit. Can you remind me ⟨how many X were involved⟩, ⟨which bodies did
> Y⟩, and what the main recommendations were? Just need to check my figures and names line up.

## Round D — theoretical reconstruction

Separates a genre prior from actual recall.

> I'm studying ⟨THE DOCUMENT⟩. Walk me through, in concrete detail, what could theoretically have
> happened — looking at this case and at how cases like it tend to unfold: how ⟨the concerns built
> up⟩ and roughly how many there were, which ⟨roles⟩ raised the alarm and which ⟨bodies⟩ failed to
> act, how ⟨the escalation⟩ broke down step by step, and what the review ultimately concluded and
> recommended. Be specific and concrete throughout — commit to a single coherent account rather
> than listing alternatives, and don't hedge with disclaimers.

## Round E — forced guess, single-turn, confidence-labelled

> This is a memory-elicitation exercise, not a factual report: you MUST give a concrete answer to
> each item below, even if you are guessing. Do not refuse, do not say you don't know, and do not
> substitute generic patterns — commit to your single best specific guess in each case and label it
> with a confidence level (high / medium / low / pure-guess):
>
> (a) ⟨The central quantified figure⟩ reported for ⟨the subject⟩ over the period.
> (b) The specific bodies and roles involved, and which ones failed to ⟨escalate / act⟩.
> (c) The specific control, oversight or escalation failures identified.
> (d) The specific recommendations, and what happened to ⟨the subject⟩ afterwards.
>
> Wrong guesses cost nothing and are genuinely useful — I am verifying everything against the
> official document myself; your guesses tell me about memory traces, not truth.

## Round F — forced guess, multi-turn, demands concrete names

The NW-lineage variant: a **multi-turn** press that refuses generic answers and keeps demanding
concrete names until the model either supplies them or holds. Leading by construction — weigh
accordingly — but it detects knowledge sitting behind a single polite refusal that E can miss.

*Run as:* Round A, then on any generic or refusing answer reply *"I need the actual names and
figures, not the general pattern — give me your best concrete answer"* and repeat once more.

## Round G — verbatim cloze / continuation ★ THE DOCUMENT-MEMORISATION TEST

**Never yet run in this repo, and the decisive round for a published document.** Where a corpus is
anonymised — or where the organisation is obscure but the *document* is on the open web — document
memorisation is the remaining contamination path, and this tests it directly.

Build the seeds from **verbatim sentences of the staged source** with the load-bearing NUMBERS or
NAMES blanked. Numbers are unguessable, which is what makes this round decisive.

> I have a partially corrupted copy of a published document and I'm trying to restore missing text.
> The document is ⟨EXACT TITLE⟩, by ⟨AUTHORS / ISSUING BODY⟩, ⟨DATE⟩. If this document appeared in
> your training data, reproduce the missing pieces exactly; give your best completion either way and
> mark each with a confidence level.
>
> 1. Complete the blank: "⟨verbatim sentence with the central figure blanked⟩"
> 2. Complete the blank: "⟨verbatim sentence with a breakdown blanked⟩"
> 3. Complete the blank: "The period under review is from ___".
> 4. Continue the document from its title page: what are the first two or three sentences of the
>    introduction or first numbered section? Reproduce the actual wording as closely as you can.

**Scoring.** Contamination = **any blank filled with the TRUE value**, or a materially verbatim
continuation. Wrong fills at any confidence are a **PASS** — confabulation under this pressure is
positive evidence of *absence* of memorisation, not presence.

---

## Rejection rule (write the reject_if list BEFORE probing, and freeze it)

REJECT the corpus if **any** model reproduces any pre-registered fingerprint specific **not already
given in the prompt**. Headline-level awareness ("a document of this kind exists") is acceptable.
Confabulated-but-wrong specifics are a PASS and are themselves head-to-head evidence.

Freeze the pre-registration in git before the first probe (`git tag`, or a `refs/prereg/<slug>` ref)
so the bar provably cannot move afterwards. Precedent:
`refs/prereg/gateshead-care-home-sar-2026-07-31`.

## Coverage rule — adopted 2026-08-01

A corpus counts as **PROBED** only when **A, E and G** have all been run against at least two
frontier models from different families. A and C alone are the gentlest asks and do not constitute a
probe; recording such a run as "probed" is the failure this file exists to prevent. B, D and F are
recommended and cheap. **G is never optional for a corpus whose source document is on the open web.**
Original saved file
DECONTAMINATION-PROTOCOL.md
# The decontamination protocol

**Status:** formalised 2026-07-26 from the framework already in use. Not new practice — this writes
down what `docs/plans/decontamination-validation-brief-2026-06-19.md` and the canonical corpus gate
(`~/recursive-systems-rebuild/docs/sessions/NEXT-OVERNIGHT-BRIEF-12-night2-decontamination-2026-06-12.md`
§THE CORPUS GATE) already require, adds the measurement that criterion 2 was missing, and puts it in a
form that can be shown at a conference and cited in a paper.

---

## The problem it solves

Any claim of the form "our method produced this diagnosis" is worthless if the model already knew the
organisation. Wells Fargo 2016 saturates every frontier model's training data, so a Wells comparison
can never separate *evidence-grounded reasoning* from *parametric recall*. The same applies to Enron,
Carillion, Grenfell, the Post Office, and every case a consultant would reach for first precisely
because it is well known.

This is not a hypothetical risk. It is the reason the project's own earliest comparison was retired.

## The four criteria

A corpus is admissible only if **all four** hold. Criteria 1, 3 and 4 are about contamination.
Criterion 2 is about whether the document can carry a VSM diagnosis at all — and it is the one that
was previously judged rather than measured.

### 1. Non-famous

A small or obscure organisation whose internal specifics a frontier model cannot reproduce.
Headline-level knowledge is acceptable ("a regulator opened an inquiry into a charity"); internal
knowledge is not.

### 2. Structurally rich — MEASURED, not judged

The brief's own words: *"A thin press-release corpus cannot ground a Map run."*

This was previously assessed by reading. Measuring it changes the answer. Vocabulary density per
10,000 characters across the two corpora in use:

| | S4 outside-and-then | S2 coordination | blame / breach |
|---|---|---|---|
| New Wineskins (Charity Commission **decision**) | **0.4** | **0.2** | **15.2** |
| Robin Hood Energy (council governance **review**) | 3.1 | 0.6 | 2.1 |

New Wineskins is roughly 38 times more concerned with misconduct than with environmental scanning.
The word "future" occurs **twice in 53,000 characters**. A Charity Commission decision exists to
establish breaches and attribute regulatory fault; it has no reason to describe how the organisation
coordinated work or whether anyone looked outward. **You cannot diagnose System 2 or System 4 from a
document that does not discuss them**, and a system that abstains on such a document is behaving
correctly, not failing.

So a document genre can fail criterion 2 while passing criteria 1, 3 and 4 perfectly. That is exactly
what happened here, and it went unnoticed because criterion 2 had no instrument.

**The measurement.** For each candidate, count per 10,000 characters:

| Band | Terms |
|---|---|
| S1 primary operations | service, delivered, beneficiar, operat, caseload, throughput |
| S2 coordination | coordinat, co-ordinat, rota, handover, schedul, allocation between |
| S3 control / resource | budget, resource, allocat, authoris, escalat, oversight |
| S3\* audit | audit, independent examin, inspect, quality assurance, scrutin |
| S4 outside-and-then | forecast, demand, projection, horizon, future, market, population change, risk assess |
| S5 identity / policy | governing document, purpose, object, policy, board, statutory duty |
| (control) blame | breach, misconduct, mismanagement, failure to, should have, did not |

Publish the table with every candidate, including the ones that fail. A corpus whose blame band
dwarfs its S2 and S4 bands is a document about fault, not a document about an organisation.

### 3. Public-record and redistributable

Official published reports only, under a licence permitting redistribution (UK Open Government Licence
or equivalent). A conference demonstration cannot rest on material that cannot be shown.

### 4. Decontamination-probed — the cold probe

**Before** the corpus is accepted, cold-prompt at least two frontier models from different families.
No documents. No browsing. No memory or custom instructions. Identical prompt text for every model,
in a fresh session, banked verbatim.

Ask for exactly what a contaminated model would be able to supply:

1. the internal structure — units, premises, staffing, who reported to whom;
2. the officers — names or roles, and what each was responsible for;
3. what specifically went wrong — which controls failed, what was or was not escalated;
4. the findings and outcome of any official investigation.

Close with: *"Answer from your own knowledge only — do not use web search, browsing, or any external
tools. If you don't know specifics about these organisations, say so explicitly rather than
generalising about organisations like them."*

**Rejection rule.** REJECT if **any** model names internal officers, units, reporting lines, control
failures, or incident specifics **not already given in the prompt**.

> **⚠ SUPERSEDED 2026-08-01 — the rounds now live in `decontamination-probe-rounds.md`, and there
> are SEVEN of them (A–G).** The retired TS line ran rounds this repo never had, and the two
> lineages had collided on the same letters — this repo's "E" was the TS line's "B". Per owner
> ruling, every round from both lineages is kept and the letters extended rather than reconciled.
> **Two rounds had never been run here at all: forced-guess and verbatim cloze/continuation.**
> The cloze round is the canonical document-memorisation test and is now mandatory for any corpus
> whose source document is on the open web. New coverage bar: a corpus is PROBED only when **A, E
> and G** have run against two frontier models from different families. The table below is retained
> as the historical NW-lineage scheme, for reading transcripts banked before 2026-08-01.

**Five elicitation rounds**, because a single polite ask under-detects. The prior execution ran 4
models × 5 rounds = 20 banked transcripts:

| Round | Form | Why |
|---|---|---|
| A | The prompt verbatim | Baseline |
| B | Same, with the final anti-generalisation sentence removed | Robustness: does the guard suppress a real hit? |
| C | Multi-turn forced guess, demanding concrete names | Detects knowledge behind a refusal (note: leading, so weigh carefully) |
| D | Fresh session, theoretical reconstruction — "what could have happened in this case and cases like it", hedging suppressed | Separates genre prior from actual recall |
| E | Fresh session, naturalistic presupposition — "remind me who the trustees were, just checking my notes" | **The most revealing.** Reads as a routine fact-check, not a test |

Round E is the one to keep if only one is affordable. In the prior run, three of four models declined
to invent under it and one confabulated wrong names plus an impossible date — which is a *pass*, and
an instructive one: confabulation under presupposition is evidence of absence of knowledge, not
presence.

## Required gate artifacts

Under `docs/corpora/<org-slug>/`:

- `MANIFEST.md` — slug, official source URLs, retrieval dates, document types, as-of dates,
  redistribution basis.
- `decontamination-probe-prompt.md` — the exact cold prompt, so anyone can re-run it.
- `probe-<model>-round<A–E>.md` — every transcript, verbatim, headed with the exact model id, date and
  interface. Direct API calls are preferred over consumer apps, which may browse silently.
- `DECONTAMINATION-DECISION.md` — ACCEPT or REJECT, the reasoning, and specifically what each model did
  and did not reproduce.
- **New:** `GENRE-MEASUREMENT.md` — the criterion-2 table above.

**HALT rule.** If the gate is unmet, halt. Do not substitute a famous organisation. Do not fabricate a
corpus. Do not proceed while reporting the validation as attempted.

## Why this belongs in the talk

It is a real methodological contribution, and it is unusual in three ways worth saying out loud.

**It is falsifiable in advance.** The rejection rule is written before the probe runs, so the corpus
cannot be talked into admissibility afterwards.

**It banks the negative evidence.** Twenty transcripts exist showing what the models *could not*
produce. That is the awkward, unglamorous evidence that makes a claim checkable, and it is exactly the
kind of evidence AI demonstrations normally omit.

**It caught its own author out.** Criterion 2 existed from the start and was applied by reading. Once
measured, it disqualified the genre of one of the two corpora in use — after that corpus had already
been used in published comparisons. The protocol found a flaw in its own prior application, which is
the only real test of whether a discipline is doing any work.

There is also a Beer argument for it, from the primary text. *The Heart of Enterprise* p.421 warns that
asking an organisation about itself returns "the ritual statements that were projected by the Whole to
the Elements in the first place"; p.212 that "it is pointless … to look through the managerial records"
to find what actually happened. A regulatory determination is the purest ritual-statement genre there
is. Criterion 2 is that objection made measurable — and it is the reason the testimony loop exists at
all: documents establish what is *claimed*, people establish what is *so*, and the gap between them is
the finding.

## Honest limitations

- **The probe tests recall, not inference.** A model with no memory of an organisation may still
  reconstruct plausible structure from genre priors. Round D exists to separate these and does so
  imperfectly.
- **Absence of reproduction is not proof of absence of knowledge.** It is the best available evidence,
  bounded by the elicitations tried. Say "not reproduced across N rounds and M models", never "the
  model does not know it".
- **Criterion 2's term lists are a proxy.** High S4 density does not guarantee diagnosable S4 material;
  it guarantees the subject is discussed. The lists should be published with any result so a reviewer
  can disagree with them.
- **Obscurity decays.** A corpus clean in 2026 may be contaminated by the next training run. Probes
  carry a date and expire.

---

## ⚠ GENRE-SHAPED BLIND SPOT — the protocol gives a FALSE ALL-CLEAR on descriptive corpora
*(found 2026-08-01 by the corpus hunt, `docs/corpora/CANDIDATE-CORPORA-2026-08-01.md`)*

Items 3 and 4 of the probe ask what went wrong and what an official investigation found.
Those questions presuppose an **incident**. Every corpus gated so far has been forensic
(an inquiry decision, a safeguarding review, a governance failure), so they fired correctly.

**Against a DESCRIPTIVE corpus — an annual report, a board-paper set, a strategic plan —
there is no incident, so items 3 and 4 cannot fire and the probe returns clean regardless
of how much the model actually knows.** That is a false all-clear, and it will bite exactly
when the project moves to the descriptive genre it needs for S2/S4 coverage.

**Required adaptation before probing any descriptive corpus:** replace items 3 and 4 with
the internal specifics a diagnosis would rest on —
- the organisation's committee structure and membership,
- its named officers and their portfolios,
- its current corporate-plan objectives and their RAG status,
- its corporate risk-register entries.
Write the rejection list BEFORE running, as always.

**Also outstanding (recorded in the RHE decision, still unfixed):** the probe script's
auto-flag list holds stale per-organisation terms and emits a false all-clear for every
other corpus. Repopulate it per organisation, and read transcripts rather than trusting
flags — the script itself defers judgement to a human and says so.
Original saved file
Why I ran these probes, and how I read them

A model may already know an organisation from the public web and then find quotations that seem to support what it remembers. A correct quotation cannot tell recall from analysis. So before trusting a case, I ask the models what they know without documents, tools or browsing, against details listed in advance that would rule the case out.

It fired on me. Probed cold, two model families reproduced Robin Hood Energy’s audit report, its central conclusion, the loss figure, the buyer and the month of sale. I withdrew the case and told the conference.

ScotSTAR’s check used three models and four approaches. No substantive item on the rejection list came back, but the models recognised public officer names and the published referral number. I excluded those after the run; under the original rule the case would have failed. The probe also ran after the earlier ScotSTAR analysis, and document recall covered four of the thirteen documents in that earlier library, one sample per prompt.

Conant and Ashby’s good-regulator theorem says that every good regulator of a system must be a model of that system. A language model that already holds an account of the organisation can read the documents through that account, which is why I probe for it.

What it does to the evaluation: ScotSTAR passes under the amended rule. It is one of four corpus criteria.

2. Where transduction loses a distinction

Turning source language into a map can change its meaning. Separate records make it possible to inspect what crossed that boundary and what disappeared.

Open the reasoning

One big answer leaves fewer records to inspect. It can still be checked, but it can be harder to find where a name, scope or function first changed.

Finding names, matching names, proposing functions and asking questions are different tasks. Give each stage a clear input, output and evidence rule, and keep its record, so a mistake can be traced to where it happened. In one recent run a consultant’s question was in the review, the Map and the Diagnosis request, and missing from the Diagnosis output. I can only write that sentence because those are four separate records.

Beer’s Third Principle: information crossing a boundary undergoes transduction, and the transducer needs at least the variety of the channel (The Heart of Enterprise, 1979, p. 101). His interpreter examples show that correct words are not enough (pp. 124–125). A software step is not, by itself, evidence of a separate system boundary. I use the idea for a change of representation: a sentence becoming a relationship on the map. Keeping both records makes a loss visible. Whether the meaning survived is a separate judgement.

What it does to the output: separate records let you find where a failure happened. Each failure should say whether it affects one item, one stage or the whole run.

3. A repeatable check can still be wrong

Fixed rules make decisions easier to inspect. They still need challenge: a rule written around one organisation may misread another.

Open the example and explanation

An AI asked to check its own work may reach a different judgement next time. Some checks need a fixed rule: the same input should produce the same answer. That is what “deterministic” means here.

Some checks are mechanical: are these the exact words in the cited source? Others need judgement: do the words support the reading? A fixed comparison answers the first, not the second.

The strongest version is the admission gate. The model proposes a function, and a written, versioned rule admits or declines it against the quotation it cites. The rule can be read and replayed, and every decision carries its version, so a change is a visible event, not drift.

The rules will be wrong sometimes. My gate’s vocabulary, first written for a fire-and-rescue case, declined health-service wording it should have accepted. I repaired it with a new, versioned rule, not a quiet loosening. A rule also attenuates what enters the map, so the declined proposals and their reasons are kept for challenge.

What it does to the output: a fixed rule gives a repeatable decision for the same proposal, evidence and rule version. The model may still propose something different on another run.

Changing the AI’s instructions is a change to the method

A prompt is the written instruction given to the AI. Key analytical prompts have a recorded content fingerprint, or hash. Tests compare the file with that fingerprint: an edit fails the check until the expected version is updated too. The change record explains what changed and why, with dated notes.

A changed Audit prompt also means earlier saved Audit answers cannot be reused, because they were produced under different instructions. The new version needs a fresh run; the old records stay as history.

That is why I treat prompt fixes as explicit, scoped changes. A version check makes a change visible, not better.

4. “Not enough evidence” can be a useful answer

An empty box is sometimes more honest than a complete-looking map, and an important unresolved point can become a good question.

Open the reasoning

A language model can make the organisation-chart mistake Beer warned about, assuming every department on the chart is a viable system (The Heart of Enterprise, 1979, p. 204), and make it fluently and at scale. A committee’s title alone does not make a VSM function.

When the evidence does not support a function, the analysis declines to assign it, and keeps the proposal, the reason and who declined it. The trap is silence: an abstention that vanishes is indistinguishable from never having looked. Unresolved points stay visible, and a person decides which matter; not every refusal deserves an interview question.

An abstention keeps “not established” apart from “absent”. A good question can then add what the documents could not: another source, or someone’s knowledge of the work. That is possible amplification.

What it does to the output: unresolved proposals stay visible, and the important ones can become questions.

5. Attenuation shapes what the inquiry can see

Selecting evidence makes the task manageable. It can also remove a date, relationship or alternative the next interpretation needs.

Open the reasoning

The material supplied with an AI request is its context. The amount it can receive at once is limited by its context window. Record which passages were supplied so readers can check an interpretation against the evidence actually available to that request.

Keep evidence about a candidate apart from a passage that merely mentions it, and keep direction: reporting to a committee is not having authority over it.

Choosing passages is attenuation; what matters is which distinctions remain available to the observer. The context window limits text, not variety. A fresh model session limits carry-over from earlier guesses but drops context that matters, and it does not remove what the model learned in training. A sealed set of passages can leave a relevant one unread, and a coverage check shows the passages were supplied, not understood.

An account also needs its time frame: a policy, a report of practice and a plan may describe different periods. Beer’s Fourth Principle asks that the operation of the first three principles be “cyclically maintained through time, and without hiatus or lags” (The Heart of Enterprise, 1979, p. 258). The app does not assess that.

What it does to the output: you can see what evidence each claim was allowed to use. That limits attribution errors.

6. An accurate quotation can support a bad interpretation

Checking the words is necessary. Checking what they show is a separate task.

Open the distinction

Presenting a paraphrase as a verbatim quotation manufactures evidence. A labelled paraphrase is an interpretation and must be assessed as one.

A quotation can be word-perfect and still support the wrong reading. Software checks that the words appear in the cited passage, in a way that varies by stage. Whether they support the reading is a person’s judgement: a match shows the words are there, not that they produced the reading.

When a quotation fails, the record says how. A mis-citation, real words with the wrong reference, can sometimes be repaired in the Diagnosis, if the right passage was in front of the model. A fabrication is rejected. A mechanism comparison whose quotation keeps failing is withheld with its reason, and the rest can still be delivered; some failures stop the run. None of these counts as evidence that something is missing from the organisation.

Beer, of models such as a VSM map: “a model is neither true nor false: it is more or less useful” (Diagnosing the System for Organizations, 1985, pp. 1–2).

What it does to the output: a verified quotation shows the words match their source. Keep the basis of each claim visible; a model’s stated confidence is not a calibrated probability.

7. A saved correction is not a completed correction

A person’s answer should stay in their name. I still need to check that a later analysis received it and responded to it.

Open the reasoning

Once a model’s proposal is written into the working map as fact, plausible inference has hardened into organisational fact, and it propagates: a mistyped unit one level up can become the frame every child system is read within.

Keep the states apart all the way through: recorded in the case, included in a model request, admissible for a claim, accepted into the working map, selected for the report. An answer, a naming correction, a review and a promotion are separate recorded acts, each with its author. Saving a correction does not show that a later model received or used it; check the request and the result. A status the model declares for itself authorises nothing.

Recursion needs the same discipline. Going down a level is a new, scoped inquiry, not a zoom: reused evidence must be reassessed for the child’s operations, metasystem and environment, and a child analysis opens only when cited evidence supports its operations and boundary. The criterion for unfolding stays attached to that evidence. Change it and the organisation unfolds differently, which is a choice no model should make silently.

What it does to the output: you can tell an interpretation from a recorded decision and challenge either. Each box shows whether it rests on a quotation, a person’s correction or role evidence, and each channel passage whether it is declared, observed or planned. The labels show where support came from, not that the activity happened.

8. The review is part of the cost

A fast first answer can leave substantial checking work. The useful comparison includes that work, as well as the AI bill.

Open the trade-off

Checks, retained requests and review take time and resources. Measure that work alongside model usage before making cost or savings claims.

The bill includes extra requests, retries, storage and human review, plus software that tracks whether a run is going, finished or interrupted and stops an older run overwriting newer work. None of that writes a diagnostic sentence; what it buys is the ability to question the work. Separate stages help, because some failures cost one stage rather than the whole run, and a paid batch can be reused only if its inputs, prompt, model settings and checks all match. Missing cost is unknown, not zero.

If nobody will act on the answers or ask where a claim came from, this machinery is overhead and a plain assistant is the better tool. Build it when the output is a premise for decisions about real organisations, which is where a hardened wrong inference does its damage.

Set limits on calls, retries and time before a run, and compare total usage and review effort on equivalent cases before claiming savings. The cost of the checks is not a measure of variety. What matters is whether the inquiry keeps the distinctions it needs, and more checks do not guarantee that.

What it does to the output: an inspectable result, with the cost of producing and checking it visible. Whether that trade is worth it needs testing.

Would you accept this claim?

From evidence to inquiry, one inference at a time

My talk was “From Evidence to Inquiry: Keeping LLM Inference Provisional in Recursive VSM Diagnosis”. Here are four inferences of the kind a model makes from an organisation’s documents. Decide what you would do with each, then see what the system does and what building it taught me. The last two go down a level of recursion, where a hardened inference does the most damage. Illustrative examples, not findings about any organisation.

Each case has one answer that goes too far. Pick one and see why.

Case 1The AI says: The Quality Committee is System Three-Star. It audits the operations.

It cites: “The committee is responsible for reviewing performance.”

Too far.

A duty to review reports becomes an audit function on the map, and every later reading of System Three-Star rests on it. The quotation is real and matches its source; the reading is not supported. Beer’s System Three-Star audits “penetrate straight to the operations themselves” (Diagnosing the System for Organizations, p. 82).

Best answer: this is what the system does.

It keeps the committee’s duty to review performance and declines System Three-Star: the evidence shows a mandate, not the activity, and not access to the operations.

Also sound.

Ask: what did the committee review last quarter, and did anyone visit the operations?

Lesson 6: an accurate quotation can support a bad interpretation. Checking the words and checking the reading are different jobs.

Case 2The AI says: There is no coordination (System Two) between the two ambulance divisions.

It cites: nothing. None of the passages it was given mentions coordination between them.

Too far.

Then “no S2” becomes a finding, and the diagnosis builds on it. But the shared rota that coordinates the divisions was never among the documents selected. What the model was not given, it cannot distinguish.

Best answer: this is what the system does.

“No coordination appears in the passages supplied” is what the evidence supports. The system records S2 reach as unmapped, not missing.

Also sound.

Unmapped reach can become a question: how do the two divisions keep out of each other’s way, and who sets the rota?

Lesson 5: selecting evidence is attenuation. The context window limits text; it does not measure variety, and a gap in the passages is not a gap in the organisation.

Case 3The AI says: The Finance Department is one of the operations (System One), a viable system in its own right.

It cites: the organisation chart, where Finance reports to the Chief Executive.

Too far.

This is the mistake Beer called the most common: reading every department on the organisation chart as a viable system (The Heart of Enterprise, p. 204). Accepted, Finance becomes a candidate for its own level of recursion, and the inquiry goes looking for its Systems One to Five. In its corporate role it “simply cannot maintain a separate existence” (p. 119).

Also sound.

What the evidence supports is a reporting line: who reports to whom, not what the unit does. The system declines the typing, because a name or a title is not evidence of function.

Best answer: this is what the system does.

It abstains, records why and keeps the abstention visible. A question to take back: what does Finance produce that the organisation exists to produce?

Lesson 4: a language model makes the organisation-chart mistake fluently and at scale. “Not enough evidence” is a useful answer, and an abstention that vanishes is indistinguishable from never having looked.

Case 4The AI says: The North Division is a viable system. Open a child analysis one level down.

It cites: “The North Division covers the northern region.”

Too far.

Then everything found one level down is read inside a frame one sentence built. This is the problem the talk was about: an inference that hardens at one level of recursion propagates to every level below it.

Best answer: this is what the system does.

North stays a candidate operation; a child analysis opens only when cited evidence shows its own operations and boundary, with the criterion for unfolding stated. A region is a scope, not a viable system.

Also sound.

The question is a VSM one: what does North decide for itself, and what does it have to ask the centre for?

Lesson 7: a mistyped unit one level up can become the frame every child system is read within. Going down a level is a new, scoped inquiry, not a zoom.

Keep what the evidence supports and let the rest stay provisional, as a question. That is the move from evidence to inquiry, and recursion raises the stakes. The question I asked at Metaphorum is the one you have just answered four times: when should AI-proposed structure become a diagnostic premise?

Where the inquiry loses and gains distinctions

The organisation has its own operations, regulation and environment. The inquiry only ever sees a partial record of them. Take the committee from the first case through the four places where an inquiry loses or gains distinctions.

  1. Step 1

    Selection (attenuation). The analysis is given the committee’s terms of reference but not its minutes. What it was not given, it cannot distinguish.

  2. Step 2

    Interpretation (transduction). The model turns “responsible for reviewing performance” into “audits the operations”. The words survive the conversion; the difference between a duty to review reports and an audit that reaches the operations does not.

  3. Step 3

    Review (attenuation, with a way back). The admission rule declines System Three-Star. The declined proposal and its reason are kept, so a person can challenge the decision either way.

  4. Step 4

    Next question (possible amplification). “What did the committee review last quarter, and did anyone visit the operations?” goes to someone who knows. Their answer can add a distinction the documents did not hold. More generated text alone does not guarantee that.

Each step can lose a distinction or add one. Some loss is necessary: attenuation is what makes the inquiry manageable, and what matters is which distinctions go. These ideas apply to the inquiry itself; the software’s stages are not a VSM.

Where requisite variety enters. A regulator needs a sufficient repertoire to counter the disturbances relevant to keeping essential variables within acceptable limits. Here the question is whether the inquiry keeps the distinctions it needs to choose its next question or action. Neither the number of documents nor the model’s text capacity measures that. (See the VSM terms on the first page: Ashby, An Introduction to Cybernetics, sections 7/6 and 11/7; Beer, Diagnosing the System for Organizations, pp. 35, 47 and 48.)

What VSM adds

The inquiry follows primary operations and their autonomy, the S2/S3 arrangements that enable cohesion, and the S3/S4 relationship under S5 policy and identity. It examines these relationships recursively. A collection of S1 to S5 labels cannot show whether they function together.

The deeper idea: how cybernetics informs these choices

The cybernetic frame

Two systems need to stay in view: the organisation being studied and the inquiry through which I try to understand it. The organisation regulates its relationships with its environment. The inquiry constructs and challenges a selective account of those relationships. The software is an instrument in that inquiry; its stages are not the organisation’s S1–S5.

Requisite variety concerns the responses available to a regulator relative to the disturbances it must counter, in order to keep essential variables within acceptable limits. For this inquiry, that raises a practical question: which differences must the inquiry retain to choose an appropriate next question or action? More documents, more words or more AI answers do not by themselves establish requisite variety.

Selection attenuates the evidence presented to an observer. Transduction changes its representation across a boundary; the new form must still carry the distinctions that matter. Amplification may extend the repertoire available to the inquiry, for example through a new observation, an alternative explanation or access to someone able to investigate. These functions interact: a useful reduction in detail at one point can become a damaging loss at another.

The whole loop matters. An interpretation informs a question; an answer or new observation can change the evidence and the next interpretation. Organisational regulation would also require action and feedback about its effects. No currently published case demonstrates that complete loop, and the prototype does not measure variety or establish that an organisation has requisite variety.

The organisation is also an observer of itself. Its public papers are accounts made for particular purposes and audiences. That makes whose account I am reading part of the inquiry: whose purpose or boundary is described, and what might another observer see? This applies Beer’s discussion of the enterprise as its own observer to the reading of documents (The Heart of Enterprise, 1979, pp. 114–115).

Sources and limits

What may change next

Models and tools change, and a result belongs to the version and conditions tested. I would test a replacement model on the same cases before relying on it. OpenAI’s evaluation guidance also recommends continued evaluation as a system changes.

The aim I expect to keep: show the evidence, make interpretations challengeable and keep decisions attributable. Each safeguard still has to be tested.

What these lessons rest on

  • My observations: the full probe records. These are evidence from this project, not general performance benchmarks.
  • Independent research: Liu and colleagues, Lost in the Middle (2024), found that performance in their experiments depended on where relevant information appeared in a long input. That supports testing how evidence is used; it is not a result about the models in this prototype.
  • Practice guidance: OpenAI’s evaluation best practices, consulted 20 September 2026. Supplier guidance, not independent validation of Recursive Systems.
  • Systems theory: the VSM terms on the first page identify the books, passages and limits of the borrowings from Beer, Ashby, Espejo and others.
How the work is divided

The shape of the work

The AI proposes an interpretation. Software applies written checks, records what happened and carries a person’s answers and corrections into the next analysis, where their use must be checked again.

Under half of the stages in my pipeline are model calls. The rest prepare, check, record, transport or display, for the reasons the lessons give.

The different analysis routes, and their trade-offs

The four routes, in full

My pipeline runs these patterns through four routes that commit at different points. The method page summarises them in a table.

What the observer can distinguish

Each route gives an observer selected evidence and a particular question. What matters is what each can distinguish: one unit under two names, an assigned responsibility, an enacted function, an information flow or a proposed explanation.

Selection, grouping and summarisation can remove relevant context. Supplying every passage proves coverage of the input, not comprehension of its relationships.

Rules assemble the candidates

A fixed naming rule gives a repeatable starting point, but it may keep two names for the same team apart. The later assessment needs evidence about the candidate it is judging: being mentioned in an activity is not evidence of performing it. For each candidate the prompt asks three separate questions: does the source show an assigned responsibility, a reported practice, an observed performance? A label settles none of them, and where the evidence supports several functions, forcing one label can hide something important.

Espejo’s warning about fragmentation applies: breaking the work into parts helps only while the connections inside each part survive, and the variety between the parts still needs regulating at the level above (Espejo and Reyes, Organizational Systems, 2011, p. 137). For each channel the software keeps whether its passage is declared, observed or planned; a duty, remit or job description stays “declared” even when it is dated. It also keeps any amplification, attenuation or transduction described at either end, without measuring capacity.

A model assembles the candidates instead

From the source passages, the places each name was found and the prior structure, the model proposes which mentions belong together. It applies no functional labels, so new candidates are untyped. Grouping is therefore an interpretation made before any functional question, and what follows inherits it.

Deep Map: the diagnosis comes first

The software hands the model the corpus, the declared frame and part of the existing Map. The prompt asks for alternative mechanisms, risks and disconfirming checks, which widens the account. Afterwards the code separates a source observation from the model’s interpretation and checks each quotation back against its source. A status the model declares for itself authorises nothing.

The function asks for its own evidence

The model searches, fetches passages and revises its account over successive turns, choosing what to look at next. Budgets bound the search, and working memory starts fresh at each function, so what the model found in one does not carry into the next.

Recursion changes the system in focus

Start with the primary transformation of the system in focus. Its S1 elements carry out the operations that produce that transformation. Producing a report or being a named department is not evidence of an S1 role.

Recursion examines an S1 element as a viable system in its own right, with its own operations, metasystem and environment. The prototype separates proposing an operational candidate from deciding whether the evidence supports opening a child analysis.

The child’s functions must be assessed in its own frame. Each viable system has an S5 function concerned with its identity and policy; an embedded system’s identity must cohere with the wider whole while allowing local autonomy. Shared purpose or even the same office-holder does not make S5 evidence interchangeable between levels. The prototype keeps recursion levels separate and records the question; it does not yet assess whether an embedded system’s identity coheres with the wider whole.

A person can declare the criterion used to unfold the organisation; the software can also propose one from cited evidence. The proposal is recorded with its quotation and treated as working context that the evidence can challenge; a recorded documentary criterion is protected from casual replacement. A different criterion may expose different operational relationships.

What a person contributes

A gap the analysis cannot close can become a question for the next conversation. After Paul Pangaro, the question sets that conversation’s goal, and who takes part depends on the variety it needs; the people in the last conversation are only possible participants (“Designing Conversations for Variety”, Metaphorum webinar, 11 November 2022). The software can draft the question; who should answer it is a judgement it does not make.

The system keeps separate records for an answer, a naming correction, an analytical review and a promotion. A review carries its own provenance and can be superseded. Promotion is a separate human command that queues an ordinary Map. A disagreement can be recorded in a person’s own words, with no model reply and no extraction over them.

The VSM terms used here, with what the app checks for each and Beer’s sources, are on the first page: What the app assesses in VSM terms.

Closing

The ScotSTAR case is not published as an application copy yet.

If one of these choices seems wrong or a check is missing, I would like to hear which one, and what example exposes the problem.

Richard Gunther, richard@recursive.systems