Report the Label, Not the F1

The only accuracy figure in the UK National Audit Office’s case study of Justice Transcribe (HC 615, 14 October 2026), the AI transcription tool now used by about 11,000 probation officers in England and Wales, is this: an HM Prison and Probation Service (HMPPS) assurance group found “some inaccuracies in 20% of final case records they examined”, mostly misspelt names and agencies. The report does not say how many records were examined (paragraph 7). It does say that HMPPS holds no measured time-saved data (paragraph 16), that the forecast of around 450,000 hours a year is modelled (paragraph 15), and that “there is no quality assurance mechanism to monitor the accuracy of the original transcript” (paragraph 21). The stated inputs, 10 minutes saved on 70 per cent of 3.7 million meetings, multiply to about 432,000 hours on the desk’s arithmetic.

That is the normal condition of government AI monitoring: a headline score, usually accuracy or F1, with no denominator, no breakdown by label and no human comparison. Six papers posted to arXiv between 24 and 27 October 2026, preprints unless stated, make the case for a different standard.

Why a high aggregate score can sit on top of a failing label

The strongest evidence is a study accepted at EMNLP Findings 2026, High Scores, Hidden Failures (arXiv:2610.35179, 26 October), in which Elena Senger, with Yuri Campbell, Rob van der Goot and Barbara Plank, tests LLM coding pipelines on four datasets with five models. Their measure of interest is “worst-label-F1”, the score on the label the model handles worst. The gap between it and micro-F1 is 30 points on one dataset, 26 on a second and 28 on a third, differences the authors call “entirely invisible in aggregate reporting”. Such pipelines, they conclude, “cannot be treated as stable measurement tools without targeted validation beyond aggregate performance”.

The pattern repeats in a shared task that reports results the way a monitoring contract should. In the ArGuard 2026 tasks on Arabic harmful-prompt detection, published at ArabicNLP 2026, Ali Khairallah and Arkaitz Zubiaga’s single fine-tuned JAIS model (arXiv:2610.36360, 26 October) ranked first on both tracks: macro-F1 0.984 on safe-versus-unsafe and 0.790 on a seven-way harm taxonomy. The headline conceals a class and a register. F1 for six of the seven classes lies between 0.73 and 0.98; the seventh, “Bully”, has precision 0.815 but recall 0.443, so it misses more than half of the true cases. Crossed with dialect, Bully scores 0.81 to 0.97 in all five regional dialects and 0.46 in Modern Standard Arabic, the register of 719 of its 908 test prompts; “the Bully weakness is not a property of the class but of Modern Standard Arabic”, the authors write. Binary accuracy by dialect runs from 93.6 per cent (Moroccan) to 99.8 per cent (Egyptian), on cells of 14 to 239 examples the paper calls indicative. None of this is visible in 0.790, and all of it matters to a regulator asking whether a moderation system treats one community’s speech differently, as AI & GPP noted of low-resource varieties in what a police AI trial measured and what it missed.

Sizheng Fan, Yinghe Sun and Rong Zhao’s study of Polymarket prices (arXiv:2610.36639, 27 October) as geopolitical early warnings notes that accuracy cannot settle the question, because most markets resolve No and accuracy stays high even when most events that happened were missed: a day before first report, prices anticipated 56 per cent of 208 unscheduled events, yet every one of the 51 scheduled central-bank decisions. Platforms, they advise, should “report recall alongside accuracy”.

Why the score at acceptance is not the score in service

Two further preprints show how far a vendor’s benchmark can sit from deployment data. EdgeDeID (arXiv:2610.33839, 24 October; under review), a 0.6-billion-parameter clinical de-identifier by Sina Daneshgar and colleagues, reaches 0.994 micro-F1 on a 100-note synthetic benchmark. On 2,434 real MIMIC intensive-care nursing notes its F1 is 0.478. The authors trace the fall to register and density: their training notes carry 10.8 identifier mentions per 1,000 characters, the nursing notes 0.92, and 155 of 200 missed names and dates came from notes where the whole output was empty or unparseable, a failure absent on the benchmark. A redaction tool bought on the strength of 0.994 would, on this evidence, leak on the text it was meant to protect.

The ArGuard results carry a smaller version of the same gap. REGLAT (arXiv:2610.37265, 27 October), a system from Ahmed M. Fetouh and colleagues, reports macro-F1 0.886 on the labelled development set and, in its abstract, 0.735 and 0.754 on the organisers’ blind test sets for the two subtasks, although its body describes only the seven-way one. A development score is a score on data the builders could see.

What “agreement with a human baseline” has to specify

The phrase “human baseline” does work only if the agreement statistic is defined. Tim Menzner and Jochen L. Leidner’s Multi-Label Agreement framework (arXiv:2610.37292, 27 October) shows how much the choice of chance model matters when labels are rare. Their worked example is four sentences, three annotators and a 31-label inventory of media-bias types: the same annotations give a pooled-marginal coefficient of 0.232, a Gwet AC1 of 0.903 and a fixed-baseline PABAK of 0.828. Twenty of the 31 labels were never selected. The authors present the framework “without claiming that one normalization is universally preferable” and recommend pre-specifying the chance model, stating whether the label inventory was fixed before annotation, and reporting items, annotators per item, raw agreement, and prevalence and agreement for every label, with confidence intervals.

For a contract the point is that a label-level figure can be defined and computed, and an aggregate cannot be read without the prevalences behind it. Full Fact’s classification of 112 fact checks, the peg for AI & GPP’s piece on why view counts mislead government monitors, describes a gold standard review of its coders’ findings but publishes no agreement statistic; the NAO’s 20 per cent has no denominator and no breakdown by error type. Neither is a baseline anyone could hold a system to.

A worked example: a benefits-letter classifier

Suppose a benefits agency buys a system that reads incoming letters and web-form messages and assigns one or more of six labels, so that a reconsideration request reaches a decision-maker before its deadline and a disclosure of risk reaches a safeguarding lead the same day. The acceptance test is a held-out sample of 2,000 items from one month’s live intake, double-coded by two caseworkers who have not seen the system’s output, with disagreements adjudicated.

Illustrative figures. Not a real system, dataset or agency; invented to show the shape of the report.

On these invented figures micro-F1 works out at about 0.87, carried by the two labels that cover 60 per cent of items. The decision turns on three rows the aggregate buries. Reconsideration requests, which carry a deadline, are missed 28 per cent of the time while the humans agree on them at 0.88, so the failure is the system’s, not the codebook’s. The safeguarding label is rare, costs most to miss, and is caught in fewer than half of cases: the ArGuard Bully class in one row. Complaints are different: the humans agree at only 0.69, so a system score of 0.58 says more about a category that needs redefining than about the system. The two critical rows would then be repeated by language and channel, as ArGuard did by dialect.

The clause that follows from the table, in the desk’s own drafting:

Performance reporting and acceptance. (a) Each reporting period the Supplier shall report, for every label in Schedule 1, on a held-out sample of not fewer than [2,000] items drawn by the Authority from live intake and double-coded by Authority staff with adjudication: the number of items; label prevalence; recall and precision against the adjudicated label; and chance-corrected system-to-coder agreement, computed with the coefficient, chance model and label inventory stated in Schedule 1 and identical to those used for coder-to-coder agreement on the same sample. (b) No aggregate measure, including accuracy, micro-F1 or macro-F1, shall substitute for (a). (c) The system fails the period if recall on any label marked harm-bearing or deadline-bearing in Schedule 1 falls below the floor stated there, or if system-to-coder agreement on any label falls more than [0.10] below coder-to-coder agreement on that label. (d) Results shall be broken down by language or dialect and by channel wherever the sample permits, with cell sizes. (e) Any change to the model, prompt, threshold or label inventory restarts acceptance under (a).

Bracketed values are for the parties to set; the tolerance is relative to how well the humans agree on each label, so a label the codebook cannot define is not scored as a system failure, and an easy label is not excused.

What the official guidance already requires, and what it leaves open

The UK Government’s Artificial Intelligence Playbook (10 February 2025), which the NAO cites at paragraph 21, tells departments to “establish performance metrics for AI systems that measure the accuracy of model outputs”, to test “in a variety of scenarios, including those capturing extreme or rare potential events”, and to put every update through quantitative testing and validation under change control. The rare-event instruction is the safeguarding row; the change-control instruction is paragraph (e). The international reference, ISO/IEC TS 4213:2022 on assessing classification performance, is a technical specification now under revision into a full standard; it is sold, at CHF 159, rather than free to read.

What the guidance leaves open is the human side: against whom accuracy is measured, how, and on which data. The design in the NAO report, HMPPS’s planned linked case audit comparing samples of transcripts, AI summaries and final records, plus the periodic review of audio the NAO suggests (paragraph 21), is the right one. Its value depends on whether the published result is a percentage or a table stating how many records, which error types, how two reviewers agreed, and whether misspelt names clustered in particular accents, as pilot users noted (paragraph 7). AI & GPP made the same argument about one threshold carrying a whole fairness case in the 0.64 setting.

Three things to do now. Write Schedule 1 before procurement: the label inventory, which labels are harm- or deadline-bearing, the agreement coefficient and its chance model, and the recall floors. Fund the double-coded sample as a line item. And ask any supplier quoting an F1 for the worst label, its prevalence and the test set’s provenance; a supplier who cannot answer has given a number, not a performance. The thing to watch is whether the national evaluation of Justice Transcribe, now under way, publishes its case audit with a denominator.

Sources

1. Productivity improvement: The Justice Transcribe AI-enabled transcription tool, HC 615 2026-27 — National Audit Office, 14 October 2026 (PDF)
2. High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis, arXiv:2610.35179 — Elena Senger, Yuri Campbell, Rob van der Goot and Barbara Plank, 26 October 2026 (preprint; accepted at EMNLP Findings 2026)
3. AK_QMUL at ArGuard Shared Tasks: A Single Fine-Tuned JAIS for Arabic Harmful-Prompt Detection, arXiv:2610.36360 — Ali Khairallah and Arkaitz Zubiaga, 26 October 2026 (preprint; published at ArabicNLP 2026, co-located with EMNLP 2026)
4. Whether Is Not When: Separating Occurrence from Timing in Polymarket Prices, arXiv:2610.36639 — Sizheng Fan, Yinghe Sun and Rong Zhao, 27 October 2026 (preprint)
5. EdgeDeID: A 0.6B Clinical De-identifier Trained Without Real Patient Data, arXiv:2610.33839 — Sina Daneshgar, Raya Namazi, Amirali Khosravi, Pouria Rouzrokh and Bardia Khosravi, 24 October 2026 (preprint; under review at ACL Rolling Review)
6. REGLAT at ArGuard Shared Tasks: An Ensemble-Based Model Integrating NER for Harmful Content Detection in Arabic, arXiv:2610.37265 — Ahmed M. Fetouh, Mariam Labib, Nsrin Ashraf, Omer Dawood and Hamada Nayel, 27 October 2026 (preprint; system description paper)
7. Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability, arXiv:2610.37292 — Tim Menzner and Jochen L. Leidner, 27 October 2026 (preprint)
8. AI-Generated Misinformation in the UK: Harm, Context, and Classification — Full Fact, undated (covers fact checks of 1 January 2025 to 31 March 2026)
9. Artificial Intelligence Playbook for the UK Government (HTML) — Government Digital Service, GOV.UK, 10 February 2025
10. ISO/IEC TS 4213:2022, Information technology — Artificial intelligence — Assessment of machine learning classification performance — ISO, October 2022, confirmed 2025 (catalogue page; full text for purchase)