In short
- Annex IV point 2(g) of Regulation (EU) 2024/1689 requires test logs and all test reports to be dated and signed by the responsible persons.
- Article 9(8) requires testing against prior defined metrics and probabilistic thresholds, so a threshold chosen after seeing the results does not meet the text.
- Article 15(3) and Article 13(3)(b)(ii) put the tested accuracy level and its metrics into the instructions for use, where a deployer and an authority can hold you to it.
- Under Annex VII point 4.4 a notified body that is not satisfied with the provider's tests carries out its own.
- Evidence has to survive: Article 18(1) keeps the technical documentation available for 10 years after placing on the market.
Evaluation work in a machine learning team usually lives in notebooks, experiment trackers and chat threads. That is where the numbers are produced, and it is not where an assessor looks for them.
Regulation (EU) 2024/1689 describes a test result that counts: fixed in advance, run, dated, signed and kept. The distance between that and a notebook is mostly paperwork, but it is paperwork that cannot be recreated honestly after the fact.
01Annex IV point 2(g) wants signed, dated reports. What does that rule out?
Annex IV point 2(g) of the EU AI Act lists what the technical documentation must say about validation and testing: the procedures used, the validation and testing data and their main characteristics, the metrics for accuracy, robustness and potentially discriminatory impacts, and “test logs and all test reports dated and signed by the responsible persons”.
That last clause rules out the most common state of evaluation evidence: a notebook re-run last week with no record of which model version, which data split and which person accepted the result. A notebook is a log at best. The Annex IV report is the document someone put their name to.
| Element | Typical notebook | What the text asks for | Provision |
|---|---|---|---|
| Metric and threshold | Chosen while reading results | Defined before the test | Article 9(8) |
| Why this metric | Default accuracy or F1 | Written appropriateness rationale | Annex IV point 4 |
| Test data | Whatever split was loaded | Described, with main characteristics | Annex IV point 2(g) |
| Subgroup results | Overall score only | Accuracy for specific groups | Annex IV point 3 |
| Sign-off | None | Dated, signed by responsible persons | Annex IV point 2(g) |
Annex IV point 4 is the requirement teams skip most readily: a description of why the chosen performance metrics are appropriate for the specific system. It is one paragraph, and it is the paragraph an assessor reads first, because every number after it depends on it. The broader set of Annex IV artefacts is covered in what an AI audit asks to see and what closes each finding.
02Thresholds belong in the test plan before the run. What proves it?
Article 9(8) of the EU AI Act requires testing “against prior defined metrics and probabilistic thresholds that are appropriate to the intended purpose”. Article 9(6) states the purpose: testing shall ensure the system performs consistently for its intended purpose and complies with the Section 2 requirements.
The practical test is chronological. The record has to show the threshold existed before the run it judges. A version-controlled test plan with a commit date, referenced from the Article 17(1)(d) procedures of the quality management system, is enough. A threshold typed into the report after the results arrived is not, however reasonable the number.
No harmonised standard yet changes this for Article 9 testing. EN 18286:2026, cited in the Official Journal on 14 October 2026 by Commission Implementing Decision (EU) 2026/2274, covers the Article 17 quality management system, not the accuracy metrics or thresholds that Article 9(8) and Article 15 require.
03The accuracy in the instructions for use is a promise. Can your tests back it?
Article 15(3) of the EU AI Act requires the levels of accuracy and the relevant accuracy metrics to be declared in the instructions for use. Article 13(3)(b)(ii) specifies that the declared figure is the level “against which the high-risk AI system has been tested and validated”, together with the circumstances that may degrade it.
That wording ties the public number to a specific test. A deployer’s claim, a market surveillance request or an incident review under the post-market monitoring plan in Article 72 will ask which report produced the declared figure, and the answer has to be a document rather than a dashboard.
04What happens when someone else re-runs the evaluation?
Under the Annex VII route of the EU AI Act, the notified body gets access to the training, validation and testing data sets under point 4.3, may require further evidence or tests under point 4.4, and runs adequate tests itself where it is not satisfied with the provider’s. Point 4.5 adds access to trained models where other means are exhausted.
| Reader | Power over the tests | Provision |
|---|---|---|
| Notified body, Annex VII route | Data access, further tests, own tests | Annex VII points 4.3 to 4.5 |
| Market surveillance authority | Documentation for 10 years | Article 18(1) |
| Authority, false or incomplete reply | Fine up to EUR 7 500 000 or 1% | Article 99(5) |
| Authority, provider obligations | Fine up to EUR 15 000 000 or 3% | Article 99(4)(a) |
A re-run is where a notebook fails. If the data split, seed, model version and threshold are not frozen in the report, the second result will differ and the provider cannot show which one is right. Which route applies to a given system is set out in the comparison of internal control and notified body assessment under Article 43.
These duties apply to Annex III systems from 2 December 2027 and to Annex I products from 2 August 2028. Reports produced now can already follow the Annex IV form, which costs nothing extra if it starts before the evidence has to be reconstructed.
05Frequently asked questions
Is a Jupyter notebook acceptable as AI Act test evidence?
Not on its own. Annex IV point 2(g) asks for test logs and test reports dated and signed by the responsible persons, with the metrics and the validation and testing data described. A notebook can be the log behind a report, but it is rarely dated, signed or frozen.
Can we change the accuracy threshold after the evaluation?
Only by running the evaluation again. Article 9(8) requires testing against prior defined metrics and probabilistic thresholds, so the record has to show the threshold existed before the run whose results it judges.
Does internal control under Annex VI mean nobody checks the tests?
No. Under internal control the provider examines its own technical documentation, and a market surveillance authority can later request it. Supplying incorrect, incomplete or misleading information in reply carries fines of up to EUR 7 500 000 or 1% of worldwide turnover under Article 99(5).
06References and method
- Regulation (EU) 2024/1689, Article 9(6) to 9(8) on testing, Article 13(3)(b)(ii) and Article 15(3) on declared accuracy, Article 17(1)(d) on test procedures in the quality management system, and Article 18(1) on the 10 year retention period.
- Regulation (EU) 2024/1689, Annex IV points 2(g), 3 and 4 for the content of the technical documentation on validation and testing, and Annex VII points 4.3 to 4.5 for notified body access to data, further tests and access to models.
- Annex IV of Regulation (EU) 2024/1689 and Article 99 in the consolidated presentation, read from the raw text on 1 November 2026 for the current wording, including the fine thresholds in Article 99(4) and 99(5).
- CEN and CENELEC, announcement of 15 October 2026 on the citation of EN 18286:2026 in the Official Journal by Implementing Decision (EU) 2026/2274, read on 1 November 2026; no harmonised standard on accuracy or testing metrics was cited at that date.
- Regulation (EU) 2026/1744, for the application dates of 2 December 2027 for Annex III systems and 2 August 2028 for Annex I products, and for the simplified technical documentation available to SMEs and SMCs under Article 11(1).