AI production assurance
Establish an actionable production baseline
Describe the actual approved state of the service so every future variation can be interpreted, assigned and decided.
Definition
The baseline describes a service, not a model file
A production baseline connects an intended purpose, executed version, population, observation period, conditions of use and accountable people. It includes the components that turn an input into a decision or action: data, rules, prompts, interfaces, human controls, third-party services and recovery mechanisms.
Identity
Make the executed version unambiguous
The inventory reconciles what was approved, deployed and observed. Every critical execution must be traceable to the relevant artefacts.
Context
Define population and window before calculating
A rate is interpretable only when its denominator, population and window are known. The baseline states included uses, exclusions, critical segments, small-volume rules and what to do when data is missing.
Measures
Choose metrics connected to a risk and a decision
Measures connect business performance to system quality, human factors, security and effects. We retain the protocol, sample, evaluator, uncertainty and limits of generalisation.
Structure of a baseline measure
| Field | Question to resolve | Expected record |
|---|---|---|
| Purpose | Which service does the measure represent? | Objective and supported decision |
| Population | For which cases is the value valid? | Segments, exclusions and volume |
| Method | How is the value produced? | Calculation, sample or review protocol |
| Limit | What does the value not establish? | Uncertainty and validity conditions |
Purpose
- Question to resolve
- Which service does the measure represent?
- Expected record
- Objective and supported decision
Population
- Question to resolve
- For which cases is the value valid?
- Expected record
- Segments, exclusions and volume
Method
- Question to resolve
- How is the value produced?
- Expected record
- Calculation, sample or review protocol
Limit
- Question to resolve
- What does the value not establish?
- Expected record
- Uncertainty and validity conditions
Limits
Write the degraded state before it is needed
The baseline establishes when the service remains acceptable, becomes restricted or must enter a safe state. The safe state may be manual operation, a previous version, reduced capability or suspension. Its feasibility must be tested; a theoretical option is not a capability.
Approval
Have the functions carrying the effects approve the baseline
The business owner validates the service and limits; technical owners attest the version and capacity to act; risk, security, data or legal functions contribute as appropriate. Approval also names who may alter the baseline.
Revision
Create a new baseline when comparison is no longer valid
A change in intended purpose, population, model, data, interface, supplier, rule or oversight may require a new baseline. The previous one is retained to reconstruct the transition, tests and decision.
Sources and references
- EU AI Act — consolidated text, Articles 9, 26, 72 and 73 (new tab)
Continuous risk management, deployer duties, post-market monitoring and serious-incident reporting.
- NIST AI 800-4 — Challenges to the Monitoring of Deployed AI Systems (new tab)
Monitoring categories and measurement challenges for AI systems in real-world operating conditions.
- NIST AI RMF 1.0 (new tab)
Govern, map, measure and manage the risks of an AI system.
- ANSSI — Generative AI system security (new tab)
Architecture, lifecycle, dependencies, pre-production tests and logging.
Frequently asked questions
Is a production baseline limited to the model?
No. It covers the socio-technical service: model, data, rules, interfaces, dependencies, population, human controls and safe state.
When is a new baseline required?
When intended purpose, population, version, data, interface, supplier or oversight makes the previous comparison invalid.
Who approves the baseline?
The functions carrying the service and its effects: business owner, technical owners and risk, security, data or legal functions as appropriate.
Production baseline
Can you identify exactly what is running?
Present the version and expected service. We assess whether the baseline already supports comparison, decision and recovery.