AI production assurance

A metric may move without the service degrading; it may also stay stable while a use, population or component changes. We first identify the object actually in service, then connect every observation to an approved limit, an empowered person and an executable action.

Assurance covers the complete socio-technical system: model, data, rules, interfaces, suppliers, people, procedures and effects. It is not the accumulation of alerts in a dashboard.

The baseline is a versioned, dated and accepted state. It makes it possible to establish what changed and whether observed results remain comparable.

• Service — intended purpose, supported decision and expected result
• Configuration — model, data, prompts, rules, interfaces and dependencies
• Scope — authorised users, populations, channels, regions and cases
• Performance — business and technical metrics with a measurement protocol
• Limits — foreseeable misuse, exclusions and degraded conditions
• Authority — people empowered to restrict, stop and authorise recovery

We select signals from the risks and intended purpose. Each domain may combine automated measures, sampling, human review, user feedback and external events.

• Function — quality of the result against the expected service
• Operations — availability, latency, capacity and degraded mode
• Human — corrections, appeals, workarounds and oversight burden
• Security — access, attacks, misuse, secrets and dependency chain
• Compliance — instructions for use, logs, controls and applicable duties
• Effects — harm, differential treatment and unexpected consequences

A control is incomplete when the team can measure but does not know who decides. Population, window and signal quality are fixed before the threshold; action and authority are fixed before the alert.

• Observe — collect a traceable signal over a defined population and window
• Compare — test the measurement against the baseline and its known limits
• Qualify — assess severity, uncertainty and the extent of the effect
• Decide — keep, restrict or stop through the empowered authority
• Verify — check the action, residual risk and conditions for recovery

An unchanged model does not guarantee an unchanged service. A change to data, prompts, interfaces, business rules, suppliers, populations or oversight may invalidate the baseline.

The need to react quickly must not destroy the information required to understand the event. The incident route separates immediate protection, fact-finding, regulatory qualification, correction and authorisation to resume.

• Detect — timestamp the signal and identify its source
• Preserve — freeze useful logs, versions, inputs and outputs
• Contain — protect people and limit or suspend use
• Qualify — assess extent, severity and causal connection
• Inform — activate applicable internal, contractual and regulatory routes
• Correct — document the cause, action and regression test
• Recover — submit the test and residual risk to the empowered authority

For high-risk AI systems, duties depend on the actual role of each entity and on the facts. The method preserves that distinction in the controls and information routes.

We deliver a versioned production baseline, the control register, results and limitations, findings, decisions and the recovery protocol. Future cadence is justified by the speed of risk, response capacity, changes and applicable duties.

• Keep — the service remains within the approved scope, with assigned reservations
• Restrict — the capability, population, channel or version is limited until evidence of return
• Stop — the system enters its defined safe state: manual mode, rollback, suspension or withdrawal