Restoring a Model Is Only Part of Restoring an AI Service

A local AI service can restart successfully and still return the wrong service to its users. The model may load while the document index belongs to an older release, the permissions cache carries obsolete decisions, or the application expects a different response format. A running process is only one part of recovery.

Consider a fictional internal knowledge assistant restored after a failed update. Its previous model checkpoint is available, but the surrounding components have changed. The operational question is whether the restored combination can answer permitted questions correctly and make its limitations visible. This essay proposes a way to investigate that question. It reports no actual outage, recovery drill, client engagement, or measured restoration time.

Define the service that must return

Start with the task people depend on. For a read-only assistant, that might mean finding a current procedure, explaining it from authorized source material, and directing unresolved questions to a responsible owner. Merely returning generated text does not satisfy that task.

List the dependencies behind it: model and tokenizer revisions, serving runtime, application code, prompt configuration, document snapshot, retrieval index, embedding model, identity provider, permissions, and source-download path. Record which components can be restored independently and which require a compatible release set. The inventory should identify real artifacts rather than a label such as latest.

NIST's contingency-planning guide connects recovery priorities with business impact, recovery strategies, exercises, and plan maintenance. It is a useful planning reference, not evidence that this proposed assistant meets a certification or that a particular recovery objective has been achieved. [1]

Preserve a compatible release set

A recovery manifest should identify what was intended to work together. Include immutable revisions or hashes where available, storage locations, configuration references, and the procedure for obtaining authorized access. Keep secrets out of a public manifest. A reference to a credential owner is safer than copying a credential into a runbook.

Preserve the previous known configuration before updating. If an embedding model changes, an older index may not belong with the new retrieval pipeline. If the application changes its schema, the old model endpoint may require an adapter or an application rollback as well. Document those relationships instead of assuming that every component can be rewound separately.

A manifest establishes identity, not fitness. Files can match their expected hashes while the combination fails the intended task. Pair artifact verification with an application-level acceptance set that covers useful answers, missing information, source attribution, and access boundaries.

Separate restoring a process from restoring trust

Use several checkpoints. The process must start. Dependencies must be reachable under the intended identity. The application must complete its bounded acceptance tasks. Only then should a designated operator decide whether the service can receive its intended workload.

Kubernetes distinguishes startup, liveness, and readiness probes for different operational purposes. That separation is useful even when a small service does not use Kubernetes: a process-start signal and a decision to accept traffic answer different questions. A probe still cannot establish answer quality by itself. [3]

A proposed readiness test for the assistant would retrieve a fictional permitted document and verify the expected source revision. It would also attempt a denied retrieval and check that restricted content never enters the model context. Returning a polite refusal after restricted text was retrieved would not establish the intended boundary.

Declare what data can be lost

Recovery time and recoverable data age need separate definitions. Choose when the recovery clock begins and when it ends. Specify which records must be durable: accepted requests, document changes, permission changes, audit events, and any review decisions may have different requirements.

A document backup from yesterday can be internally consistent but unsuitable if today's withdrawal of access is missing. Treat security-state restoration as a deliberate decision. Historical permissions should not silently replace the current authoritative policy. If current authorization cannot be established, keep affected operations unavailable while the uncertainty is resolved.

Do not announce a universal target without the service owner's input and actual measurements. A proposed objective is a planning field. A restoration duration is an observation from a defined rehearsal. Neither is proof that every future disruption will behave the same way.

Test rollback as a new operating configuration

Rolling back a model does not necessarily reproduce identical output. Runtime, hardware, scheduling, input formatting, and other conditions can change behavior. vLLM's reproducibility guidance explicitly bounds reproducibility by conditions including hardware and software version. Preserve those conditions rather than treating a seed as the whole experiment. [2]

Assess the restored configuration with the same task rubric and source snapshot used for the decision. Exact text equality may be appropriate for a deterministic transformation, while source support and task completion may be the relevant criteria for a generative explanation. Declare the comparison method before examining the restored answers.

If recovery needs a new model or runtime, call it a changed configuration. Record what was substituted and what remains unverified. This keeps an emergency workaround from quietly acquiring the evidence status of the previous release.

Rehearse the handoff and the stop decision

A useful exercise includes a point where recovery must stop. Missing artifacts, unresolved permissions, incompatible indexes, or failed acceptance tasks should lead to a visible hold state. The operator needs an accountable escalation route and a way to communicate which tasks remain unavailable.

Capture actions, timestamps, artifact checks, acceptance outputs, and decisions. Record a failed step even if a later workaround succeeds. A rehearsal that exposes a missing dependency produces actionable evidence; replacing that evidence with a success summary loses its main value.

The companion recovery-rehearsal Design study turns this method into a proposed isolated exercise. The research preparation pack supplies a checklist and blank evidence record. No rehearsal has been executed. The next useful milestone is an attributable run that another operator can inspect and repeat.