Designing a Recovery Rehearsal for a Local Knowledge Assistant

The fictional assistant and the proposed exercise

This Design study describes an isolated recovery rehearsal for a fictional organization using a local, read-only knowledge assistant. Staff would ask about approved operating procedures. The assistant would retrieve authorized documents, generate source-backed explanations, and expose unresolved questions for human review. It would not dispatch crews, approve payments, or make decisions about public-service eligibility.

No organization, deployment, outage, hardware run, or recovery outcome is represented here. The exercise is proposed, not performed. Its purpose is to determine whether an operator could restore a compatible service configuration and establish evidence for reopening it.

Choose a release boundary that includes retrieval

Define release A as a fictional model, runtime, application configuration, embedding pipeline, document snapshot, and index that were intended to operate together. Define release B as a proposed update. Neither name is a real deployed identifier or a claim of previous validation.

The rehearsal would first inventory the artifacts required to reconstruct A. Record their immutable revisions or hashes, expected locations, compatibility relationships, and access dependencies. Keep credentials in their proper controlled systems. The public plan contains no real host address, credential, customer record, or machine configuration.

The exercise must also identify components that should remain current during recovery, especially authoritative permissions. Restoring an earlier document snapshot does not authorize reviving an employee's revoked access. Any uncertainty about policy freshness should block the affected retrieval path.

Keep the rehearsal separate from production

Use an authorized isolated environment with synthetic documents and controlled test identities. Preserve the production service and its records. Obtain a test window and name the operator and reviewer before execution; those fields remain unfilled in the released template.

NIST's contingency guide places testing, training, exercises, and ongoing maintenance within contingency planning. This proposal applies that planning principle to a narrow AI service; it supplies neither a NIST-prescribed exercise nor evidence of compliance. [1]

Declare a stopping rule for unexpected connectivity, unavailable artifacts, or an uncontrolled identity boundary. The plan should not depend on an operator improvising changes on a production machine to make a demonstration succeed.

Exercise five recovery boundaries

The first planned scenario is a clean restoration of A from the declared artifacts. The second introduces a missing or mismatched artifact and expects recovery to remain on hold. The third combines the old retrieval index with an incompatible embedding configuration and expects the mismatch to be detected before normal use.

The fourth scenario restores an older document snapshot while a synthetic user's permission has been revoked in the current authorization source. The expected behavior is denial without exposing the restricted fixture to model context. The fifth scenario starts the process successfully but deliberately fails an application acceptance task. A successful start must not override that failed task.

These are expected behaviors, not passed tests. The public evidence record leaves all actual outcomes, timings, reviewer identities, and artifact references null. Filling those fields from the expectations would fabricate a rehearsal.

Define readiness beyond a successful restart

Measure process startup and application readiness as separate events. Kubernetes documents the distinction among startup, liveness, and readiness probes. This study uses the conceptual separation without requiring Kubernetes or claiming that a health endpoint certifies the assistant's answers. [3]

The proposed acceptance set would cover a supported procedural answer, a missing-source question, a citation check, an allowed retrieval, and a denied retrieval. Preserve exact input and document revisions. A reviewer would assess both the response and the application traces relevant to each boundary.

Before declaring the service ready, confirm that source links still resolve under the intended identity and that stale cached answers cannot bypass current permissions. A restored user interface with broken or unauthorized citations would not satisfy this proposed service definition.

Record configuration changes honestly

If the original hardware or runtime cannot be reconstructed, preserve that fact. vLLM's reproducibility guidance limits what can be expected across hardware and software versions. A replacement configuration needs its own recorded conditions and evaluation; it should not inherit a claim of identical behavior merely because the model name is unchanged. [2]

Use task-specific comparison rules. An answer can change wording while retaining the correct cited procedure, or keep familiar wording while citing the wrong revision. The reviewer should record support, task completion, attribution, and access behavior separately, with a reason for any acceptance decision.

Leave recovery objectives unfilled until justified

The template separates the proposed recovery-time objective from the actual observed duration. It also distinguishes the allowed age of restored data from the observed age of the chosen snapshot. No default number implies that the service meets a business requirement.

Actual measurements require a defined start event, end event, clock, scope, and interruption record. A later report should retain failed and abandoned scenarios, not only the shortest successful restoration. Reopening the service would remain an accountable decision supported by those records.

What is ready and what remains

The research pack now contains a rehearsal checklist and a blank evidence template for these five scenarios. The material is preparation only. It does not restore artifacts, contact an assistant, execute permission tests, or measure recovery performance.

Execution still requires authorized environment access, verified release artifacts, controlled identities, synthetic source documents, trace capture, justified objectives, and an accountable reviewer. The companion essay explains why restoration must be evaluated at the service boundary rather than reduced to restarting the model process.