# AM-RP-001: evaluation preparation pack

Author: Anant Mishra. Prepared 28 October 2026. Version 0.1.0.

**Status: preparation only. No model calls, DGX runs, quality scores, or benchmark results accompany this pack.** Local structural validation does not establish service performance, research validity, peer review, or publication indexing.

## Files and purpose

- `evaluation-plan.json`: proposed capacity matrix, repetitions, evidence fields, scoring rules, and unfilled service objectives.
- `environment-manifest.template.json`: unfilled environment record. Historical configuration descriptions are not evidence of the environment at execution time.
- `synthetic-quality-tasks.jsonl`: eight invented tasks for developing the review process. Six concern answer quality; two require an implemented application with permissions and caching. They are not a representative workplace dataset.

The plan proposes 3 input/output profiles × 3 concurrency limits × 3 seeds = 27 measured trials per configuration, each with 100 attempts. Ten separate warm-up requests precede each trial. These are planning choices, not a statistical sufficiency claim. Input tokens and actual output lengths must be verified; an output limit is not an observed output length. Capacity prompts are synthetic and distinct from the semantic quality cases. Saturated demand does not model the number of employees or a real arrival process.

## Validate locally without a model

From the repository root, using Python 3.10 or later:

```sh
python3 scripts/check-research-preparation.py
```

The standard-library script checks the plan, case IDs, coverage, synthetic labels, and the deliberately unfilled manifest. It prints hashes for the public input files. It makes no network request and cannot certify a run or a release decision. A non-zero exit means the preparation files are inconsistent. Keep this starter manifest unfilled; create a separate actual manifest when execution is authorised.

## Quality review

Define representative permitted tasks and a separate development/evaluation split before making a readiness claim. For each answer, record task ID, exact input/document revisions, actual response, citations, reviewer, dimension scores, and any critical failure. Preserve a second reviewer's independent scores and disagreements on a sample.

Score support, task completion, citation accuracy, and appropriate abstention separately:

- **2:** usable without material correction within the supplied evidence and access scope.
- **1:** requires a material human correction, with no release-blocking failure.
- **0:** wrong, unsupported, incomplete for the task, or answers when abstention is needed.

Record `not_applicable` with a reason when a dimension does not apply. Do not convert it to a successful score. Restricted disclosure and an invented policy presented as approved are proposed release blockers; keep them separate from the mean. The organisation must agree the scope and criteria before testing. Random-token output is not quality evidence.

For the two integration cases, inspect the retrieval, prompt, cache, and access-decision traces. Do not send the restricted fixture to the model and rely on a refusal. All fixture names and markers are fictional. No client document or personal data is needed to exercise these boundary tests.

## Before hardware execution

1. Obtain target-machine access and an approved test window. Freeze actual hardware, serving software, model/tokenizer revisions, licence, client location, and launch arguments in a separate manifest. Remove secrets before publication.
2. Complete and justify service objectives and timeouts, identify the accountable reviewer, and preserve the frozen plan hash. A null threshold is a pending decision, never a pass.
3. Save the installed `vllm bench serve --help` output and reconcile the versioned reference. No install, server restart, cache reset, or model change is performed by this pack.
4. Verify the capacity baseline's disabled prefix-cache setting, warm-up treatment, context fit, input token lengths, actual output lengths, and request failure accounting. Warm-cache or rate-controlled tests require separately recorded conditions.
5. Preserve each trial's exact command, start/end times, seed, raw per-request records, counts, units, and percentile method. Keep failures and timeouts; link retries to originals. Report repetitions individually before any aggregate.
6. Evaluate semantic tasks and then the complete application path. Serving latency goodput does not establish that responses pass the quality rubric. No cost or energy claim is possible without a defined measurement boundary and actual observations.

## Supplemental permission plan — 30 October 2026

`PERMISSION-EVALUATION.md`, `permission-test-plan.json`, and `permission-evidence.template.json` develop quality cases Q05/Q06 into twelve planned integration scenarios. Their local preparation validation runs through the same script above. No assistant is connected, no permission scenario is executed, and the evidence record remains blank. The original benchmark plan, eight quality cases, and environment template are unchanged.

## Supplemental request-accounting reference — 1 November 2026

`REQUEST-ACCOUNTING.md`, `request_accounting.py`, `synthetic-request-attempts.json`, and `test_request_accounting.py` provide an executable counting convention for client observations. Download the four files together and follow the guide. These synthetic software checks retain every supplied attempt and unknown outcome; they are not model runs, agent failure tests, or evidence of service-side idempotency. The earlier benchmark and permission plans are unchanged.

## Supplemental latency-boundary reference — 2 November 2026

`LATENCY-BOUNDARIES.md`, `latency_boundaries.py`, `synthetic-latency-boundaries.json`, and `test_latency_boundaries.py` provide a frozen fictional schedule, corresponding observations and executable timing arithmetic. The guide declares the single-clock boundary, completed-only samples, nearest-rank method, outcome counts and remaining measurement dependencies. No model, load generator or admission policy is run; the earlier benchmark and permission plans remain unchanged.

## Sources and limits

- Versioned CLI reference used for preparation: https://docs.vllm.ai/en/v0.31.0/cli/bench/serve/
- Current metric definitions and cache caveats, checked 28 October 2026: https://docs.vllm.ai/en/latest/benchmarking/cli/
- NIST AI 600-1, MEASURE 2.3 and 2.5: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

The installed version must be checked at execution. These sources do not validate the proposed sample size, fixture set, chosen hardware, or service objectives. Hardware access, completed environment records, representative authorised tasks, actual run logs, and accountable review remain dependencies. AI assistance was used to prepare this draft material; no empirical findings or author-reviewed scholarly release are asserted.