Designing a Fair Test for a Local AI Service

The proposed service and the decision

Consider a fictional organisation preparing a read-only assistant for its internal procedures. Employees would ask questions, receive source-backed explanations, and escalate unresolved issues to a human. The proposed local inference service would generate answers; the application would own authentication, permissions, retrieval, and presentation.

The decision is whether a specific configuration deserves a bounded application pilot. This design study sets out the evidence needed to make that decision. It reports no hardware experiment, customer engagement, or measured improvement. Every workload size below is a planning assumption.

Separate two experiments that answer different questions

The first evaluation asks whether the application produces useful, permitted answers. It needs realistic semantic tasks, a fixed document set, an access matrix, expected behaviour, and a review rubric. The second asks how the serving process behaves under controlled demand. It needs recorded prompt lengths, output limits, arrival patterns, cache conditions, and failures.

Random-token prompts can help control a capacity experiment. Their generated text cannot establish whether an employee received a correct explanation. Conversely, a small hand-scored task set cannot establish sustained multi-user capacity. A final application-level evaluation must connect quality with the full response path under intended conditions.

NIST AI 600-1 supports empirical capability assessment and verification of generated citations. It supplies evaluation principles; it does not endorse this workload or provide results for it. [3]

Declare a modest workload matrix

The proposed capacity matrix has three profiles: 512 input tokens with a 128-token output limit; 2,048 input tokens with a 256-token output limit; and 8,192 input tokens with a 256-token output limit. These probe increasing context size while keeping the comparison understandable. They are not claims about normal employee behaviour.

For each profile, the initial plan uses maximum in-flight request limits of one, four, and eight, with 100 attempted requests per trial and three repetitions. A release report would disclose the actual output lengths and completed counts. The proposed sample size is an initial engineering choice, not a guarantee of statistical precision.

A saturated, concurrency-limited run asks how the server behaves when work is continuously available. It does not reproduce a workplace's arrival pattern. Rate-controlled trials should follow when observed or justified demand assumptions exist. Preserve those as separate experiments rather than treating a concurrency limit as a measured number of users.

Freeze what changes the comparison

Before any hardware run, complete the environment manifest: hardware, operating system, driver, serving version, model and tokenizer revisions, quantisation, context limit, launch arguments, and the relevant licence. Record the benchmark-client version and network location as well.

The command reference used to prepare this pack is versioned vLLM v0.31.0 documentation. It documents controls for concurrency, random workload lengths, detailed result saving, and latency objectives for goodput. This is a reference version, not a claim that the user's hardware currently runs it. The operator must reconcile the installed command help before execution. [1]

Keep warm-up separate from measured trials. Use a predeclared cache policy and record any restart or reset. vLLM's current benchmark guide warns that repeating prompts against retained prefix caches can change throughput. A repeatable seed alone therefore does not make two trials comparable. [2]

Keep an evidence record for every attempt

Each trial should retain a plan version, configuration hash, UTC timestamps, command, seed, warm-up record, cache state, and raw output. Each attempted request needs a stable identifier and an outcome. Preserve errors and timeouts alongside successful responses.

Report attempted, successful, failed, and timed-out counts before presenting latency percentiles. Keep the percentile method, sample size, and measurement boundary visible. Do not combine the successful requests from different repetitions into one flattering headline while discarding a failed run.

The report should distinguish serving goodput, based on declared latency objectives, from answers that also pass the task rubric. A latency-qualified response can still contain the wrong instruction. For this application, useful completion requires both acceptable content and acceptable service behaviour.

Prepare quality cases without pretending they are field data

The linked preparation pack contains eight small synthetic cases. They cover a supported answer, missing information, contradictory versions, a citation mismatch, a restricted document, revoked access to a cached answer, an instruction embedded in source text, and an incomplete source excerpt.

Each case includes fictional context, expected behaviour, and a scoring note. Permission cases name the expected application behaviour: blocked content must never reach the model. They are specifications for an integration test, not evidence that a model prompt can enforce access.

The pack also includes a proposed run plan, an unfilled environment manifest, and a review guide. A local validator checks file structure, case coverage, and plan consistency without contacting a model. That check establishes that the preparation material is readable and internally consistent. It establishes no answer quality or hardware performance.

What would justify moving forward

To execute this design, an operator needs access to the target machine, a verified configuration, an approved test window, completed service objectives, and an accountable reviewer for quality judgements. Representative authorised tasks must replace or extend the starter examples before any production-readiness claim.

A later report should publish the permitted inputs, raw run evidence, scoring decisions, failed cases, and limitations. If a configuration fails an agreed gate, retain that result and either narrow the scope or revise the system. No configuration is declared a winner here.

The local LLM inference protocol links the preparation files and keeps this work under its accurate status: experiment preparation, with measurements pending. The companion acceptance scorecard essay explains how those records should inform a release decision.