Designing a Queue Test for a Shared Local AI Service

A fictional shared service and a narrow decision

This Design study proposes a queue evaluation for a fictional organization sharing a local AI service across public-infrastructure planning teams. Authorized staff would request short procedural explanations or longer document summaries. The service would support internal analysis; it would not make eligibility decisions, dispatch field crews, or issue public instructions.

No organization, project, deployment, or demand trace is represented here. The question is narrower: how should a test reveal work waiting before dispatch, work waiting inside the service, and work that never completes? A fast generation rate alone would not settle that question.

Freeze the arrival schedule before admission

Prepare a finite list of fictional arrivals with stable request IDs, scheduled offsets, and workload-class labels. Use a spread-out schedule and a separate burst schedule. Both are engineering assumptions until supported by permitted operational evidence. Keep each schedule unchanged when comparing candidate admission rules.

Record the intended arrival even if the load generator cannot dispatch it. Its capacity limit is part of the test environment. Replacing a missed scheduled arrival with a later timestamp would conceal generator waiting and make the candidate configurations difficult to compare.

Grafana k6's open-model documentation describes arrivals that are scheduled independently of earlier completion. That provides the conceptual distinction for this design; it does not establish that a particular schedule reproduces real demand. [2]

Compare admission behaviour under the same conditions

The first proposed candidate is a bounded first-come, first-served queue with a maximum waiting age. A second candidate adds explicit per-principal admission limits while preserving the declared workload classes. Queue size, age limits, and principal budgets remain decisions for an accountable service owner. This study supplies no calibrated capacity recommendation.

Compare the candidates on identical model, runtime, prompt set, cache policy, arrival schedule, and generator resources. Report each repetition separately. Keep semantic answer evaluation independent: accepting requests promptly does not show that their answers are supported by the source documents.

The earlier local-service evaluation design covers workload profiles and environment evidence. This study adds the admission and timing-boundary question; it does not replace the original benchmark plan or claim another hardware run.

Instrument both sides of dispatch

The client would retain planned arrival, dispatch, first qualifying streamed output, and completion or observation-end events. Derive client intervals using one monotonic clock. State what qualifies as first output, especially if the serving interface emits content and other stream events separately.

The current vLLM benchmark documentation exposes client queue-time reporting and a schedule-relative end-to-end metric under finite request rates. Verify the installed client's support and saved fields before execution, rather than assuming that a command from an earlier version captures them. [1]

Capture server-side evidence separately when investigating where dispatched work waits. vLLM documents waiting, prefill, and decode metrics at the server. An aggregate server histogram cannot identify the generator-side interval for a particular request. [3]

Match diagnostic records through stable IDs and documented boundaries. Avoid joining unsynchronized absolute timestamps or assigning every unexplained millisecond to the GPU. Any diagnostic instrumentation overhead also belongs in the run record.

Preserve every planned request at the cutoff

Classify the frozen schedule at a declared observation cutoff. The proposed categories are completed, timed out at the client, explicitly rejected, not dispatched, and still in flight at the cutoff. Partial output before a timeout remains partial; it cannot create an invented completion timestamp.

A completed-only latency chart must state its sample count and sit beside the other outcome counts. A candidate that rejects most demand could look quick among its remaining completions. The comparison therefore needs both admission coverage and conditional latency, without treating a fast rejection as a successful answer.

The service owner must also define cancellation behaviour and verify it in the implementation. A client deadline does not establish that the server released resources. This proposal includes no cancellation or overload test result.

Use synthetic arithmetic to check the report first

A small downloadable reference supplies six invented timing records. Three complete; the remaining three represent a timeout, a rejection, and a request not dispatched. One completed record has a 600-millisecond dispatch-to-first-output interval and a 2,400-millisecond schedule-to-first-output interval. Both follow directly from its fictional timestamps.

The reference calculates the two boundaries without guessing server queue time. Its completed-only summaries name the population and nearest-rank percentile method. Software tests exercise timing order, missing events, invalid values, outcome partitions, and the distinction between observed duration and completed-response latency.

These checks validate arithmetic and input handling only. They do not simulate a queue scheduler, call vLLM, execute the two admission candidates, evaluate answer quality, or establish tail-latency performance. The fixed fixture is not a load test.

Make the release decision answerable to evidence

A later comparison would need actual schedule-versus-dispatch traces, server evidence where relevant, outcome counts, workload-specific distributions, repeatability records, and reviewer-approved service objectives. It should show which requests missed their intended window and whether one principal or class absorbed disproportionate delay.

Only that evidence could justify selecting an admission policy or narrowing the supported workload. None is inferred from the synthetic timing example. The research protocol and reproducibility files identify the remaining execution dependencies, while the companion essay explains how to read the resulting report.