Where the Waiting Starts: Reading a Shared AI Service’s Latency

A shared AI service can look quick in a benchmark while requests spend substantial time waiting to be sent. The report may be accurate about the interval it measures. The problem begins when that interval is presented as the whole experience.

Consider a fictional request scheduled at the start of a test. The client waits for a concurrency slot, dispatches it later, and then receives streamed output. Starting the stopwatch at dispatch leaves the earlier wait outside that particular metric. Before interpreting an average or a percentile, ask where its clock starts.

This essay develops that measurement boundary for shared local AI services. The examples are synthetic. They report no hardware benchmark, deployment, or observed employee demand.

Name the interval before naming the percentile

Record four events on one instrument's monotonic clock: the planned arrival, actual dispatch, first qualifying output, and completed response. Planned arrival describes the test schedule. Dispatch describes when the client actually sends the request. Their difference is generator-side waiting, which can include a concurrency gate or a delayed generator.

The current vLLM benchmark guide distinguishes client queue time from dispatch-relative latency. It also documents schedule-relative end-to-end reporting for finite request rates. These options matter: a modern tool may expose the missing interval, but the report still needs to select and explain it. Verify the installed version before relying on a current documentation feature. [1]

A scheduled timestamp is not automatically a person's button-click timestamp. Authentication, document retrieval, and interface work may sit outside the benchmark. Describe the complete application journey separately when making a user-experience claim.

Work through one small example

In the fictional timing reference accompanying this article, request L01 is scheduled at 0 milliseconds, dispatched at 1,800, produces its first output at 2,400, and completes at 3,000. All four numbers are invented offsets on the same clock.

Those durations answer different questions about the same invented record. The 1,800-millisecond difference belongs to the generator-side interval. It is not evidence of server queue time, GPU execution time, or a real user's waiting time.

Choose how demand arrives

A test that starts another request only when a previous one finishes has a different demand model from a test with independently scheduled arrivals. Grafana k6 documents this as the distinction between closed and open workload models, including the risk of omitting demand when slow responses themselves reduce new arrivals. [2]

A fixed-client test is useful when that is the intended question. An independently paced schedule is useful when investigating what happens as work keeps arriving. Neither becomes a model of a workplace merely by being labelled realistic.

Declare the intended schedule and compare it with actual dispatches. If the generator falls behind or cannot start a scheduled request, retain that record. Quietly moving its planned arrival forward changes the experiment and removes the wait from view.

Keep client waiting separate from server waiting

Once dispatched, a request can encounter transport, frontend processing, server scheduling, and generation. A long client-observed wait does not identify which component caused it.

vLLM's production metrics include a server-side waiting-phase histogram alongside prefill and decode metrics. These describe engine-side boundaries; they do not measure time spent before the benchmark client dispatched a request. [3]

Use linked request traces when investigating a particular delay. Do not subtract unrelated percentile summaries and call the remainder network overhead. Different requests can occupy the tail of each distribution. Cross-machine timestamps also need a declared synchronization method; a shared field name does not make their clocks interchangeable.

Show the population behind each percentile

A completion-latency percentile usually needs a defined set of completed responses. Rejected requests and timeouts do not acquire completion times simply because a chart expects a number. A timeout is an observation boundary, not proof that remote computation stopped.

The released reference fixture has six scheduled requests: three complete, one times out, one receives a rejection, and one is not dispatched. Its completion-latency summaries therefore contain three samples. The other outcomes remain visible as counts; none is silently turned into a zero or an invented completion.

The reference explicitly uses the nearest-rank percentile convention. With three values, its p95 selects the largest value. That is an arithmetic property of a tiny synthetic set, not a stable estimate of a service's tail. Keep the sample count, selection rule, units, and percentile method beside any reported statistic.

Examine who experiences the delay

A pooled report can obscure different operating conditions. Separate short lookups from long summaries when those are distinct workloads. Preserve input length, actual output length, cache condition, and arrival pattern. Compare the same defined population across configurations.

Also inspect the age of queued work and the distribution of waiting across authorized principals or work classes. A proposed admission rule may improve one group's experience while delaying another. A global average cannot decide whether that trade-off is acceptable.

For sovereign computing, local model execution is only one operating decision. The organization also owns admission rules, workload priorities, trace retention, and the response when demand exceeds capacity. Those decisions need evidence rather than an attractive throughput headline.

Leave the report with a concrete decision

Present arrival and dispatch counts, outcome counts, declared timing boundaries, conditional latency summaries, and relevant workload slices together. State the observation cutoff and show what remains unresolved. Then identify the next decision: change the admission policy, narrow the supported task, investigate a component, or run another controlled comparison.

The companion queue-test Design study specifies that proposed comparison. The local-inference research pack supplies executable timing arithmetic and synthetic software checks. Neither is a completed model experiment. The immediate improvement is a report whose stopwatch and denominator can be inspected.