Before You Buy a GPU: The Decisions That Make Private AI Work

A private AI project often begins with a hardware conversation. How much memory? Which GPU? How large a model can it hold? Those are reasonable questions. They become expensive when nobody has agreed what the system must actually do.

Consider an internal assistant that searches operating procedures and drafts answers for staff. A convincing demonstration might involve one person asking three questions. The working system must deal with old documents, restricted folders, impatient users, busy periods, and questions it should decline to answer. That is the workload worth designing for.

The first purchase decision should follow a written operating brief. Here is what that brief needs to settle.

Start with the work people need to finish

Choose one task with a clear owner. Document search, invoice extraction, and software assistance have different demands. A search assistant needs reliable retrieval and source attribution. An extraction service needs predictable fields and a way to reject an uncertain result. A coding assistant needs evaluation against the actual languages and repositories its users work in.

Write down a useful answer, an unacceptable answer, and an appropriate escalation. Then assemble representative examples that you have permission to use. Include missing information and awkward cases. A system that works only when the question is perfectly phrased will send the hard work back to its users.

These examples become an acceptance set. They give engineering and leadership a shared basis for deciding whether the pilot deserves a larger investment.

Define what control means for this organisation

Keeping model inference on a local machine changes where that computation happens. The surrounding system still includes document storage, identity services, retrieval indexes, monitoring, backups, software updates, and the people who administer them.

Map those flows before using the word sovereign. Where can a prompt travel? Which logs retain it? Can an administrator read retrieved documents? What happens to a backup? Who approves a model or dependency update? The answers should be concrete enough to inspect.

NIST's Generative AI Profile provides a useful risk-management reference across design, development, use, and evaluation. It is a voluntary framework, rather than a certificate that makes a particular deployment safe. The practical task is to translate relevant risks into controls with named owners. [2]

Measure the experience under a realistic load

A tokens-per-second number leaves several questions unanswered. How long did the user wait before the first output appeared? How many other requests were running? How long was the input? Did the answer finish successfully? Was its content useful?

The vLLM benchmark documentation distinguishes time to first token, inter-token latency, and request-level time per output token. It also explains that similarly named metrics can use different measurement points or aggregation methods. Comparisons need the definitions and conditions alongside the numbers. [1]

A useful pilot report should include a workload description, input and output lengths, concurrency, successful and failed requests, quality checks, and latency distributions. Record the exact model revision and serving configuration. Show slower requests as well as the average. An impressive average offers little comfort to the person whose request repeatedly stalls.

Give documents the same access rules as their owners

An assistant can become a convenient route around existing permissions if retrieval treats every indexed file as available to every user. The access decision has to happen before restricted content enters the prompt.

For the operating-procedure assistant, that means carrying document permissions into retrieval and testing them with real access scenarios. A branch employee should not receive a management document simply because the wording matches a question. A departed employee's access should end with their account. Prompt logs and cached answers deserve the same review.

These are proposed design controls. Their effectiveness must be tested in the implemented system; writing them into an architecture document does not establish that they work.

Price the service you will have to operate

The hardware invoice is one line in the operating model. Engineering time, monitoring, patching, recovery, storage, power, and unused capacity also matter. The comparison should use the same expected workload and quality requirements for each option.

It is entirely possible for a team to value tighter data control enough to accept a higher operating cost. It is also possible for an intermittent workload to make dedicated hardware difficult to justify. State the trade-off directly. Avoid a savings claim until a measured baseline and a complete cost model support it.

Include a recovery drill in the pilot. If a machine fails, which work stops, what can be restored, and who takes responsibility? An AI service becomes part of normal operations the moment people begin to depend on it.

Leave the pilot with a decision

A pilot should end with one of three choices: proceed, revise, or stop. Agree the criteria before the demonstration. For example, a team might require acceptable answers on its approved evaluation set, a latency target at an expected peak load, verified permission boundaries, and a recovery procedure another operator can follow.

The thresholds depend on the task. Their purpose is to make the decision inspectable. If the system misses a target, record what happened and whether a change is worth testing. That produces more useful knowledge than quietly changing the test until the dashboard turns green.

The accompanying local LLM inference research protocol sets out a proposed way to investigate these questions. The measurements still need to be made. For anyone considering private AI now, the immediate step is straightforward: write the workload, control requirements, acceptance tests, and ownership model on one page. The hardware discussion will then have something solid to work from.