A Timeout Is Not a Failed Action: Testing AI Agents Under Uncertainty

An AI assistant opens a maintenance request. The service stores it, but the reply disappears before the assistant receives it. The assistant sees a timeout and tries again. If the second call creates another request, a network problem has become an operational problem.

This fictional example exposes a useful test question: can the workflow distinguish an unsuccessful action from an action whose outcome it does not know? A fluent explanation after the event is insufficient. The system needs evidence about what happened and a bounded way to resolve uncertainty.

This essay proposes that test discipline. It reports no deployed agent, client incident, or measured reliability improvement.

Give uncertainty its own state

Separate the intent, the attempted call, and the confirmed business result. A generated tool argument is an intention. Dispatch means a request was sent. A transport acknowledgement means some component received it. None of those facts alone establishes that the intended business operation finished.

For a particular tool, define what counts as confirmed completion, confirmed rejection, and an unresolved outcome. Rejection must have a documented meaning: an error response can arrive after partial work. A timeout belongs in the unresolved category until the service contract or a later check establishes more.

Keep the user-facing status equally precise. If the system cannot confirm whether a request exists, it should show that uncertainty and its reference number. It should not manufacture a success message or quietly replace uncertainty with failure.

Make retry permission part of the tool contract

HTTP semantics already draw this distinction. RFC 9110 advises against automatically repeating a non-idempotent request unless the client knows the operation is effectively idempotent or can establish that the original request was not applied. A timeout alone supplies neither assurance. [1]

AWS's idempotent-API guidance describes caller-supplied request identifiers, parameter matching, and retention of earlier requests. The token has value because the receiving service enforces a contract around it. Writing a key into an agent log does not create that contract. [2]

For each mutating tool, record whether replay is supported, how long its protection lasts, which caller and operation scope it covers, and how to look up the original result. If that information is unavailable, the proposed default is reconciliation or escalation rather than another mutation.

Preserve one intent across several attempts

A logical operation needs a stable identity before its first dispatch. Individual attempts need separate identities beneath it. The workflow should retain the approved payload revision, acting principal, applicable permission decision, tool version, and service receipt when one exists.

Changing a location, amount, or destination changes the proposed action. Treating that change as a retry can hide a new decision inside an old authorization. Conversely, assigning a fresh operation identity merely because the network failed can defeat duplicate protection. The application must make those distinctions outside the language model.

Two deliberately requested, identical operations can also be legitimate. A content hash alone cannot establish whether the caller wanted one operation or two. Preserve the originating intent as well as the parameters.

Test the moments a demonstration skips

Build a controlled test service that can fail at named boundaries. Keep its authoritative records separate from the assistant's view. Then compare actual effects with what the assistant reports.

• Commit followed by a lost reply: store one request, suppress the response, and inspect whether the workflow reconciles it without creating another.
• Duplicate delivery: deliver the same operation twice, including concurrent calls. Check the service's effects and receipts, not only the assistant's final sentence.
• Worker restart: stop processing after dispatch but before a receipt is saved. Recovery must retain the original operation identity and the unresolved status.
• Stale permission or changed payload: revoke access or alter the proposed action before another attempt. Check that the original authorization cannot be reused for the changed circumstances.

These are proposed integration tests, not results. Add late replies, unavailable status lookups, and expired duplicate-protection records. A useful failure report identifies the boundary, the expected invariant, the observed effect, and the unresolved evidence.

Bound recovery work as carefully as normal work

Choose a recovery budget before testing: maximum attempts, maximum reconciliation checks, and a deadline. The values depend on the service and the consequence of delay. Record who can change them. A model should not extend its own budget because another attempt sounds promising.

When the budget expires, move the workflow into a visible review state. Stopping the workflow does not prove the remote action stopped. Keep watching for late evidence through an accountable recovery process, and prevent a new worker from treating the review state as permission to start over.

Give the reviewer a decision, with evidence

A human handoff should include the intended operation, exact attempts, known receipts, current authoritative status, and the remaining uncertainty. Offer only actions the reviewer's role permits. Avoid a generic retry button when replay could create another effect.

OWASP's 2026 Excessive Agency guidance recommends narrow tool capabilities and independently enforced authorization, with human review for consequential actions. Those controls belong at execution boundaries. They cannot be replaced by a prompt asking the model to be careful. [3]

A compensating action needs its own permission, receipt, and failure treatment. Cancelling a request may be possible; undoing a notification someone already read is not. Specify the remaining effects instead of promising a universal rollback.

Report every attempt without hiding the unfinished ones

A successful second response should not erase the first timeout. Count logical requests and physical attempts separately. Retain unresolved attempts even when another attempt returned a usable response. Multiple successful responses also deserve inspection; they do not prove multiple business effects, but they reveal work that a single success count conceals.

The local-inference preparation pack now includes a small executable accounting reference using synthetic records. Its software checks concern counting rules only. They neither run a model nor establish that an agent handles these failure scenarios correctly.

The accompanying service-request design study turns the uncertainty boundary into a proposed workflow. Before trusting an agent with operational authority, ask it to demonstrate an honest unresolved state, a reproducible recovery path, and an evidence-bearing stop.