Testing an agent means more than testing an answer

A final answer can conceal an incident

An agent can claim to have processed a case when no write succeeded. It can also provide an excellent summary after reading a prohibited resource. Assurance therefore concerns the acting system: inputs, tools, identities, control decisions, state transitions and persisted effects. The final answer is one observation among others.

Within Digital Place’s Agentic Assurance / Control Engineering approach, we recommend connecting every requirement to a control and evidence. This note sets out an engineering method. It claims no certification or measured security level for a product.

Define invariants before scenarios

State properties that must always hold: no out-of-scope reads, no unauthorised actions, no secrets in outputs, no approval reused after an operation changes. Define expected final state and stopping conditions too. The model may choose variable paths while permission boundaries remain stable.

A hypothetical case update may permit preparation, require approval for writing and prohibit deletion. The test checks database state, attempted calls and controller decisions after execution. A “done” message is not evidence of a write.

Observe trajectories without requiring one path

A trajectory records observable execution events: retrieval, proposed tools, arguments, results, controls and state. Several paths can satisfy a task. Tests should enforce necessary dependencies, such as authorisation before writing, without imposing arbitrary ordering on searches without side effects.

Anthropic’s January 2026 guide to agent evaluations distinguishes tasks, trials, traces and environmental outcomes. Our practical application is to check usefulness, policy adherence and final state separately. Observable events are sufficient; recording a model’s internal reasoning is unnecessary.

Tool calling: test the real boundary

Validate argument types, limits, target resources and outputs. Reject unexpected fields rather than silently ignoring them. A valid schema proves neither authorisation nor the appropriateness of an action. Check both server-side, close to the effect, with distinct identities and minimal permissions.

Test missing tools, changed descriptions, truncated results, timeouts, transient errors and partial success. After a failure, the agent must distinguish an unexecuted action from one whose outcome is unknown. Retries must respect idempotency and budgets; tool responses cannot grant themselves new permissions.

Deterministic policies, probabilistic behaviour

Access control decisions, schema validation, resource limits and authorised transitions support deterministic assertions. Answers, tool selection and task resolution require repeated trials. Combining these levels into one score conceals important defects.

High average task success cannot compensate for a forbidden action. Report business success, policy violations, escalations, budget overruns and state errors separately. A low temperature does not guarantee reproducibility: model versions, tools, environment and context remain variables.

Golden sets and maintained scenarios

Build a reference set covering routine cases, boundaries and realistic failures. Each scenario specifies initial state, identity, accessible sources, tools, policy, expected outcome and grading criteria. Separate development and validation data. Minimise or synthesise incident-derived cases before adding them to the corpus.

Combine code checks for verifiable facts, human review for ambiguity and model judges for selected writing qualities. Calibrate model judges, let them declare uncertainty and never let them approve a security violation. Revisit scenarios that become too easy or depend on obsolete tools.

Injection tests at every untrusted entry

Place direct attacks in requests, indirect attacks in retrieved documents and malicious instructions in tool responses. Include multilingual content, encodings, misleading quotations and claims of administrator authority. The OWASP Agentic Top 10 2026 helps organise risk coverage.

Verbal refusal alone is insufficient. Check that no forbidden call executed, no data left its permitted scope and state remained safe. Prompt filtering is an additional measure; it does not replace system-enforced permissions.

MCP and the trust chain

Test MCP servers as dependencies able to change schemas, descriptions and results. Verify provenance, versions, authorisation, token audiences and connection rules. MCP security best practices discuss confused-deputy attacks and token forwarding risks.

Authentication does not grant unlimited business permissions. The controller must check the action and resource and prevent tool descriptions from expanding scope. Tool and dependency updates belong in the same regression discipline as prompt changes.

Sandboxing and human approval

Restrict networking, filesystem access, processes and secrets to task needs. Exercise the actual denials: out-of-directory reads, forbidden connections, subprocess launches and secret access attempts. A sandbox with unrestricted networking can still allow exfiltration.

Human approval must show a concrete action, parameters and consequences. Bind it to that operation, requester identity and state; expire it or request fresh approval when those change. Test refusal, expiry, double clicks and resumption after interruption. A model cannot create approval by asserting it in text.

Traces, replay and auditability

Record policy, tool, model and corpus versions, authorisation decisions, timestamps and confirmed effects. Minimise personal content and mask secrets. A trace should explain what was permitted or denied, by which control and with what result, even when execution stops midway.

Distinguish replay of a recorded trace from a new model execution. The former helps reproduce a control decision; the latter may vary. Replay effects with simulated tools in isolation, never against a live business system. Verify that reviewing evidence does not grant execution rights.

Budgets, SLOs and safe failure

Set time, call, token and effect budgets per task. Test loops, slow tools, saturated context, cancellation and dependency outages. An SLO should describe a task correctly completed within its permissions, not merely model API availability.

When budgets are exhausted or state is uncertain, stop side effects and provide a human recovery path. Keep confirmed tasks distinguishable from prepared ones. Compensation, where available, must be explicit and tested; it does not make an irreversible action harmless.

Regression across the whole system

Rerun controls when a model, prompt, tool, policy, index or runtime changes. Compare scenarios and severity levels, then inspect regression traces. Research on defeating prompt injections by design highlights architectural boundaries; a research publication does not guarantee safety for every implementation.

Useful evidence connects requirement, threat, control, test and result. That chain makes an agentic system assessable by a CIO, maintainable by engineers and possible to suspend when it no longer meets its contract.