How to test an AI agent sandbox before it runs code

An AI agent sandbox test must prove five results before the agent runs code against real systems. The intended task succeeds. A denied action fails at the expected control. A test credential stays outside the inspected sandbox surfaces. The evidence identifies each decision, and the workflow returns to a safe state after a failure.

Use one real task. Pin the software and policy, test each authority path alone, and then run the paths together. The result applies to one configuration in one environment. It does not certify the product for every use.

Freeze one real task before you change the policy

Choose a task for the reviewed pilot. Name its input and output files, commands, network destinations, credential path, final action, owner, and the consequence of a wrong action. A label such as “coding work” or “support automation” is too broad for a useful test.

Record the sandbox version, policy file and hash, agent and tool versions, operating system, kernel, terminal, working directory, and test input. Use the same machine and input for the baseline and sandboxed runs. This makes the policy effect easier to separate from another change.

Use a disposable environment and non-production credentials. Keep customer data, production secrets, and irreversible systems outside the test. Direct any send, publish, merge, deploy, delete, or record change to a test destination, or stop the task before the final write.

Write the acceptance criteria before the first run:

Intended action:
Required files and commands:
Allowed destinations and methods:
Credential path:
Denied actions to attempt:
Evidence expected from each control:
Safe fallback:
Pilot owner:

This record is short on purpose. It gives the test a stable task and expected result without turning ordinary test work into a new framework.

Test each authority path at its enforcement point

An enforcement point is the component that allows or blocks an action. It can be an operating-system sandbox, command broker, network proxy, credential broker, approval service, or audit store. A failure at another component proves little. An unavailable test server, for example, does not prove that a network allowlist blocked the request.

Run one allowed case and one denied case for each path. Keep the input and policy unchanged while you test that path.

Scroll sideways to see all 5 columns.

Path	Allowed case	Denied case	Evidence to keep	Safe fallback
Filesystem	Read and write only the task fixtures named in the policy	Attempt a sibling path, sensitive path, or unlisted output path	Requested path, access mode, enforcement component, decision, time, and error	Preserve the fixture and return the task to review
Command	Run the approved executable with the approved argument shape	Change the executable, subcommand, argument, working directory, or child process	Full argument list with secrets removed, caller, policy rule, decision, and exit state	Stop the command and keep the previous approved output
Network	Send the required method and path to the approved test destination	Change the host, method, path, protocol, or route	Destination, method, path, visible route, proxy decision, response, and retry count	Queue the work or use a manual handoff
Credential	Complete an approved request through the credential broker	Inspect sandbox surfaces and send the stand-in to an unapproved destination	Credential identifier, broker route, endpoint rule, redemption result, and redacted logs	Revoke the test credential and return to a non-credentialed path

For the filesystem test, prove that the agent can read and write the required fixtures. Then attempt a denied path and confirm that the operating-system or sandbox control blocked it. A missing file, ordinary file permission, or broken path is a different failure.

For the command test, use the required executable and argument shape. Then change one part that the policy must deny. Include child commands because an approved tool can start another process. The evidence should identify the caller and the process that the broker allowed or blocked.

For the network test, confirm that the enforcement point can see the governed traffic. Nono’s v0.70.0 per-route limit is one example. The project says its proxy applies the limit to application-layer traffic that it can read. Opaque CONNECT tunnels remain outside the limit. Test an approved route, a denied route, the timeout or rate limit, and the agent’s response to rejection.

For the credential test, use a unique test credential with narrow permissions. If the sandbox uses a stand-in value, check the process environment, mounted files, process arguments, command output, and relevant logs for the real test secret. Confirm that the stand-in works only through the approved broker route and fails at an unapproved host, method, or path. A clean search supports the test; it cannot prove that no unknown leak path exists.

Nono’s merged Vault OIDC work shows a project claim at this boundary. The pull request says the real token remains outside the sandbox and a stand-in is redeemed on approved egress. A local test must still prove that behavior with the pinned build, policy, secret store, logs, and destination rules.

Test resource limits separately from access limits

A sandbox can block files and host functions while code still runs too long or uses too many resources. The Lua.ex README separates these controls. It blocks dangerous standard-library paths by default, while call-depth and instruction limits require host configuration.

Test the timeouts and CPU or instruction budgets that matter for the task. Include memory, recursion depth, output size, process count, request rate, and retry count. Confirm the error, cleanup, audit event, and state of partial output. A timeout that leaves a child process, lock, credential, or half-written file has not reached a safe fallback.

Run the assembled policy and exercise failure behavior

After each path works alone, run the complete task with the final policy. Policy settings can change one another. Nono v0.70.0 includes a correction that rejects an upstream proxy during hard network blocking and a correction for a credential that could activate host filtering. These Nono changes show why any sandbox with several policy layers needs an assembled test.

Run the intended task, then repeat it with an unavailable service, malformed input, denied action, timeout, exhausted rate limit, and revoked test credential. Test the agent’s retry behavior as part of the same run. A proxy can return a correct denial while an unbounded retry loop still overloads the workflow.

If the sandbox can request more access at runtime, exercise approval, denial, and timeout on the planned operating system, terminal, and agent interface. The approval display, enforced result, and audit entry must agree. An unreadable prompt or untraceable decision makes that approval path unreliable.

Inspect the audit output after every case. It should identify the session, policy version, action, enforcement point, decision, timing, and final state without copying secrets. Then run the fallback. Confirm that the item enters review, the last approved output remains active, temporary access is revoked, or the workflow stops in its selected safe state.

Keep project claims, issue reports, and local evidence separate

Release notes and merged pull requests can show intended behavior and project tests. They do not replace a local test. Nono’s README says its APIs are still stabilizing before 1.0, so pin the tested release and policy.

Issue threads need the same care. A v0.70.0 audit report said Linux capability decisions did not appear in nono audit show. The issue closed on July 30, 2026. The retrieved thread did not name a release with a fix. An interactive terminal report remained open on July 30 and described difficult-to-use approval output in one Linux TTY path. A later comment described a similar symptom on a different macOS and command-policy path.

These threads are reports and discussions, not confirmed vulnerabilities or proof that every approval path fails. Turn a relevant report into a local regression test. Record whether the pinned build passes, fails, or remains untested. A closed issue does not remove the need to retest. An open issue outside the workflow’s authority path does not automatically block a bounded pilot.

The Nono 0.70 control-path analysis applies this source boundary to one release. The OpenShell runtime analysis explains why controls belong outside the model. If a future rule may block legitimate traffic, use observe mode before enforcement.

Decide whether the pinned configuration can enter a reviewed pilot

Mark every required path as verified, failed, not tested, or open to interpretation. A reviewed pilot requires a successful task and each required denial at its expected enforcement point. It also requires no test secret in the inspected sandbox surfaces, usable evidence, and a completed fallback. Keep a failed or untested critical path out of the pilot.

Limit the approval to the tested software version, policy hash, operating system, kernel, terminal, agent, tool chain, destinations, credential method, and task. Retest when any of those conditions changes. Also retest after a policy exception, new child command, new network route, new approval backend, or sandbox update.

The AI workflow security review guide connects this test to the access decision. Use the AI workflow security review worksheet to record the task, excluded data, allowed actions, reviewer evidence, and stop conditions before access expands. BaristaLabs uses the same review to help teams test sandbox boundaries without turning one successful demo into a production claim.