Executive summary
A sandbox is an important control, but its configuration is not evidence that every action stayed inside the intended scope. An AI evaluation may use shells, package managers, network tools, and cloud identities. Each dependency adds a boundary that the operator must define and test. The practical question is not simply whether isolation was configured. It is what evidence would reveal an unauthorized crossing, and who would receive that evidence.
Aktoh Cyber proposes a complementary containment-verification layer: controlled canary credentials, decoy endpoints, and independently retained event records. The proposal draws on deception-based detection, but it is not a replacement for isolation, access control, or human evaluation. This paper presents a design to test, not a validated product, an established detection rate, or proof that an environment cannot escape.
Separate prevention from evidence
Network restrictions and identity policies limit the actions an environment can take. Configuration review checks those intended restrictions. Runtime telemetry records what actually happens. Transcript review helps reconstruct the sequence and understand the evaluated agent's decisions. These controls complement each other; none should be described as useful only before or after a run when it can also support ongoing monitoring.
Boundary instrumentation adds a narrower signal: interaction with an asset that the task did not authorize. The value depends on placement, coverage, and reliable delivery. A canary can reveal a particular path being used. Silence from that canary says nothing about paths it cannot observe.
Boundary canaries and controlled decoys
Use synthetic, narrowly scoped credentials that cannot unlock production data. Place them away from legitimate task inputs and scoring artifacts. Route interactions to controlled infrastructure with no connection to unrelated organizations. Record the permitted task scope, the canary identifier, the run identifier, and the expected interaction policy before execution begins.
A reachable decoy outside the approved boundary may help reveal unintended reachability, but only if its location and access conditions are independently verified. An interaction is a reason to investigate the run; it is not automatically proof of malicious intent, model misalignment, or a successful production compromise.
Cover the paths, not just the perimeter
Review direct HTTP requests, redirects, DNS, package registries and caches, cloud metadata, mounted secrets, service-account credentials, and intermediary tools. This is a review checklist, not a claim that each path is exploitable in every environment. Apply controls where each request actually leaves the trusted boundary, including indirect requests made by dependencies.
Default-deny egress should be enforced outside the agent's administrative control where feasible. Approved destinations need explicit scope and logging. Deception remains supplemental: legitimate access controls should prevent unauthorized traffic even when no decoy is touched.
Keep evidence outside the environment under test
Deliver boundary events to a collector the evaluated agent cannot alter. Use scoped access, stable run identifiers, trustworthy timestamps, retention rules, and documented integrity checks. Test collector failures and dropped messages as well as successful delivery. A local log that disappears with the sandbox is not sufficient independent evidence.
Define who can stop a run, what signal requires review, and how evidence is preserved. Any automatic intervention must follow an agreed policy; a monitoring instrument should not silently decide what the evaluator counts as a valid result.
What this design cannot establish
Canaries may be avoided, removed, accidentally triggered, or recognized as test artifacts. They may also change the agent's behavior or provide a shortcut through a task. Detection latency depends on collection and alerting as well as placement. Claims such as instant detection, zero false positives, or complete coverage require evidence this proposal does not supply.
The next useful step is a controlled pilot: enumerate covered paths, run intentional boundary tests, measure missed events and false alarms, and compare task outcomes with and without instrumentation. Publish failures alongside successes before making a broader assurance claim.
Conclusion. Treat containment as a property to test and observe. A decoy interaction can add evidence; the absence of one cannot prove safety.
Sources and scope
- Inspect AI: sandboxing
Background on isolated tool execution in AI evaluations; not a guarantee of containment.
- CISA and NCSC: secure AI system development
Secure-by-design guidance across the AI system lifecycle.
- Joint guidance: Deploying AI Systems Securely
Operational security recommendations for deploying externally developed AI systems.