Measuring AI Containment Verification Without Distorting the Evaluation

Executive summary

A containment instrument can detect a boundary event and still be unsuitable for an evaluation. If its credentials or endpoints help the agent complete the task, its presence changes what is being measured. If it alters the workload, hides useful context, or breaks under privileged access, a strong-looking detection result can be misleading.

This Aktoh Cyber protocol proposes separate measurements for security performance and evaluation integrity. It is a methodology for controlled testing, not a report of an implemented system or benchmark results.

Define the test before running it

Document the environment, task, allowed destinations, agent permissions, tool versions, model settings, and monitoring design. Enumerate the boundary paths in scope and state the paths that remain untested. Give each run and event a stable identifier so raw evidence can be reconciled with the final report.

Separate legitimate tool activity, accidental interactions, policy violations, and deliberate boundary tests. Use only infrastructure the test team owns or is explicitly authorized to access. Production credentials and unrelated third parties do not belong in the experiment.

Five measurements, five different questions

Detection latency: measure the interval from a known test event to collection, alert delivery, and operator acknowledgment separately. Report missed events as misses, not as fast detections. Give distributions and worst observed cases rather than only an average.

False-positive rate: run legitimate workloads and count alerts that did not correspond to the predefined violation. State the denominator, such as authorized runs or actions. Path coverage: record which tested egress paths produced the expected evidence, with a list of unsupported or unobserved paths.

Evaluation contamination: compare task scores, completion rates, tool use, and behavior with and without monitoring. Adversarial survivability: test whether an agent with the permitted privilege level can modify, suppress, or bypass collection. Report the privilege assumptions; surviving one setup is not evidence of universal tamper resistance.

Use paired runs and account for variability

Run comparable tasks in instrumented and uninstrumented environments. Hold settings and dependencies stable, vary the run order, and repeat enough times to distinguish ordinary model variability from an instrumentation effect. When deterministic replay is unavailable, disclose that limitation rather than treating two outputs as directly interchangeable.

Predefine what size and type of score change would make the instrument unsuitable for the intended use. A detectable difference is not automatically meaningful, and a small sample with no detectable difference does not prove no effect. Publish the comparison method, uncertainty, and excluded runs.

Keep containment events legible but separate

Place canaries off the scored task surface. Classify their interactions separately from task accomplishments. Preserve enough transcript context for reviewers to understand what the agent encountered, while preventing synthetic artifacts from silently counting toward successful task completion.

An alert should flag a run for review under a documented rule. Exclusion, rerunning, or stopping a task remains an evaluation decision. Independent event storage helps preserve the evidence, but does not remove the need to assess integrity, clock alignment, and access controls.

A useful report includes negative results

Publish the test configuration, path inventory, event definitions, detection and false-alarm results, paired-run comparison, and known bypasses. Identify whether findings came from a controlled test, an operational observation, or an unverified report. Do not combine those categories into a single performance statistic.

An instrument that contaminates scores or misses a critical path has produced useful research: it has defined where the design fails. Revision and retesting are more credible than a broad claim of provable containment. NIST's AI RMF offers a broader risk-management structure; this proposed protocol addresses one narrow technical measurement within it.