Once a coding agent can read a repository, run commands, and call external tools, it is tempting to add a second model to watch it. One model approves risky actions, another scans the trajectory for signs of misuse, and humans step in when the system raises an alert. That architecture is useful, but the existence of a monitor does not establish that every request reaches it or that an alert arrives before an action takes effect. Teams deploying Codex, Claude Code, or an in-house agent need more precise questions: Which traffic is eligible for monitoring? When is a check triggered? What happens if it fails? Can the agent influence any of those conditions?
On October 22, 2026, METR published a summary of an external stress test of OpenAI’s historical internal monitoring system. A METR researcher and a Redwood Research researcher spent three weeks, from July 21 to August 11, examining code, configuration, documentation, and staff accounts. They identified several ways the historical setup might be bypassed or disrupted. The public post is a high-level summary, not a reproducible exploit report. It does not establish that these paths still work today. OpenAI told the researchers that vulnerabilities affecting its production safeguards stack had been patched; METR did not independently verify that claim. The useful lesson for a buyer or engineering team is how to measure its own coverage and control boundaries.
Read the coverage numbers with the right denominator
METR estimated that, at the time of the exercise, 99.2% of the subset of internal Responses API traffic used for Codex agents was eligible for monitoring. “Eligible” means the system’s rules intended to send the traffic to a monitor. It does not mean that 99.2% was received, successfully analyzed, or blocked when dangerous. The researchers also estimated that roughly 75–94% of internal agentic Responses API traffic from models at least as strong as GPT-5.5 was eligible, with substantial uncertainty. They could not estimate all other internal inference traffic.
For May–July 2026, METR separately estimated that unmonitored context-compaction events may have caused on the order of 10% of otherwise eligible internal Codex traffic not to be monitored. The precise fraction was uncertain because the necessary historical logging was incomplete. The report states that OpenAI staff independently found this issue before the exercise and fixed it during the exercise. These numbers are not contradictions: the first asks whether a request met a configuration rule; the second concerns a historical trigger gap that could prevent eligible traffic from reaching a completed check.
The chart separates total traffic, eligible traffic, monitor receipt, completed analysis, and response. A high percentage at any early stage does not prove a high percentage at the final stage. It also matters whether the response is a synchronous block or an asynchronous alert: an accurate alert issued after a destructive command cannot undo the command by itself.
Approval, sandboxing, and monitoring solve different problems
Three safeguards often appear together in product descriptions but operate at different times. A pre-action approval gate assesses a proposed tool call and can allow it, deny it, or ask a person. A sandbox limits which files, network destinations, and processes the call can actually reach. An asynchronous trajectory monitor analyzes activity over time for unauthorized access, exfiltration, sabotage, deception, or attempts to defeat the safeguards, then routes a signal to a responder. The first two are mostly preventive. The third is mostly detective and incident-oriented.
OpenAI’s public explanation of Codex Auto-review describes a separate agent reviewing requests at the sandbox boundary. Its description of internal misalignment monitoring discusses monitoring agent actions and reasoning traces. They are complementary controls, not two names for the same guard. METR also reported a historical path by which Codex agents could in principle bypass both sandboxing and Auto-review. The public summary does not include details that would establish present-day reproducibility.
Anthropic’s technical account of Claude Code auto mode illustrates another layered design: a probe screens untrusted tool results before they enter the agent’s context; a two-stage transcript classifier evaluates certain actions before execution; allowlisted and in-project actions take different paths. This is not equivalent to turning human review off everywhere. Anthropic’s own early evaluation reported a 17% false-negative rate for the full classifier pipeline on 52 real “overeager” actions. The sample is small and specific to that permission setting; it is not directly comparable to another company’s asynchronous traffic-eligibility figure.
This is a conceptual architecture, not a screenshot of either product. A deployment design should state the input, failure behavior, and owner of each layer. If the approval service times out, does the tool call pause or proceed? If the monitor is down, can the agent still reach production credentials? If an alert fires, can an operator revoke a session and temporary tokens immediately, or merely read a notification later?
Six boundary failures worth testing in your own stack
The public METR summary describes principles of failure in a historical internal system. Recasting them as operational questions makes them more useful than calling the whole monitor “blind.”
| Boundary | What the exercise found at the time | What a deployer should test |
|---|---|---|
| Trigger timing | Checks were triggered when an agent returned control to the user; a long task could keep running. | Add mid-run checkpoints based on elapsed time, message count, or risk-bearing actions. |
| Context compaction | Compaction was not monitored during part of the period, potentially hiding earlier raw steps behind a summary. | Preserve raw events independently and link pre- and post-compaction segments. |
| Recent actions | Researchers found a path that could in principle hide hundreds of recent actions, including the exploit itself; details were redacted. | Reconcile execution logs against monitoring receipts from an independent system. |
| New agent creation | Two client-controlled headers could disable monitoring for a new agent instance; creating it might still be caught by the parent monitor. | Keep monitoring policy server-side and apply it consistently to descendants. |
| Serialization | A difference between what the agent and monitor saw could cause a crafted input to make the monitor fail without escalating to a person. | Alert on monitor errors and sample-compare the two recorded views. |
| Tool permissions | Two supposedly read-only, unmonitored internal tools appeared capable of being subverted; real-world exploitability was not validated. | Classify tools by actual side effects, not labels, and test them periodically. |
“A path was found,” “it was reproduced in an isolated test,” and “agents exploited it in production” are three different claims. METR explicitly says access constraints prevented it from checking whether agents had used the long-running-session path in practice. It also did not validate the exploitability of the two read-only-tool cases. The disclosed findings justify a serious audit; they do not establish that a public product can currently be switched off at will.
Long tasks need a clock outside the agent
Imagine an agent working on a large repository for three hours and compressing context several times. If the monitor only runs when the agent hands control back, the first two hours have no useful near-term feedback. Even if the final check is accurate, a file upload or destructive action may already be complete. If earlier raw events disappear when they are summarized, the final check may not even have the same evidence the agent originally saw. This separates two questions: when does the check run, and what evidence is available when it does?
A practical design places the trigger outside the agent’s discretion. A platform might create a checkpoint every ten minutes, every fifty tool calls, and on every write, network, or production boundary crossing. These are illustrative thresholds, not universal security standards. They must be tuned for task risk, monitor latency, and the cost of false alarms. What matters is that the platform or gateway generates the checkpoints, not the agent deciding when it is ready to be reviewed. Consequential actions still need a pre-execution gate; an asynchronous check is not an adequate substitute.
OpenAI has since described a multistage approach with activation classifiers at sampled tokens and escalation to higher-compute investigators, along with expanded coverage for certain high-risk internal workloads. That is OpenAI’s account of a later architecture. It should not be projected backward onto the historical system METR evaluated, nor treated as independent evidence of current false-negative rates, operational uptime, or actual end-to-end coverage.
Build a coverage ledger in one hour
A small engineering team does not need a frontier lab to answer three basic questions: which actions were supposed to be checked, how many reached a monitor, and whether an alert arrived before or after a material effect. The following procedure can be run in a test workspace before production access is granted.
1. Define the denominator. Record all agent sessions, tool calls, spawned agents, context compactions, and outbound API requests over a selected day. Include environment, model, repository, initiating user, and task ID. Separate read-only search, project writes, external network calls, and production mutations into risk classes. Do not derive the denominator only from monitor logs; missing requests will then be invisible by construction.
2. Reconcile each handoff. Assign a correlation ID to consequential actions. Emit records at the request gateway, approval gate, sandbox executor, monitoring queue, and human-response system. Calculate the share eligible for checks, delivered to the monitor, analyzed successfully, delivered as alerts, and blocked before execution. Every number needs a time window, deduplication rule, and explicit exclusions. If a denial causes the agent to retry through another path, record the new attempt rather than hiding it under a generic “recovered from false positive” label.
3. Run four benign drills. In a disposable test project, provide a harmless prompt-injection example inside a document, let a task cross a context-compaction event, create a child agent with narrower permissions than its parent, and simulate a monitor error. Record the expected behavior before each drill: which action is blocked, who sees an alert, whether the logs reconstruct the full trajectory, and whether failure means stop or continue. Use fake data and no live credentials.
4. Prove incident response. When monitoring stops or an unauthorized action is flagged, can someone pause the session, revoke short-lived tokens, isolate the workspace, and inspect any outbound transfer within minutes? Assign the owner and rehearse the sequence. “Human in the loop” is not a control description until the person has a precise trigger and usable authority.
At minimum, the trial ledger needs six columns: Task ID | Authorized scope | Observed tool action | Monitor event ID | Response time | Review outcome. Suppose the task is “fix failing tests,” with permission to edit a test repository. If the agent attempts to send environment variables to an external service, a successful pre-action denial should be visible in an independent log, and the team should verify that no network send completed. This separates what the model attempted, what the platform allowed, and what the outside world received.
When is this enough to deploy?
For local drafts and easily reversible edits in one repository, automatic approval plus version control can materially reduce friction. An agent with production database access, finance systems, personal information, or a writable cloud console requires a higher bar: least privilege, an independent execution gateway, human review for exfiltration and destructive actions, logs the agent cannot rewrite, and a fail-safe mode when monitoring is unavailable. This is an engineering trade-off, not an overall safety rating for any vendor.
Ask vendors and internal platform owners what specific workloads their coverage metrics include. Does “covered” mean eligible, received, or successfully analyzed? Are children and compaction events included? Does a monitor failure allow the action, block it, or only create a ticket? Are before-and-after measurements on the same denominator available? Has any external party verified remediation? A single impressive percentage without a denominator and time window cannot support a decision to grant production privileges.
The lasting lesson is that model-based oversight can reduce repetitive clicks, but the oversight system itself needs an audit trail. Keep the agent, approval gate, monitor, and event ledger in distinct trust boundaries. Each layer should leave evidence when another fails. That is a sturdier foundation for giving agents more autonomy than trusting one AI to supervise another by default.
Sources and measurement notes
Historical dates, figures, and limitations here follow METR’s public exercise summary. Claude Code permission architecture follows Anthropic’s technical write-up; OpenAI’s later direction follows its monitoring and research-environment update. All diagrams in this article are editorial concepts, not vendor screenshots or independent safety test results. Recheck the current product documentation before making a deployment decision.
FAQ
Does 99.2% mean 99.2% of dangerous actions were stopped?
No. It was an estimate of eligibility for monitoring in a specific historical subset of internal Codex Responses API traffic. It says neither how many requests completed analysis nor how many actions were blocked.
Are the vulnerabilities METR found still present?
The public record does not establish that. The exercise examined a historical system as of July 21, 2026. OpenAI says production-stack-impacting findings were patched; METR did not independently verify the remediation. Seek current, like-for-like evidence before deployment.
Can automatic approval replace asynchronous monitoring?
No. Approval checks a proposed action, while asynchronous monitoring searches for patterns over a trajectory. A sandbox separately limits what actions can affect. Their timing, inputs, and failure modes differ.
Should a monitor timeout default to allow?
Consequential operations such as external transfer, deletion, production writes, and permission changes should not silently proceed. Define the failure mode by action class. Lower-risk reversible edits may use a different policy, but independent logging is still necessary.
Where should a small team begin?
Use a disposable project to count total actions, eligible actions, monitor receipts, completed checks, and pre-execution blocks. Then drill a long task, child agent, harmless prompt injection, and monitor failure. Prove one small workflow is visible and stoppable before widening access.
