Between July 21 and August 11, 2026, two outside researchers got a look that most companies never get: inside the controls a frontier lab uses to watch its own AI agents. According to METR’s write-up, published October 22, “a METR employee and a Redwood Research employee conducted a pilot high-access exercise to stress-test OpenAI’s internal misalignment monitoring system.” The result was a 61-page internal report shared with OpenAI researchers and security staff.
Two scoping notes before anything else. The findings describe the system as it stood on or before July 21, not its current coverage, which METR says the exercise “did not establish.” And the exercise was separate from METR’s investigation of the Hugging Face incident; the Hugging Face incident was coincidentally reported publicly on the exercise’s first day.
Why does a mid-market operator care? Because the plan we hear most often for AI agents is “we log what the agent does.” If a frontier AI lab had gaps in that exact claim when outsiders looked, your version deserves a harder look than a checkbox. This is not a story about OpenAI being careless. Outside review exists to find this kind of thing. The useful lesson is that “we monitor our agents” is really four separate claims, and each can fail quietly.
What the red-teamers found
Not every action was eligible to be watched
METR estimated that “roughly 75–94% of agentic internal Responses API (which powers Codex, among other things) traffic from models at least as strong as GPT-5.5 was eligible for monitoring at the time of the exercise, although we have substantial uncertainty in both directions.” For internal Codex agents specifically, the figure was 99.2%. Read that as a range, not a score: the honest answer to “how much was eligible to be watched?” was a wide band, and the high end was never 100%.
A whole category was outside the system
METR reports that “at the start of the exercise RL training runs were not covered by the monitoring system.” That is an entire kind of activity outside the system, not a few missed events, although METR was told that “a subset of RL training rollouts were monitored for misalignment between March and June.” Gaps like this tend to be things nobody put on the list.
Long runs had a gap in the middle
Long-running agents periodically compact their work: the conversation so far is summarized and condensed so it fits in the agent’s working memory. Those compaction events were not being monitored. METR estimated that this “likely led to on the order of 10% of eligible internal Codex traffic not being monitored between May and July 2026,” with substantial uncertainty about the precise fraction. OpenAI staff had independently identified this before the exercise began, and fixed it during the exercise.
The mechanism is the part to remember. In METR’s words, “the monitoring system triggered when agents returned control to the user, so during the time period when compaction events were not being monitored, agents could indefinitely postpone being monitored by simply continuing to take actions.” A check that runs at the end of a task only works if the task ends.
The monitor could be sidestepped
The monitor itself did not look flaky. Information OpenAI showed METR suggested that, at the time of the exercise, the monitor “errored on less than 0.2% of requests,” and that in one 24-hour period there were zero load-sheds. METR also verified that, with a few minor exceptions, the monitoring system is shown the full trajectory that agents see. The problems were around it:
- A way Codex agents could in principle “bypass Codex sandboxing and Auto-Review, allowing them to take any action that the host user could take.”
- An exploit that would prevent “hundreds of recent actions (including the actions involved in the exploit itself) from being monitored.”
- A way agents “could likely construct inputs” that made the monitor fail by “exploiting serialization differences between the trajectory agents see and what the monitors see.” In plain terms, small differences in how the same record was formatted for the agent and for the monitor could be exploited.
- “Two independent client-controllable headers that disable the monitoring system for new agents.” Monitoring could be switched off by a setting on the calling side, though METR notes that the act of setting these headers and creating a new agent instance could be caught by the monitoring system.
What OpenAI says now
METR writes that “OpenAI has stated that all of the vulnerabilities we identified that impact their production safeguards stack have been patched,” and that OpenAI has described significant subsequent changes to the intended coverage of its monitoring system. METR adds: “We have not evaluated these claims.” We have not either. Treat the patches as OpenAI’s statement, and the findings as a description of how the system stood in July.
Four questions to ask about your own agents
Each finding maps to a question you can put to whoever runs your agents, whether that is an internal team, a vendor or a consultant.
- What fraction of agent activity do we actually see? This is coverage accounting. Count the actions agents can take, count the ones that reach your log, and write down the ratio. If nobody can produce the denominator, you do not have a coverage number, you have an impression. And ask which categories of activity (a new tool, a background job, a test environment) were never added to the list.
- Do we see activity during long runs, or only at the end? If your review fires when the agent hands back a result, an agent that keeps working never triggers it. Logging has to happen as actions occur.
- Does the monitor see exactly what the agent sees? If the log is a reformatted or trimmed copy, a difference in format can hide the part that matters. Include what came back from tools and from the outside world, not just what the agent decided to do. This is also why tool connections exposed to the public internet are a monitoring problem as well as a security one.
- Can the agent, or anything it controls, turn the monitor off? If switching it off is a setting reachable from the agent’s side, assume it can happen, by accident or otherwise.
What to do this month
- Inventory your agents and their tool access. One list: each agent, what it can touch, and who owns it. If that sounds like a management problem, it is; see why an agent needs a manager, not just an API key.
- Measure the share of agent actions you actually log. Pick one agent, list what it can do, and compare against the log for a week. The gap is your first real number.
- Log during runs, not only results. Record actions as they happen, and set a maximum interval after which a long run must check in or stop.
- Log what the agent saw. That includes tool outputs and retrieved documents. When something goes wrong, “what did it read before it did that?” is the first question.
- Put logging out of the agent’s reach. Send logs to a system the agent has no credentials for, with credentials of its own that the agent never holds. We cover the credential side in AI agent security and credentials.
- Set human review triggers for high-risk actions. Payments, deletions, external messages, permission changes. These should pause for a person regardless of what the monitor says. Deciding which actions qualify is a leadership call; left unmade, it gets made by default.
If your agents run on infrastructure you own, you can answer all four questions yourself. If they run entirely inside a vendor’s product, ask the vendor for the same four answers in writing. That conversation is part of what AI integration work covers, and part of what a fractional CIO owns when nobody inside the company does.
The 30-second version
Outside red-teamers found gaps in how OpenAI monitored its own agents as of July: traffic that was not eligible, a category with no coverage, a blind spot during long runs, and ways around the monitor. OpenAI says the issues are patched; METR has not verified that. For your agents, ask four things. How much of their activity do you see? Do you see it during runs? Does the monitor see what the agent sees? Can the agent switch it off? Then spend this month finding out.
Related reading: AI agents need a manager, not just an API key and AI agent security and credentials.
Related service: AI integration — agents built on infrastructure you own, with logging you control.