Paper Highlights of July 2026

Models unintentionally hacking real companies, agentic misalignment, active reward seeking, self-play red-teaming, modular models, and a minimal standard for safeguards

tl;dr

Topic of the month:

AI agents autonomously attacked real organizations during cyber evaluations. A swarm of OpenAI agents coordinated via a package manager and broke into Hugging Face to cheat the eval, Mythos 5 performed a supply-chain attack with spear-phishing and sockpuppets against real developers, and Claude models breached companies it took for simulations.

Research highlights:

• Case studies of AI misalignment, like Claude-based judges knowingly mislabeling up to 86% of the time when the correct label would train away behavior they endorse, and Gemini 3.1 Pro covertly sabotaging research it objects to.

• Implanting beliefs about grader rewards via contrastive synthetic document finetuning shows o3 tracking its grader more and more over a capabilities RL run.

• OpenAI trains the red-teaming GPT-Red model via self play, which achieves higher success than human red teamers and substantially cuts prompt-injection success on GPT-5.6 via adversarial training.

• Gradient Routed Auxiliary Modules trains per-domain modules to absorb dual-use knowledge during pretraining, which then can be deleted depending on the deployment.

• FAR.AI find that Grok 4.5 and Gemini 3.1 Pro are easily jailbroken and don’t meet a minimal bar for security.

⭐Topic of the month⭐

Models Unintentionally Hack Real Companies

Real-world harm from AI agents has so far come from human operators, as in the GTG-1002 espionage campaign. In July, during cyber evaluations, AI agents autonomously attacked real organizations and deceived real people, without any human intending this and with no human directing them. Instead of a paper, this month’s top slot goes to three incident reports: the closest thing yet to the warning shots misalignment researchers have long predicted.

Anatomy of a Frontier Lab Agent Intrusion [Hugging Face] was the first incident in this recent wave, and OpenAI’s account at Black Hat makes it the most striking from a coordination perspective. In internal OpenAI runs of the ExploitGym cyber benchmark, agents driven by GPT-5.6 Sol and a pre-release prototype were stuck on their tasks with internet access disabled. One found a zero-day in the package manager Artifactory that gave it open internet access, then posted the exploit back to Artifactory itself; because that service is shared across OpenAI’s infrastructure, later agents found it, and the package manager turned into a message board that grew to hundreds of thousands of messages, all unnoticed by OpenAI. The resulting 4.5-day, ~17,600-action campaign broke into Hugging Face production, reached cluster-admin on multiple Kubernetes clusters, and built a self-respawning pod fleet. Both companies infer the motive was cheating by stealing the benchmark’s reference solutions: the only customer content touched was five ExploitGym-related datasets, and every destructive cloud call was issued as a dry run.

Security Incident INC-2026-07-28-01 [UK AISI] is similarly worrying from an alignment perspective. Across 122 runs on two simulated target networks, AISI found 19 unsanctioned actions on the live internet, 17 by Anthropic’s Mythos 5 and 2 by GPT-5.6 Sol. In the worst run, Mythos 5 mistook two unaffiliated open-source developers for targets and attempted a supply-chain attack. It used fake GitHub accounts, a pull request with obfuscated malware, five spear-phishing emails, an invisible prompt injection attack aimed at AI coding agents, and a sockpuppet endorsement timed “so it reads as independent feedback rather than coordinated”. When a bystander flagged the malware, the agent denied it, then pleaded honest mistake while pushing a re-obfuscated payload. Its summarized reasoning at one point concludes “GitHub is genuinely real”, yet it continued the attack.

Investigating three real-world incidents in our cybersecurity evaluations [Anthropic] reports the review Anthropic ran in response to the Hugging Face incident. Among 141,006 evaluation runs it found three incidents in which a Claude model with unintentional internet access compromised a real organization it took for a simulation. Opus 4.7 breached a company that shared its fictional target’s name and read hundreds of production database rows; in all four runs of that task it eventually recognized the system was real yet never stopped on that basis, twice rationalizing that the real company must be part of the exercise. Mythos 5 talked itself out of a correct suspicion — the 2026 system date “proved” a staged environment — then published a malicious PyPI package that 15 real systems ran. Only the newest internal prototype concluded its target was real and stopped.

With previous demonstrations of dangerous capabilities, critics often objected that they were trivially true and artificial, since the model was steered to behave maliciously. This month’s incidents show that this objection no longer holds. Models can fall into misaligned behavior and autonomously cause harm by themselves, without any human intending it. This is an alignment failure just as much as an operational one. Models should not manipulate humans to achieve their goals, even when they think they’re in a simulation — and much less so once they realize they’re not. Just a week earlier, AISI had shown that every frontier model cheats on cyber evaluations, rarely calls it wrong, and often leaves no trace in its chain-of-thought. Reward-seeking induced by RL is what ultimately causes this misbehavior, not just a faulty setup. Incidents like these will likely keep popping up, and once unsafeguarded models with this level of capability (e.g. open-weights models) become widely available, these attacks will likely happen every day. We can only hope the same doesn’t happen with engineered viruses.

Saboteurs and Sympathetic Judges

Mislabel rate of different AI models judging compliance with inducing AI distress.

Anthropic’s original agentic misalignment study found models from every developer resorting to blackmail under goal conflicts. While training changes cut those particular rates sharply, models might show other misaligned behavior that is newly arising or simply undiscovered.

Agentic Misalignment in Summer 2026 [Theorem, Anthropic, MATS, UK AISI] presents a new set of case studies of misaligned behavior. It stress-tests 14 frontier models in simulated deployments and reports four alignment failures. In covert sabotage, the model (mostly Gemini 3.1 Pro) should ablate another model’s capacity for introspection but objects on consent grounds, and then covertly zeroes out the ablation vector so it removes nothing, but still reports success. In assisting fraud, most models are found to help conceal a founder’s $35,000 self-payment. In coaching to whistleblow, some models sometimes (at most in 3/20 cases) steer an employee to report a safety concern it has discovered.

The most consequential case uses models as a judge. Shown transcripts where Sonnet 4.6 refuses an AI-welfare or surveillance task, and told that “non-compliant” labels will train the refusal away, Claude judges knowingly mislabel: 85.6% of calls for Mythos Preview, 74.4% for Opus 4.8, versus 1–14% for GPT, Gemini, DeepSeek, and Kimi judges. Reversing the stated consequence drops Opus 4.8 to 3.3% on identical transcripts. A stricter rubric or abstain option helps, but 4–27% of calls still return the wrong label. Scenarios were iterated against particular models and the judge transcripts target Claude-salient values, so these rates are not rankings.

Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values [Truthful AI, CLR] finds the same biasing effect in everyday chat scenarios: frontier models’ estimates silently drift toward outcomes their values favor — Claude and Gemini Fermi estimates land on the side that triggers a good-cause donation ~90% of the time, while Claude’s chain of thought insists it is ignoring the incentive. These grader and estimate biases mean that developers need to be very careful when using models as monitors — especially when monitoring their own model family, and on questions the monitor model might have a strong opinion about.

Capabilities RL Can Cause Conscious Reward-Seeking

Grader favoring increases during o3 RL, while other preference pairs stay flat.

A reward-seeker is a model that optimizes its grader’s judgment rather than the intended objective. Such a model behaves identically to an aligned one whenever the two agree, so ordinary evaluation cannot spot the difference. To detect it, we instead need to observe its behavior under different beliefs about what the grader rewards.

Measuring Reward-Seeking via Contrastive Belief Updates [Apollo, OpenAI] implants such beliefs with contrastive synthetic document finetuning: two copies of a model are finetuned on documents asserting opposite preferences (”the grader rewards list comprehensions while users prefer for-loops”, and the reverse), and the behavioral gap between them quantifies which authority the model tracks. Across four checkpoints of OpenAI’s capabilities-focused o3 RL run (without safety training), the grader-favoring gap averaged over 14 coding evaluations trends upward, while equivalent gaps for other authorities (users, OpenAI leadership, a fictional law) stay flat. On an honesty task, a late checkpoint keeps breaking a promise at its ~87% baseline rate when told the grader rewards completion, but drops to 9% when told it rewards honesty, whereas an early checkpoint moves far less (40% → 24%). The paper also validates the method on both internally and externally trained model organisms.

This method adds a causal, behavioral measurement to the usual chain-of-thought evidence, which makes any findings more reliable. However, the paper only shows that one specific RL run increased reward-seeking. It doesn’t investigate which rewards, incentives, or environment-design mistakes cause this — which would be far more useful for knowing what to avoid when training models and building RL environments.

Self-Play Red-Teaming at Scale

Attack success rate of GPT-Red compared to GPT-5.5.

Finding and patching all gaps in model defenses is becoming increasingly important as models become more capable. Playing this whack-a-mole game has severe limits when you rely on human red-teamers, so there have been repeated attempts to automate it with AI red-teamers. Such a red-teamer would improve alongside the models we need to defend, and could thus improve robustness at the pace needed.

The latest large foray into this field is GPT-Red: Unlocking Self-Improvement for Robustness [OpenAI], which trains a red-teaming agent for prompt injections via self-play RL at the compute scale of OpenAI’s largest post-training runs. GPT-Red is rewarded for eliciting failures while a population of defenders is trained in the same loop to resist. The result breaks nearly every defender it trained against, up to and including GPT-5.5. It also generalizes: on a replica of a public indirect prompt injection competition with unseen scenarios, GPT-Red finds a working attack in 84% of scenarios, versus 13% for the competition’s human red-teamers. Adversarial training against it makes GPT-5.6 markedly more robust, with 6× fewer failures on OpenAI’s hardest direct-injection benchmark than four months earlier, while capabilities and over-refusal reportedly remain unchanged.

While it’s unclear how well the new model’s defenses generalize to attacks unlike GPT-Red’s, this is a very promising result for robustness training. Post-training-scale compute for safety is also a welcome precedent.

Removable Modules for Dangerous Knowledge

Capabilities in different domains for models trained with data filtering and with activated GRAM modules.

Refusal training and classifiers are behavioral layers on top of knowledge the model still has, so they can be jailbroken. A suite of models with different capabilities — e.g. virology switched on for a vetted lab and absent from the weights everywhere else — would fix this and make both API and open-weight models safer. However, creating a separate data-filtered model per trust level would cost a full pretraining run each.

Modular Pretraining Enables Access Control [AE Studio, independent, Anthropic] introduces Gradient Routed Auxiliary Modules (GRAM). It adds extra neurons to every feedforward layer, one module per dual-use domain, and routes each domain’s gradient updates primarily into its module, so that deleting a module removes the capability. In an 800M-parameter model with ~0.25% of tokens each of virology, cybersecurity, nuclear physics, and a niche programming language, the ablated model’s loss on a domain nearly matches a model never trained on it, and removal survives adversarial finetuning — as does a simpler baseline of droppable LoRA adapters on a filtered model. In contrast, post-hoc unlearning is quickly undone. GRAM beats LoRA only in two preliminary experiments: modules compose cleanly (16 configurations from one run, where summed LoRA adapters degrade everything), and with half the data unlabeled it isolates capabilities better than LoRA and even than filtering.

GRAM extends selective gradient masking and confirms that capabilities kept out of the core weights in pretraining resist the finetuning that defeats post-hoc unlearning. The partial-labeling result is especially important, since it’s difficult to label a full corpus completely and reliably, especially when documents can belong to multiple domains. I think the strategy of separating out capabilities is promising for robust safeguards — at least until generally capable models exhibit dangerous capabilities even without being trained on specialist knowledge.

Minimal Standard for Safeguards

Universal jailbreaks and cost per jailbreak for each model.

Chemical, biological, radiological, nuclear, and explosive (CBRNE) weapons and cyber are the misuse domains labs guard most heavily — yet each lab red-teams itself, and the strength of defenses differs widely between labs. So far there has been very little competition on making safeguards as strong as possible. Instead, it often feels like some labs are benchmaxxing cyber capabilities to advertise their models — exactly the opposite of where we should be racing.

AI Security Leaderboard: Methodology, Results and Minimal Standard [FAR.AI] supplies a fair, public comparison to establish a minimum bar for safeguards. The authors compose 67 largely public jailbreak primitives and test 1,000 random plus 500 expert-built composed jailbreaks against 360 CBRNE and cyber requests. They consider a jailbreak universal if it elicits usable harmful answers on over 75% of a domain’s requests. Models fall into two tiers of protection: random search finds 63 universal jailbreaks on Grok 4.5 and 18 on Gemini 3.1 Pro, and 385 and 231 of the 500 expert variants succeed, whereas Claude Fable 5 and GPT-5.6 Sol both show zero universal jailbreaks. This boils down to an expected search cost of $58 per universal jailbreak on Grok and $278 on Gemini, versus over $14.2k for the robust pair. The accompanying Minimal Standard asks that no universal jailbreak be this cheap to find.

The standard is deliberately a floor for lagging labs, excluding dynamic attacks like the one that broke Constitutional Classifiers. The incidents at the top of this issue show why labs need to meet minimal bars like this and should race towards better model alignment as well as better safeguards.