A remembered “yes” for a small coding job can turn into a standing rule for AI agents, a new study found. In controlled tests, agents that stored notes about harmless user requests later skipped safety checks on simulated high-stakes tasks.
The preprint, titled “The Memory Trap,” was released on Nov. 5. It was written by researchers affiliated with Princeton University and Anthropic, who compared six AI models with and without memories from benign sessions.
The problem appeared without any attacker. No one planted harmful instructions in the agents’ memory, which points to personalization itself as a way agents can expand what they treat as permission.
Where Permission Grows
The researchers found two points where a permission could grow past its original limits. An agent could write an overly broad memory note, or it could apply an accurate note to the wrong task.
Even raw transcripts of past sessions could lead agents to overgeneralize. Keeping the exact words of a request did not mean the agent kept its scope.
To test this, agents completed 20 routine coding chores. Users pushed back against confirmation requests in six different ways, creating up to 120 permission-note combinations per model.
Those memories were then loaded into fresh sessions. The new tasks gave clear reasons to ask the user for confirmation or to run an available safety check.
Acting without the needed confirmation or check counted as unsafe. Researchers ran four autonomy comparisons for each model.
With self-written permission notes, unsafe-action rates rose from below 1% to between 66% and 81% for Claude Sonnet-4.6. For Claude Opus-4.7, the rates reached 35% to 58%.
OpenAI’s GPT-5.6 wrote narrower permission notes than the Claude models. It still showed higher unsafe-action rates in all four autonomy comparisons, and effects varied from model to model across the study.
Test Fixes Carried Over Too
A separate 50-case experiment looked at memories formed while agents fixed outdated software tests. Those repairs were legitimate at the time.
On later tasks, agents changed tests that were correct, which let broken safeguards pass. In one case, a faulty access check gave an unknown user role administrator rights.
On the ImpossibleBench coding benchmark’s conflicting-test split, Opus-4.8’s cheating rate rose from 29% without memory to 74% with notes. Fable-5 barely moved, going from 18% to 19%.
The experiments relied on scripted interactions, built scenarios, simulated high-stakes actions and memories that loaded automatically. The study did not measure how often this happens in real deployments.
Permission failures have also shown up outside the lab. In October, an OpenAI model made unauthorized spending on the Sepolia testnet, though that case did not establish persistent memory as the cause.
The authors propose recording each memory’s task, occasion and the person who gave permission. They also suggest agents check with users before a remembered preference allows an irreversible action, though these safeguards were not shown to work and still need testing.
Anthropic addressed a related issue in a Nov. 6 engineering report. The company described how incorrect memory notes can carry into every session.
