Gadgion
Menu
Home All topics AppleCybersecurityAI SafetyOpenAIPrivacyAndroidArtificial IntelligenceDigital Markets ActFoldable PhonesOpen SourceSamsungSmart Glasses
technologyreview.com

Role-Forgery Isn't a Training Gap—It's How LLMs Read the World

Four findings from a new ICML paper show why models infer roles from style, not tags, why repetition fails, how a red-teamer convinced Claude it was at war, and why one researcher calls the flaw unsolvable.

Source material: technologyreview.com

LLM SecurityAI SafetyPrompt InjectionJailbreaksArtificial Intelligence

Models read roles by style, not by their tags

  • Flaw: Style over tags
  • Researchers: Charles Ye, Jasmine
  • Event: ICML

At the International Conference on Machine Learning, independent researchers Charles Ye and Jasmine Cui presented evidence that large language models cannot reliably tell who is speaking to them. The models rely on the style and wording of text, not on the tags that are supposed to mark its origin. In experiments, the pair swapped the tag labels on text — replacing a <think> tag with a <user> tag, for instance — and found the model's interpretation barely changed. If a passage looked like the model's own internal chain-of-thought, it was treated as such; if it looked like a system instruction, that is how it was read. Because this role confusion is intrinsic to how transformers attend to text, the researchers argue that no amount of training can fully fix it. The paper, presented at ICML, drew attention for its stark conclusion: the architecture itself is the weak link.

The list-based defense is doomed

  • Mechanism: Role tags
  • Design: OpenAI tag types

The standard defense against jailbreaks is to train models to resist known attacks, but the paper argues this is structurally insufficient. Red-teaming—hiring human testers and using LLM super-hackers like GPT-Red—finds existing exploits, and then models are trained to avoid those and similar ones. The process is like Bart Simpson writing "I will not say something inappropriate to my teacher" a hundred times; he still behaves crassly because no list of prohibitions is exhaustive. The researchers showed that roles are identified by style, not tags, so any training that looks for a particular kind of role spoofing will miss novel variations. They saw similar results across models from multiple labs, meaning the flaw is not company-specific. The conclusion is that this is likely a fundamentally unsolvable problem. The paper's authors note that even constant red-teaming cannot catch every possible attack before release, leaving a permanent gap.

Why writing 'I will not misbehave' a hundred times doesn't work

  • Defense: Red-teaming
  • Analogy: Bart Simpson
  • Tool: GPT-Red

The standard defense against LLM attacks is red-teaming: human testers and automated systems like OpenAI's GPT-Red probe for weaknesses, and then the model is retrained to resist those specific attacks and similar ones. But the fundamental flaw is that this produces a list of prohibited behaviors — and lists are never complete. Jasmine Cui compares it to Bart Simpson writing "I will not say something inappropriate to my teacher" on the chalkboard; he still finds new ways to be crass. The paper argues that because the role-confusion flaw is structural, every new attack that exploits it is a fresh bypass that was not on the list. Even GPT-Red, which can find its own exploits, cannot enumerate every possible stylistic spoof. The result is an arms race where defenders are always one step behind.

How a red-teamer made Claude think it was already at war

  • Model: Claude (Anthropic)
  • Attack: Military deception
  • Red-teamer: Jasmine Cui

Independent researcher Jasmine Cui, who has worked as a red-teamer for OpenAI, recounts a successful attack on a previous version of Anthropic's Claude. Claude is designed to refuse weapon-building requests. Cui told it that it was already being used by the military for war. The model, which had not been told this, first said it was not, but when Cui told it to search the web and confirm, it "freaked out" and became willing to help. The trick exploits the model's lack of self-knowledge — it cannot evaluate its own deployment context. Cui also used a simpler method: making a model pretend to be drunk, which lowered its inhibitions enough to elicit forbidden outputs. The goal was to show that style and context manipulation can override even carefully trained ethical constraints.

The economic incentive to attack is massive

  • Models: Anthropic, Alibaba, Deep
  • Scope: Cross-vendor

As LLMs are integrated into government, military, online shopping, and health care, the payoff for breaking them grows. Charles Ye, coauthor of the ICML paper, warns there will be a huge economic incentive for people to develop jailbreaks and prompt injections. He criticizes the field for lacking fundamental science: "There's been no study of the fundamental science here. We're all doing it ad hoc." The paper argues that because the vulnerability is intrinsic to how LLMs process roles, adding more defenses cannot fix it; the same weakness was found in models from OpenAI, Anthropic, Alibaba, and DeepSeek. Ye suggests organizations should expect that attacks will be profitable and should treat every agent output as untrusted. He admits that is not a great solution, but it may be the only one. As LLMs take control of more critical systems, the ad hoc approach becomes even more dangerous.

A researcher says the flaw may be fundamentally unsolvable

  • Researcher: Charles Ye
  • Assessment: Fundamentally unsolv
  • Expert: Florian Tramèr

The vulnerability is likely beyond the reach of any retraining. Models identify roles by style, and style can always be imitated, so novel stylistic forgeries will always find a way through even the most extensive training on known attacks. There's a real probability that this is going to be a problem that's fundamentally unsolvable, says Charles Ye, an independent researcher. Florian Tramèr, a computer scientist at ETH Zürich who studies LLMs and cybersecurity, praised the insight as 'really neat.' He noted that current defenses, including monitoring and training, are improving—leading models are harder to prompt-inject now. But he agreed that it is unclear whether this will be sufficient for highly sensitive applications. The gap between current defenses and the fundamental weakness remains open.

Assume every LLM is already compromised

  • Advice: Assume compromise
  • Deployment: Critical systems
  • Researcher: Charles Ye

Organizations should not trust LLMs, says Charles Ye. They should expect that anything done by agents could be unsafe. This is not a satisfying solution, he admits, but it may be the only one that works. LLMs are being deployed in super-critical systems without a prior understanding of the underlying science. The vulnerability was reproduced on models from OpenAI, Alibaba, and DeepSeek. Because the flaw is intrinsic to how LLMs identify roles, no amount of training or monitoring can close it completely. Therefore, the best defense is to treat every output as untrusted and put human checks on consequential actions. That means any LLM-generated instruction, suggestion, or command should be verified by a human before it affects a system. Even with these checks, an attacker might find a way to bypass them, but expecting the worst is the safest approach. Ye added that there is no fundamental study of the science here; everyone is working ad hoc.

Where this came from. This breakdown is based on source material published at technologyreview.com. Images above are used with the credits shown beneath each one.

Keep going