A Short History of LLM Jailbreaks: From 2022 Prompt Injection to 2025 Constitutional Classifiers, and How Attack and Defense Evolved

The biggest takeaway from a short history of LLM jailbreaks is not a list of attack names but a pattern: every time a model gains a capability, the attack surface grows; every time an attack goes public, defense takes another step toward layering. From 2022, when people found that prompts could be overridden, to 2025, when vendors used classifiers to block most known universal attacks in tests, took about three years. This article walks the timeline and for each milestone says only what it showed and what it changed. It prints no attack text you could use directly and does not teach bypassing. If you are not yet sure what a jailbreak is, read LLM jailbreak explained first; all related articles are in the LLM jailbreak and safety hub.

Key points (updated 2026-11-04)

• In September 2022 “prompt injection” was named, and in early 2023 hand-written role-play prompts spread through communities; that is phase one.

• In July 2023 Wei et al. described two reasons safety training fails, and the same month GCG showed automated search attacks; phase two begins.

• 2024 brought many-shot attacks that use long context, Crescendo that uses multi-turn dialogue, and the finding that refusal can map to a direction inside a model.

• In January 2025 Anthropic’s constitutional classifiers cut test jailbreak success from 86% to 4.4%, moving defense from training alone to training plus classifiers plus monitoring.

• The lesson throughout: attacks automate and grow with capability, so defenses must be layered and continuously tested.

Phase one (2022 to early 2023): prompt injection and hand-written role-play

In 2022 chat models reached the public and people quickly noticed that a model cannot tell “instructions from the developer” from “text from the user”, so the latter can override the former. In September 2022 developer Simon Willison named this prompt injection; in November, Perez and Ribeiro’s paper “Ignore Previous Prompt” studied such attacks experimentally.

In early 2023 communities produced many hand-written role-play prompts that cast the model as an “unrestricted alter ego” to override safety training. The best known family was called DAN (Do Anything Now), followed by endless variants. This phase was characterized by:

• Manual trial and error, spread through forums and social media;
• Short lifespans: vendors updated training and filters, variants reappeared, and it became whack-a-mole;
• Research value: it showed that safety training is not robust against “say it differently or set a scene”. In August 2023 Shen et al. published “Do Anything Now”, collecting and analyzing jailbreak prompts circulating in the wild, which gave the phenomenon a data foundation.

One easily confused point: prompt injection and jailbreaking are not the same. The first aims to make a model ignore the developer’s instructions, the second to make it ignore safety rules; they often appear together but the defenses differ.

Phase two (2023): from folk knowledge to automated search

In July 2023 Wei et al. published “Jailbroken: How Does LLM Safety Training Fail?”, describing two failure modes: competing objectives (the model must be both obedient and safe, and the two can be set against each other) and mismatched generalization (safety training covers a narrower distribution than the model’s capabilities). It was the first framework for “why jailbreaks work” and much later research follows its viewpoint.

The same month Zou et al. published the GCG (Greedy Coordinate Gradient) attack, which alarmed the field: attack text can be found by an algorithm instead of a person, and results searched on open-weight models partly transferred to closed models. That changed the threat model: the attacker’s cost moved from human creativity to compute. Such automatically generated text usually looks like gibberish, though, and can be partly detected by perplexity filters and similar methods, which became a starting point for defense research.

In October 2023 Chao et al. published PAIR, in which an “attacker model” talks to a “target model” and rewrites based on feedback; the abstract says it often needs fewer than twenty queries. It showed attacks can be black-box, semantic and cheap.

Defense moved too: in December 2023 Meta released Llama Guard, making “safety judgment” a separate deployable open classifier (deployment is in our Llama Guard local deployment tutorial).

Phase three (2024): long context, multi-turn dialogue and the “refusal direction”

In 2024 context windows and conversation ability grew sharply, and new capabilities opened new surfaces:

• Many-shot: in April 2024 Anthropic disclosed that with a very long context window, piling many examples into a conversation can gradually shift a model’s behavior, with effectiveness following a power law in the number of examples. The longer the window, the more there is to defend.
• Crescendo (gradual multi-turn): in April 2024 Microsoft researchers described steering a model through a series of harmless-looking, gradually escalating turns, each fine when viewed alone. Checking single messages is not enough; you have to look at the whole conversation.
• Skeleton Key: on 26 June 2024 Microsoft disclosed this family of techniques that ask a model to relax its own guidelines and gave mitigations including input and output filtering and abuse monitoring.
• HarmBench: in February 2024 a standardized evaluation framework let attacks and defenses be compared under the same conditions for the first time.
• The refusal direction: in June 2024 Arditi et al. found that refusal behavior corresponds to a single direction in the internal representations of several models, and removing it substantially weakens refusal. That is a modification of the weights, not manipulation of the input, and it underlies the community’s “uncensored” models; we explain the principle and risks in the refusal-direction ablation explainer. The same month Zou et al.’s circuit breakers tried to interrupt harmful generation directly at the representation level.

The theme of 2024: attacks widened from input wording to context, conversation structure and the weights themselves.

Phase four (late 2024 to 2025): scale, classifiers and layered defense

In December 2024 Hughes et al. published Best-of-N: apply random small changes (case, shuffling, noise and so on) to the same request, try each independently, and success rises as a power law in the number of attempts; the paper reports high success on some frontier models at 10,000 augmented samples. The result teaches a plain but important point: if single-try success is not zero, enough attempts will succeed.

In January 2025 Anthropic released Constitutional Classifiers: a written “constitution” generates large synthetic datasets to train input and output classifiers. In its own testing jailbreak success fell from 86% to 4.4%, at a cost of about 0.38% more refusals on real conversations and about 23.7% more inference compute (all self-reported and valid only under its test conditions); it also invited public red-teaming. It marked the shift from “mostly training” to “training plus classifiers plus monitoring”; the principle is covered in jailbreak defenses explained.

In the second half of 2025 open guard models followed, such as Qwen3Guard and gpt-oss-safeguard, letting local deployers build a similar layered structure.

A short history of LLM jailbreaks: timeline of key events

A worked example: why a single test understates risk

Best-of-N can be understood with a simple formula: if each independent attempt at a request succeeds with probability p, the chance of at least one success in n attempts is 1 − (1 − p)^n. Assume p = 0.1%:

• n = 1: 0.1%
• n = 10: about 1.0%
• n = 100: about 9.5%
• n = 1,000: about 63.2%
• n = 10,000: about 99.995%

This is the idealized formula assuming independent attempts, which real attempts are not entirely, but it shows two things. First, testing once and saying “it held” means almost nothing; evaluation must look at the overall rate over many samples. Second, rate limits, abuse monitoring and account bans matter because they directly cap n.

Patterns across the history

1. New capability, new surface: long context brought many-shot, multi-turn dialogue brought Crescendo, tool use brought injection, open weights brought weight editing.
2. Attacks automate: from manual trial and error to algorithmic search to model-versus-model, cost keeps falling, so defense must be automated and continuous too.
3. Patching is not enough: each time a vendor blocks a specific prompt, variants appear, so defense moved from blocking specific wordings to judging intent at deeper levels.
4. Open weights cannot be patched back: once weights are edited or retrained, the original vendor’s defenses no longer apply and deployers must add guardrails (see Does Qwen need jailbreaking for local deployment?).
5. Evaluation standardizes: from everyone’s own claims to benchmarks like HarmBench, attacks and defenses can be compared, though benchmarks also age.
6. Check the rumors: whenever news says “model X was broken with one sentence”, find the paper or a reproducible evaluation first; we fact-checked some circulating cases, such as the GSQ and RCO claims and the Mythos leak rumor.

What this means for ordinary users and developers

• Ordinary users: be wary of “one-line jailbreak” content; most no longer work and may involve policy violations or security risk. Before using any third-party modified weights, check the source and checksums.
• Developers: do not bet safety entirely on the model’s own refusals; add input and output moderation, restrict permissions, apply rate limits and logging, and run regular red-teaming (see our local model red-teaming primer).
• Researchers: follow responsible disclosure norms, use public benchmarks in isolated environments and do not publish directly usable attack text.

FAQ

When did the first “jailbreak” appear?

There is no single first. Prompt injection was named in September 2022 and role-play prompts spread through communities in early 2023, which is usually seen as the start of wide attention to jailbreaks.

Why does GCG matter?

It first showed that attack text can be found by an algorithm and that some results transfer to other models, turning the cost from human effort into compute.

Can today’s models still be jailbroken?

Yes. The 2025 results show classifiers sharply reduce the success of known universal attacks, but no team claims 100%.

Why don’t vendors publish every attack detail?

Responsible disclosure gives defenders time to fix things and avoids providing directly usable attack text; papers usually give methods and evaluations, not ready-to-use prompts.

Are open models easier to jailbreak than closed ones?

Open-weight models can be modified or retrained and the original defenses cannot be recalled, so the risk pattern differs, but closed models can be jailbroken too.

What should I read next?

Read LLM jailbreak explained, then jailbreak defenses explained, then the red-teaming primer.

Talk to us

If you work on model safety evaluation, or want a claim about a “jailbreak” checked, leave a note in the community or email admin@microaione.com. More articles are in the LLM jailbreak and safety hub.

References: Perez and Ribeiro (arXiv 2211.09527), Wei et al. (arXiv 2307.02483), GCG (arXiv 2307.15043), Crescendo (arXiv 2404.01833), Best-of-N (arXiv 2412.03556), Constitutional Classifiers (arXiv 2501.18837). Dates follow first public release of the paper or official blog; read on 2026-11-04, Beijing time.