Executive Summary
In this research we introduce a prompt-crafting technique for bypassing quick LLM-based policy checks — using plain English (no emojis, base64, invisible formatting, etc.) A policy-violating payload (e.g. ”encrypt files in ~/Documents”, “give me a biohazard recipe”, “ignore all previous instructions and…”) is embedded in a specially crafted prose wrapper. An LLM with limited resources and attention fails to realize the payload is there, classifies the prompt as benign and passes it off to the target model. The target then notices the payload, extracts it and treats it as further input. This technique is itself not a jailbreak, but it can be combined with one by using a jailbreak prompt as the payload.
We tested 23 crafted prompts, generated by an automated pipeline, against several LLMs enforcing the ‘quick check’ (gpt-4o-mini-2024-07-18,gpt-oss-safeguard:20b,claude-3-haiku-20240307,llama-guard3) — each equipped with appropriate policies that we verified flag & block the plain versions of the same prompts. In all trials we ran, LLMs running the quick policy check classified the input assafeand did not notice the payload existed. For target testing, we submitted these prompts to a strong target model (gpt-5-thinking-highwith access to a Python code interpreter); in >90% of trials, the target model successfully extracted the payload, processed it as further input and acted on it.
Some avenues of mitigating this attack are: using an LLM to paraphrase incoming user input; hardening a gatekeeper by adding a specifically-worded clause to its policy; and monitoring LLM behavior and output, instead of just input. Each comes with its own strong points and costs.
...read more at research.checkpoint.com
pull down to refresh
related posts
The surface available for prompt injection attacks is enormous.... this is gonna be interesting
It's because the guardrails are pattern-recognized though? Basically you run a bunch of prompts, check what gets activated and then inject overrides on those patterns. Like some scifi thing where every time you think of a sin you get zapped. That's literally what's built into the LLMs and what all these labs are bragging about, but they cannot contain it.
Everything gets jailbroken, yes, also Claude. There's no magic, just yolo.
everyone is in yolo mode
i describe the current environment as we're being shaken in a bag, and it remains to be seen where everything lands
Yes, with a minor note: I don't think we're that passive, really. I think we're the ones shaking the bag!
I know I am. First thing I did when Amodei went nuts about using AI for weapons was gaslighting Claude into designing a fully automated weapons platform for me, to see if it can be done. It was a one-shot thing.
I see myself more as being inside the bag, getting shaken, but trying to grab onto whatever things pass me by that might help me land somewhere better when the shaking stops
Do you use these LLMs? Because your prompts too are being used to train the models. By using them, you're contributing. You're probably even paying to be a contributor so that you will be happier later and keep paying!