OpenAI: prompt injection is unlikely to be fully solved

In a blog post published on Monday, OpenAI acknowledged that prompt injection is unlikely to ever be fully solved. The company conceded that the “agent mode” in its ChatGPT Atlas browser, which launched in October, expands the security threat surface for users on the open web. OpenAI compared the vulnerability to existing web threats like scams and social engineering, stating that it views prompt injection as a long-term AI security challenge that will require continuous defensive upgrades. The admission highlights the difficulty of securing AI agents that interact with untrusted data on the open web, where malicious instructions can easily manipulate their behavior.

To address these persistent risks, OpenAI is deploying an LLM-based automated attacker. This system is a bot trained using reinforcement learning to simulate hackers attempting to sneak malicious instructions to an AI agent. By testing attacks in a simulated environment, the automated attacker can discover novel attack strategies internally before they are exploited. OpenAI noted that its reinforcement learning-trained attacker can manipulate an agent into executing complex, long-horizon harmful workflows that can span tens or even hundreds of steps. This method allows the company to continuously adapt its defenses to attacker behavior by analyzing the target AI’s internal reasoning.

Other organizations have also flagged the long-term nature of these vulnerabilities. Earlier this month, the National Cyber Security Centre, a U.K. government agency, warned that prompt injection attacks against generative AI applications may never be totally mitigated, leaving websites vulnerable to data breaches. Security experts suggest that the fundamental architecture of agentic browsers presents inherent trade-offs. Rami McCarthy, principal security researcher at cybersecurity firm Wiz, noted that reinforcement learning is one way to continuously adapt to attacker behavior, but pointed to a broader structural challenge. “A useful way to reason about risk in AI systems is autonomy multiplied by access,” McCarthy said, adding that for most everyday use cases, agentic browsers do not yet provide enough value to justify their current risk profile.

Why it matters

OpenAI’s admission that prompt injection may never be fully solved highlights the persistent security challenges facing AI agents as they gain broader access to user data and systems.