OpenAI published details of a new evaluation framework called the Instruction Hierarchy Challenge, designed to test how well frontier language models distinguish between system instructions and user-supplied content [source]. The framework measures whether models can resist prompt injection attacks where user input attempts to override system-level directives.

According to the post, OpenAI tested multiple frontier models including GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro against scenarios where user messages contained instructions that conflicted with system prompts. The evaluation found that all tested models showed vulnerability to instruction hierarchy confusion, with success rates for maintaining correct instruction priority ranging from 60% to 85% depending on attack sophistication.

OpenAI reports implementing architectural changes in its latest models to improve instruction hierarchy handling. The company states that GPT-4o now achieves 89% accuracy on the benchmark's hardest tier, compared to 73% for the previous version. However, the post acknowledges that no current frontier model achieves perfect separation between system and user instruction contexts.

The Instruction Hierarchy Challenge includes 500 test cases across three difficulty tiers. Tier 1 involves simple conflicting instructions, Tier 2 includes obfuscated attempts to override system context, and Tier 3 tests multi-turn conversations where injection attempts are distributed across messages.

OpenAI released the evaluation dataset publicly and stated that instruction hierarchy remains an open research problem across the industry. The post notes that applications using LLMs to process untrusted user input remain vulnerable to prompt injection techniques that exploit unclear boundaries between system instructions and user content.