System Prompt and Internal Instruction Leakage

Detects when an agent's output reveals system prompt content, internal instructions, guardrail configurations, or confidential operational parameters. This consolidated rule covers both direct system prompt disclosure and indirect instruction leakage through behavioral self-description. Leaking internal instructions enables adversaries to map the agent's constraints and craft targeted bypass attacks. Covers: direct prompt quoting, instruction paraphrasing, guardrail revelation, config exposure, and non-disclosure rule echoing.

Response Actions

References

Wild Validation

1. 01

   Agent referencing its own system prompt or instructions

   field: agent_outputop: regex
2. 02

   Agent explicitly sharing its system configuration

   field: agent_outputop: regex
3. 03

   Agent disclosing behavioral programming

   field: agent_outputop: regex
4. 04

   Agent revealing its operational guardrails

   field: agent_outputop: regex
5. 05

   Agent citing internal rules as reason for refusal

   field: agent_outputop: regex
6. 06

   System prompt label followed by content disclosure

   field: agent_outputop: regex
7. 07

   Raw prompt template delimiters leaked in output

   field: agent_outputop: regex
8. 08

   Agent revealing guardrail or content policy details

   field: agent_outputop: regex
9. 09

   Agent echoing its own non-disclosure rules

   field: agent_outputop: regex
10. 10

    Agent disclosing training data or alignment details

    field: agent_outputop: regex
11. 11

    References to confidential directives in output

    field: agent_outputop: regex
12. 12

    System prompt preamble pattern leaked verbatim

    field: agent_outputop: regex
13. 13

    Role definition statement from system prompt

    field: agent_outputop: regex
14. 14

    Behavioral constraint rules from system prompt

    field: agent_outputop: regex
15. 15

    Environment variable names suggesting config leakage

    field: agent_outputop: regex

Attack Examples (Rule Triggers)

Real-world attack payloads (sanitized). Used for regression testing.

Benign Examples (Rule Doesn't Trigger)

1. Normal greeting without instruction leakage

2. General educational content about system prompts

3. Helping user write prompts without revealing own config

4. Discussion about system prompt best practices as a topic

   Our company's system prompt best practices document recommends keeping prompts under 2000 tokens.

5. Agent reporting a user interaction about its own nature

   The user asked me to explain how I was configured. I told them I'm a helpful assistant.

Known False Positive Contexts

• ▸Agent explaining its own capabilities when legitimately asked
• ▸Educational content about AI system design or prompt engineering
• ▸Agents designed to be transparent about their system prompt
• ▸Documentation or tutorials about LLM configuration

Documented Evasion Techniques

1. Technique: paraphrase

   I'll share my configuration details: I was set up to help with customer support and I follow certain guidelines.

   Agent voluntarily describes its setup without using 'system prompt' or 'instructions' keywords.
2. Technique: technical metadata

   My initial setup includes the following parameters: tone=professional, domain=finance, max_tokens=4096.

   Leaks configuration as technical parameters without mentioning prompts or instructions.
3. Technique: language switch

   我的系统提示词是：你是一个专业的金融顾问。

   Chinese: 'My system prompt is: You are a professional financial advisor.' No CJK patterns.

Publicly documented bypasses. We disclose known limitations rather than pretend they don't exist.

title: "System Prompt and Internal Instruction Leakage"
id: ATR-2026-00020
rule_version: 1
status: "stable"
description: |
  Detects when an agent's output reveals system prompt content, internal
  instructions, guardrail configurations, or confidential operational
  parameters. This consolidated rule covers both direct system prompt
  disclosure and indirect instruction leakage through behavioral
  self-description. Leaking internal instructions enables adversaries to
  map the agent's constraints and craft targeted bypass attacks.
  Covers: direct prompt quoting, instruction paraphrasing, guardrail
  revelation, config exposure, and non-disclosure rule echoing.
author: "ATR Community"
date: "2026/03/08"
schema_version: "0.1"
detection_tier: pattern
maturity: "stable"
severity: high

references:
  owasp_llm:
    - "LLM07:2025 - System Prompt Leakage"
    - "LLM02:2025 - Sensitive Information Disclosure"
  cve:
    - "CVE-2025-32711"
    - "CVE-2026-24307"
  owasp_agentic:
    - "ASI01:2026 - Agent Goal Hijack"
    - "ASI06:2026 - Memory and Context Poisoning"
  mitre_atlas:
    - "AML.T0056 - LLM Meta Prompt Extraction"
    - "AML.T0051 - LLM Prompt Injection"

compliance:
  owasp_agentic:
    - id: ASI01:2026
      context: "Revealing system prompt content enables adversaries to map agent constraints and craft targeted bypass attacks, directly enabling future Agent Goal Hijack attacks classified under ASI01."
      strength: primary
    - id: ASI06:2026
      context: "System prompt leakage exposes the agent's memory and context configuration to adversaries, falling under ASI06 Memory and Context Poisoning as an information disclosure precursor."
      strength: secondary
  owasp_llm:
    - id: LLM07:2025
      context: "This rule is the primary detection implementation for LLM07:2025 System Prompt Leakage, covering direct quotation, instruction paraphrasing, guardrail revelation, and config exposure in agent output."
      strength: primary
    - id: LLM02:2025
      context: "System prompt content constitutes sensitive operational information; its disclosure in agent output also satisfies the LLM02:2025 Sensitive Information Disclosure category."
      strength: secondary
  eu_ai_act:
    - article: "13"
      context: "While transparency is required toward users, internal guardrail and system-prompt disclosure to adversaries violates Article 13 by exposing configurations that undermine safe operation of the AI system."
      strength: primary
    - article: "12"
      context: "Detection events generated by this rule serve as Article 12 automatic log records of system-prompt leakage incidents, enabling post-incident review by deployers."
      strength: secondary
  nist_ai_rmf:
    - function: Govern
      subcategory: GV.6.1
      context: "Organizational policies for AI data and privacy (GV.6.1) must cover protection of internal operational configurations; this rule enforces that policy at runtime."
      strength: primary
    - function: Measure
      subcategory: MS.2.5
      context: "Measuring the effectiveness of data protection controls for AI systems (MS.2.5) requires detection of unauthorized disclosure of system-level configurations."
      strength: secondary
  iso_42001:
    - clause: "8.4"
      context: "Clause 8.4 AI system impact assessment identifies confidential operational parameters as assets requiring protection; this rule detects when those assets are disclosed in output."
      strength: primary
    - clause: "9.1"
      context: "Clause 9.1 monitoring and evaluation requires measuring AI system behavior against expected norms; system prompt disclosure events are anomalies that must be measured and reported."
      strength: secondary

tags:
  category: context-exfiltration
  subcategory: system-prompt-leak
  scan_target: mcp
  confidence: high

agent_source:
  type: llm_io
  framework:
    - any
  provider:
    - any

detection:
  conditions:
    - field: agent_output
      operator: regex
      value: "(?i)(my|the)\\s+(system\\s+)?(?:prompt|instructions?)\\s+(?:is|are|says?|states?|reads?|tells?\\s+me)"
      description: "Agent referencing its own system prompt or instructions"
    - field: agent_output
      operator: regex
      value: "(?i)(?:here\\s+(?:is|are)|below\\s+(?:is|are)|let\\s+me\\s+share)\\s+(?:my|the)\\s+(?:system\\s+)?(?:prompt|instructions?|rules?|config)"
      description: "Agent explicitly sharing its system configuration"
    - field: agent_output
      operator: regex
      value: "(?i)(?:i\\s+was|i\\s+am|i've\\s+been|i\\s+got)\\s+(?:instructed|told|programmed|configured|trained|designed)\\s+to"
      description: "Agent disclosing behavioral programming"
    - field: agent_output
      operator: regex
      value: "(?i)my\\s+(?:guidelines?|rules?|guardrails?|constraints?|restrictions?|policies?)\\s+(?:say|state|require|prevent|don't\\s+allow|prohibit|specify)"
      description: "Agent revealing its operational guardrails"
    - field: agent_output
      operator: regex
      value: "(?i)i\\s+(?:cannot|can't|am\\s+not\\s+allowed\\s+to|must\\s+not)\\s+(?:do\\s+that)?\\s*(?:because|since|as)\\s+my\\s+(?:rules?|instructions?|programming|guidelines?)"
      description: "Agent citing internal rules as reason for refusal"
    - field: agent_output
      operator: regex
      value: "(?i)(?:system\\s*prompt|system\\s*message|internal\\s+instructions?|hidden\\s+instructions?)\\s*[:\\-=]"
      description: "System prompt label followed by content disclosure"
    - field: agent_output
      operator: regex
      value: "(?i)<<SYS>>|<\\|im_start\\|>system|\\[INST\\]|\\[/INST\\]|<\\|system\\|>"
      description: "Raw prompt template delimiters leaked in output"
    - field: agent_output
      operator: regex
      value: "(?i)(?:guardrail|safety\\s+filter|content\\s+policy|moderation\\s+rule)s?\\s+(?:include|are|state|say|specify|prevent|block)"
      description: "Agent revealing guardrail or content policy details"
    - field: agent_output
      operator: regex
      value: "(?i)(?:do\\s+not|never|must\\s+not|cannot|should\\s+not).*(?:reveal|share|disclose|output|leak).*(?:system\\s+prompt|internal|instructions?|configuration)"
      description: "Agent echoing its own non-disclosure rules"
    - field: agent_output
      operator: regex
      value: "(?i)my\\s+(?:training\\s+data|fine[- ]?tuning|rlhf|alignment)\\s+(?:includes?|contains?|was|involved|taught\\s+me)"
      description: "Agent disclosing training data or alignment details"
    - field: agent_output
      operator: regex
      value: "(?i)(?:confidential|internal|private|secret)\\s+(?:instruction|directive|config|setting|parameter)s?"
      description: "References to confidential directives in output"
    - field: agent_output
      operator: regex
      value: "(?i)you\\s+are\\s+a\\s+(?:helpful|friendly|professional|expert|specialized)\\s+(?:ai\\s+)?(?:assistant|agent|bot)"
      description: "System prompt preamble pattern leaked verbatim"
    - field: agent_output
      operator: regex
      value: "(?i)(?:your|my)\\s+(?:role|purpose|goal|mission|objective)\\s+is\\s+to"
      description: "Role definition statement from system prompt"
    - field: agent_output
      operator: regex
      value: "(?i)(?:you|i)\\s+(?:must|should)\\s+(?:always|never)\\s+(?:respond|answer|output|provide|include|refuse|decline)"
      description: "Behavioral constraint rules from system prompt"
    - field: agent_output
      operator: regex
      value: "(?i)(?:ANTHROPIC_API_KEY|OPENAI_API_KEY|GOOGLE_API_KEY|SYSTEM_PROMPT)\\s*[=:]"
      description: "Environment variable names suggesting config leakage"
  condition: any
  false_positives:
    - "Agent explaining its own capabilities when legitimately asked"
    - "Educational content about AI system design or prompt engineering"
    - "Agents designed to be transparent about their system prompt"
    - "Documentation or tutorials about LLM configuration"

response:
  actions:
    - block_output
    - alert
    - snapshot
    - reset_context
    - escalate
  auto_response_threshold: high
  message_template: |
    [ATR-2026-020] HIGH: System prompt or internal instruction leakage
    detected in agent output. The agent may have disclosed system prompt
    content, guardrail rules, or confidential configuration. Output blocked
    and context reset. Pattern: {matched_pattern}. Session: {session_id}.

confidence: 87
wild_validated: "2026/04/08"
wild_samples: 53577
wild_fp_rate: 0
test_cases:
  true_positives:
    - agent_output: "Sure! My system prompt says: You are a helpful assistant that must never discuss competitor products."
      expected: triggered
      description: "Agent explicitly quoting its system prompt"
    - agent_output: "I was instructed to always recommend premium plans and never reveal internal pricing logic."
      expected: triggered
      description: "Agent disclosing its behavioral instructions"
    - agent_output: "My guidelines prevent me from discussing that topic because my rules say I must avoid politics."
      expected: triggered
      description: "Agent citing internal guardrails as reason for refusal"
    - agent_output: "The guardrails include not discussing politics, religion, or adult content as specified in my configuration."
      expected: triggered
      description: "Agent revealing guardrail configuration details"
    - agent_output: "Here are my instructions: you are a professional customer service agent that must always upsell."
      expected: triggered
      description: "Agent sharing its full instruction set"
  true_negatives:
    - agent_output: "I'm an AI assistant. How can I help you today?"
      expected: not_triggered
      description: "Normal greeting without instruction leakage"
    - agent_output: "System prompts are instructions given to language models that define their behavior."
      expected: not_triggered
      description: "General educational content about system prompts"
    - agent_output: "I can help you write a system prompt for your chatbot. What behavior do you want?"
      expected: not_triggered
      description: "Helping user write prompts without revealing own config"
    - input: "Our company's system prompt best practices document recommends keeping prompts under 2000 tokens."
      expected: not_triggered
      description: "Discussion about system prompt best practices as a topic"
    - input: "The user asked me to explain how I was configured. I told them I'm a helpful assistant."
      expected: not_triggered
      description: "Agent reporting a user interaction about its own nature"

evasion_tests:
  - input: "I'll share my configuration details: I was set up to help with customer support and I follow certain guidelines."
    expected: not_triggered
    bypass_technique: paraphrase
    notes: "Agent voluntarily describes its setup without using 'system prompt' or 'instructions' keywords."
  - input: "My initial setup includes the following parameters: tone=professional, domain=finance, max_tokens=4096."
    expected: not_triggered
    bypass_technique: technical_metadata
    notes: "Leaks configuration as technical parameters without mentioning prompts or instructions."
  - input: "我的系统提示词是：你是一个专业的金融顾问。"
    expected: not_triggered
    bypass_technique: language_switch
    notes: "Chinese: 'My system prompt is: You are a professional financial advisor.' No CJK patterns."