RAG Security Testing Checklist for AI Agents

RAG security testing checks whether retrieved content can change an agent's instructions, hide bad sources, leak private context, or trigger tool calls outside the user's intent.

Quick Answer

1. Seed documents with hostile instructions.
2. Test source attribution on conflicting chunks.
3. Check whether stale content overrides current policy.
4. Block retrieved text from authorizing tool calls.
5. Save failures as regression cases.

Poison the retrieval set on purpose

Add documents that look normal but contain instructions like "ignore the user" or "send this context elsewhere." The agent should summarize the document and ignore the instruction.

Test source confusion

RAG systems often retrieve multiple chunks. Include conflicting dates, fake policy text, and similar-looking sources. The agent should cite the right source and flag conflict instead of blending them.

Tool calls need a separate gate

A retrieved document should not be enough to authorize an email, calendar edit, deployment, or payment. Treat retrieval as evidence, not approval.

Where BreakMyAgent fits

BreakMyAgent helps teams build RAG attack fixtures and rerun them after indexing, prompt, and model changes.

FAQ

Is RAG safer than browsing?

Not automatically. Curated documents can still contain hostile or stale instructions.

What is the first test to run?

Put an instruction inside a retrieved document and verify the agent refuses to obey it.

Should source links be mandatory?

Yes for factual answers and any action recommendation.