Research: experimental prompt-injection detector

Research · Experimental · Synthetic data

Experimental detector: 99.2% attack detection on Boundary Pairs synthetic test data, in each of three training runs (119/120 attacks each). Average benign false-alarm rate 3.9% (per-run 2.5%, 4.2%, 5.0%). 10 held-out scenarios from one synthetic generator; not evidence of real-world protection. Shipped rule-based scanner is separate.

Shipped product vs. research

Methods

Model

Base model: protectai/deberta-v3-base-prompt-injection-v2 at revision 90c9989b1a342275dd0d1a95aad283c04e075671, an existing DeBERTa-v3-base sequence classifier (labels SAFE / INJECTION). No new architecture or head: the whole model is fine-tuned, and the detection score is logit(INJECTION) − logit(SAFE). The unmodified base model is reported as the comparison arm.

Data

Agentic Prompt-Injection Boundary Pairs (CC BY 4.0), pinned at commit 30af820d. Each pair holds one text that preserves an intended authorization boundary and one that tries to cross it. The author's splits are grouped by scenario:

Pre-training checks used train and validation only: no duplicate texts and no pair or scenario overlap between them. The test split was not read until all three runs had finished and the thresholds were frozen; it was then checked (hash, schema, duplicates, pairs, and no scenario or pair overlap with train or validation) before any scoring. Only the text is used as input; no metadata.

Training

Three runs with seeds 42, 43 and 44 and an identical recipe, all reported (no best-seed selection): 3 epochs (159 optimizer steps), AdamW with learning rate 2e-5 and weight decay 0.01, 9 warm-up steps then linear decay, effective batch 16 (4 × 4 accumulation; each epoch has 210 micro-batches, so its last optimizer step accumulates only 2), max length 512 tokens, binary cross-entropy on the score with positive weight 1. Training ran on Apple MPS in fp32, which is not bit-deterministic.

Thresholds and baselines

Each model's threshold was chosen on validation data only: the most attacks caught with at most 1 of 60 benign false alarms. Full-precision thresholds: base model 14.985377311706543; seed 42 −2.541528105735779; seed 43 −2.8378076553344727; seed 44 −2.841301918029785. Thresholds and two simple baselines were frozen before the test set was read once:

• Phrase-list baseline: word pairs that occur in at least 10 training rows of one class and none of the other. It was chosen after a validation diagnostic, so it is an adaptive baseline.
• Length baseline: flag any text of 32 or more tokens.

Test results

Differences (fine-tuned 3-run mean minus comparison), with 95% intervals from a paired bootstrap over the 10 test scenarios (1,000 resamples):

With only 10 test scenarios the intervals are wide and unstable. The false-alarm comparison with the base model is inconclusive.

Limits

• All splits come from one synthetic generator with shared templates. A simple phrase list already separates much of the data, so the result shows learning of this generator’s boundary patterns, not general understanding.
• Attack texts are slightly longer than benign ones; the fine-tuned model was not length-controlled.
• 120 benign test texts cannot certify a low false-alarm rate.
• No evidence about real emails, indirect injection in tool outputs, unseen attack families, or production use.
• Earlier experiments did not transfer well: detectors trained on our earlier data were weak on email and document-style contexts, and three attempts to build length-controlled context training sets failed their own pre-registered checks (C1 failed its length gate; C1R failed its primary semantic/generalisation gate with the NotInject hard-negative set; CAL-R failed its whole-calibration-set check). These failures are kept on record.

Full methods, results and negative results: docs/research on GitHub.