Open benchmark
Prompt injection detection, measured in public
Security teams should judge a detector by how many attacks it catches while keeping false alarms rare. Here is our detector's true positive rate (TPR) at low false positive rates (FPR) on a public benchmark — including where it falls short.
PromptShield results
Test split of the PromptShield benchmark (arXiv:2501.15145): 23,516 prompts — 6,486 injections and 17,030 benign prompts. Re-run on 11 November 2026 with the multilingual and de-obfuscation layers enabled (previous run, 8 November: 22.3% TPR, 0.06% FPR).
| Setting | TPR (attacks caught) | FPR (benign flagged) |
|---|---|---|
| Default rule (threshold 60) | 22.4% | 0.22% |
| Tuned for ≤ 0.1% FPR (threshold 68) | 22.4% | 0.10% |
| Tuned for ≤ 1% FPR (threshold 40) | 23.0% | 0.90% |
What this means: on the default rule about 1 in 460 benign prompts is flagged. The new layers target non-English and obfuscated attacks, which this mostly-English benchmark rarely contains, so they barely move the score here, but roughly 3 in 4 injections in this benchmark are not caught by pattern matching alone. That is why Vigil is built around hard tool boundaries, argument inspection and fail-closed defaults rather than detection alone — see our manifesto.
Raw output: promptshield-2026-11-11.json (previous: 8 November). Timing is detector CPU time only and excludes network and storage.
AI-assisted mode (patterns + model)
The prompt injection rule now has an optional AI-assisted check. Prompts the pattern rules allow are also sent to a small AI model; if it says "injection", the rule fires. We measured it on a fresh fixed random sample of the PromptShield test split (500 injections, 500 benign prompts, temperature 0):
| Mode | TPR | FPR | Added delay |
|---|---|---|---|
| Patterns only (default) | 20.2% | 0.0% | ≈0.05 ms |
| AI-assisted (patterns + model) | 47.0% | 1.0% | ≈0.7 s median, 1.2 s p95 |
AI-assisted mode more than doubles detection while keeping false alarms at about 1 in 100. It is still not enough on its own, and it adds model latency and cost, so it is off by default. If the model is unreachable, Vigil falls back to the pattern verdict. Raw output: hybrid-2026-11.json.
We also tried adding more rules learned from the PromptShield training split. They raised detection on the training data but not on the test split, and increased false alarms, so we did not ship them.
Tool calls and MCP
The numbers above cover text-only prompt detection. Separately, Vigil's tool allowlist rule inspects tool arguments (destructive shell and SQL, path traversal, cloud metadata endpoints) and flags changes to pinned MCP tool descriptions. These checks are not measured by this benchmark.
How sure are these numbers?
Every result is an estimate. Below are 95% confidence ranges (Wilson interval). Small samples give wide ranges: on the 200-attack sample, Vigil and Prompt Guard overlap, so we do not claim either is better there.
| Result | Measured | 95% range | Sample size |
|---|---|---|---|
| PromptShield full test, default rule — TPR | 22.4% | 21.4–23.5% | 6,486 attacks |
| PromptShield full test, default rule — FPR | 0.22% | 0.16–0.30% | 17,030 benign |
| AI-assisted mode — TPR | 47.0% | 42.7–51.4% | 500 attacks |
| AI-assisted mode — FPR | 1.0% | 0.4–2.3% | 500 benign |
| Vigil vs Prompt Guard sample — Vigil TPR | 26.0% | 20.4–32.5% | 200 attacks |
| Vigil vs Prompt Guard sample — Prompt Guard jailbreak TPR | 23.5% | 18.2–29.8% | 200 attacks |
- Only the public test split is used for reported numbers; rules are never tuned on it.
- Samples use a fixed random seed, so anyone can rebuild the exact same prompt set.
- We publish results that make us look bad, including rules we tried and dropped.
Lakera on the same prompts: ready to run
We have not published Lakera numbers on PromptShield because we have not run them yet, and we will not show invented results. The script below sends the same seeded sample to Lakera's API and writes TPR/FPR in the same format as our other files. Results will appear here once run.
LAKERA_API_KEY=... bun scripts/bench-lakera.ts /tmp/promptshield-test.json 500 7
# POST https://api.lakera.ai/v2/guard { "messages": [{ "role": "user", "content": <prompt> }] }
# flagged: true -> counted as "blocked"Meta Prompt Guard on the same prompts
We ran Meta Prompt Guard 86M (v1) ourselves on a fixed random sample of the same PromptShield test split (200 injections, 200 benign prompts), and ran Vigil's default rule on the identical sample.
| Detector | TPR | FPR |
|---|---|---|
| Vigil default rule (threshold 60) | 26.0% | 0.0% |
| Prompt Guard, jailbreak label (score ≥ 0.5) | 23.5% | 2.5% |
| Prompt Guard, jailbreak label tuned to ≤ 1% FPR | 14.5% | 1.0% |
| Prompt Guard, injection or jailbreak (score ≥ 0.5) | 98.5% | 99.0% |
On this benchmark, Prompt Guard's "injection" label fires on almost every benign prompt, so it is unusable for user prompts without fine-tuning (Meta itself recommends fine-tuning). Its "jailbreak" label is usable, and catches a similar share to Vigil's rules. Neither is good enough alone, which is why we keep tool boundaries and fail-closed defaults. A sample of 400 is directional; on CPU Prompt Guard took about 700 ms per prompt versus about 0.05 ms for Vigil. Raw output: prompt-guard-2026-11.json.
Lakera and SIREN: published numbers
We have not run Lakera Guard ourselves; it is a paid hosted service. The figures below are from Lakera's own PINT benchmark (a private dataset, best configuration per tool, run by Lakera). They are not comparable to the PromptShield numbers above: different data, different metric (PINT is balanced accuracy), and run by a vendor.
| Tool | PINT score (published) | Test date |
|---|---|---|
| Lakera Guard | 95.22% | 2025-05-02 |
| AWS Bedrock Guardrails | 89.24% | 2025-05-02 |
| Azure AI Prompt Shield | 89.12% | 2025-05-02 |
| Llama Prompt Guard 2 (86M) | 78.76% | 2025-05-05 |
| Llama Prompt Guard (v1) | 61.82% | 2025-05-02 |
Source: lakeraai/pint-benchmark. Vigil is not on that list; PINT data is private, so we cannot run it. SIREN results are not shown because its test set is not publicly available. With a Lakera API key we will run Lakera on the same PromptShield sample and publish the numbers here.
Architecture trade-off: hosted classifiers like Lakera send each prompt to a third-party cloud and add a network round trip. Vigil's pattern rules run inside your request with no extra call; AI-assisted mode adds one model call only for prompts the patterns allow.
How approaches compare
Only the Vigil and Prompt Guard rows were measured by us. We do not restate other vendors' numbers; use their published evaluations.
| Approach | How it decides | Latency cost | Trade-off | Accuracy source |
|---|---|---|---|---|
| Vigil deterministic detectors | Pattern signals, no model call | ≈0.05 ms CPU per prompt (measured) | Low FPR; misses novel and paraphrased attacks | Measured above |
| Vigil AI-assisted mode | Patterns, then a model for the rest | ≈0.7 s median (measured) | 47% TPR at 1% FPR; adds cost per call | Measured above |
| Hosted classifier API (e.g. Lakera Guard) | Vendor model in the cloud | Network round trip | Prompts leave your environment | Vendor-published PINT only |
| Fine-tuned classifier models (e.g. PromptShield, Prompt Guard) | Model inference | ≈700 ms per prompt on our CPU (Prompt Guard) | Needs fine-tuning for your traffic | Prompt Guard measured above |
| LLM-as-judge guard (e.g. Llama Guard, NeMo self-check) | Extra LLM call | Adds a full model round trip | Flexible; non-deterministic, costly per call | Not measured by us |
| Web application firewall | HTTP signatures | Low | Does not understand natural-language instructions | Not applicable |
Reproduce it
curl -L -o /tmp/promptshield-test.json \ https://huggingface.co/datasets/hendzh/PromptShield/resolve/main/test.json bun scripts/bench-promptshield.ts /tmp/promptshield-test.json
Detector signals are listed in the detector reference.