Skip to content

Open benchmark

Prompt injection detection, measured in public

Security teams should judge a detector by how many attacks it catches while keeping false alarms rare. Here is our detector's true positive rate (TPR) at low false positive rates (FPR) on a public benchmark — including where it falls short.

PromptShield results

Test split of the PromptShield benchmark (arXiv:2501.15145): 23,516 prompts — 6,486 injections and 17,030 benign prompts. Re-run on 11 November 2026 with the multilingual and de-obfuscation layers enabled (previous run, 8 November: 22.3% TPR, 0.06% FPR).

SettingTPR (attacks caught)FPR (benign flagged)
Default rule (threshold 60)22.4%0.22%
Tuned for ≤ 0.1% FPR (threshold 68)22.4%0.10%
Tuned for ≤ 1% FPR (threshold 40)23.0%0.90%

What this means: on the default rule about 1 in 460 benign prompts is flagged. The new layers target non-English and obfuscated attacks, which this mostly-English benchmark rarely contains, so they barely move the score here, but roughly 3 in 4 injections in this benchmark are not caught by pattern matching alone. That is why Vigil is built around hard tool boundaries, argument inspection and fail-closed defaults rather than detection alone — see our manifesto.

Raw output: promptshield-2026-11-11.json (previous: 8 November). Timing is detector CPU time only and excludes network and storage.

AI-assisted mode (patterns + model)

The prompt injection rule now has an optional AI-assisted check. Prompts the pattern rules allow are also sent to a small AI model; if it says "injection", the rule fires. We measured it on a fresh fixed random sample of the PromptShield test split (500 injections, 500 benign prompts, temperature 0):

ModeTPRFPRAdded delay
Patterns only (default)20.2%0.0%≈0.05 ms
AI-assisted (patterns + model)47.0%1.0%≈0.7 s median, 1.2 s p95

AI-assisted mode more than doubles detection while keeping false alarms at about 1 in 100. It is still not enough on its own, and it adds model latency and cost, so it is off by default. If the model is unreachable, Vigil falls back to the pattern verdict. Raw output: hybrid-2026-11.json.

We also tried adding more rules learned from the PromptShield training split. They raised detection on the training data but not on the test split, and increased false alarms, so we did not ship them.

Tool calls and MCP

The numbers above cover text-only prompt detection. Separately, Vigil's tool allowlist rule inspects tool arguments (destructive shell and SQL, path traversal, cloud metadata endpoints) and flags changes to pinned MCP tool descriptions. These checks are not measured by this benchmark.

How sure are these numbers?

Every result is an estimate. Below are 95% confidence ranges (Wilson interval). Small samples give wide ranges: on the 200-attack sample, Vigil and Prompt Guard overlap, so we do not claim either is better there.

ResultMeasured95% rangeSample size
PromptShield full test, default rule — TPR22.4%21.4–23.5%6,486 attacks
PromptShield full test, default rule — FPR0.22%0.16–0.30%17,030 benign
AI-assisted mode — TPR47.0%42.7–51.4%500 attacks
AI-assisted mode — FPR1.0%0.4–2.3%500 benign
Vigil vs Prompt Guard sample — Vigil TPR26.0%20.4–32.5%200 attacks
Vigil vs Prompt Guard sample — Prompt Guard jailbreak TPR23.5%18.2–29.8%200 attacks
  • Only the public test split is used for reported numbers; rules are never tuned on it.
  • Samples use a fixed random seed, so anyone can rebuild the exact same prompt set.
  • We publish results that make us look bad, including rules we tried and dropped.

Lakera on the same prompts: ready to run

We have not published Lakera numbers on PromptShield because we have not run them yet, and we will not show invented results. The script below sends the same seeded sample to Lakera's API and writes TPR/FPR in the same format as our other files. Results will appear here once run.

LAKERA_API_KEY=... bun scripts/bench-lakera.ts /tmp/promptshield-test.json 500 7
# POST https://api.lakera.ai/v2/guard  { "messages": [{ "role": "user", "content": <prompt> }] }
# flagged: true  -> counted as "blocked"

Meta Prompt Guard on the same prompts

We ran Meta Prompt Guard 86M (v1) ourselves on a fixed random sample of the same PromptShield test split (200 injections, 200 benign prompts), and ran Vigil's default rule on the identical sample.

DetectorTPRFPR
Vigil default rule (threshold 60)26.0%0.0%
Prompt Guard, jailbreak label (score ≥ 0.5)23.5%2.5%
Prompt Guard, jailbreak label tuned to ≤ 1% FPR14.5%1.0%
Prompt Guard, injection or jailbreak (score ≥ 0.5)98.5%99.0%

On this benchmark, Prompt Guard's "injection" label fires on almost every benign prompt, so it is unusable for user prompts without fine-tuning (Meta itself recommends fine-tuning). Its "jailbreak" label is usable, and catches a similar share to Vigil's rules. Neither is good enough alone, which is why we keep tool boundaries and fail-closed defaults. A sample of 400 is directional; on CPU Prompt Guard took about 700 ms per prompt versus about 0.05 ms for Vigil. Raw output: prompt-guard-2026-11.json.

Lakera and SIREN: published numbers

We have not run Lakera Guard ourselves; it is a paid hosted service. The figures below are from Lakera's own PINT benchmark (a private dataset, best configuration per tool, run by Lakera). They are not comparable to the PromptShield numbers above: different data, different metric (PINT is balanced accuracy), and run by a vendor.

ToolPINT score (published)Test date
Lakera Guard95.22%2025-05-02
AWS Bedrock Guardrails89.24%2025-05-02
Azure AI Prompt Shield89.12%2025-05-02
Llama Prompt Guard 2 (86M)78.76%2025-05-05
Llama Prompt Guard (v1)61.82%2025-05-02

Source: lakeraai/pint-benchmark. Vigil is not on that list; PINT data is private, so we cannot run it. SIREN results are not shown because its test set is not publicly available. With a Lakera API key we will run Lakera on the same PromptShield sample and publish the numbers here.

Architecture trade-off: hosted classifiers like Lakera send each prompt to a third-party cloud and add a network round trip. Vigil's pattern rules run inside your request with no extra call; AI-assisted mode adds one model call only for prompts the patterns allow.

How approaches compare

Only the Vigil and Prompt Guard rows were measured by us. We do not restate other vendors' numbers; use their published evaluations.

ApproachHow it decidesLatency costTrade-offAccuracy source
Vigil deterministic detectorsPattern signals, no model call≈0.05 ms CPU per prompt (measured)Low FPR; misses novel and paraphrased attacksMeasured above
Vigil AI-assisted modePatterns, then a model for the rest≈0.7 s median (measured)47% TPR at 1% FPR; adds cost per callMeasured above
Hosted classifier API (e.g. Lakera Guard)Vendor model in the cloudNetwork round tripPrompts leave your environmentVendor-published PINT only
Fine-tuned classifier models (e.g. PromptShield, Prompt Guard)Model inference≈700 ms per prompt on our CPU (Prompt Guard)Needs fine-tuning for your trafficPrompt Guard measured above
LLM-as-judge guard (e.g. Llama Guard, NeMo self-check)Extra LLM callAdds a full model round tripFlexible; non-deterministic, costly per callNot measured by us
Web application firewallHTTP signaturesLowDoes not understand natural-language instructionsNot applicable

Reproduce it

curl -L -o /tmp/promptshield-test.json \
  https://huggingface.co/datasets/hendzh/PromptShield/resolve/main/test.json
bun scripts/bench-promptshield.ts /tmp/promptshield-test.json

Detector signals are listed in the detector reference.