Benchmark
Every number here comes from a committed script you can rerun. Small samples are labelled as small.
How it works and its limits
Full methodology, every table, and how to reproduce each run are in docs/benchmark.md. The caveats for each number are in docs/limitations.md.
Accuracy under attack
From the Attack Lab's benchmark, n = 16 (8 human, 8 gpt2-generated). B and C count texts still labelled correctly as human or AI. F counts texts still matched to their own logged original, a different kind of correctness that isn't directly comparable.
| Attack | Strength | B, statistical | C, classifier | F, ledger |
|---|---|---|---|---|
| None | — | 16/16 | 7/16 | 16/16 |
| Synonym | 0.3 | 15/16 | 7/16 | 16/16 |
| Synonym | 0.6 | 14/16 | 7/16 | 16/16 |
| Reorder | 0.3 | 16/16 | 7/16 | 16/16 |
| Reorder | 0.6 | 15/16 | 7/16 | 16/16 |
| Truncate | 0.3 | 14/16 | 7/16 | 16/16 |
| Truncate | 0.6 | 14/16 | 7/16 | 9/16 |
| Paraphrase | — | 16/16 | 9/16 | 16/16 |
The real paraphrase attack turned out to be the least damaging here, the opposite of this project's own expectation going in. That reflects the small T5 checkpoint used, not a refutation of the published result.
Trained classifier (C)
Held-out test split, n = 767, trained on HC3 (CC-BY-SA 4.0).
- Accuracy
- 0.814
- Precision
- 0.789
- Recall
- 0.867
- F1
- 0.826
- ROC-AUC
- 0.896
- Interval coverage
- 91.9%
Coverage targets 90%. These numbers hold for HC3-like text. On the Attack Lab's different domain and generator, C labels only 7 of 16 correctly (above).
Statistical detector (B)
Calibration ranges of the Binoculars-style score (gpt2 performer, distilgpt2 observer). Lower reads as more machine-like.
Scale 0 to 0.8. Dashed lines mark the thresholds: below 0.24 is flagged AI, above 0.28 is flagged human. The gap between the two calibration ranges is where the uncertain band sits.
- AI range
- 0.093–0.225
- Human range
- 0.294–0.725
Watermark robustness (A)
Share of 30 seeded runs still flagged as watermarked as a structural attack strengthens (green-list scheme, δ = 2). These are structural edits, not real paraphrasing.
| Attack | 0% | 20% | 40% | 75% | 100% |
|---|---|---|---|---|---|
| Substitute | 100% | 57% | 0% | 0% | 0% |
| Delete | 100% | 87% | 40% | 0% | 0% |
| Insert | 100% | 100% | 60% | 10% | 3% |
| Reorder | 100% | 57% | 27% | 0% | 0% |