Methods
Evaluation and reproducibility
SarcBench separates its historical pilot from the corrected protocol used for future model evaluations.
Dataset
SarcBench v0.1 contains 25 released examples, 125 hidden scored questions, and 25 hidden reserve questions. It covers intended meaning, target identification, sentiment reversal, sincere controls, and context dependence. The item quality fields are author checklists, not independent human validation.
Historical pilot
The May 2026 evaluation used five OpenRouter calls per model, temperature 0.7, top-p 0.95, and a system plus user prompt. The archived prompt accidentally repeated option E. Output limits were raised for a corrected rerun, but 69 length-limited responses remained and were counted wrong. The historical leaderboard is preserved with that limitation.
Current protocol
New models use v0.1.1-openrouter: one A–F option each, strict Final Answer parsing, five independent calls, max_completion_tokens 4096, pinned provider routing, and private-data routing controls. Technical failures or truncation make a model run unpublishable.
Scoring
Avg@5 is mean accuracy across five complete calls. Maj@5 uses the modal answer for each question, with tied modes scored wrong. Category scores, strict-format failures, consistency, costs, token use, latency, and seeded bootstrap confidence intervals are also recorded.
Reproducibility
Every new run records dataset and prompt hashes, exact settings, requested and returned model IDs, provider metadata, costs, environment versions, and artifact hashes. Private questions and raw outputs remain organizer-only; public manifests provide counts and cryptographic hashes.
Run and inspect the evaluator
The public repository contains the installable evaluator, immutable protocol files, schemas, historical audit, organizer guide, and test suite.
View code on GitHub