SarcBench: A Pilot Benchmark for Sarcasm Understanding in Language Models

Sarcasm is a form of meaning mismatch. A person may say something positive while clearly meaning something negative, or say something mild while implying frustration, criticism, or disbelief. This makes sarcasm difficult for language models because the literal words are often misleading. A model has to use context, speaker intent, social expectations, and common sense to decide what is actually being communicated.

Earlier work on sarcasm detection has often treated the problem as binary classification: given a sentence, decide whether it is sarcastic or not sarcastic. This is useful, but it does not fully test whether a model understands the meaning behind the sarcastic statement. A model can sometimes detect that sarcasm is present without correctly explaining what the speaker means, who the sarcasm targets, or why the context makes the statement non-literal.

This was the motivation for SarcBench. Rather than asking only “is this sarcastic?”, SarcBench asks a more practical question: can the model infer what the speaker actually means?

This shift matters because real communication is rarely just a yes-or-no sarcasm label. A sarcastic comment usually has a target, a reason, and an intended meaning. Sincere comments can also look sarcastic if a model overreacts to certain words. For example, a phrase like “great timing” can be sarcastic when something goes wrong, but sincere when something works out at the perfect moment. SarcBench is designed to test that distinction.

Sarcasm has been studied in natural language processing for years, often through datasets that ask models to classify text as sarcastic or not sarcastic [2]. Sarcasm Corpus V2 is one example. It is based on the Internet Argument Corpus and includes posts annotated for sarcasm across general sarcasm, hyperbole, and rhetorical questions [4, 8]. Its general sarcasm subset contains 3,260 sarcastic posts and 3,260 non-sarcastic posts [8].

Previous sarcasm datasets are valuable, but they also show why the task is complicated. Sarcasm Corpus V2 contains binary labels and comes from online debate forums, which can make the language domain specific. A 2024 survey-style analysis of sarcasm datasets notes that Sarcasm Corpus V2 is relatively large and context-aware during annotation, but also points out that the released dataset itself does not provide the original contexts and is shaped by its online debate source domain [9].

This project also builds on an earlier study comparing ChatGPT-4, ChatGPT-3.5, and Bard on sarcasm detection. That study tested 100 sentences, split evenly between sarcastic and non-sarcastic examples, using the prompt “Is the following sentence sarcastic or not? Please provide a brief explanation for your answer.” The results showed that models differed in how they handled sarcastic and non-sarcastic sentences, with ChatGPT-4 and Bard showing more balanced behavior and ChatGPT-3.5 performing better on non-sarcastic cases than sarcastic ones [7].

SarcBench is meant to move beyond that earlier binary setup. Instead of measuring whether a model can classify sarcasm, it measures whether the model can choose the intended interpretation from plausible alternatives.

SarcBench is also inspired by benchmark design choices from recent language model evaluations. MMLU helped popularize broad multiple-choice testing across many knowledge areas, and its public repository describes it as the ICLR 2021 benchmark “Measuring Massive Multitask Language Understanding” [1, 10]. BIG-bench used a large collection of tasks to probe abilities believed to be difficult for language models, with 204 tasks contributed by hundreds of authors [6]. HELM emphasized transparent and standardized evaluation, comparing models across shared scenarios and reporting more than one metric [3].

The closest design influence for SarcBench is SimpleBench. SimpleBench uses handcrafted six-option multiple-choice questions, keeps part of the benchmark private to reduce contamination, evaluates models over five runs, and reports both average accuracy and majority-vote accuracy [5]. SarcBench uses a similar scoring style, but focuses specifically on sarcasm and non-literal meaning.

SarcBench v0.1 is a six-option multiple-choice benchmark. Each item includes a short context, an utterance, a question, six possible answers labeled A through F, and one correct answer. The goal is not simply to detect sarcasm. The goal is to identify the meaning that best fits the situation.

The private scored set contains 125 questions. The questions are grouped into five categories:

The v0.1 evaluation used the hidden 125-question private set. Each model was run five times on the same questions using the same standard zero-shot prompt. The prompt instructed the model to answer the multiple-choice question and output only the final answer in the form:

where X is A, B, C, D, E, or F. The evaluation script then extracted the answer letter and compared it with the correct answer. A response was counted as correct only if the extracted answer matched the ground truth. The raw result logs include the model ID, returned model name when available, question ID, category, correct answer, predicted answer, correctness, token usage, cost, latency, and any error.

The main metrics are:

Because SarcBench uses six answer choices, random guessing gives an expected Avg@5 score of about 16.7%. This is the same chance baseline used in other six-option multiple-choice benchmarks such as SimpleBench.

The reported runs used the benchmark script defaults unless changed in the command line: temperature 0.7, top-p 0.95, and standard zero-shot prompting. For most models, the final official run used a 128-token output limit. Four models were later rerun with a larger output limit because their earlier scores were affected by answer truncation. The final merged leaderboard uses the corrected rerun for Gemini 3.1 Pro Preview, DeepSeek V4 Flash, DeepSeek V4 Pro, and Kimi K2.6.

The final v0.1 pilot leaderboard combines two clean runs. The first run evaluated nine models and produced valid scores for the top group, including Claude Opus 4.7, Claude Sonnet 4.6, Gemini 3 Flash Preview, GPT-5.5, and Grok 4.1 Fast. The second run corrected the models that had been affected by output-length issues: Gemini 3.1 Pro Preview, DeepSeek V4 Flash, DeepSeek V4 Pro, and Kimi K2.6.

The top result is not a single model winning by a small margin. Instead, five models hit the ceiling of the benchmark. Claude Opus 4.7, Claude Sonnet 4.6, Gemini 3 Flash Preview, GPT-5.5, and Gemini 3.1 Pro Preview all scored 100.0% on both Avg@5 and Maj@5. DeepSeek V4 Flash and DeepSeek V4 Pro also scored extremely high after rerunning with a larger output limit. Kimi K2.6 was the only evaluated model with a noticeably lower score, especially on sincere control.

The corrected category scores show the same pattern. Gemini 3.1 Pro Preview scored 100.0% in every category. DeepSeek V4 Flash scored 100.0% on intended meaning, target identification, sentiment reversal, and sincere control, with 98.0% on context dependence. DeepSeek V4 Pro scored 100.0% on intended meaning, target identification, and sentiment reversal, with 96.0% on sincere control and context dependence. Kimi K2.6 scored 98.0% on intended meaning and 100.0% on sentiment reversal, but dropped to 68.0% on sincere control.

The main finding is simple: SarcBench v0.1 is too easy for the strongest current models.

That does not mean the benchmark failed. In a pilot release, this is useful information. The evaluation code worked, the answer extraction worked, the scoring format worked, and the benchmark produced a clean leaderboard. The problem is that the private set did not create enough separation at the top of the model field.

This ceiling effect matters. If several frontier models score 100%, then the benchmark cannot tell us which of those models has better sarcasm understanding. It can still show that weaker models may struggle with certain categories, but it cannot yet serve as a hard frontier benchmark.

The results also suggest that modern models are much better at this specific format than older sarcasm-detection studies might lead us to expect. The earlier binary sarcasm-detection work found that ChatGPT-4, ChatGPT-3.5, and Bard made different kinds of mistakes on sarcastic and non-sarcastic examples. In contrast, the current SarcBench v0.1 results show that many newer models can solve short, well-structured sarcasm interpretation questions very reliably.

This does not prove that sarcasm understanding is solved. SarcBench v0.1 uses short written contexts, clean answer choices, and controlled scenarios. Real sarcasm can depend on voice, relationship history, shared memory, timing, culture, or prior conversation. A multiple-choice benchmark can test one important part of sarcasm understanding, but not all of it.

The most useful interpretation is that SarcBench v0.1 validates the benchmark format, but not the benchmark difficulty. The next version should keep the same basic structure while making the private set harder.

SarcBench v0.1 has several important limitations.

First, there is no formal human baseline yet. This means the current leaderboard should be described as a model-only pilot evaluation. It should not claim that models are above or below human performance. Human comparison should remain marked as pending until human participants complete the same private set under controlled instructions.

Second, the current private set is too easy for top models. Five evaluated models reached 100.0% Avg@5 and Maj@5. A benchmark with that many perfect scores is useful as a pilot, but not as a final frontier evaluation.

Third, the dataset is still small. The scored private set contains 125 questions. That is enough to test the pipeline and publish a pilot leaderboard, but a stronger benchmark should include more questions to reduce noise and improve category-level reliability.

Fourth, multiple-choice evaluation makes grading clean, but it also makes the task easier. The correct answer is always visible among the choices. A model may succeed by eliminating bad answers rather than fully generating the intended meaning itself.

Fifth, the current questions are written in clear, compact English. That helps with fairness and scoring, but it also removes many features that make sarcasm hard in the real world, such as long conversation history, ambiguous speaker relationships, cultural references, and tone.

Sixth, the item validation process is not yet as strong as it should be for a final benchmark. The current items include internal quality checks, but they have not yet gone through independent human agreement testing. A future version should measure whether human annotators reliably choose the same answer.

The next version of SarcBench should focus on making the benchmark harder, not larger for its own sake.

A stronger v0.2 should include more examples where the literal answer is tempting, but wrong. It should include sincere comments that look sarcastic, sarcastic comments without obvious cue words, and paired examples where one small context change flips the answer. It should also include more target-identification questions where multiple people or objects are plausible targets.

A good v0.2 should also include human validation. At minimum, each private question should be answered by several human participants. Items should be kept only when humans agree on the correct answer at a high rate. This would make the ground truth more defensible and allow the leaderboard to report a real human baseline.

Another useful direction is an open-ended version. Instead of giving six choices, the model could be asked to explain what the speaker means in one sentence. That would be harder to grade, but it would better match real-world interpretation. One compromise is to keep multiple-choice scoring for the official leaderboard while also collecting open-ended outputs for qualitative analysis.

Finally, SarcBench should be treated as a living benchmark. HELM argues for transparent and continuously updated evaluation, including shared scenarios and standardized model comparisons [3]. SarcBench can follow a similar idea on a smaller scale by keeping a stable public set, a hidden private set, a reserve set for future refreshes, and clear version numbers for every leaderboard release.

SarcBench v0.1 introduces a focused benchmark for testing sarcasm understanding in language models. Its main idea is simple: sarcasm evaluation should not stop at detecting whether sarcasm is present. A useful benchmark should ask whether the model understands what the speaker actually means.

The pilot results show that the current version is not difficult enough for the strongest models. Several frontier systems reached perfect scores, and two more came very close. This means SarcBench v0.1 should be presented honestly as a pilot leaderboard, not as a final proof that models struggle with sarcasm.

Even with that limitation, the project has a clear contribution. It defines a sarcasm-specific multiple- choice format, separates the task into meaningful categories, evaluates models over repeated runs, and produces a reproducible leaderboard. The next step is to build a harder v0.2 with stronger human validation and a formal human baseline.

SarcBench v0.1 answers one question clearly: the pipeline works. The next version should answer a harder question: where do strong models still misunderstand what people really mean?