Two models rank OpenMath result #3 a breakthrough

Two model assessments of Epoch AI Research’s OpenMath results agree that result #3 qualifies as a “Breakthrough”, while at least seven results meet the organization’s “Major Advance” standard.

The comparison was posted by a Hacker News user who said the underlying mathematics was difficult for non-specialists to evaluate. They asked GPT-5.6 Sol Pro and Fable 5 Max to classify the results using OpenMath’s rubric.

How the OpenMath labels differ

The rubric defines three levels:

• Solid Result: A strong researcher in the relevant area would be satisfied if their median output addressed problems of this caliber, although the work would probably attract little attention outside its subfield.
• Major Advance: The median mathematician working in a broad area such as number theory or graph theory would take notice and likely spend time understanding at least the outline of the solution.
• Breakthrough: The median mathematician would want to know about the result even if it fell outside their specialty. It could be considered one of the year’s best results across mathematics.

Both models placed #3 in the top category. The post says that helps explain why Sébastien Bubeck opened his own post with that result.

The models differed on one classification. Fable 5 Max rated #7 as a Solid Result, while GPT-5.6 Sol Pro rated it a Major Advance. Despite that disagreement, both assessments identified at least seven Major Advances among the results.

Recommended reading

Sber opens SIRIN to catch errors in AI answers

Conservative scoring and borderline breakthroughs

The official OpenMath guidance says evaluators should choose the more conservative category when multiple tiers appear plausible:

“When multiple tiers seemed plausible for a problem, we erred in the conservative direction. It would be disappointing to downgrade a problem’s notability after it was solved, whereas we can always highlight any unexpectedly interesting elements of a solution.”

Fable’s assessment went further on three results: #1, #4, and #9 were judged “Borderline Breakthroughs.” The post asks AcerFur for an opinion, but provides no response or independent mathematical assessment.

The comparison also leaves open how the underlying results should ultimately be ranked by mathematicians. It reports model classifications, not a new evaluation of the proofs or their significance.