A 10,000-agent AI effort may have cracked a Millennium Prize problem and opened a deeper debate over how mathematics gets done.

Mathematicians at OpenAI showed that the Navier-Stokes equations, which describe how fluids flow, can sometimes “blow up.” But the massive result is not without controversy. On the morning of Tuesday, ...