“We do not claim to fully understand the proof yet.” That sentence comes from the committee OpenAI appointed to vet its claimed Yang-Mills result, as quoted by Scientific American. The group, led by the physicist David Gross, had just affirmed that the proof of one half of the problem checks out in Lean, a programming language that checks mathematical work. It still couldn't say how the proof works.
We read the newest AI-for-science material: that article, five Anthropic posts, an OpenAI wet-lab benchmark, TechCrunch on a startup called PSI, and a blog post from a blogger whose AI agents flagged a Maya site missing from Mexico's official register. The headlines run well ahead of all of it. So we took six popular claims and checked each against what the sources actually say.
Claim 1: AI is now discovering new science on its own
Verdict: not on its own. The best cases are real, and in every one a person chose the question, ran the checks, or did both.
Start with biology. Anthropic says Claude agents found an enzyme system with CRISPR-like repeats, using roughly 950 agents, 210 million tokens and 21 hours (Anthropic). One agent spotted the repeat pattern. Anthropic's scientists supplied the starting prompt and did all of the lab work, most of Claude's candidates were eliminated in review, and the write-up is a preprint. Work on the system's primary function is still ongoing. The post says Claude “appears to be the first to notice” the system's defining features, and that Claude “can autonomously detect anomalies”: fair for the search stage, not for the rest. Feng Zhang of MIT and the Broad Institute, quoted in the same post after reading the preprint, called it “an exciting example of how AI agents can contribute to biological discovery” and said the finding “merits further investigation.”
The physicist Matthew Schwartz, a visiting researcher at Anthropic, is more candid than most. In a guest post for Anthropic he reports 30 integrals run end to end with Claude, 15 reproducing known results and 15 never computed before (Anthropic). He calls the good targets “Claude-shaped” problems, and he adds that “it can take a lot of hand-holding to get them to produce anything of scientific value.” After he brought in outside experts, he found that “in almost all cases, Claude was technically correct, but the result was not all that interesting until the expert helped steer us.” Claude, he says, “seems to favor old debates, highly cited but long forgotten.” Judgment about what's worth solving stayed with the humans.
Then PSI. TechCrunch reports that David Turturean, a member of the technical staff at the startup lab Physical Superintelligence (PSI), used early access to Anthropic's Fable 5.2 model to solve an 18-year-old problem from quantum computing, the Aaronson-Ambainis conjecture (TechCrunch). Scott Aaronson, who posed the problem, writes that he doesn't quite understand the proof but believes it's correct because it was autoformalized in Lean and explained by a team of top researchers PSI shared it with. Turturean's own description is modest: “It's a negative result that allows people to know, okay, we're no longer looking there.” The solution appears to close one road to quantum speedups, not all of them.
We'd call this a fast, tireless collaborator that needs someone else to pick the problem. That's a lot. It isn't autonomy, and the companies' own posts say so if you read to the caveats.
Claim 2: The results are verified
Verdict: partly. A formal checker is a strong test in mathematics. Elsewhere “verified” can mean anything from a simulation to a register lookup.
In mathematics, Lean can confirm mechanically that a proof follows from the statement it was given. That's a strong check, and it's why the PSI and OpenAI results are being taken seriously at all. But Scientific American's reporting shows the limits. OpenAI's claim covers the existence half of the Yang-Mills problem; the separate mass-gap half is unresolved. The committee used OpenAI's own Astra model to help digest the Lean code, and says outright that it doesn't claim to understand the proof yet.
Then there's openness. Scientific American notes that the shepherding approach contradicts transparency recommendations from an OpenAI-affiliated advisory group and has riled some mathematicians who see it as gatekeeping. The Lean proof is public now. NYU's Scott Armstrong said he was glad it was finally out and added: “This review can be done more efficiently and quickly in the open” (Scientific American). We agree with him. A proof that only OpenAI's appointed committee had seen was a claim waiting for review, and the review can finally start.
Outside mathematics there's no Lean. PSI's CEO Matt Pines told TechCrunch that physics differs from math because “nature gets a vote in your models of reality,” and that machine-driven theoretical physics is “not there yet, but it's on its way.” Schwartz found that “the automated checks still can't be trusted.” And the Maya post, which we'll come back to, got an official reply: the Mexican institute found no registered site at the reported location. That confirms an absence from a register. It doesn't confirm a site.

Claim 3: Agents can run a lab
Verdict: not yet. They can plan, design and analyze. People still pipette and solder.
Anthropic's electronics post is the strongest recent example, and it's worth reading in full. Working with the board maker Diode (Anthropic says it has a commercial relationship with the company), Claude (the Fable 5.2 model) designed a USB-C speakerphone, a voice-assistant hub and a handheld virtual pet, from schematic to layout to firmware (Anthropic). Diode's engineers manufactured the boards from Claude's design files and built and tested the prototypes by hand, because Claude wasn't connected to test equipment or to the machines that make circuit boards. The devices haven't been through electrical validation, antenna tuning, thermal testing or certification. The post also quotes benchmark gains (69% on Diode's internal PCB Gym layout test, up from 25% for the previous model), which are Anthropic's numbers on a test run at its request, and it shows cost only on a chart axis, not in the text.
On the biology side, OpenAI has published BioBasicsBench, a 100-task benchmark that it says measures how well AI agents interpret raw experimental data and handle everyday wet-lab science (OpenAI). It's a vendor-built test, and a benchmark is a claim until people outside the vendor run it.
Anthropic's usage study of 207,049 scientific sessions on consumer plans shows the same shape in the wild. Data gathering is 28.6% of researchers' time but only 11.5% of sessions on Claude, and the gap is widest where there are physical barriers like collecting clinical or biochemical data (Anthropic). The study's own words: “physical bottlenecks remain powerful.”
Claim 4: Anyone can do this now
Verdict: anyone can start. Getting something that survives takes someone who knows the field.
The Maya story is the cleanest test. Pietro Miotti, who says he's not an archaeologist, pointed 48 agents at about 1,200 square kilometers of public NASA lidar and got a hilltop site with a plaza and a pyramid, then an official reply, in under 72 hours (his post). His method line is admirably plain: “There was no special model or classifier involved: the agents literally looked at the pictures.”
Then the correction, which sits at the top of the page now. “The agents missed it while going through the literature, so they rediscovered it rather than discovered it.” The site was already mapped in a 2025 paper by Estrada-Belli and colleagues; it just wasn't in the Mexican register yet. Credit to Miotti for putting that first. He also says he doesn't speak Spanish and trusted the agents' translation of his report, the Classic-period story is “speculation, from a long chat with the agents,” and the reconstruction image is AI-generated. No ground survey is described.
Our reading: it's a good demonstration of cheap screening and a poor demonstration of discovery. Looking is now nearly free. Knowing whether anyone has looked before still takes a specialist, or at least a literature search the agents didn't do properly. PSI's route was different and instructive: it held the result until Anthropic's model was nearly ready for release, and used the time to show it to researchers who could explain it. Jim Halverson, a Northeastern professor advising PSI, told TechCrunch that “Best practices are still emerging.”
Claim 5: Scientists are about to be automated
Verdict: unproven, and the loudest claims come from people with a stake in them.
Turturean told TechCrunch “I'm totally accepting the idea that this might be the last paper I ever put out.” Pines said that six to nine months ago, when talking to investors, he “had to sort of stretch” to say theoretical physics would soon be mostly the domain of machines, and that it's “not there yet.” Anthropic's survey says about 40% of 528 researchers expect AI to carry out full research projects on its own by 2030, but they were recruited from applicants to Anthropic's own AI for Science grant programs, so they're likely heavier AI users than most (Anthropic).
The modeled numbers are smaller. Anthropic's modeling says current capabilities, widely adopted, could take on 27 to 52% of the researcher time needed for a new study, which implies output rising 1.35x in the arts up to 2.02x in statistics. The report's hypothetical scenario in which AI automates all non-physical work reaches about 15x in statistics and computer science, but only 3 to 4x in bench sciences like biology and chemistry, where physical steps stay the bottleneck. That's a scenario, not a measurement. Even the modest case is a big change for a field. It isn't replacement, and the report notes that AI work today is mostly assistive or shared. On who gets the credit, Schwartz puts it better than we would: it “won't be resolved by pretending the human contribution was the typing.”
Claim 6: If the AI's model fits the data, we understand the system
Verdict: false, and the authors building these systems say so.
A guest post from a Tübingen group (Jakob Macke, Daniel Gedon and Stefan Wahl, all Anthropic AI for Science grantees) on teaching AI to build scientific models makes the argument carefully. Many models fit the same data, so the useful output is a spread of plausible models and a list of what they agree on (Anthropic). They concede that these systems “have mostly been tested on rediscovering models that scientists already know,” with sparse evidence on problems that have no known answer. And they write: “An AI system that merely hands you the model might give you an answer, but it does not give you understanding.”
We think that last sentence should be taped above every lab bench and every student's desk. A fit is a number. An explanation is something a person can use to predict what happens next.
A routine for researchers and students
None of the above says don't use these tools. Anthropic's usage data suggests scientists already use them where they work best, mostly analysis and writing. Here's the routine we'd follow, built from what went wrong in the cases above.

- Give agents work whose output you can check yourself: data analysis, porting code between languages, screening papers or images. Claude Code and Claude Cowork are built for that kind of job, and our terminal coding agents comparison covers the field.
- Decide before you start what counts as a check and who does it. Not the same model. A formal verifier, a second analysis, a bench experiment or a colleague.
- When an agent says something is new, search the literature yourself. A dedicated tool such as Elicit, Consensus, SciSpace or ResearchRabbit takes minutes, and the Maya case shows the cost of skipping it. Our ranking of AI research assistants has the longer list.
- Keep the physical step human and logged. If a result depends on a measurement, a sample or a site visit, that's where it gets tested.
- Say what the agent did when you write it up, and what you checked. Schwartz is blunt that the human contribution wasn't the typing.
- Students: build your first models yourself. The Tübingen authors write that “it is the process of constructing it that provides the insight.” Use an agent to critique your model, not to hand you one.
If you want the older, better-tested example of AI doing biology, it's still AlphaFold, which predicts structures that labs then test. For literature work and agent-style research there are also Edison Scientific, Scientific Agent Skills for equipping your own agents, and the Top 250 AI Tools for Researchers board. For the general-purpose assistants, ChatGPT vs Claude vs Gemini sets ChatGPT against its two main rivals, while the AI research assistants and AI agents boards rank the specialist tools.
Where we land, and what would change our minds
Real: agents are now useful, tireless analysts and engineers. The electronics prototypes work, the enzyme lead is interesting, and a Lean-checked proof from a model is a real result, whoever ends up understanding it first. Schwartz's 15 new integrals are real results in a narrow field.
Hype: autonomy, “verified” used loosely, and a coming wave of automated scientists. Nearly every impressive result above has a human in the loop doing the part that decides whether it's true, and the humans say so. We'd also like the labs to print the dollar cost next to the benchmark score. Credits and token counts hide what a discovery costs, and 210 million tokens for one enzyme search is a number a lab budget cares about.
Two things would move us. One is mathematicians outside OpenAI's committee reading the Yang-Mills Lean code and reporting that they understand it. The other is a peer-reviewed version of the enzyme work that says what the system does. Until then we're treating both as promising and unfinished.







