How scientists could check an AI’s route to a forgotten theorem

Since the previous research note · Earlier expectation not tested

Track continuity

Since the previous research note: When AI answers questions about the universe, who checks its work?

Previous expectation: A broader, human-graded cosmology test could reveal whether AI judges reliably evaluate research assistants' answers.

What this study adds: This conceptual preprint proposes a different check for scientific AI: remove a known result, test its reconstruction, and audit for leakage. It reports no test of AI judges or the earlier cosmology benchmark.

---

What changed in this study

A correct proof does not reveal how an AI reached it. A model asked to prove a familiar theorem might reason from basic rules—or retrieve the result through a remembered lemma, a paraphrase, or another chain of connected facts. The answer could look identical either way.

A position paper posted to arXiv on August 25, 2025, proposes a way to make that test harder to fake: deliberately target a result for removal, then ask the model to reconstruct it. The idea repurposes machine unlearning—the effort to make a model stop using specified information—as a probe of scientific reasoning. The paper outlines a possible benchmark; it does not report a completed test showing that an AI rediscovered a result.

A familiar answer can hide its route

Reproducing a known result is an ambiguous measure of capability. A model may produce a valid proof or working algorithm because it has learned the answer or pieces that lead to it. Fluency and correctness alone do not distinguish that route from building the result from permitted starting points.

The proposal treats unlearning as a kind of ablation: intentionally removing a target and relevant paths back to it, then testing what the model can reconstruct. In ordinary settings, unlearning is studied for goals such as privacy, copyright, or safety. Here, the proposed purpose is epistemic: to examine whether a system can generate a result after its familiar routes to that result have been weakened.

The routes matter. A theorem may depend on intermediate lemmas; the same knowledge may appear in paraphrased form or be reachable through several reasoning steps. A model that no longer gives the target when asked directly might still recover it through one of those alternatives. The paper therefore treats the quality of unlearning as part of the difficulty of the rediscovery task. The more faithfully the target knowledge and its routes are removed, the more informative a reconstruction attempt could be.

That condition is also a major challenge. Apparent forgetting may be only surface-level suppression: the model seems unable to recall something under one prompt, yet another prompt can bring it back. Knowledge can be entangled with other material, and later prompting or fine-tuning may restore use of content that seemed forgotten. A weak removal step would make success hard to interpret because the model might still have access to the answer indirectly.

Proofs and programs offer different checks

For mathematics, the paper sketches a task in which a target theorem is removed and the model is asked to prove it again from allowed axioms and rules of inference. A proof assistant can check whether the submitted proof follows those rules and accepts the theorem. This verifies the proof’s formal validity; it does not, by itself, show that the model avoided a hidden route to the result.

Algorithms provide another possible starting point. The paper gives the Knuth–Morris–Pratt string-matching algorithm as an example: a model would be asked to implement it after the algorithm and prerequisite knowledge had been targeted for unlearning. A hidden test suite could check whether the implementation behaves correctly on cases not disclosed to the model. Passing would establish performance on those tests, not how the model arrived at its solution.

These checks answer a necessary but limited question: is the result valid by an external standard? Proof acceptance and software tests make correctness less dependent on a human grader’s impression. They do not settle whether the model reconstructed the result or drew on material that survived the removal step. For that, the proposed evaluation needs separate audits for leakage.

Audits help make success interpretable

The paper proposes probing for leakage with paraphrases, multi-step routes, and prompts that ask for the same answer in another form, drawing on material connected to the target. If the model can recover the supposedly forgotten result through those routes, a correct final proof or implementation is not persuasive evidence of rediscovery. The benchmark would need to treat such recovery as a failure of the test’s premise.

A second control asks whether unlearning has damaged unrelated abilities. If a model performs worse across tasks that have nothing to do with the target, then a failed reconstruction could reflect broad degradation rather than a specific loss of the relevant knowledge. The paper suggests checking performance on unrelated benchmarks as one way to see whether general capability remains intact.

Together, verification, leakage audits, and checks on unrelated performance would make results easier to interpret. Each addresses a different ambiguity: whether the answer is correct, whether the target remains reachable through another route, and whether the removal process has impaired the model more broadly. The proposal does not present results from a systematic empirical validation of this combination.

Mathematics and algorithms are suggested as pilot domains because their outputs can be checked against explicit standards: proof assistants for proofs, hidden tests for implementations. The paper also points toward possible extensions in sciences such as physics and chemistry, where a model might be asked to re-derive an established result from more basic knowledge. Whether the same approach can produce a reliable test in those less formally bounded settings remains an open research question.

Where this research could lead

A dependable rediscovery benchmark could eventually help a research team choose which AI system merits investigation on a new scientific question. Two systems might offer equally polished answers; repeated reconstruction of withheld results, checked independently and paired with audits for detectable leakage, could provide a more concrete basis for judging whether either is doing more than retrieving familiar material.

That decision aid depends on the paper’s proposed combination of deliberate unlearning, independent verification, and leakage checks. Proof assistants and test suites already exist to check particular proofs and programs; they are not themselves benchmarks showing that an AI can rebuild knowledge after its routes to it have been removed. Extending that kind of evidence to genuinely new scientific discovery would require independent validation across tasks and domains.

A nearer milestone would be a completed mathematics or algorithms pilot that reports verified task success alongside leakage findings and checks that unrelated abilities remain intact. Such results could show whether the proposed test is workable in domains with clear correctness checks. They would be an early step toward evaluating scientific AI, not proof that it can make open-ended discoveries.

Key Sources

• Primary source: Unlearning as Ablation: Toward a Falsifiable Benchmark for Generative Scientific Discovery
  arXiv · Preprint · 2025-08-25