Research Signal – Source date 2022-06-01
A neural network can reveal a promising pattern without giving scientists an understandable account of how it found it. That creates an apparent problem for research. If the reasoning is inaccessible, what makes a discovery reached with its help worth believing?
The 2022 preprint Deep Learning Opacity in Scientific Discovery argues that the answer depends on the model’s place in the research process. This is a philosophical argument illustrated by two cases from scientific literature, one involving mathematical knots and the other involving aftershock locations. Its contribution is an account of how difficult-to-interpret AI can participate in discoveries that scientists have grounds to accept.
The underlying puzzle is subtler than hidden computer code. A trained network’s weights—the numerical values governing its connections—can be available for inspection, and its calculations can be precisely specified. Yet those calculations may still resist a meaningful scientific interpretation. Knowing which numbers were multiplied does not necessarily explain which mathematical relationship or physical process the model has captured.
The paper therefore asks readers to look beyond the network itself. A model’s output may be one step in an investigation whose eventual result rests on reasoning and evidence developed elsewhere in that process.
A reason to investigate and a reason to believe
An intriguing pattern can make a question worth pursuing before anyone knows whether the suggested answer is correct. Researchers then need grounds for accepting the claim they develop. These are different jobs, even when they occur within the same investigation.
Philosophers describe the distinction as the “context of discovery” and the “context of justification.” The first concerns how researchers arrive at ideas; the second concerns what warrants accepting them. An opaque model can contribute to the first by exposing a relationship, directing attention toward particular inputs, or producing a pattern that invites explanation.
Consider a network trained to predict one property from several others. Successful prediction gives researchers a reason to examine possible connections. They can study how its behavior changes with the inputs, refine their question, and formulate a claim that can be assessed using the standards of their discipline. In mathematics, that may involve proof. In empirical science, it involves support for claims about the world.
The author argues that this wider sequence matters when judging the two cases. The resulting findings need not depend for their justification on understanding the network’s internal reasoning or treating its predictive accuracy as proof. The model can be useful even when the account scientists ultimately defend takes a very different form from its calculations.
Narrowing the search in mathematical knots
The mathematical example concerns relationships between geometric properties of knots and a numerical property called the signature. Mathematical knots are closed loops studied through properties that remain meaningful across different descriptions of the same knot. Finding connections between different kinds of properties can expose structure that was previously difficult to see.
In the case discussed by the preprint, researchers trained a neural network to predict knot signatures from geometric quantities known as hyperbolic invariants. The prediction task provided a way to investigate a suspected connection between geometric and algebraic descriptions.
The next step made the exercise more useful than a prediction score alone. Researchers used an input-sensitivity procedure to identify which geometric quantities contributed most to the model’s predictive performance. The procedure examined gradients, which describe how a calculated quantity changes as an input changes. This helped focus attention on a smaller set of properties when formulating a conjecture.
That narrowing matters because an open-ended mathematical search can contain many plausible relationships. A suggestion about which quantities deserve attention gives mathematicians a more manageable problem. It does not require them to translate every layer of the network into a coherent mathematical explanation.
Nor should sensitivity be mistaken for the theorem itself. An input that matters to a model is a candidate for closer mathematical examination. The value of the technique lies in making that examination more focused, helping researchers move from a broad suspicion toward a relationship precise enough to investigate formally.
Reading patterns in aftershock locations
The aftershock case changes the kind of information researchers inspect. Here, the model’s useful output is a spatial pattern rather than a shortlist of mathematical inputs.
As described in the preprint, researchers represented earthquake events using a grid of spatial cells and trained a deep learning model to investigate how aftershock locations relate to mainshocks. Its predictions produced probability patterns indicating where aftershocks were more or less likely within the representation.
Those patterns could then be compared with spatial distributions associated with known geophysical properties. The comparison offered a way to ask which physical quantities might help account for the distribution of aftershocks and guide revisions to existing theory.
The scientific work therefore continued through a translation between two kinds of description. The network supplied a distribution of predicted probabilities. Researchers examined how that distribution corresponded to patterns associated with physical properties. A resemblance could direct attention toward a candidate explanation and help make a theoretical question more specific.
In the author’s account, this is a route by which predictive modeling can contribute to theory development. It concerns understanding aftershock patterns; it should not be read as a demonstration of operational earthquake warnings or reduced earthquake harm.
Partial insight still matters
There is a substantive objection to drawing too sharp a line around the knot example. Identifying influential inputs already looks like a form of interpretation. The preprint acknowledges that the saliency procedure can be viewed as an interpretive step.
That concession helps clarify what kind of understanding is at issue. Researchers may gain useful insight into what a network responds to without acquiring an intelligible account of the whole network’s reasoning. Such partial insight can be enough to guide a search. The case therefore supports a more specific conclusion than the claim that interpretation played no part at all.
The two examples also have different standards of success. A mathematical conjecture and a proposed account of aftershock distributions face different demands for justification. Their shared use of deep learning does not turn them into a common benchmark for AI-assisted science.
The paper’s argument consequently has limits. It does not establish that every claim reached with an opaque model can be justified independently of that model. Nor does it make interpretability unnecessary across research or high-stakes decisions, where accountability and an explanation of a particular output may themselves be essential. The cases show why the role of opacity must be assessed in the actual investigation, rather than decided from the network’s complexity alone.
What this could change
From this 2022 perspective, the possibility is that deep learning could expand the range of relationships scientists can productively investigate. In mathematics, directing attention toward a small set of relevant properties could help researchers uncover connections that become precise conjectures and, if successful, provable results. The gain would be access to mathematical structure that a less focused search might struggle to reveal.
In physical science, model-generated patterns could help researchers recognize promising connections between observations and candidate physical explanations. If those connections survive independent scientific assessment, they could support better accounts of phenomena such as aftershock distributions.
These possibilities depend on researchers being able to carry the work beyond a model’s suggestive output. When that bridge can be built, an AI system whose reasoning remains difficult to interpret could still help enlarge what humans can explain and defend as scientific knowledge.
Key Sources
- Primary source: Deep Learning Opacity in Scientific Discovery
arXiv · Preprint · 2022-06-01
Keep exploring