npj Computational Materials
2026
Shared — curated by The Gufo editorial team
Surprisingly high redundancy in electronic structure data across materials explained by low intrinsic dimensionality
Sazzad Hossain*, Ponkrshnan Thiagarajan, Shashank Pathrudkar, Stephanie Taylor, Abhijeet Sadashiv Gangan, Amartya S. Banerjee, Susanta Ghosh
Received: 2026-10-22 · Published: 2026-10-19
© 2026 The Author(s) and the original publisher.
Abstract
Abstract Machine learning (ML) models for electronic structure typically rely on large datasets generated by computationally expensive Kohn-Sham density functional theory calculations, as it is not known a priori which portions of the data are essential for accurate learning. Here, we reveal significant redundancies in electronic structure datasets across diverse material systems and attribute them to the low intrinsic dimensionality of the underlying data. We show that even random pruning can substantially reduce dataset size with minimal degradation in predictive accuracy. Moreover, a state-of-the-art coverage-based pruning strategy that samples data across all learning difficulties almost always preserves chemical accuracy and maintains model generalizability while using up to two orders of magnitude less data and reducing training time by a factor of three or more. We further demonstrate that the essential electronic structure information lies on a low-dimensional, non-linear manifold, providing a potential geometric explanation for the observed prunability. These observations are consistent with the predominance of local atomic environments in determining electronic properties, as suggested by nearsightedness arguments, and indicate that large-scale datasets may contain highly overlapping information. Our findings challenge the prevailing assumption that such extensive datasets are necessary for accurate ML-based electronic structure predictions and open a path toward identifying minimal, representative datasets for each material class.
Summary
Machine learning models that predict how electrons are arranged in materials usually need huge training datasets generated by expensive quantum calculations, since researchers typically do not know ahead of time which data points actually matter. This paper investigates whether these large datasets are truly necessary, finding that they contain substantial redundancy across many different material systems.
The authors show that even randomly removing large portions of the data barely hurts prediction accuracy, and that a smarter pruning method, which samples examples across all levels of learning difficulty, can preserve chemical accuracy while using far less data and training much faster. They trace this redundancy to the fact that the essential electronic structure information actually lies on a low-dimensional, curved surface within the larger dataset. This fits with the idea that local atomic surroundings, rather than the full extended system, largely determine electronic properties, meaning large datasets often repeat overlapping information rather than adding new insight.
Plain-language summary by The Gufo; the paper is the authors’ own text.
Key findings
- Electronic structure datasets across diverse materials contain significant redundancy, allowing substantial size reduction without major accuracy loss.
- Random pruning of training data can greatly shrink dataset size while only minimally affecting predictive accuracy.
- A coverage-based pruning strategy that samples across all learning difficulties maintains chemical accuracy and model generalizability while using up to two orders of magnitude less data and cutting training time by a factor of three or more.
- The essential electronic structure information resides on a low-dimensional, non-linear manifold, offering a geometric explanation for this prunability and aligning with nearsightedness arguments favoring local atomic environments.
