Mouhamadou Makhtar Fall,
Samia Ainouz,
Pierre-Jean Lapray and
Jean-Philippe Tarel
Abstract
Larger learning rates are frequently associated with better generalization, as they encourage exploration of flatter minima in the loss landscape. In this work, we show that the large learning rate phase produces predictive distributions that naturally capture dataset-dependent uncertainty and structured relationships between classes.
Across several datasets, we observe that models trained with a large learning rate exhibit significantly lower calibration error early in training, indicating that large learning rate dynamics generate probability distributions that are more faithful to the intrinsic uncertainty of the data labels.
Building on this observation, we introduce Learning Rate Informed Priors (LRIP), a simple training strategy that reuses the predictive distributions obtained at the end of the large learning rate phase as improved supervision targets. LRIP replaces uniform or heuristic smoothing with sample-dependent priors extracted directly from the model when its predictions are the most calibrated.
Extensive experiments across different modalities and learning-rate schedules demonstrate that LRIP consistently improves calibration while maintaining competitive accuracy. These findings highlight a new perspective on the role of large learning rates: beyond optimization benefits, they induce informative predictive uncertainty that can be directly leveraged to build more reliable and better-calibrated models.
Reference
@InProceedings{ jpt-icpr26b,
author = {Fall, M.-M. and Ainouz, S. and Lapray, P.-J. and Tarel, J.-P.},
title = {Learning Rate Informed Priors for Neural Network Calibration},
booktitle = {International Conference on Pattern Recognition (ICPR'26), Part VI},
date = {August 17-22},
address = {Lyon, France},
pages = {112-126},
year = {2026},
url = {http://perso.lcpc.fr/tarel.jean-philippe/publis/icpr26b.html}
}