Statistical physics has studied exactly solvable models of neural networks since more than four decades. In this talk, we will put this line of work in perspective of recent empirical observations stemming from deep learning. We will describe several types of phase transition that appear in the limit of large sizes as a function of the amount of data. Discontinuous phase transitions are linked to adjacent algorithmic hardness. This so-called hard phase influences the behaviour of gradient-descent-like algorithms. We show a case where the hardness is mitigated by overparametrization proposing that the benefits of overparametrization may be linked to the usage of a certain type of algorithms. We then discuss the overconfidence of overparametrized neural networks and evaluate methods to mitigate it, and calibrate the uncertainty.