We consider the general class of time-homogeneous stochastic dynamical systems,both discrete and continuous, and study the problem of learning a representationof the state that faithfully captures its dynamics. This is instrumental to learn thetransfer operator of the system, that in turn can be used for numerous tasks, suchas forecasting and interpreting the system dynamics. We show that the searchfor a good representation can be cast as an optimization problem over neuralnetworks. Our approach is supported by recent results in statistical learning theory,highlighting the role of approximation error and metric distortion in the context oftransfer operator regression. The objective function we propose is associated withprojection operators from the representation space to the data space, overcomesmetric distortion, and can be empirically estimated from data. In the discrete timesetting, we further derive a relaxed objective function that is differentiable andnumerically well-conditioned. We compare our method against state-of-the-artapproaches on different datasets, showing better performance across the board