High-dimensional nonlinear systems such as atmospheric or oceanic flows present a computational challenge for data assimilation (DA) algorithms such as Kalman filters. A potential solution is to rely on a reduced-order model (ROM) of the dynamics. However, ROMs are prone to large errors, which negatively affects the accuracy of the resulting forecast. Here, we introduce the reinforcement learning reduced-order estimator (RL-ROE), a ROM-based data assimilation algorithm in which the correction term that takes in the measurement data is given by a nonlinear stochastic policy trained through reinforcement learning. The flexibility of the nonlinear policy enables the RL-ROE to compensate for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. We show that the trained RL-ROE is able to outperform a Kalman filter designed using the same ROM, and displays robust estimation performance with respect to different reference trajectories and initial state estimates.