Deep generative models learn the distribution of training data, enabling to recognise the structures and patterns in it without requiring labels. Likelihood-based generative models, such as Variational Autoencoders (VAEs), flow-based models and autoregressive models, allow inferring the log-likelihood of a given data point and sampling from the learned distribution. A well-known fact about all of these models is that they can give higher log-likelihood values for structured out-of-distribution (OOD) data than for in-distribution data that they were trained on, rendering likelihood-based OOD detection infeasible. We provide further evidence for the hypothesis that this is due to a strong dependence on the counter-intuitive nature of volumes in the high-dimensional spaces under which one chooses to represent the input data, and provide theoretical results illustrating that the gradient of the log-likelihood is invariant under this choice of representation. We then present a first gradient-based anomaly detection method which exploits our theoretical results. Experimentally, our proposed method performs well on image-based OOD detection, illustrating its potential.