Diffusion models are generative models that have recently demonstrated impressive performances in terms of sampling quality and density estimation in high dimensions. They rely on a forward continuous diffusion process and a backward continuous denoising process, which can be described by a time-dependent vector field and is used as a generative model. In the original formulation of the diffusion model, this vector field is assumed to be the score function (i.e. it is the gradient of the log-probability at a given time in the diffusion process). Curiously, on the practical side, most studies on diffusion models implement this vector field as a neural network function and do not constrain it be the gradient of some energy function (that is, most studies do not constrain the vector field to be conservative). Even though some studies investigated empirically whether such a constraint will lead to a performance gain with contradicting results, they lack analytical evidence. Here, we provide three analytical results regarding the extent of the modeling freedom of this vector field. Firstly, we show that to obtain exact density estimation and exact sampling, it is neither necessary nor sufficient to assume the vector field to be conservative. Secondly, we derive the full (gauge) freedom satisfied by the vector field. Finally, we show that when it comes to inferring local information of the data manifold, conservativity is sufficient. In particular, we provide a novel algorithm to infer the intrinsic dimensionality of manifolds based on diffusion models.