Crystal Structure Prediction (CSP) is crucial in various scientific disciplines. Existing learning-based generative approaches seldom capture the full symmetries of the crystal structure distribution---the invariance of translation, rotation, and periodicity. In this paper, we propose DiffCSP, a novel diffusion method to learn the stable structure distribution from data, incorporating the above symmetries. To be specific, DiffCSP jointly generates the lattice and the fractional coordinates of all atoms by employing a periodic-E(3)-equivariant denoising model to better model the crystal geometry. Notably, DiffCSP leverages fractional coordinates other than traditional Cartesian coordinates to represent crystals, remarkably promoting the diffusion and the generation process of atom positions. Extensive experiments on crystal structure prediction verify the effectiveness of DiffCSP against existing learning-based counterparts.