Molecular representation learning is a fundamental task for AI-based drug design and discovery. Self-supervised contrastive learning on molecular graphs, which aims to learn good representations via semantic-preserving transformations, is an attractive framework for this task. However, it is relatively under-explored to design such transformations for molecules under consideration of their chemical semantics. In this paper, we consider fragmentation which decomposes a molecule into a set of chemically meaningful fragments (e.g., functional groups) as the semantic-preserving transformation. Here, we also utilize the 3D geometric views of molecules as another source of such transformation. Based on these molecule-specialized semantic-preserving transformations, we propose fragment-based multi-view molecular contrastive learning (FragCL), an effective framework that learns chemically meaningful molecular representations. Through extensive experiments, we demonstrate that our framework outperforms prior molecular representation learning methods across various molecular property prediction tasks.