Recent advances in image de-raining have focused on training powerful models on mixed multiple datasets comprising diverse rain types and backgrounds. However, this approach tends to overlook the inherent differences between datasets, resulting in suboptimal optimization and poor generalization. To address this limitation, we propose an approach to learn instance-specific de-raining models by exploring meaningful representations that characterize both the rain and background components in rainy images. Leveraging these representations as instructive guidance, we put forth a Context-based Instance-specific Modulation (CoI-M) mechanism which can modulate CNN- or Transformer-based models. Furthermore, we develop a rain-/detail-aware contrastive learning strategy to help extract joint rain-/detail-aware instance-specific representations. By integrating CoI-M with the rain-/detail-aware Contrastive learning, we develop CoIC, an innovative and effective algorithm for training models on mixed datasets. Moreover, CoIC offers insight into modeling relationships of datasets, quantitatively assessing the impact of rain and details on restoration, and revealing different behaviors of models given diverse inputs. Extensive experiments validate the effectiveness of CoIC in boosting the de-raining ability of CNN- and Transformer-based models, as well as significantly improving their generalization ability.