Jointly harnessing complementary features of multi-modal input data in a commonlatent space has been found to be beneficial long ago. However, the influence ofeach modality on the model’s decision remains a puzzle. This study proposesa deep learning framework for the modality-level interpretation of multimodalearth observation data in an end-to-end fashion. While leveraging an explainablemachine learning method, namely Occlusion Sensitivity, the proposed frameworkinvestigates the influence of modalities under an early-fusion scenario in whichthe modalities are fused before the learning process. We show that the task ofwilderness mapping largely benefits from auxiliary data such as land cover andnight time light data.