Deep learning-based methods are state-of-the-art methods for the semantic segmentation of dwellings. However, their performance can severely drop when they are used outside of the trained domain, which is often the case for rapid segmentation tasks that appear as a consequence of forced population displacement in cases of disasters, conflicts or political instabilities. Unsupervised Domain Adaptation has been proposed as a possible solution for such an issue as it tries to adapt a classifier trained on a specified domain with labels to help predict in a different domain without labels. Inspired by recent success of optimal transport in the context of domain adaptation, we propose a new unsupervised domain adaptation technique for semantic segmentation (SegJUMBOT). This method addresses the domain shift problem by leveraging the unbalanced minibatch-based optimal transport framework in the case of semantic segmentation of large remote sensing datasets. We apply our novel methodology to a challenging adaptation problem where we leverage a standard building detection dataset (INRIA Aerial Image Labelling Dataset) acquired over European cities to detect footprint of buildings in the Bangladesh site of Kutupalong. In our experiments, our method compares positively to other optimal transport based methods.