Poverty maps—spatial representations of economic wealth—are essential tools for governments and NGOs to adequately allocate infrastructure and services in places in need. They also help to better understand social phenomena such as human mobility and segregation, and environmental problems induced by urbanization. Traditionally, such maps are inferred from Census and survey data, which are expensive and collected occasionally; thus, they commonly provide outdated and low-resolution socioeconomic information, especially in developing countries. Remotely sensed data combined with advanced machine learning methods provided a recent breakthrough in poverty map inference. However, these models are not optimized to produce accountable results that guarantee accurate predictions to all sub-populations including the rich and the poor or the urban and rural divide. In this paper, we touch upon the opportunities that multimodal data can offer to solve these issues, as well as the challenges of working with noisy, biased and sparse datasets for predicting high-resolution poverty maps in Sierra Leone.