Transfer learning, which involves adapting a pre-trained model to perform a downstream task, is a widely used paradigm in machine learning. However, traditional transfer learning methods are typically designed for scenarios where fine-tuning data is abundant. Adapting such methods to the few-shot regime can be challenging because the quantity of data is limited compared to the model's capacity. In this work, we present a method called Adaptive Weight-space Ensembling (AWE) that demonstrates the effectiveness of weight-space ensembling, originally designed for large-scale data, in the few-shot setting. We achieve this by leveraging patterns in oracle weight-space ensembling to develop an adaptive ensembling method that can easily be deployed in practice. Our method achieves state-of-the-art results by more than 2% on average on standard few-shot setting benchmarks.