Artificial neural networks have demonstrated exemplary learning capabilities in a wide range of tasks, including computer vision, natural language processing and, most recently, graph-based learning. Many of the advances in deep learning have been made possible by the large design-space for neural network architectures. We believe that this diversity in architectures may lead to novel and emergent learning capabilities, especially when architectures are connected into a collective system. In this work, we outline a form of neural network collectives (NNC), motivated by recent work in the field of collective intelligence, and give details about the specific sub-components that an NNC may have.