The stochastic block model (SBM) is widely studied as a benchmark for graph clustering aka community detection. In practice, graph data often come with node attributes that bear additional information about the communities. Previous works modelled such data by considering that the node attributes are generated from the node community memberships. In this work, motivated by recent surge of works in signal processing using deep neural networks as priors, we propose to model the communities as being determined by the node attributes rather than the opposite. We define the corresponding model; that we call the neural-prior SBM. We propose an algorithm, stemming from statistical physics, based on a combination of belief propagation and approximate message passing. We argue it achieves Bayes-optimal performance for the considered setting. The proposed model and algorithm can hence be used as a benchmark for both theory and algorithms. To illustrate this, we compare the optimal performances to the performance of a simple graph convolution network.