The most dominant mode of oceanic climate variability on an interdecadal scale is the El Ni\~no Southern Oscillation (ENSO). ENSO events are characterized by anomalous sea surface temperatures (SSTs) in the equatorial Pacific affecting global climate, ecosystem, and society. The spatial structure and dynamics of SST fields associated to ENSO however show strong variability between different events, known as ENSO diversity. Whether this diversity consists of distinct types or merely a continuous process is to date an open question.Using Gaussian mixture variational autoencoders, we analyze the multi-modality of the non-linear low dimensional representation of SST anomalies in the tropical Pacific. We show that hard classification approaches are not suitable to describe ENSO diversity. Analyzing the low dimensional representations allows us to identify two main factors describing the spatial variability of ENSO, namely the average maximum temperature anomaly and the zonal gradient. We suggest a weighting of ENSO events based on these factors to yield more expressive composites that describe the variability of ENSO.