The upcoming Daniel K. Inouye Solar Telescope (DKIST) will produce unprecedented high-cadence, high-resolution, and multi-line spectropolarimetric observations of the Sun. New computational techniques are needed to infer the state of the Sun's atmosphere from these observations. Deep learning is a promising approach to this spectropolarimetric inversion problem that can both provide real-time visualizations to astronomers and potentially improve upon existing algorithms by combining spatial, temporal, and multi-spectral information. Here we investigate group equivariant deep learning as a method for inferring the three-dimensional photospheric structures, training on magnetohydrodynamic (MHD) simulations of two types of solar features: sunspots and active regions. Our results demonstrate that including multiple lines improves the mean relative error from 18.6% to 14.4%, averaged over all MHD state variables, and that using group equivariant convolution architectures further improves the mean relative error to 12.5%.