Scattering scanning near-field optical microscopy (s-SNOM) provides spectroscopic imaging from molecular to quantum materials with few nanometer deep subdiffraction limited spatial resolution. However, in its conventional implementation s-SNOM is slow to effectively acquire a series of spatio-spectral images, especially with large fields of view. This problem is further exacerbated for weak resonance contrast or when using light sources with limited spectral irradiance. Indeed, the generally limited signal-to-noise ratio prevents sampling a weak signal at the Nyquist sampling rate. Here, we demonstrate how acquisition time and sampling rate can be significantly reduced by using compressed sampling, matrix completion, and adaptive random sampling, while maintaining or even enhancing the physical or chemical image content. We use fully sampled real data sets of molecular, biological, and quantum materials as ground-truth physical data and show how deep under-sampling with a corresponding reduction of acquisition time by 1 order of magnitude or more retains the core s-SNOM image information. We demonstrate that a sampling rate of up to 6× smaller than the Nyquist criterion can be applied, which would provide a 30-fold reduction in the data required under typical experimental conditions. Our smart s-SNOM approach is generally applicable and provides systematic full spatio-spectral s-SNOM imaging with a large field of view at high spectral resolution and reduced acquisition time.