The broken inversion symmetry and strong spin-orbit coupling of transition metal dichalcogenides and the resulting spin-valley locking are the foundation for potential two-dimensional spin devices and valleytronics. However, ultrafast valley polarization is difficult to characterize due to the need to image both intra- and inter-valley scattering across the Brillouin zone. Here, we present an all-optical method for measuring the rate of ultrafast spin- and momentum-conserving scattering through circularly polarized time-resolved pump-probe second-harmonic spectroscopy. With Wollaston-prism-based detection for facile and simultaneous measurement of both K and K’ valley populations, using the example of exfoliated tungsten diselenide monolayers we resolve intra-valley and inter-valley scattering rates of 460 fs and 20 fs, respectively. These ultrafast time scales indicate rapid room temperature valley depolarization through either intervalley exchange coupling or the Elliot-Yafet mechanism, and are consistent with recent time-resolved molybdenum diselenide and tungsten disulfide results.