The interaction of light with a metal mediated by surface plasmon polaritons provides for sub-diffraction limited optical confinement and control. While the relationship of the linear plasmon response to the underlying elementary electronic excitations of the metal is well understood in general, the corresponding ultrafast and nonlinear plasmon interactions could provide further enhanced functionalities. However, while the ultrafast and nonlinear optics of metals is an advanced field, the understanding of the related plasmonic properties is less developed. Here we discuss ultrafast and nonlinear wave-mixing properties of metals and metallic nanostructures in terms of the elementary optical interactions related to electronic band structure, plasmon resonances, and geometric selection rules. These properties form the fundamental basis of the nonlinear plasmonic light-matter interaction. The understanding of these fundamental properties, together with the ability to measure and control the typically fast femtosecond intrinsic and extrinsic dephasing times, is important for the development of applications such as enhanced nano-imaging, coherent control of individual quantum systems, strong light-matter interaction and extreme nonlinear optics, and nano-photonic devices.