Yep that's the sort of manual passive spectrometer i had in mind.
You could certainly make a spectrometer for long wave infrared too, tho at that point you can't use CMOS/CCD image sensors to detect the light as they are not sensitive to it. Pure silicon and germanium become transparent at those wavelengths, this makes them great for making infrared lenses, not so great for detecting it cause it just passes trough. A sutiable detector would likely be a microbolometer array such as is used in thermal cameras, a vidicon tube (CRT like device that works as a camera), or a suitable photomultiplier tube on a moving sled.
Tho not that you would see a whole lot of interesting things in long wave infrared. Anything close to room temperature glows brightly in it and emitter lamps designed for this wavelength are mostly just glorified heating elements. LEDs are incapable of producing this long wavelength of IR (Well apart from black body radiation from the LED being warm)
Similar story for going into the short wavelength UV. The longer wavelength UV works fine with normal lenses, but the shorter wavelength high energy UV needs optics made from special materials. But at least detecting UV is easy because it not only has enough energy to easily knock off electrons in pretty much all semiconductors, but it even has enough energy to rip apart molecules. This high energy UV is what gives you sun burns. The highest energy UV even rips oxygen in the air apart and creates ozone, you really don't want that shining on you, go any shorter in wavelength and you get Xrays and you really don't want to be exposed to those. But LEDs are unable to produce short wavelength UV