You have to cool a lot to reduce noise significantly. No problem for peltier coolers. There is a huge problem that's hard to address- condensation. Get some condensation and you have current leakage. The device has to be in either vacuum or dry air/gas. It can be done- look at cooled astro cameras, but it's a big jump in terms of mechanical design. If you evaculate the housing, you can back-fill it with argon (common from any welding shop), which has poor thermal conductivity, making it easier and more efficient to control the device temperature. The right desiccant pack is also needed. I think GR filled some of their reference capacitors with dry nitrogen, so it's nothing new.