I'm almost a decade out of date; I built my last silent workstation almost a decade ago now.
My favourite fan was 120×120×25mm Scythe Gentle Typhoon, aka Nidec D1225C12B5AP, but I've also used various Noctua 80×80×25mm and 120×120×25mm fans.
Noctua NF-A12×25 PWM is considered the best all-around fan right now. For CPU heatsinks, I like huge Noctua towers (NH-D12 and NH-D15 series). Again, not up to date on what's best right now.
For "silent" operation, I run these fans at 500 RPM to 800 RPM. The larger the fan, the better: the area of the airflow is larger, so that for the same volume of air moved, the velocity is lower. Smaller than 80×80mm fans often
whine –– often simply because of the velocity of air in the small cylindrical area –– to move useful amounts of air, so I like to design my cooling around larger fans. I only trust reviews so much; it all bogs down to testing different fans in the same setup, to see which works for me best.
I didn't build my cases scientifically, and instead approached it empirically. I used the largest heatsinks and fans I could fit, then scattered lots of thermal sensors inside the enclosure, and measured the temperatures as I reduced the fan speeds. Today, I'd definitely get a thermal camera for this, using
something that is transparent at the 3µm – 14µm wavelengths for one side of the enclosure; PMMA and Plexiglas are not that, they're quite opaque at 3µm to 25µm.
If you use an application to control your fans, you can have them predict extra thermal load from increased CPU and I/O load. Other than that, having enough thermal mass (which is just mass for pure-metal heatsinks) smooths out any sudden spikes. You'll want a thermal sensor at the base of your CPU heatsink (assuming a large tower-like heatsink with heatpipes), so that when it shows an increase in temperature, you act on it immediately; whereas for ambient temperature changes, you react more slowly. I know of only empirical tuning of this, i.e. adjusting the rates at which the controller reacts to increased thermal loads, and recording the ensuing temperature curves; and changing until it satisfies your needs.
This is also exactly why I suggest USB-based DIY fan controller: it won't crash even if your CPU does, but you can still easily create a small UI widget to switch between "be quiet" and other modes like "number crunching" where you want the fans to be more aggressive. For logging the temperatures, you can use a microSD card socket, either via SPI or via SDMMC if the MCU has that peripheral.