I have used all of them, and I think 4.3 or 4.5 is probably the most comfy, naked, anyway. And you don't want to go taller (naked) if durability matters.
Aside from height, another important factor for durability is the press force. The stiffer the switch, the lower the cycles, and it is quite dramatic.
I wouldn't call it a hard rule... you have to check the datasheets.... but IME the lowest force and highest press cycles come in black. White is in the middle. Red is on the stiffer end. Then there are other colors between those.
I find if you are not using any enclosure or button cap, that the lowest force feel fine.
Using the 5mm's naked AND with the higher stiffness, these buttons can quickly become faulty, internally, with no external damage. Worse than simply quitting, they will start to function, intermittently. Esp on a widget with a lot of buttons on it, this will be a big annoyance.