Its always good to keep in mind, its more a scale of how much you want to prioritise it,
For this project the critical signals like RAM and the USB PHY I would say yes, The SD card comes down to your max read rate,
Match only to what the specs require + some margin, if you can go better then fine, but too many get caught up on 0.001mm when the spec was 2mm length matching,
Make sure high speed signals have an unbroken ground path, you can jump layers with vias but you need a via for the ground aswell then, path of least impedance rules here, either below or beside,
Ground via placement kind of follows the above, make sure the path to the chips ground plane is convenient for the signals,
Keep decoupling caps close to the load, and make them connect to the capacitor and the supply on different traces if possible
I personally stop caring about impedance below smallest wavelength / 20 length traces,