If you have reusable (high-level) code (known good and unit tested) that uses fixed interfaces to access device-specific resources, the overhead is minimal and it all gets compiled into very efficient code (C++ templates). You only have to implement those fixed interface methods for the resources you actually use when porting to a new device.
Your libraries are split into generic code (using fixed interfaces) and device specific reusable code (implementing these fixed interfaces) - per device. You harvest the generic code from existing projects and you write the device-specific code as needed - and/or reuse what you already have.
I guess you could also make this work in C, just a little messier and less optimal IMO.