I'm starting to wonder if a VERY weak pullup on device side temporarily in place of the 1.5K pullups would work?
200k-1m pullup on device D+/- line, when plugged in with be pulled down by the host's 15k. So at 3.3V, when a device is plugged in, I would see the line go down to under .2V or so, easily low enough to trigger a low state / zero in IDR.
Then once I detect a low I can go turn on the 1.5K pullup to set FullSpeed/Highspeed mode, shut down CAN, and start the USB module yea? I'd need to see if a 200-1m pullup would mess anything up but I kinda doubt it.
Since it's a very weak pullup I think it would probably be a fair idea to "debounce" it to make sure it wasn't just a transient error. After being sure it's really driven low enough (plugged in), shut the weak pullup off.
Thoughts?
EDIT: And for anyone in the future that might stumble over this... The "connectivity" line of STM32 parts do not have this CAN/USB simultaneous issue, there is separate RAM for each module. I could step up to a STM32F105/7 part and not have to deal with any of this. For this project, the much smaller and cheaper chips are more ideal for me, provided there is a solution that may work.