I was first under the impression that AON6276 80V 100A would be enough, but as im typing this out it's obvious that some headroom is required for the amperage at least. Thank you a lot, this is the exact thing I meant with a second pair of eyes looking over the design.
The 100A I
D rating for the AON6276 isn't a suggestion that the device can actually handle 100A in a real application. That rating assumes the device's case temperature is fixed (usually at 25C), which basically means you have an ideal heatsink (not happening in real life). The vast majority of FETs will have such a rating, as a way to compare one FET against another, but the actual current it will be able to handle in the application will always be much lower.
The I
DSM ratings are much more realistic, as it assumes a fixed ambient temperature, not a fixed case temperature. But even that is a crude approximation, the real current handling will depend a great deal on the PCB layout and how that PCB transfers heat to ambient.
And of course, even if ESC as a whole provides 100A, the average current through each FET will be less than that. And of course there's switching loss to consider too, which could be even greater than conduction loss, depending on your PWM frequency and choice of FETs....
If you know you're going to do many iterations, I wouldn't get too bogged down in thermal design just yet. You should focus on developing the signalling and control aspects, then scale up power later on. I think you will also need stronger gate drivers than the FD6288.
On the topic of signalling and control, it looks like you intend to sense backemf and battery current, as opposed to the reference design which senses backemf and motor current with a single shunt. The latter provides a current measurement which is useful for motor control, while your current sense circuit would only be useful for monitoring average current draw, not really useful for determining the motor current. I would not suggest replicating their current sensing method either, though. At high operating currents, a single shunt will lead to lots of issues with layout parasitics, ringing, etc. I would have separate shunts on each leg (at least two, if not all three), either on the source of the low side FETs or in series with the inverter outputs (requires current shunt amplifiers which can withstand a lot of common mode bounce, look up the INA240A devices from TI for example). IMO at high power levels having extra capabilities for sensing the motor state is worth it. You probably wouldn't regret adding interfaces for hall sensors or quadrature encoders as well (even if you don't intend to use them in the long run, they may prove invaluable while developing the control firmware).