The STM32F4 is more than capable of doing this. I made a dual stereo USB soundcard with the STM32F411, which is a simpler slower chip without the LCD and lacks some of the interconnectivity peripherals. It is more than capable of driving both audio channels,
as well as doing software mixing of a center and a subwoofer (IIR low-pass filter at 48kHz) channel, using hardware floating point. I haven't fussed with the USB enumeration profiles to make do some more sophisticated stuff I had planned, but it goes to show there is plenty of room to spare.
Moreover, the I2S peripheral also has DMA which I haven't used yet. So this chip is also serving 4x 48kHz interrupts at this moment, which is very wasteful but it works.
It shows how much room is left for processing other stuff like LCDs, even at 100MHz. But if you can budget in a STM32F42x or STM32F43x, you can do some wicked stuff at 180MHz and it's DMA2D engine.
You don't need much buffers at all for USB audio. Audio is sent using USB isochronous transfers, which are only 192 bytes (2 channels x 16-bit x 48 samples) in size in my case. 44.1kHz is annoying to deal with because it's a non-multiple of 1000, so rather pick 48kHz. The USB sends Isochronous packets at 1kHz and synchronizes it using SOF tokens on the bus. SOF is a cheap 1kHz time base for USB, and is used to latch the buffer to the audio codec. This is useful because there will always be clock differences even when using the I2S PLL, and if your word sample frequency is off by 1-2Hz that means you're dropping (or interpolating) 1-2 samples each second. For me this is inaudible and just a consequence of dealing with USB.
I've first tried finishing off the buffer and then swapping to the new buffer at the audio codec level, but that had some audio distortion that only popped up every minute or so. I think it was because the sample clock was a few Hz to slow, and so eventually it was 1 complete buffer of 48 samples behind which was eventually completely dropped.
I can say the ST code examples were helpful, but because I was in a hurry I also borrowed some parts of
this project