Have you thought about using a complete Linux PC as your sound device?
With USB OTG it can be used as a USB device to your windows USB Host, and there is all kinds of software for audio manipulation for Linux available. You can do most software development on a regular Linux box, or emulator or VM in Windows.
A potential problem with with any digital audio is latency. An AES/EBU or S/Pdif stream has very low latency, but when you add buffering of data, extra buffers for changing protocols such as going to USB, and buffers needed to do actual filtering, then they all add up.
Relating to this question also "minidsp" popped up in my memory:
https://www.minidsp.com/It's just an example, other companies make similar products.