When scopes want to go fast, they often interleave 2 ADCs.
As above, consider the 600 MHz Teensy 4.1 - it's FAST! And it uses the Arduino infrastructure with rewritten libraries to duplicate Arduino functions - teensyduino
Wow, that is superb project.
Linux is not designed to be real-time os (so why do you try to use it as a such), as Torvalds said (approximately) (edit. It seems there is now RTLinux or equivalent kernel extension.). One route if one still insist to use RPi as ease of implementation is to go with something that turns it 'baremetal dev board' with full hw access like Ultibo project. Maybe there is similar project in language of your choice.