The Fourier transform of a boxcar in frequency is a sinc(t). The only way to eliminate the ringing in time is to not have a sharp edge. In seismic work cosine taper edges are typical, but the filter is being specified and applied in the frequency domain. The usual specification is a trapezoid, f0, f1, f2, f3. The actual implementation commonly uses a cosine taper for the slopes of the trapezoid. Filters to flatten the spectrum such as the classic Wiener prediction error filter are designed differently.
All the work is being done on recorded data. So it's a lot easier to do basic stuff. Of course, when you want to create a 3D image from 10 TB of data, it gets rather more demanding. At a very basic level each sample in the output volume requires a mathematical summation of a 150K-500K input samples. That is a week or two with tens of thousands of CPU cores.
Good phase response is not particularly a problem so long as one is able to accept the latency. That's a problem in audio, but not in a DSO.
I'd appreciate an explanation of your estimate of the number of taps. That's a lot of zeros in the transfer function. I've been reading "VLSI Digital Signal Processing Systems" by K.K. Parhi and "FPGA-based Implementation of signal Processing Systems" by Roger Woods et al to help me make the transition from DSP in recorded time to DSP in real time. Aside from having very different constraints, the terminology in the seismic and EE communities is completely different.
Thomas H. Lee presents an example of an analog maximally flat phase low pass filter in "Planar Microwave Engineering" so I don't see any serious obstacle other than the mathematics of the Fourier transform.
The Gaussian taper pass band edge gets a lot of lip service in EE, but rather less use. However, sech(x) is symmetric in time and frequency, so it is a good candidate for consideration,
If you are using regular sampling, a low pass anti-alias filter is an absolute necessity. The aliasing arises because the Fourier transform of a spike series is a spike series. If the sampling is sufficiently random, then the transform of the sampling interval is a spike in frequency and aliasing doesn't occur.
I spent most of my time from 2013 to 2016 studying compressive sensing. In the process I read "A Mathematical Introduction to Compressive Sensing" by Foucart and Rauhut twice and "A Wavelet Tour of Signal Processing" by Mallat once.followed by the original papers by Candes, Donoho, Tanner et al. In total about 3000 pages of the most complex mathematics I've ever read. I had to read F&R twice bacause I really needed the mathematical foundations presented by Mallat. Subsequently as a consequence of some papers by Donoho I read quite a bit from "Convex Polytopes" by Grunbaum and "Lectures on Polytopes" by Ziegler to gain a better understanding of a fast algorithm for solving Ax=y using an L1 norm.
I plan to revisit all that at some point, but I need to master the FPGA implementation of FIR and IIR filters at high clock rates first. I've bought a Tek 11801 and four 20 GHz dual channel, 13 ps rise time sampling heads so I can measure bit skew rather than rely on Vivado and Quartus to calculate it correctly. Just constructing an 8 line fixture with lines matched to a few ps is going to be a challenge.
The HMCAD1520 offers 8, 12 and 14 bit sampling at different clock rates, so filtering that in an FPGA real estate efficient manner is going to be a challenge. An additional requirement is arbitrary, user specified filter pipeline as the LeCroy offers. So I will be attempting to use the partial reconfiguration feature of the Zynq line.
My current focus is the FPGA input to DDR section. I am investigating the anti-alias filter aspect to the extent that I must know what the signal passband looks like to do the post ADC processing, but I'm not going past the filter shape into the details of the actual analog filter implementation, attenuator responses, etc. I'm trying to eat an elephant, so I'm taking it one bite at a time.
It appears that we share a common interest with enough overlap in skill sets to be able to communicate, but with many critical skills the other lacks. So I'm hopeful we can collaborate on making this happen sooner rather than later.
I have a large DSP library going back to the "The Interpolation, Extrapolation and Smoothing of Stationary Time Series" by Norbert Wiener which is where DSP starts and was trained by a member of Wiener's Geophysical Analysis Group. Of all the books, I think "An Introduction to Digital Signal Processing" by John H. Karl is probably the best general presentation. The classic text is "Geophysical Signal Analysis" by Robinson and Treitel, the most prominent members of Wiener's GAG. They literally wrote the book on DSP in the 50's and 60's in the form of a series of professional papers which were published as "The Robinson and Treitel Reader" by Seismograph Service Corporation. "Geophysical Signal Analysis" is those papers reworked into a book. R&T focuses quite a lot on the problem of water layer reverberation as that was the driving application in seismic work. Hence, my suggestion of Karl instead.
In closing, the screen refresh rate is limited by the display. Even at 120 Hz that's an eternity compared to the data sample rates.