There's the Nano, a crap open source 'scope so why not do the same thing with higher spec' hardware?
The nano already costs $100, and doesn't have a real scope AFE and has a pathetic sample rate, and only one channel. Put real spec hardware in there, and you are increasing the cost a lot.
Hmm! Good point. Do you really think that SIGNIFICANT savings could be made in the hardware (without cutting corners) to justify the effort of a home brew software firmware? I think Rigol do a pretty good job of squeezing every last cent out of there product (performance wise) and think they would be a hard act to beat.
No, I don't think you can make the equivalent of the 50 MHz Rigol for less than the market price, just based on parts cost alone, at least not in the volume a small open hardware project would get.
The bottom end of the market is under tremendous price pressure. Manufacturers are basically selling their cheapest models near cost (at their much larger volumes) and making their profit and recovering the development costs on the premium for higher end features -- extra channels, extra bandwidth, mixed signal, and so forth all cost proportionally more than the base model.
So I think the only place you could attack the scope market is to aim higher, and especially with analysis tools. If you had a few really good mixed signal designers who had been made independently wealthy and wanted to build an open hardware oscilloscope platform in their free time, you might be able to design something like a 4 channel, 200 MHz mixed signal scope where the production costs were less than a comparable commercial unit. They key then would be to be able to add in all of the bonus firmware stuff for free: the measurements, the pass/fail analysis, and the protocol decoders.
If you did that right, you might end up with something that had the hardware cost and specs of the 200 MHz/4 channel Rigol, but all the unlock codes from the Agilent 3000X. Of course, you aren't going to get the obscene waveform update rate and real-time advanced triggering and protocol decoding without the agilent ASIC.
There are two problems with this: it takes a really talented team a long time to design something that has the functionality of a higher end scope. You have to find those guys and convince them to work for free. Second, the scope you build is still going to end up costing considerably more than the $400 a lot of hobbyists already balk at paying. You might end up with the most powerful $1600 scope that nobody buys because the target audience can't afford it, and the people who can afford it and want the features are more interested in getting a name brand scope with support than a hackable open platform. Plus you still have to contend with the problem that the big players are sitting there with their ASICs that you can't match in terms of performance.