I tend to be less optimistic for reverse-engineering the Windows drivers, and more optimistic in reversing/extending the DOS software.
The Windows software is optimized towards GSM functionality. The RF functionality is very basic, essentially very little more than "tune path 1,2 to txfreq x, rxfreq y, with attenuations a,b" is exposed. Frequency Sweeping (necessary for the spectrum analyzer) is not exposed, for example. (Or I missed it.)
The DOS OPSW, at least the main software (i.e. not the tools), is a "modular" 32-bit system (using the PharLap extender; it's basically simulates a Win32 environment in DOS, so it shares the PE file format and some more things). Many symbols are exported, helping disassembly. For example the GSM functionality is a bunch of DLLs that get runtime-linked and provide the GSM features. It should be possible to add custom modules that neatly integrate into the main UI.,
Also there is, as far as I know, only a single ASIC in the system - the DDC chip. Everything else is "just" FPGAs (which tend do only do glue-logic/interfacing, and not a lot of functionality), or DSPs (which custom code, of course).
So, writing software (running under Linux for example) that uses the device can be accomplished. Sure it's harder than just hacking existing software. But this wouldn't be the first device where a linux-based replacement firmware was written for. Examples are d-Box II, gaming consoles, iPod, ... It's possible, it's just a lot of work.