Yep, slow sweep and bad spurs are the biggest issues. Phase noise might be another. It's the sheer variety of spurs that adds excitement, coupled with the fact that spurs in your SDR/SA would sometimes be difficult or impractical to tell from spurs in your DUT (especially if the SA spur drowns out the DUT spur).
How severe these limitations are depends on the application, of course. For standards, emissions, or performance testing, not knowing equipment spurs from DUT spurs is a showstopper. In other applications, the shortcomings of SDR can even turn into strengths! At work I deal with devices that have uplink and downlink frequencies separated by up to 50MHz, but I can use a $20 1MHz SDR to get presence+timing information out of both directions of traffic by putting the antenna somewhat near the device. The un-tuned signal (~12kHz wide) overloads the receiver and leaks all the way across the 50MHz into the 1MHz capture bandwidth, interleaving with the in-band signal but showing convenient contrast in amplitude. The SDR dongles are cheap and light enough to hand out and they make it easier for the SW guys to figure out "is this thing on?" type problems all on their own
Anyway, here's an example of possibly non-obvious trouble that spurs can cause.
Red, White, and Blue: the SA44B economy spectrum analyzer looking at a BG7TBL broadband noise source with 0dB, 20dB, and 40dB attenuation. It has two LO frequencies. Different LO = different spurs, so it can reject spurs that it sees in one and not the other. A step up from SDR, but a step down from a proper SA.
Black & Amber: a fixer-upper SA with proper mechanical attenuation relays, YIG preselector, and YIG LO looking at the same BG7TBL. The calibration is about 10dB off, but the point is the overall shape of the noise and how different it is from what the SA44B shows.
One of them lies, despite the attention its designers paid to its shortcomings, and one of them tells the truth. Don't get me wrong, it's not a dig at SignalHound -- their BB60C would have had no problems with this signal, and they call out the broadband measurement problem specifically in the SA44B datasheet -- but it *is* a limitation of the architecture that SDRs would share (except worse).
That said, I'd love to see some good SA/VNA software for SDRs. A SA/VNA that you can afford has infinitely better performance than one you can't.