I have a question along the same lines as the OP, and my main hobby has turned to building SDR hardware and some cabling in software beyond the driver level to DSP code.
Let's say I'm looking at a signal in a basic SDR program, I.e sdr#, sdruno, hell even gnu radio fft. How do I make educated guesses as to go about decoding? I know how to tell between AM based and FM mod but what about phased based, what characteristics are unique to any phased base mod? Next is it look, ook, ask, fsk, psk, QAM??? After you figure that out then you can use programs and math to figure out the symbol rate, if it's qam1,2,4 etc... But I never know where to start when looking at a signal, I just pipe it through every decoder I have and hope something sticks. Some times research gets the job done that is looking through local fcc data etc.. but that's not helpful for personal ism type devices, SATs, amateur tv and packet, etc. For example there's a constant 1240mhz broadcast about 3mhz wide and almost flat, as in a 10db rise horizontal for 3mhz the 10db fall back to the noise floor. I can only pick this up with a patch pointed within a few degrees at the exact right height. It's always there it always looks the same no one on Reddit seems to know what it is, maybe it's noise idk, but where do you start when research fails?