There's a good reason why people doing audio and video processing in DSPs are few and far between. It's hard!
Specifically, there are a couple of quite separate disciplines with which you need to be intimately familiar.
Digital filtering is all about maths. You'll need a really solid understanding of wave theory, information theory, Fourier transforms, working in both time and frequency domains, and converting between the two.
Then, you'll also need an equally solid understanding of how to program a CPU, DSP or FPGA to implement the algorithms you're interested in. This is, again, maths as much as anything else, with a focus on numerical methods and good, down-to-earth pragmatism, as well as all the I/O and application code that you can probably learn about on the job if you haven't already.
Speaking as one myself, I'm not entirely sure this level of specialisation is actually a good fit for being a small business owner as well - depending, of course, on what you want that business to be. My customers generally need a complete product designing, not just a chunk of code, so I have to have a broad (though admittedly shallower) set of skills.
If you want to be one of those perpetually busy consultants, then by all means go ahead. You'll be a technical specialist first and foremost, and provided you can find customers that keep you busy, you could make a good living at it.
Tip: make sure you retain ownership of as much IP that you create as you possibly can. 'Write once, licence many times' is a much better business model than 'write once, avoid ever writing again in order to avoid potential infringement lawsuit'.