My supply is very sound and I've taken extra measures to make sure it is properly damped.
Okay, fine. How damped is it? If you've "taken measures", show us the measurements!

The problem is when the comparator transitions. The edge is extremely sharp and radiates EMI into a high impedance part of my circuit.
Show the measurements.
There is nothing about a TL082's capability that I would call "extremely sharp", or "radiating", or "EMI".
I have to assume what you really mean is: the voltage is
capacitively coupled into a nearby high impedance node.
These terms (EMI, radiating, sharp, coupling) all assume something about the nature of the circuit that created the signal. EMI is usually a lower impedance, common mode signal. Radiation is a high frequency, modest impedance, propagating signal. "Extremely sharp" suggests, at most, tens of nanoseconds risetime.
"Capacitively coupled" simply means you have an effective C-R (high pass) network, where the C might be the accidental capacitance between wires or traces, and R is the impedance of the affected node.
As the origins differ, so do the solutions: for this case, simply reduce or eliminate the coupling capacitance. Or reduce the impedance of the affected node, or both.
A simple way to reduce coupling capacitance is to reroute wires/traces to greater separation distances, and add ground traces or fill between them (shielding). A simple way to reduce node impedance, at the frequencies where it matters, is by adding a bypass capacitor. At high frequencies, this leads to a capacitive divider effect, so that the error signal will not be eliminated, but will be significantly reduced.
Whereas, EMI usually needs a common mode choke, and local filtering; and radiation needs filtering and shielding. A TL082 might be capable of producing signals of this nature if it's badly abused, say by placing a resonant circuit in the power supply pins, or something; but this should be prevented by your "properly damped" supply. But again, you haven't told us a lick about your system, so for all we know, it might well be resonating!
I have done experiments to slow it down and the EMI disappears if I insert a 4.7uf cap between the comparator output and ground. I was looking to see if there are other solutions to this problem because I don't like putting a huge cap on the output.
Well, slowing down the output is one way, but this directly impairs the performance of the circuit in the first place (and, as you note, isn't friendly to the op-amp!).
But now you need to tell us more about your circuit: its application (what, precisely, do you actually need from it?), what the nearby circuit is, and so on.
Tim