From your description it sounds to me like your circuit is still oscillating somewhat.
Look at my modification of Hero999's modification of your schematic and put the capacitors in. If it still oscillates try increasing the value of the capacitors from 100 nF to 200 nF. See the quote from the Data Sheet below.
The circuit does not work for the _presence_ of a magnetic field. The mere presence of a constant magnetic field does not induce any voltage in a coil. The field must be _changing_ for a voltage to be induced. See Faraday's Law of Induction. The circuit responds to induced voltage in the coil, so if the field is not changing there is nothing to respond to. If the LED is on fully or dimly, then there is either enough changing magnetic field to induce voltage in the coil, or the circuit is oscillating, or both. When the magnet is stuck to the coil, it is possible that this changes the tendency of the circuit to oscillate, so the LED goes out.
One solution to oscillation is to add capacitors. You should have a local decoupling capacitor of 100 nF or so as near as possible to the chip, across the chip's Vcc and Vss (supply and ground) pins 3 and 12, and also have capacitors from the input pins 6 and 7 to ground. (See the quote from the Data Sheet below.)
Wiring should be direct and short and your breadboard should be in good condition.
Also, if you are still using the LM339 quad comparator, the 3 unused comparators in the chip can cause oscillations if they are left floating (no input or output). The Data Sheet for the TI LM339 says
The LM139 series are high gain, wide bandwidth devices which, like most comparators, can easily oscillate if the
output lead is inadvertently allowed to capacitively couple to the inputs via stray capacitance. This shows up only
during the output voltage transition intervals as the comparator changes states. Power supply bypassing is not
required to solve this problem. Standard PC board layout is helpful as it reduces stray input-output coupling.
Reducing this input resistors to < 10 k? reduces the feedback signal levels and finally, adding even a small
amount (1 to 10 mV) of positive feedback (hysteresis) causes such a rapid transition that oscillations due to stray
feedback are not possible. Simply socketing the IC and attaching resistors to the pins will cause input-output
oscillations during the small transition intervals unless hysteresis is used. If the input signal is a pulse waveform,
with relatively fast rise and fall times, hysteresis is not required.
All pins of any unused comparators should be tied to the negative supply.
If you have an oscilloscope you should be able to see if oscillations are keeping your LED glowing.
ETA: I just tested the circuit with LM339. It oscillated, keeping the LED on as you describe. Without using any capacitors, simply tying all remaining pins of the 3 unused comparators (2, 4, 5, 8, 9, 10, 11, 13, 14) to Ground killed all my oscillations and now the circuit works as you probably expect it to.