We don't know what sort of LED the original poster is using, as they've not posted a part number, description or schematic. 320mA would be perfectly reasonable for a high powered LED and many have a built-in resistor or driver, so no series resistor is required.
The reason for the massive initial current draw is the large capacitor charging up, when the switch is pressed. In reality, the peak current could be much higher, than that indicated by the meter: possibly hundreds of amps. It can shortern the life of the switch contacts. Add a 10Ohm resistor in series with the switch, to limit the current to a safe level.