Your load is effectively R = V/I = 5/0.5 = 10 ohms. You want to smooth over 100ms, so say you want a time constant of 3 times tau for your capacitors. tau = R*C. Looks like you might need about 30mF.
Try a 47mF or higher super cap. Then put a few electrolytic caps in parallel (several hundred microfarads), and then a ceramic 0.1uF cap. Super caps tend to have high ESR and probably will not be able to deliver pulses of current. Hence, the extra types of other caps.