Here's some info for you:
7805 regulators need to always have the input voltage about 1.5v higher than the output voltage if you want the output voltage to be stable in time. For LEDs it doesn't matter that much, but it's good to remember for other projects.
Since the output voltage is 5v, this means you should pick the capacitor to be big enough so that the regulator will always see at least 7.5v
You have a 12v AC transformer, which means that after you rectify it to DC using a bridge rectifier, you get a peak DC voltage of 12v x 1.414 = 17v, and then you also have to take out the voltage drop on the two diodes that are always conducting inside the bridge rectifier, so you're left with a peak DC voltage of about 15v.
Now you need to think how much current those leds would use and then you can choose a capacitor to keep the minimum DC voltage always above 7.5v ... you can figure out how much you need with this formula that approximates how much is needed:
C = Current / [ 2 x Mains Frequency x ( Vdc peak - Vdc min) ]
Let's go with 0.5A (i'm sure those 50 leds use much less) and let's say you want at least 8v and for safety, let's say you'll have up to 14v DC ... in which case, you'd need to use C = 0.5A / ( 2 x 50Hz x (14v-8v) ) = 0.5 / 100 x 6 = 0.000833 Farads or 833 uF .. so the next value up of 1000uF would be good enough for up to 0.5A in all cases.
You could use a very large capacitor but then the minimum voltage will always stay closer to 15v and you'd only dissipate more energy in the linear regulator ...
Remember, all above 5v is dissipated on the regulator as heat, and the IC can't dissipate more than about 1w without a heatsink... so for example if you have 15v in and 5v out , at 0.5A you'll have (15-5) x 0.5) = 5 watts of power dissipated as heat.