You don't have a temperature problem ... 40c is not really that hot.
You can reduce the temperature through other means, like better placement of the regulator IC - if you place it more towards the center of the board, the center pad will transfer heat and the heat will radiate in all directions of the circuit board over a larger area, so the chip will potentially be cooler. In your layout, you have the IC in that bottom right corner, so the heat can only go in two directions.
You can also get better heat transfer with thicker copper, ex 2oz copper on top would be better than 1oz or less.
I don't see 3 electrolytic panasonic capacitors in your schematic and pictures of the board.
Those electrolytic capacitors you chose aren't great. Surface mount electrolytics in general aren't great. Your 47uF 50v capacitors have a ripple current of 195mA and 0.68 ohm ESR (not that important for "buffering" output of a bridge rectifier).
The Rubycon ones I linked to (47uF 50v) are rated for 295mA current and the ESR is half at around 0.3 ohm. They're also rated for 7000h at 105c compared to Panasonic's 2000h @ 105c
The polymer ones I linked to are 6.3mm x 8mm and have current rating of 2.1A and ESR in the 30-40mOhm range. They're rated for 2000h@105c but it's different rating compared to electrolytics, it's more like over 30k hours of electrolytic equivalent.
With electrolytic capacitors you can approximate double the lifetime for every 10c decrease in temperature, so for example at 45c ambient temperature a 7000h @ 105c would be 14k at 95, 32k at 85, 64k@75, 128k at 65m, 256k@55, 512k@45c
With polymers it's more like lifetime = original lifetime x 10
(rating - ambient)/20 so for example at 45c internal temperature, you'd get 2000h * 10
(105-45)/20 = 2000h * 10
3 = 2000*1000 = 2,000,000 hours
They also take much less space compared to your surface mount capacitors if layed flat on the board, you could have two of them with the leads on the center of the board like this
[=] C1
[=]
+ -
+ -
[=]
[=] C2
Don't even need holes if you don't want the back of the circuit board to be broken, you could solder the leads to pads and lock the capacitors in place with a bit of glue or threadlocker liquid.
The surface mount bridge rectifier you chose seems to be rated for 1A or some lower value, you may want to see how how the current goes, as you say you'll need more than 500mA on the output
Yes, you need 3.3v trace and everything to be thicker/wider, going through a via is bad idea.
The sizes of the input capacitors seem a bit off.. I would expect that 1uF ceramic to be 0603 X7R and the 10uF to be 1206 or 1210 and rated for at least 75v, 100v would be preferred.... maybe something like this :
https://www.lcsc.com/product-detail/Multilayer-Ceramic-Capacitors-MLCC-SMD-SMT_Murata-Electronics-GRM32EC72A106KE05L_C576517.htmlYour design is also kind of weird because you have the AC input of your rectifier towards the middle ... you're running AC voltage close to 3.3v output (may be on separate layers or with enough distance but still odd ... think about if there's some water getting inside the device, is your ac separate enough from output?
The forward voltage of a diode varies with the current ... a diode may drop 0.8v at 1A but may drop only 0.4v at 50mA ... your problem is at low loads, when your high consumption device is turned off... the transformer's output will be higher, even higher if your mains voltage is high, and the bridge rectifier will drop less voltage so they all add up and you end up with high voltage. Adding one or two diodes won't do much because they'll also drop little voltage due to low current.
See for example
https://www.comchiptech.com/admin/files/product/DF2005S-G%20Thru340691.%20DF210S-G%20RevF.pdf - look at Figure 4 on page 2, you have nearly 0.6v at <100mA, and up to 1v at 1A. This forward voltage also changes with temperature but that's less important.
If you do use a zener diode, you'd probably want to add a resistor after the capacitors to limit the current going through the zener diode when you have those excess voltages otherwise the diode could be damaged. Don't know what value, something small