If the transformer is overheating, likely reason is that the rectifier diode is s/c. These are a special type designed for the voltage involved. Of course the transformer insulation might be damaged if the overheating is severe.
The rectifier, along with the capacitor wired in series, is a voltage doubler. If you're wondering how you achieve a voltage doubler with only one diode, well actuallly you don't. The magnetron acts as the second diode.
Usual warning about some really dangerous voltages in these thngs. Don't forget to dump any residual charge on the cap before handling, with a well-insulated tool.