Ebay is the best bet. For HV transformers, neon-sign transformers are generally the most available but this depends a lot on how popular neon signs are in your area, as well as in some cases regulations that restrict voltage/power, or mandate internal groundfault sensing electronics.
Oil burner ignition transformers are also good for small coils, but again availability depends a lot on the local market.
The only widely available species of HV transformer is the microwave oven transformer, but their relatively low voltage and high current capability make them difficult and more dangerous to use.
For capacitors, the usual method is to build an array of lower voltage caps, generally referred to as "MMC" - Multi-Mini Cap.