Don't forget your local GoodWill or Thrift store. They seem to vary a lot by how much electronics they sell but my local one has provided all sorts of sources of used components and even some test equipment for very cheap!
Old style AC power plugs - the big heavy ones can often be bought for $1-2 and provide many of the parts you need to experiment building your own linear power supply - transformer, caps, diodes, etc. The higher quality laptop power adapters can also often be bought for a couple of dollars. They are switching power supplies and can be a good source of inductors and other parts as well. Just don't mess with those power plugs while they're plugged in until you know what you're doing and heed the advice about shorting the big caps with a screwdriver before touching them.