I agree with dumper diving, scavenging parts from old PCBs. If you have little money, but lots of time and interest, then this can be a good source of some types of parts.
One of the first things you will need is a power supply. You could scavenge 5 and 3.3 V fixed output regulators from old PCBs and use any wall-wart with 9-12 V or so output to power them. But even this is hardly worth it. For $1 (shipped!) on ebay you can get a power module that neatly plugs into the two power strips on either side of a typical solderless breadboard, giving you 5 and 3.3 V rails. Search on ebay: "Breadboard Power Supply". You can also get a bundle of a breadboard, the supply, and jumper wires for around $5, like this
http://www.ebay.com/itm/331329599179 (I do not endorse the seller, it's just a search hit).
I LOVE surplus stores. Sadly, the two I knew of in my area (Ottawa, Canada) closed up shop and I don't know of any others around.