It depends on what kind of parts your after.
eBay can be good (not for production components of course) to get you started with a kit of through hole resistors, capacitors, basic transistors, diodes etc. I bought an assortment of 2500 1/4W resistors for about $10 or something on eBay. They even sell books of SMT resistors (from 1ohm to 10Mohm) and capacitors for around $15-$20 or so.
If your looking for very specific items, I always use element14 and RS components just because they are convenient, but you normally pay a little more for that convenience. But again, it depends what kind of parts your after.
There are a few hobby stores around which are worth a look, but often because you are purchasing low volumes, parts can be expensive, but they are still worth a look.
Just google these:
Little Bird Electronics
Sparkfun
Australian Robotics
Jaycar
Altronics
Pololu Robotics and Electronics
Ocean Controls
Wiltronics