My experience so far in life is limited to repair and recycle, and I'm only just starting to get into the design side. Your question was very broad, so I'll try to spread it out into chunks that are easier to focus on.
Tools:I'm going to assume you've got a basic assortment of screwdrivers, pliers, wire strippers and cutters, and the like. If not just speak up, this is something that doesn't get mentioned often, and could probably bear some in-depth coverage.
Equipment:Solderless Breadboard Using point to point wiring to make everything nearly cost me my sanity! This was the single most important thing for me. I picked up a pair of 830 point boards with a jumper wire assortment for ~$20US.
Soldering Iron I'm using a twenty year old 'hot nail' from Radio Shack, and will someday upgrade to a Hakko 937 or one of it's clones. I've read that it's much easier to learn to solder with a good temp controlled iron, and I believe it. I use my iron for desoldering salvage parts for the time being, so I can get away with an old cheapie to some extent. Fine pitch surface mount soldering with a 'hot nail' would be somewhere in between a bad idea and impossible for all but the most skilled. The cheapest irons are available in a hardware store here for about $12US and a decent Hakko clone capable of lead free soldering should be available for less than $100US.
Multimeter This is the place to learn about multimeters! I won't repeat the advice already here, but this is definitely on the list of things needed. I'd advise doing some research to narrow down what you need, and getting the best one you can afford that does what you want. I'd say that multimeters can roughly be categorized into <$30US, $30-$150US, and awesome. Each range has it's uses, and you may find deals all around the price ranges.
Power Supply My approach here was to salvage an old computer power supply. I'm sure you know to be careful around mains voltage and big capacitors, but it's worth a mention for others who might read this.
Every power supply I've dealt with has a list of it's max amperage ratings per voltage rail on a sticker on the case, so it's easy to see what you're going to wind up with. Given you'll be dealing with audio, it's worth noting that computer supplies generally only support a few hundred mA on their -5 and -12 rails, so you'll probably have to buy or build something yourself for a good split supply. Commercial power supplies are available starting from a bit over $100US, and it's possible to get lucky at a swap meet or on ebay.
Oscilloscope Like the multimeter, this is the place to learn. I don't have one myself (yet), but I'd advise researching the benefits and drawbacks of used analog vs. new DSO. If you're on a budget, you can get a lot done in the audio ranges with an older analog, but it's buyer beware out there on the auction sites. Older and used scopes can come your way for any price down to 'free to a good home', but for something new you're looking at a minimum of ~$400US for a standalone, bench scope. There are USB scopes and scopemeters out there for less, but they compromise somewhat heavily on their specs for their affordability (in my opinion).
Function Generator I know little about the ins-and-outs of function generators, so I won't say much. In a pinch, you can use your computer with various software for simple audio generation, just beware of it's limitations. I haven't researched prices, as this is farther down my list of priorities.
Parts and consumables:SolderThere are a heap of different types of solder, and in your case you may be forces to use lead free soon. Others may have more advice to give here, I'm using scrap lengths of rosin core industrial wiring solder I get from an electrician friend. Not recommended.
DesolderingDesoldering can be accomplished with a suction bulb, suction pen, desoldering wick, or suction machine. Many would advise picking up a pen and desoldering wick, and this is plenty to get started. Bulbs are a bit kludgy to work with, and the dedicated machines are usually part of an expensive soldering station.
Iron TipsIf you decide to go with Hakko, for example, there is a huge assortment of tips available for different needs. If you get a Hakko clone, try to ensure it's capable of using Hakko replacement tips. Hakko makes very good, long lasting tips, and this is a good way to squeeze some value out of a cheap clone.
Tip CleanerMany use the moist sponge method, some report awesome results using the flux impregnated Hakko brillo pad looking things. I use flux soaked steel wool, and I don't recommend it.
FluxRosin core solder, when melted, boils off the flux in it's core. This is the smoke you see when soldering, for the most part. When there is no flux present, solder tends to become clumpy and resist sticking to anything but the iron. Flux is essentially the 'water' in watercolor painting, you put it where you want the solder to go. You can get flux in many different forms: 'big bucket of flux', small tin, flux pen, needle tipped squeeze bottle. I use a small tin myself, but I primarily do desoldering work thus far. Watching some video tutorials on soldering will help determine what you need for your own uses. Just be sure to stay away from 'acid' flux, it's intended for pipes and such and will eat plastics.
Wire You can't have too much wire! You can easily salvage more wire than you'll ever use from construction site waste, and buy only what you specifically need. One good example here is household telephone wiring. Older wiring when I live used 24AWG solid core wires, so I saved about six feet of a ten pair bundle and I've got enough breadboard wire to last forever. Various sizes of insulation become useful for temporarily connecting parts with leads, you just slip the leads in each end and if you used the right size insulation the connection will stay together hands free. There are many great improvisation opportunities here. Or perhaps, I'm just a bit nuts.
Discrete Components Oh, you could buy them, but I recommend at least a short salvage operation. It seems like all the folks who learned hands-on first started by taking things apart. It's when curiosity kicks in that you start learning things. Why is there fuzzy plastic debris on the end of every screw I remove? How could this thing even work with a spider egg sac on the IC?! What's the series resistance of that spider egg sac? Grab the ohmmeter! If you start pulling discrete components it's a decent introduction to solder behavior as well, and a relatively painless way to learn how not to overheat parts.
Stuff to take apart Ask everyone you know to save you broken things! Everyone! You'll have a never ending stream of awesome built up after a while. If you're just getting started you can try any swap meet, flea market, thrift store, etc. Try to aim for older and larger things when possible, as you won't have much to do taking apart, say, an ipod. You're looking for '80s and '90s consumer electronics, old stereos are a good place to start.
Bah, I suck at organization. Anyhow, I hope this helps some. I'll be here to continue the dialogue if you have more to ask, I'm just not sure where to start rambling.