I have been messing around with electronics since '69 and I STILL don't have a benchtop power supply. Mostly I use wall-warts. The only reason I might by 'something' is the requirement for both +15 and -15 for traditional op amp circuits. At the moment, I have a DIN rail power block to provide those voltages.
Once you move past DC circuits which you can analyze with a multimeter, you can't go far without a scope. I built my first scope back in '56 or '57 from plans in the ARRL Handbook. It wasn't much but it worked. I have been through a couple of cheap scopes and I scored a Tek 465 (350 MHz dual channel) about 12 years ago. Yesterday I ordered the 4 channel Rigol DS1054Z. Today, this is where I would start. Yes, it is going to chew up a chunk of your budget but you will need it just about every single day. I use it far more than my Fluke DMM.
Before you can get to a realistic shopping list, you really do need to consider what you want to do.
If you just want to 'learn electronics', I am going to suggest you go with the Digilentinc.com Analog Discovery 2. This is a lab in a tiny box. It's amazing how much electronics you can do with this $300 gadget. It satisfies your need for a scope, it has 2 variable voltage power supplies (very limited current), an arbitrary waveform generator, digital inputs and outputs (16 channels) and a lot of capability built into the software. Like the Network Analyzer which sweeps across a frequency range to measure phase and frequency response of the device under test. This is NEAT when you start playing around with tuned circuits - filters and such. Digilent also has some online courses and they are excellent. Unless you have a specific need for something else, starting here is a really good idea. You can learn a lot from just this one item!
Caveat: This device is low voltage and low current. It can't power a PC, it can't read house voltages, there's a whole lot of limitations. None of which matters! Most of electronics IS low voltage and low current. At least in my little corner.
Anyway, something to think about!
http://store.digilentinc.com/analog-discovery-2-100msps-usb-oscilloscope-logic-analyzer-and-variable-power-supply/The analog parts kit, scope probes and BNC adapter board are all useful.
And, yes, I did buy one a couple of years ago. I like it a lot. Of course, mine is the older V1. The V2 is classier and the adjustable supplies MIGHT be useful when working on 3.3V logic or something.
Set up a PC in your lab area with at least two screens. One should be large (mine is 27") the other can be large or small, mine is just 19". Pop up the datasheets in the small monitor and run the Analog Discovery in the main screen. If you get into programming, you will do the same things: Datasheet in the small monitor, code IDE on the large monitor. I can't imagine a lab without a high power PC but, when things get tight, a laptop will do. My PC is set for dual boot: Win 10 and Debian Linux. I prefer to use Linux for most programming projects. OTOH, some software only runs on Windows. I'm agnostic, I don't care which OS I use. BTW, I would prefer 3 screens - all 27". Budget gets in the way...