I have an EE degree from one of those Ivy League colleges (U of Pennsylvania, 1981), after waltzing through my primary education. It was plenty challenging, though not always in interesting ways. There were "honors" classes, grad classes open to undergrads, work and/or research opportunities. The hardest part was fitting in the things I wanted to learn, in a way that still resulted in a degree in 4 years (EE and CS were quite separate in those days.) But an Ivy EE degree is (was?) not particularly hobbyist-satisfying. There is a lot of theory, and math, and physics; stuff that fills in the gaps that would otherwise be left (as Ian says.) Not much "actually making stuff."
OTOH, the idea that I'd be able to take extra classes every semester (as I'd done in HS) fell by the wayside REALLY quick... (On the third hand, there was a semester or two where physics, math, and EE classes were all actually using the same math, and that was pretty cool.)