While it might not be the best idea, as the prices keep shooting up, I started by building a good old 486 PC-Compatible.
As Halcyon said, there are many ways you can get started, each dependent on your skill level.
If you're looking for a good place to get started for little experience in EE, and you just want a machine to toy around with, I would base one off the Slot 1 platform with a Pentium 3. Get a decent card like a GeForce 2mx, or if you want a Cadillac, 1 or 2 Voodoo 2's with a decent AGP 2D card. Something like a cheap AWE64 or Sound Blaster Live! would do for sound. The idea is to get used to building old machines, and managing the software on them.
With this machine you can easily play most DOS and Windows 95/98 game that are not affected by high CPU speeds. Stuff like Doom, Quake, Duke3D, and Half-Life will all run great on a machine like this, and the parts are relatively cheap on eBay.
If you're looking for a solder together, get a neat machine kit, then that's a bit harder. Altair 8800 kits would be my first choice, but they are piss expensive, as well as trying to do something like MTM Scientific's IBM PC 5150 clone. Apple 1 clone boards are a bit on the cheaper and easier side, last I recall, but they don't do a whole lot, having only what is effectively a serial video terminal strapped to the machine, meaning you can't do any sort of graphics, and in terms of games, you better like chess.
For more classic micros, C64s are always easy to pick up, as well as Atari computers, like the 8-bit, and maybe even the ST. If you're in the UK, you can probably find a ZX Spectrum for the pile of lint in your pocket.
If you want further inspiration, I'd check out the machines built by people like LGR and PhilsComputerLab on YouTube. Those two, especially the latter, tend to have very very good reviews and perspectives on the machines they cover and build.
Home computers were a thing from the 1970's to today, and with no real limit to what I can consider vintage, there are a plethora of options.