I think it's more software related than hardware though by no means is hardware not susceptible to aging.
The way the operating systems are written and the way other programs interact with them has a lot to do with it. For instance, even if you uninstall a program from your OS, there's still some sort of "leftovers" in the registry or in the filesystem. Even though that program isn't there, remnants of it still remain. There's so much that we don't see in operating systems and programs that goes on behind the scenes. Junk winds up in folders we don't know exist and thus we get that "slow-down".
Also as time progresses, the requirements of software grow. Software is kind of a "gas". It expands to whatever container it is in. As the hardware (container), get's better so does the requirements of the software (gas)..Weird analogy, but just popped in my head.
I think hardware can "slow-down" but not in the traditional sense. Computer hardware more or less works or doesn't work. Processors have lots of redundancy just incase one little transistor dies, the whole processor doesn't go to waste...or atleast that's what I've read. But it's not like the chip it's self is getting slow. Probably the only thing that can be hardware and will slow down your computer is your hard drive. This can happen in multiple ways. On a more software-oriented side, when bits are saved to the disc, they can get spread out all over the hard drive platter. So when the read head seeks this information, sometimes it takes a while for it to physically get to all of the magnetic bits and read them. Additionally, you've probably heard of a "bad sector" in your hard drive. That is one of the few instances where hardware does get slow over time. A bad sector means an actual part of the hardware is now unusable for whatever reason. So yes, that could definitely slow you down. But with flash based storage becoming the norm..slowly..that won't be as significant of a problem.
On that subject, yes silicon does wear down. NAND storage does actually have limitations. It's state can't be changed more than oh a few billion times before it wears down. Yes it can happen, but usually the effects aren't even significant enough for us to notice.