+1 to the above
Plug in USB. Measure voltages, see if the 5V is getting from the connector into the board. Easy way is to keep the black probe of your multimeter clipped to the metal shell of a connector and then hunt with the red probe, starting from all of the pins on the USB connector.
Many USB devices will have a fuse OR a resistor acting as a fuse. These might be blown.
I doubt the main caps will have shorted, I've never seen that failure mode (outside of "I exploded the whole board"). If the main caps have shorted then you'll be able to observe little to no voltage on the USB connector (the host USB device will/should limit the current if it's USB compliant).
So, in that case I could plug it in my laptop and then test it (I don't own a bench power supply yet, so if this is the case it would be convenient)?
Yes. Your laptop should have polyfuses (or similar active protections) in the USB ports to prevent too much current exiting its ports. Otherwise every faulty USB device would damage your laptop.
I don't know why you would bother with a bench power supply here. If the device is drawing a reasonable amount of current then your laptop will supply it fine, if it's trying to draw too much then your laptop will refuse to supply it (eg voltage will drop OR it will get turned off, possible repeatedly cycled on/off/on/off). A bench power supply could let you do the same (with more effort) or it could let you provide more current... but that's not a safe diagnostic method
As for the caps, I believe these are already polymer solid ones (this unit was made in 2001). So, as you said, I would also expect them to go out in quite a loud fashion instead of dying quietly of old age, right? [EDIT: after some research, I found out that these can indeed degrade and fail with time without any physical signs of damage]
Again I think this is unlikely. There are hundreds of unlikely things I could list, I'm not sure where the focus on electrolytic caps came from.
Measure more, guess less.
If it was indeed a blown resistor, would I need the schematics of this PCB to be able to identify it (if I wanted to avoid testing them all one by one XD)? Or should I look for a "000" resistor (which I could assume to be the fuse)?
No need for schematics. It will be wired very close to the USB port and you can use your multimeter on continuity buzz mode to find it (edit: whilst the unit is unplugged from power! otherwise the multimeter will be confused). It will almost always be the first thing wired to the +5V pin of the port.
Yes 00, 000 or similar if it's a resistor. If it's a fuse style package then it might be a different colour or have a letter written on it instead of a number.
Stop asking questions and go have a look
Plugging it into USB power and probing with a DMM is unlikely to cause any harm as long as you are careful to (1) keep the multimeter on volts (not amps) mode and (2) don't short nearby pins/pads out.
Once you have more information we may be able to help you more. At the moment all we can do is guess. Power supplies are always a good place to start looking.