As explained above, a good way of improving reliability for a system is to build it with high quality (ie genuine) components from mainstream manufacturers, and ensure that all components are operated well within their SOAR (safe operating area). You also need to consider the operating environment: that might include protection against water ingress, for example. Also, rarely, fire resistance might be important if the system has a vital safety role. The environment also includes the electrical environment; noisy and hostile environments like cars require the electronics to incorporate protection circuits on its inputs and outputs. This typically includes over-voltage protection and filtering.
Beyond that we need to consider measures to improve resilience. Redundancy is a popular and effective way to achieve that, but it is expensive and doubles the maintenance burden. Where redundancy is used, common mode failures may be a risk, so highly resilient systems will use redundancy in which there is diversity as well. That is, the main and standby systems are functionally identical but implemented differently - for example by two completely unrelated development teams working to the same specification.
Crucial to high reliability for complex systems - especially software systems - is high quality testing (a whole subject in itself).
Talking of testing, a redundant system is only effective if the fault detection works reliably. That is, failures in the main system need to be detected in order to initiate a changeover to the standby system. And this is more complicated than you might think: the standby system must itself be tested rigorously in order to ensure it is available should the main system fail. Not all faults in the main or the standby system will be detected by the fault detection system; this is known as "fault coverage" and obviously needs to be as high as possible. One way of managing this is to switch between the main and standby systems once every 24 hours (say). This exposes both systems to user workload, which is often the most thorough test regime, and if a fault is detected there is a very high probability that the changeover can switch back to a working system, bearing in mind it was working fine a few seconds ago.
When a failure does occur, good reliability does rely on a short downtime. That means having spares in stock ready for a rapid swap-out. Remember, if the main system fails and the standby system is now in use, you no longer have any redundancy until the failed system is repaired.
There is a lot more to be said, and feel free to ask more questions, but I expect you've got enough to think about for now!
EDIT: There is also the concept of software reliability. Software doesn't fail in the same way as hardware, nor is it repaired in the same way, but bugs can manifest as a failure in the system from the viewpoint of the user, so it does contribute to the overall performance and the 'perceived reliability' of the system.