Going down that line of thought you immediately get a few problems:
1. Paradoxes - depending on how time travel or the Universe works/would work, then: X is bad, you go back in time, kill X (or set him on a nicer path where he finishes art school and becomes a mediocre artist instead of legendary baddie), problem caused by X is no longer. This sets the world on a completely different course where odds are that you were never born and even if you were, you don't know about the problem X caused, since it never happened. As such you have no reason to go to the past and kill/reeducate X. Grandfather paradox.
2. Is it reasonable to roll the dice? All that has led up to any point in history is a series of events that had some chance to not happen. Even if you go back in time, and do literally nothing but have history unfold again, will it unfold in the exact same way? Step on the wrong bug, it won't eat a tick in the forest that will in 1673 bite and infect some kid that will eventually grow up to be Super X.
Also, any single noble act can have absolutely devastating consequences. Killing X can, in the short run have a good impact (war doesn't happen), in the long run, have an absolutely catastrophic effect (tensions erupt in a different way and instead of X rising to power, Y rises to power in a different country and creates a dystopian horror state that will encompass the whole world and last for 10000 years).
Butterfly effect. This also likely destroys your civilization. Assuming that you have access to time travel technology, odds are that civilization is fairly advanced. Are you going to risk that when you have no way of knowing how even the tiniest change will affect stuff?
3. Ethical concerns - your reasons for altering the past may be noble, but someone's won't be. If time travel was invented, it would immediately become the most secreted and regulated technology. The amount of damage that can be done by someone going 5 years back is immense. 50 years is world changing.
There's more, but these are a few quick ones I can think of.