It is much more nuanced than that, and other causes are more significant. Read Piketty; his thesis has been lauded by everybody from all walks of life. The only other comparable book is "Without Hot Air", about energy.
The rich elites are very adept at getting the naive to believe that immigrants are the cause of relative poverty, and to ignore who "trickle down economics" impoverishes and benefits. Not difficult to see why!
It is much easier to get people to hate those they can see and who look "different", than it is to get them to hate people they can't see.
It depends on where the immigrants are from.
If people move to a rich country from a low wage economy, it drives down wages, making local people poorer. The migrants are willing to work for less then the locals and have less understanding of the law i.e. employment laws, so rich companies can take advantage of them. If they're there illegally, then even better: pay cash, no tax, national insurance or minimum wage to worry about!
If people move from a rich country, to a poorer one, they bring money with them, along with a rise in living standards for the local population. That doesn't happen very often though, because few people from rich countries want to live in poorer ones, but we see it more though tourism, although it's often temporary and more seasonal.
In the UK, the rich actively encouraged mass immigration from low wage economies because it benefits them. They don't care about the poor. This is why every government we've had since 1997 has has done the same and people who oppose mass immigration from poor countries are demonised by the mainstream media.
Like mass immigration, over-tourism tourism causes problems for the local population. People tend to bring their culture with them. Locals living near popular tourist destinations understandably get annoyed by rude, rowdy, drunk holiday makers who leave rubbish behind. Yes, they benefit the local economy, but there are locals who would rather be poorer and not have to deal with tourists.