Generally speaking, there are absolutely no issues or downsides to using these adapters. In fact a lot of the big brands like HP and Dell ship these kinds of adapters with some of their commercial-grade monitors.
That's actually a fairly misleading statement.
HDMI-DVI is generally fine, because that is only a passive plug adaptor (plus you lose things like sound when converting from HDMI to DVI, because DVI doesn't have these).
However, DisplayPort <-> HDMI/DVI or anything to VGA can and does have issues, depending on the adapter.
These are active (there is some sort of converter IC inside) and depending on what you have bought it may or may not work, may not support the resolution/refresh rates you want, the image quality may suck (e.g. when converting to VGA).
Worse, some cheap ones may even fry your HDMI port. Many draw current from the 5V pin in the HDMI socket, which is rated for only about 50mA - and there are some HDMI-VGA converters that need over 100mA to function. I have encountered this problem on both my RPi and my TV set-top box HDMI outputs.
Better designed adapters use ICs that are less power hungry and/or have a connector for external power but some crappy ones don't - usually with a bit of care you can crack the case open, break the power connection to the HDMI plug and feed in 5V from a USB port to solve the problem. The ones converting from DisplayPort are generally better because DP is able to source more current and these issues don't happen there.
Another problem you are likely to encounter is that many of these adapter dongles don't translate/pass DDC info, so your PC/signal source will not see the capabilities of the monitor connected but of the adapter. Most often it isn't a big problem but it can happily break stuff like VR helmets - the computer will claim it is not connected because it can't detect its DDC info.
Also if your are planning to play Blurays, you will have problems with HDCP that these dongles don't support (for obvious reasons).
If you can avoid having to use such adapters it is always better.