A few things to look into:
It looks like you are using IDC ribbon cable for the link.
Your pinout shows ground, data_h, unused wire, data_l, unused wire, etc.
This will not give you a 100 Ohm differential impedance and your common mode impedance to data_h will be much lower than to data_l.
Additionally you do not mention end termination. Since this is a bi-directional link you will need to have termination on both ends.
This will still leave little stubs for each board, but they look to be about 1 inch, so are likely OK.
Signal integrity will definitely be a thing, since your transceivers are rated at 200 Mbps. The UARTs will do some filtering, if they are designed correctly, but you can't feed them complete garbage.
Running LVDS between boards is doable, but you have relatively low common mode range for the receivers, in this case -1 to 3.4V which is actually pretty good.
Nonetheless, any ground shift between boards can generate false data reception.
Best practices would be to transformer couple, but that requires an encoding scheme like 8B10B. Common mode chokes could be used instead, but are not as effective with multi-drop, since you can't put termination behind every single one.
If this needs to run in a very noisy environment, you might want to switch to RS485 transceivers.
To debug this, I would suggest taking a direct measurement on the UART lines and if you see something unexpected, look at the LVDS lines.
Probing needs to be good, i.e. fast enough probe and scope with low impedance grounding of the probe. Throw away the 6 inch ground lead...
Edit: I just noticed the analog switch you use to connect a termination resistor. Assuming that you only have the last one turned on, you should be OK. However, this further reduces your common mode range as the chip will clip negative excursions.