I did some experimentation. I taped the magnetometer breakout board down to the bench to stop it moving (and thus giving false positives), and waved the accelerometer breakout board around the magnetometer in various positions to see what effect it had on the readings.
The largest effect was when placing the pin headers of the accelerometer board at a perpendicular angle to the edge of the magnetometer board, within about 5mm distance. You can see on the graph below (orange line at the top) that it skewed the calculated azimuth figure by quite a few degrees.
To test whether it was fields from the power or transmissions on the jumper wires going to the accelerometer board, I then unplugged it and simply waved around the board alone. Same result! So, it would appear to be the 0.1" pin headers I have attached to the board.
I checked my parts bin, and I don't have any straight 0.1" header strips that are non-ferrous.
The only non-ferrous headers I have are right-angle, which aren't suitable for my ultimate intended usage. I bought both types from the same source, so I guess you can't tell what you're going to get with the cheap stuff.
So is a solution to this problem going to be to ensure I use non-ferrous headers on these breakout boards? e.g. copper alloy or bronze? Also, it seems provide most skew when the pin headers are moving, so I guess when both boards are fixed, that will mitigate any effects somewhat.