SPI pins are super sensitive to any interference, meaning that, if I touch any of the SPI pins except the power and ground, it completely bricks the matrix
You are not supposed to touch the pins at all. When you touch the pins, you apply a few hundreds volts at low current. Those 200-300V are induced from the surroundings and from the Y capacitors in the power supply. Usually there are protection input diodes that limits the hundreds of volts to only Vcc+0.7V and GND-0.7V, but it is never a good idea to abuse them.
Depending on grounding, and the desktop/laptop/power supply, you may even permanently damage your circuits.
- You may try removing the capacitors connected to the quartz (in case the oscillator stops when you touch the circuit, also check the oscillator fuses are the same and set properly - unlikely that this is the cause, but possible).
- Another thing you may try is to add 50...100 ohms resistors in series with the SPI lines, to reduce reflections in the SPI wires.
- Make sure the wires are well connected and are making good contact, have less than 1 ohm resistance (DuPont wires and other cheap Aliexpresss wires are famous for making very bad contacts sometimes)
Find an isolated plank to put on the floor, or sit on a wooden chair and lift your feet from the floor first, then touch the GND of the Atmega circuit (second), then touch a SPI wire (third - but again, you are not supposed to touch the wires, in the first place).
After all the above, does the display still goes erratic when SPI is touched?
(If the cause was over-voltage induced by your body, by isolating yourself from ground/floor/other masive objects, you should see less interference when touching the SPI)