One more update (I need to go on date night with my wife after this one).
I tried changing the gain to 10x and that greatly reduced the size of the spike. Now the spike is only a -0.25 V spike. I suspect if you know more about op amps this might help you know if the problem is from noise on the v+ line or from the op amp inputs. The voltage differences on the input lines would be so small (0.025V) I'd have trouble seeing them on my scope, but that is something I can try.
As a new owner of a
high speed power supply, I do wonder if what you are seeing is a problem borne from a "slow" power supply. Surely a USB power supply qualifies as slow, but so also might your bench power supply. If the (about 3ms) event which is happening every 100ms is actually transmitting a wifi or other radio signal (and this could make sense? I thought only APs send 100ms beacon packets, but maybe clients do something similar), then the huge (relative to the idle) current draw could cause a voltage drop like this.
A quick way to test this theory would be to power from an actual battery (which is even faster than a high speed power supply). Probably both with and without extra capacitors.
As a quick rule of thumb for thinking about how fast or slow a bench power supply is, note that most (even high end) power supplies are rated to return to the requested voltage (after a pulsed load) after a period of something like 100 µs (or much slower)
for a load change 50% - 100% of the current load. The aforementioned high speed power supply (and others like it) return to the requested voltage in less than 40 µs
for a load change of 1000%. And in practice they are often still faster than this. I presume an actual battery is even faster.
In any case, this may be totally the wrong route to investigate but it should be a quick and easy test.