However, instead of using pogo pins for this application, I recommend using machined socket receptacles and pins.
I am strongly inclined to agree.
Like others, I have a concern about the quality of the contact using pogo pins. A firm plug-and-socket arrangement gives me much more confidence about the connection.
While you say the pogo pins are making contact, it is clear that there is something problematic with them since replacement with hard wiring results in correct operation.
If you want to track down why there is a problem with the pogo pins, I would suggest you try replacing one pogo pin at a time with a wire (or vice versa) to find a pin which is involved in the problem. Once you have found one, then leave that connection hard wired and try the others, just in case there is more than one pin causing issues.
Since I would guess it is likely there is only one pin involved, I would then get really up close and personal with it. Microscope, photographs, low speed and high speed signal measurement - including transients, whatever you can think of and maybe, just maybe, you will find that Eureka moment.
If you just want to fix it, I'd try the plug and socket idea.