I'm not really looking to test robustness of the CAN interface, so I'm not keen on any methods that have the potential to damage my test setup (yes, I know CAN transceivers have protection against things like short circuit and transients, but I'd rather not test them

).
Get one of the many CAN to USB interfaces with their free entry level PC application. Every one I've seen allows you to create various kinds of soft errors (the data format kind that aren't just shorting CANH or CANL to something). Peak and Kvaser are two names to investigate. I went through this several years ago, nearly built my own from scratch, but in the end it was cheaper and faster to buy off the shelf.
That sounds great, but having previously seen prices for Peak and Kvaser stuff, I almost certainly don't have the budget for such things. This is an amateur project, after all (as evidenced by my test setup

).
By the way, how would be they creating soft-error traffic? Don't CAN controllers (whether MCU peripheral or discrete) not allow you to create invalid CAN frames? Like, with bad CRCs, incorrect bit stuffing, etc; because those features are all handled automatically, and not exposed to the user. I guess maybe they are forming their CAN frames 'by hand'?
A good way is to connect CAN-L to a metal file and drag the CAN-H line lightly across the file teeth.
This creates very short intermittent shorts on the bus that might only corrupt only a few bits in a CAN message rather than kill the bus completely for the duration of holding a pushbutton.
Not sure if serious...

I do have a large metal file in the toolbox, though...
But would there be a different approach to constructing a way of intermittently shorting the bus? Maybe an N-channel MOSFET between H and L, with gate triggered for a millisecond or two pulse on a frequent randomised basis by RNG on transmitting micro?