Turns out the root of a lot of the scopes problems are caused by that special IC connected to the FPGA.
I don't know why earlier testing failed, when I used the FPGA commands for reading the samples directly, without the translation via the special IC, but I changed the code to do it without them and now it works. Also removed the translation of the volt/div setting via this IC, and that solves the occasional zeroing of the traces.
On earlier tests I could see the traces flash as a straight line on the bottom of the scope once in a while.
The tests I ran today showed that at random the readings form the special IC fail. The software then returns a zero, and when multiplying with this value the result is zero

Also found that the brightness translation is based on a simple formula: (brightness * 560) + 4000.
Another finding is that the FPGA command 0x0D is used for setting the sample clock. For the long time base settings in the original code this is always set with 2000, which seems to be equal to 50KSa/s. For the other time base settings the values go from 99999999, 39999999, 19999999, ..... 99, 39, 19, 9, 3, 1, 0, ... 0.
The zero value is for 200MSa/s.
This means it should be possible to select a higher clock for software/timer based sampling. But this needs to be tested.
The last thing I have to look into regarding this special IC is parameter 0x11. This is used for reading the sample data and has to do with the trigger point. Failure at this point will result in the trigger point not being on the correct spot, so I would like to get rid of this translation too, but it is a bit harder to determine the range of the input values, since it is read from the FPGA after a trigger has been detected.
So a bit more testing and researching to do.