See:
Your problem is probe compensation. The input of a probe is not just a 1 Mohm resistor, but there is some capacitance (in the order of 20 pF) in parallel. At higher frequencies (above a few tens of kHz or so), this capacitance becomes dominant. This does not matter for a 1x probe: it is 'just' a fancy specially engineered piece of screened coax. A 10x probe attenuates the signal by putting a 9 Mohm resistor in series, however. The capacitance in parallel with this 9 Mohm resistor has to match the capacitor in the scope (20 pF / 9 in my example). It becomes a capacitive divider at higher frequencies. Adjusting the cap matches the attenuation of this capacitive divider to the attenuation of the resistive part, so low and high frequencies are attenuated equally.
So turning the trimmer should allow you to switch the probe from undercompensated to overcompensated, with correctly compensated in between. Why doesn't this work?
- Maybe you're twisting the wrong trimmer. Some probes feature a LF and a HF compensation (ignore the HF compensation for now). Is there another trimmer in either the probe body and the part near the BNC connector?
- Probes are designed for a limited range of input capacitances. Maybe the input capacitance of your scope is outside this range? This is not likely unless the probe in designed for high frequencies well beyond 100 MHz, the probe is very old, or the scope has a capacitance much higher than 20 pF. You should be able to find the input capacitance of the scope either on the front panel or in the manual. You may be able to find the input capacitance of the probe in the datasheet if it's a brand name probe.
- The probe may also be broken.
What happens if you connect it to your other recently acquired scope and try to adjust it?