dB represents logarithmic ratio, relative to some value. It looks that for your case it talks about dBm - dB relative to 1 mW.
Assuming that dBm is measured as RMS power, if you want to get RMS voltage you can calculate it as follows:
Urms = sqrt((10^(dBm/10) / 1000) * Z0),
where Z0 is impedance, in your example Z0=8 Ω
For Z0=8 Ω:
-26 dBm = 4.483 mVrms
-25.6 dBm = 4.694 mVrms
0 dBm = 89.443 mVrms
+13.3 dBm = 413.566 mVrms
Also note that this is voltage for a loaded output.
For peak sine voltage you can replace Z0 in equation with 2*Z0, but for other waveform it depends on exact waveform.
The above is incorrect.
To the OP:
Context is key, and the above calculations didn't consider that. You can see that by just visual inspection of those calculated results, such differences in value clearly make no sense for an audio amp.
There's a typo in those instructions in your attachment, and the author cannot have meant -26 dBm when referring to an audio amplifier.
It's most likely a reference to dBu, since there is most likely a 600 ohm audio input.
-26 dBu would be 39 mV RMS, which is the same as 55 mV peak or 110 mV peak-to-peak sinewave.
In other words, the instructions are advising you to apply a 600 Hz sine wave of 39 mV RMS at the input (measure that with a 'scope or an AC voltmeter (some DMMs can do it) that works at audio frequencies, because it's likely that the signal generator equipment may display a different output level, so it needs to be measured).
At the output it really doesn't matter what the actual output is, because it's dependent on the volume setting, i.e. it is up to the user to set it. All that matters, is that relative values across different frequencies are correct.
So, the procedure would be to set the volume to some nominal value (you can connect a speaker, or a dummy speaker load, or even leave the output disconnected, although for best accuracy you will want to connect a speaker or dummy speaker load), and then measure the output, and call it 0 dB.
Next, you'd change your input frequency (but keep it at the 39 mV RMS level!) and then see if the output varies to the amount in those instructions.
Here's an example.
Let's say with a 600 Hz sine wave input at 39 mV RMS, you measure an output of A volts, where A could be (say) 0.5V or 0.6V or 1V or whatever (it can be RMS or peak or peak-to-peak, no difference), it doesn't matter. Let's say A is 0.123 V RMS. That's going to be your 0 dB reference.
Now, if the instructions advise the output at 6 kHz to be D dB where D is +9.1 dB on the HF output, then you'll need to look for a voltage of A * 10^(D/20) which is 0.123 * 10^(+9.1/20) which is 0.123 * 10^0.455 which is 0.123 * 2.85 and that's 0.35 V RMS.
So, if you'd measured 0.123 V at the LF output at 600 Hz, then you'd want to see 0.35 V output at the HF output at 6 kHz.
I may have made a mistake in the above info, apologies if I did, but I think it's mostly correct, and is broadly along the lines of what's actually intended by the instructions that you showed.