So now I really do not know what to do
Start by working out what you expect to be able measure, simple numerical input and output voltages. Then check each item works on its own. Then gradually connect them together and verify measurements correspond to predictions.
Does the AD work? Connect the AWG outputs to the scope inputs.
Does the amplifier work? Use it in a normal audio amplifier configuration.
What is the amplifier maximum power output, into what resistance? Use that to calculate the maximum output voltage, RMS. Compare that with the amplifier peak-peak output voltage you previously measured. (Remember for a sine wave, V
pp = 2.8V
rms). Was the amplifier clipping?
Connect the AWG outputs to the amplifier input, and listen to the output. Does the volume sound sensible and undistorted? Measure the output voltage (RMS) with a voltmeter, not the AD. Compare that with the voltage calculated above. Make sure the measured voltage is much lower than the maximum voltage. If not, reduce the AWG output voltage.
Connect the scope probe inner to an amplifier output, but do not connect the probe shield to anything. What does the scope show? A sine wave, or a clipped sine wave etc.
Finally, only when all that looks right, do a distortion measurement.