Calculating THD+N which one is the proper way?

[gf]

Having more samples of the same spectrum can be useful in some calculations, apart from being useful to display/zoom. Some time ago a method to compute precise estimations of the frequency of harmonics relied on zero padding to get enough bins near the peak, then fit a parabole thru the highest bin and neighbours and identify the vertex of such parabole as the harmonic frequency. If I remember correctly, 5x-10x padding was used.

Quadratic interpolation (to locate a peak) between three adjacent frequency points in logarithmic (decibel) space is exact only when used with a Gaussian spectrum analysis filter¹⁾. When used with any other window function, it's just an approximation with limited accuracy. Then, interpolated frequency samples (with narrower spacing) are certainly helpful to improve the approximation accuracy of the quadratic interpolation.

¹⁾ The Fourier transform of a Gaussian function is again a Gaussian, and the logarithm of a Gaussian is a quadratic parabola.

[bson]

Now that you're around, i'm currently measuring SNR, SINAD, THD, THD+N and SFDR. I also considered calculating ENOB, but i'm not sure if it's something worth calculating when the oscilloscope i take the signal data from uses an 8-bit ADC.

Addition increases precision.  So if you add up 256 8-bit values, you get a 16-bit sum, with 16 bits of precision.  This also applies to the mean, but it's really the addition part of the mean that produces the increase. Dividing the 16-bit sum into a mean, by 256, would give you 8 bits of integer and 8 bits of fractional part.  But the division part of the mean isn't needed to increase precision - all summing will do it.  Including things like low-pass filters.

To calculate THD+N you remove the fundamental.  The easiest is to capture both the fundamental input and the output.  Then you can multiply the fundamental or divide the output to normalize their power/voltage levels (by the amplifier or circuit gain), take an FFT of both, and subtract the fundamental input spectrum bin by bin from the output.  Then sum up the power/voltage of the remaining bins, and there's your answer.

If the output fundamental is lower than the input (after normalization) you have gain compression.  This needs to be characterized and corrected for, or it will show up as THD+N or even mask it (by the fundamental contributing negative power in a portion of the spectrum).