Your KdB value should be something like 0.1, not 0.28, if I'm understanding this correctly. It's the error between your device and the reference and it looks pretty small to me. The sign matters. However, I see that the discrepancy is not linear at the ends of your scale and that may be a problem.
Then your 'CLIN' is the amount of correction you need to apply, with CLIN = 1.0000 meaning no correction. You are calculating and then entering a new gain constant and the equation is just giving you the new gain constant by taking the old one and multiplying it by the deviation.
So if KdB = 0.1, then 10^(0.1/20) = 1.01157945, meaning you are going to increase the gain constant by about 1.1%. You'll still have issues at the -16 and +10dBM levels.