Harmonic mixing might be a solution to another one of my problems (which is that I can't generate enough power for the LO past 20 GHz).
I programmed my device to use harmonic mixing past 13.3 GHz. In theory, a double balanced mixer can be ran at odd harmonics (if you look at the harmonics of a square wave). If you turn on the diodes hard enough, there should be enough 3rd harmonic to mix higher frequencies down.
Based on my oscilloscope capture, it seems like the insertion loss increases by about 13 dB. That doesn't really matter though because the isolation is nowhere near good enough right now to see the effects of that.
Anyways, if I do decide to try higher frequencies, harmonic mixing is probably necessary since getting the 15dBm or so at 40 GHz to drive the mixers is going to be challenging (but 15dBm can easily be generated at 13.3 GHz, even with FR4). The XX1000 chip is also interesting (it is a frequency doubler IC which fits nicely with the LMX2820 frequency range), and could be used to increase the range with harmonic mixing. The 13.3 GHz cutoff could also be moved up since I only see issues with LO power at 18 GHz.
I also need to see if my homemade mixer design works with harmonic mixing too.
Picture 1 shows the difference between non-harmonic mixing and harmonic mixing at 13.3GHz. The signal on the left is the IF at 13.3 GHz (no harmonic mixing). The signal on the right is the IF at a little higher than 13.3 GHz (with 3x harmonic mixing). The signal on the left is about 4.3 times larger, which translates to 12.7 dB.
Picture 2 shows a measurement of a 30dB attenuator with harmonic mixing. Everything works.
Picture 3 shows the isolation. It is impossible to tell where the harmonic mixing starts (unlike with old spectrum analyzers, where you can clearly see from the noise floor where harmonic mixing is used). This is because my setup sucks in terms of shielding, and there is a lot of leakage. I think I will have to redesign everything to make the shielding better, but that'll be for when I make version 2 of the VNA.
Picture 4 shows the difference between a warmed up and cold measurement. Before, there would be 10dB difference at 23 GHz because the LO power is too low to drive the mixer diodes properly. Now, there is 1dB of difference between the warmed up and cold measurement. Also, I untwisted and twisted the cable, so there is probably some error due to that. To measure it, I calibrated the device while it was hot, and then I waited for everything to cool down. I quickly powered it back up and loaded the calibration file, and then I ran another measurement to get the difference (though I probably shouldn't have unscrewed the cable). I think I probably should buy a proper torque wrench someday and not just use my fingers.