I'm pretty sure I posted some timings on here a view months ago that very much reflect your findings, using a 1050ti, 1070 and 1080.
One thing to note,as I'm sure you know, is that as soon as you do any effects, even as simple as text annotations, for that segment all bets are off. For simple cut editing without fancy transitions, PIPs, split screens, or other effects etc, there's no practical difference that I was able to determine.
It does also very much depend on the software you're using, something that works well in , say Adobe Premiere might not work well in Vegas and vice versa.
Also a setup that works great in 1080p does not mean it'll work well in 4k: the GPU encoders and decoders appear to be optimised for certain standard definitions in my experience.