If you're serious about it, then I would just pick up a cheap 120-ish GB SSD for about £30-40 and a USB 3 to SATA adapter. Not much more expensive than SD cards and will ultimately save you considerable time with transfers, compatibility and boot/RW speeds.
I am very serious about it. That is a good idea, because it should be reasonably quick as well.
It might be possible to use an ESata connection, to directly plug the SSD's in.
SSD's have been falling in price a lot over the years. They are probably cheap enough now, to do exactly what you just said.
(The other attraction of SD cards, was that you can get them for say £10 each, with reasonably high capacities).
What I've already done, is use the USB flash pen solution. Allowing me to try out different versions of Linux, without committing my self to changing the main HDD's operating system. You don't even necessarily need more than one flash pen, as you can fairly quickly re-purpose the same one, time and time again.
BUT the flash pen solution (with a USB2 port, rather than a quicker USB3 port), was too slow. Just about fast enough to use, but too slow, really.
Anyway, you have convinced me to use SSD's, in the future. As you say, they are cheap enough now, to use like that now. I can either use USB3 as you suggested, ESata or some kind of rapid Sata plug in system (like you have on many servers, on the front of the rack server. The caddie systems, are available from computer suppliers, for about £50 to £75, for the cheaper ones, with about 4 slots).