Resurrecting this as I've revisited this topic a week ago while getting started with an ESP32.
I think I got a solution to my initial problem: Eclipse + PlatformIO + OpenOCD with a hardware debugger of your choice.
PlatformIO has a debug option, but it's a monthly plan, so I did not even test it.
You can use any IDE you want with PIO, but Eclipse was my choice as I've used it for >10 years.
PIO allows you to use standard C/C++ code and move slowly away from the Arduino environment as your project grows. But you can still use the extensive Arduino library set, which is not worth reinventing. Especially the TFT/EPD libraries.
Debugging is a mixed bag, I'm currently using a Bus Pirate with a custom-built OpenOCD. It's slow and works only 50% of the time. But when it works, it's priceless: you can inspect all variables, step through code, see all the threads, just like you would on a computer. But it took me >12 hours to set everything up, mostly through trial-and-error. And the platform (ESP32) is still in development, not as stable as the other ones.
I've got an Olimex Tiny H on the way as well as a JLink clone, hopefully these will be faster, more reliable and better supported.
Portability is decent: Eclipse CDT is stock, PlatformIO is easy to install, the project configuration files (makefiles and tooling) use a path system that make them usable across machines. Versioning control is done on bitbucket.org , as they have free private repositories. I guess you could host your own, a server costs 5$/mo at digitalocean and you can do a lot with it.
Ease of use - currently the best compromise I could find. You can start a project in Arduino (if you want a dirty prototype), create a new project with "pio init" and copy the arduino files over. Maybe make a small change to the platformio.ini file and then just click "Build" and "Upload". Unfortunately you have no direct control over the Makefile, AFAIK. But that's beyond my paygrade anyway...