mEDBG is stupidly slow, since it is only a Full Speed device.
I wouldn't expect "Full speed" to be that much of a limitation (~10Mbps!), though the mEDBG is pretty slow.
I'd offer other explanations:
1) Other resource limitations of the ATmega32u4 limit performance.
2) CMSIS/DAP might be inherently inefficient (or badly implemented) over high-transaction-cost paths (for example, we've noticed that TI's CMSIS/DAP implementation in their MSP432 launchpad is significantly slower than the TI proprietary debug protocol.)
3) No one has expended much effort toward improving the performance of what is supposed to be a second-class debug product anyway.
(I suspect (3) is the most likely)
Do you know of any open source CMSIS/DAP to SWD implementations?