#1 reason for avoiding numerous and large layers of frameworks and libraries is not performance, although that is sometimes important, too. Code writability, readability and maintainability are the actual reasons. You can do a surprisingly large project in just 2000 lines of code within five-ten .c files, so that the whole project is within grasp of a single developer, and another one can catch up quickly, and modifications do not require "paradigm changes" because of lack of arbitrary limitations added by "frameworks".
For a beginner, this is even more important. A 100-megabyte SDK, where Hello World requires a 2000-line logging library, and LED blinker requires ten files with BSP generator programs, leads to the fact that the beginner has no idea about anything, can only run the existing example, and if everything happens to go well, might be able to modify some parts of the example.
Hence, I second the suggestion: keep the Arduino so far; learn to program it bare metal. Basically, this means downloading and reading the AVR datasheet and following it. Because bare metal AVR is simple, it's a step of correct size for you: not too big at once, but not insignificant or completely without challenges, either. After that, you can pick any popular ARM Cortex-M0, M3 or M4 based microcontroller and keep going. But that doesn't need to happen right now.