The point of most recommendations re: Arduino is that you have to start somewhere. Since everything that can be done with an Arduino has already been done and is documented on the Internet, you have plenty of help both in terms of hardware and programming. Don't dismiss this lightly!
Yes, I started on bare iron, the Arduino didn't come along for 40 years or so. If I need something done 'right now', I reach for an Arduino UNO. If I want it to run FAST or I need more hardware features, I grab something else. The original mbed is my #2 choice followed by various LPC2xxx ARM chips. If I have to, I'll use STM32F chips but I'm not proficient.
If I were a little more confident of the learning curve, I would join forum user Danadak in recommending the Cypress PSOCs. The programming infrastructure is magnificent. You basically drag and drop gadgets on a sketch and then generate the program. The IDE creates all of the required code to use the devices and they even create main.c. Your job is to flesh out the details but you don't need to worry about the underlying peripheral code. It is already done - you just use the predefined functions and macros. Documentation is outstanding. Cypress has done a terrific job with this product line.
I like the PSOC 4s a lot and there are quite a few videos - search for 'psoc video'.
Alas, that is probably a step too far at this point. The Arduino is the way to start.