I can't remember any more how much of the Adafruit libraries I actually used. I had graphic libraries for windows, linux and sgi which I developed graphics primitives for drawing and filling points, lines, rectangles, circles, etc. using OpenGL, DirectX and Xt/Xm. Really, I just trimmed down what I already had to fit into a microcontroller. I know that probably doesn't help you much but for an Arduino, I would probably just stick with the Adafruit libraries.
Like I said before, I would adjust the Adafruit graphics initialization to dim down the OLED intensity. They have it set pretty much to max just burning the OLED. You can cut the brightness in half and it's still bight enough for anything indoors.
The biggest issue with some other graphic controllers is that they require that the host microcontroller maintains a copy of the graphics memory (which can be very large). Too big in fact to fit in small microcontrolles RAM space. Essentially, you're building the graphics screen in the microcontollers memory and then transferring the memory to the display.