If you really want to use 3V+ Vf LEDs, why not simply boost from 2xAA to 5V, and run the MCU, logic and LEDs from 5V? USB power can feed in to the regulator output via a simple Schottky diode, with a circuit to activate the boost chip's inhibit pin when USB 5V is present, and if you want to get fancy, you could use a LiPO for the battery and add a LiPO charger IC. The less the voltage drop across a LED's series resistor, the more small variations in Vf affect the LED current and thus the brightness.
If you dont drop 20% in the resistors, and you want them all to have the same brightness, you'll have to hand match the LEDs for Vf @5mA If, and use matched ones for each row. Ideally you'd match all the rows, but you can trim the brightness on a per-row basis via the PWM.
An alternative on the LED side of things would be to use
Neopixels. They are smart, fully dimmable RGB or RGBW LEDs with a daisy-chained serial interface, and run from a 5V power bus. Available in a variety of form-factors including individual SMD or thru-hole, and prepackaged matrixes, bars and flex strips. You could do all sorts of effects, e.g. making the colours reflect the time of day - dim cool blue-white (simulated moonlight) at night, reds and oranges for dawn and disk and bright yellow-white for daytime.
Put a ground plane under and round the switching regulator module, 'stitch' the ground tracks to it with plenty of vias, and keep logic level tracks away from it. Otherwise it will probably give you hell with lots of EMI.