I will add my few pence of idea as well.
Use a thinner board, and make the red solder mask on the topside have clear sections where the led displays are going to be. Place a white screenprint around the areas where the digits are going to show, and a grid of white to delimit the digits. Instead of placing the leds on the front, place them on the back so they show through the clear pcb material, using the material as a diffuser. The white mask will help with digit reading, and will also enable you to hide the traces that you will need to common the columns. Rows can be routed on the back side, so that the front panel is bigger display wise, has easy to read masks and has no visible display when off aside from the legends. If you do not want to have multiple vias under the soldermask and screenprint, then you can just use 0R jumpers on the back side to provide the required crossings. Might be cheaper than vias, but more parts to lay out.
Just a thought, I have used thin clear pcb material as a display cover for HP led displays that run on 5mA, they are visible through it when on, and off is a milky white. The on is somewhat diffused, but perfectly legible. It will mean using a thinner board material, or at least a clear one with a reasonable light transmission.