FR-4 is both quite absorptive and scattering. That is, it makes a terrible window, and a slightly better diffuser. Both act to reduce the brightness of your LEDs.
Standard solutions exist, fortunately: bottom-firing LEDs, placed over holes in the PCB; light pipes, and transparent (or diffused or tinted or..) overlays are excellent, and quite affordable (at least in small production quantities; YMMV for one-offs).
Very thin PCB would be competitive, but is mechanically questionable, and special-order. Flex PCB is also quite available, but as the substrate is yellow to brown (depending on thickness), it wouldn't be as practical a transmission medium. But you can bend it into whatever orientation you like, again making standard top-firing LEDs quite reasonable.
You can kind of combine the two and get overlays with LED dice integrated, too.
Tim