Just get light pipes (I’ve used the round ones that just press into a hole) that have a matte finish, press them in, and then put black tubing onto them to prevent bleed from adjacent LEDs. For this application you wouldn’t want them too thin.
You can also use tubing without a light pipe (in which case opaque white is ideal) to act as a light guide and baffle. Then you need some kind of diffuser on the panel. One thing I experimented with was to drill small holes in an aluminum front panel, then use a heat gun to heat the aluminum. Then I pressed a hot melt glue stick into the back of the panel, causing it to melt into the holes. After cooling, I used a scalpel to trim the excess from the front. Then I hit the front with hot air again, just for a second, to smooth the cut surface again. The end result is fantastic, using standard milky hot melt glue. This combined with a 3D-printed baffle produces superb results with RGB LEDs.