AFAIK there is no math involved; either you find the number (simply given in percentage!) in the diffuser material datasheet, or if they don't publish it, you have to guess or measure.
Very good diffusers can indeed approach 90% transmission while still diffusing properly but that would be an expensive special product; this would also require special coatings since you lose almost 10% total already in the two air-plastic interfaces even with a perfect clear sheet with no absorption at all.
Usually there is a tradeoff between the quality of mixing and transmission. Mixing the different color LEDs require that the material internally reflects the light in random directions number of times and this involves losses.
A large part of the loss is due to diffuser doing it job, diffusing the light also back to the LED side. Adding reflective coating (aluminum tape, chrome paint, bright white paint...) on that side with just large enough holes for the LEDs help, but it's still the dominant loss source because there will be a mismatch in refractive index so each pass of photons to that reflective coating and back to the diffuser "core" loses energy.
Sanding plain clear Plexiglas (PMMA) sheet with a coarse sandpaper is indeed in my experience surprisingly good, at least it's better than just using opal white PMMA, these do good job at diffusing but with the cost of being horribly lossy.