As you have become aware, there is usually no such thing as a single audio level. The normal approach for level meters is some form of averaging. Back in the day, a VU meter would have mechanical ballistics - the needle would respond fairly rapidly going up, with some sluggishness (low pass filtering - of the needle movement, not the audio), and the needle would fall back down more slowly, giving the user time to see what was going on. The user did the averaging and such with intuition.
One very important question is whether you want to measure an average audio level, or peak levels. In the digital realm, peaks are very important, because they're going to determine whether you have, or will have, clipping. It is an objective measurement of a specific level. The time element here would to hold the indication long enough for it to be useful.
Average level, on the other hand, is generally used for more subjective purposes, e.g. "Is it loud enough?" Here, the usual approach is to convert to RMS, and again there is a time element in how quickly the average is made. The levels reported such an averaging meter are usually divided into ranges, at least two, or better three, or more, that can be considered something like "signal present," "low," "normal," and "high," perhaps with a separate peak indicator that means "dangerous." This is what you'd normally find on a mixer, for example, in one form or another.
Without knowing what you need to accomplish, I can't suggest a circuit. I would suggest you investigate peak detectors, though - that might be what you'd want.