Well, they're all polarized, at least.
As far as electrolytic, the tantalum oxide I think is produced electrolytically (anodizing, as with aluminum and niobium types as well), but the electrolyte is cleaned out and replaced with a heavy-ish layer of MnO2 ("solid electrolyte") and a thinner coating of silver (and/or graphite?? I forget). I forget if polymer types are anodized in place (thus obtaining oxidation from the solid polymer electrolyte), or if they are formed before the polymer is applied.
So, as far as what is "electrolytic" about them: it's probably the least questionable to call "wet", formed types (aluminum electrolytic, wet slug tantalum, and maybe I'm missing a few?) electrolytic, in that they involve an electrolyte, and their nature depends upon an electrolytic reaction, both during manufacture, and in maintenance of operation.
As for [dry] tantalums and polymers, it's somewhat more questionable to call them "electrolytic", as manufacture involves an electrolytic step, but they do not use an electrolyte, at least in the conventional sense. (They probably should still be, as electrolytic reaction is still possible -- tantalums can be "reformed" to a small extent, though it's my understanding that it's more of a self healing process (which should exhibit a minuscule decrease in capacitance?). So, does it look like a goose, or quack like a duck? I'm not sure. I'm not sure offhand if polymers exhibit similar reactions; it's my understanding they behave pretty much like film caps, and don't really reform or degrade.)
Electric bilayer (supercaps) types, conversely, are specifically designed not to be electrolytic. That's why they all have such a consistently low voltage rating: it causes breakdown of the electrolyte. But, they are still "wet", in that an ionic liquid -- an electrolyte -- is involved.
To extend the nit-picking even further... if the condition of "electrolytic" depends only on using an electrolyte in the design, or an electrolytic step in the manufacture, then shouldn't any capacitor which utilizes an electrolytic reaction in the active region, be labeled as such? If so: this would include the opposite process, that is, applying a metal contact to an insulating dielectric. An example would be [usually "electroless", but hey, redox is redox, right?..] silver or nickel plating, say for ceramic (the macroscopic kinds) or mica types. I doubt this is actually used in any current production capacitors (vacuum deposition seems much more likely, as I know is the case for film caps), but the possibility gives an interesting twist in exploring the scope of possible definitions in this conversation.
So...
TLDR: it probably is more correct to say all polarized caps (and "bipolar", which are just polarized pairs, connected back-to-back internally) are electrolytic. How correct it is is probably a matter of interpretation, with some being more likely than others. So, if you forget sometimes, that's OK. And you can never go wrong by being specific about the type, without having to worry about the definition (a tantalum cap is a tantalum cap, whether the oxide was grown by anodizing or by virgins warming billets of compressed tantalum between their soft, thermally enhanced thighs). (Wait, what were we talking about again?)
Tim