I'm not an expert, but looking at SDS and the example of the fire at the selenium rectifier factory, it is rare for irreversible symptoms to occur, and with the appropriate knowledge, I don't think the toxicity to the human body is something to be feared.
Mercury is contained in common products used by the general public, so there are established routes for disposing of these, but selenium is not expected to be handled by the general public, and the lack of a route makes it difficult to dispose of.
Swollen lithium polymer batteries are also cursed items that are difficult for individuals to dispose of correctly if you try to follow the rules.
Some manufacturers will take back lenses if you send them back when you dispose of them, but only if they were purchased from that manufacturer.
That is the situation under the laws and rules in my country, but what are the rules in your country?