Manganese is an important micronutrient and one of the arguments to eat nuts and whole grain.
Yet, at the same time, it's labelled "toxic heavy metal". This group of toxic heavy metals is really interesting, many are important nutrients present in healthy foods and sold as dietary supplements. Lead and mercury are quite exceptional for two reasons, first they have no positive effects so are unnecessary, and secondly quite small amounts can be really dangerous.
For manganese, WebMD says 11mg/day orally, or 55 µg/day by IV, is the safe upper limit. For inhalation, they just warn against long-term inhalation.
So basically, if you eat the batteries, or grind them into a fine powder and breath that in for long periods of time, you may get manganese poisoning. Normally, manganese poses little danger to general public.
On the other hand, eating batteries is really dangerous because of the corrosive KOH, combined with the fact the battery causes corrosive electrolysis within the body.