You'll have a lot fewer problems with alignment tool fragility if you check the force to turn the trimmer isn't excessive using a regular screwdriver before using a non-metallic one.
Cores are a different matter. Traditionally, a strand of elastic was often stretched through the former to provide friction before the threaded core was originally inserted, and if that elastic has degraded to goo, the core may be very well locked in place, so attacking it with a metal tool is likely to result in it fracturing. There's also the possibility its been locked with hot wax, and again, attempting to move it cold will probably be a disaster.
Hollow cores with a hexagonal center hole are some of the worst - the wedging action of turning a hex key in the hole is extreme and ferrites (and other ceramic magnetic materials) are brittle with relatively low tensile strength. Old iron dust cores are probably the worst - they crumble at the slightest provocation.
For slotted cores, a soft brass blade, filed to an exact (but not tight) fit, with all edges and corners 'broken' on a fine stone is probably the safest compromise between strength and kindness to the core, for initial adjustment. For hex cores, using anything except a plastic hex trim tool is a last resort. Before reaching for the Allen keys, try heat and solvents as trying to shift a stuck core with a metal tool seldom goes well, and even if you have an exact replacement core handy, it may be impossible to remove the broken fragments of the stuck core without wrecking the former.