I apologize for not being more forthcoming about the device. I'm involved with others on something that might actually be a product some day, and just can't say more right now.
This makes it hard to suggest anything at all. However a gear box should not be causing you problems if in fact there is no significant load on the motor. I just reread your post and just realized that I might have assumed that the spinning rod was some sort of drive shaft (inline with the motor). If it is mounted perpendicular to the motor or gear box shaft, you will have a very unbalanced load which will lead to all sorts of issues.
No, you were right the first time. The rod is inline with the motor. However, even so, it turns out to be heavier along one side, so there is a balance problem I need to deal with. I've mounted it at the very center of two flanges, but that assembly always rolls to one position, and is clearly unbalanced. I can add weight to make it statically balanced, but I think it will not remain balanced when spinning - because the added weights, which stick out farther from the center of rotation, would be relatively heavier when it spins up. Something about moment of inertia, r-squared, and such, which I don't understand.
Or I can mount it off-center on the flanges, but so that it's statically balanced at rest, but again, when it spins up I think it will be out of balance again because the mass isn't evenly distributed around the center of mass and center of rotation. It seems to me that there must be a mounting position on the flanges that would produce a balanced assembly when spinning at 10 Hz (600 rpm), but I don't know how to find that position. But at this point I don't know if the out-of-balance condition is typical, or just a bad copy.
What bothers me here is that you say the RPM varies widely which it shouldn't do with a stable power supply and an even load. If you are getting significant speed variations I suspect that you have not considered the load being driven, it might not be as trivial as you think. This comes back to the issue of a gear box, sometimes you need a gear box to better match inertia. IF this is going to be a product for sale you might want to seriously consider hiring a consultant or cranking through the numbers yourself. A brushed motor may or may not be the right solution here and frankly a gear box may be required no mater the motor technology used.
The speed variation was after it broke and I removed the gear box. So it was completely unloaded at that point, and I was just trying to see how slow I could make it run by adjusting the voltage alone. But below about 4V, it would speed up and slow down with no change in voltage at all. So it seems the brushes are not right, and whatever broke the gears also messed up the motor.
I hope the new, larger geared motor will work well enough. It certainly would be more convenient to just turn on the power and not have to add a controller circuit. But the rod-and-flanges assembly imbalance may be a deal breaker for the whole thing.