Any single point cutter will work, particularly if done in a rigid setup. Of course, step drills are in that category, but you eliminated them from consideration.
When working on aircraft sheet metal, a wider included angle (i.e., 135° vs. 118°) is often used. That helps with small holes. In fact, they are sometimes referred to as "aircraft drills." You can get sets in number, letter, and other dimensions.