Steve's question shows the hole has to be quite accurate.
Whilst a hole-saw will rough out the bulk of the metal, even the rigid-bodied type will NOT give an accurate hole! They are not designed to do that, they leave a rough-walled hole, and tend to run eccentrically. Also, most of the readily available ones are sized for standard water-pipe and electrical conduit material in applications not needed a close push fit.
The older type of hole-saw, consisting of a ring of bandsaw-blade clipped into in a groove in the face of a large bush, is even less accurate.
Do use hole-saws, but be aware of their limitations, and use one generously under-size to leave accurate finishing by the various techniques described above.
A step-drill in a decent machine-tool may give the fit you need for the particular work-piece: try on an off-cut first.
A cone-drill can be used with care to give a tight fit, but obviously the hole will be tapered and diameter control is difficult. The significance of the taper will be controlled by the material thickness and the fit required.
Step-drills and cone-drills are not desperately costly from places like Screwfix and Toolstation (which also sell hole-saws), and very useful tools indeed. Assuming appropriately matching diameters of work and tool and the fit requirements, they can leave a finished-size hole needing no more than as the drawings say, removal of burrs and sharp edges.
I use all of the above for various tasks, obviously depending on material and purpose, and have even roughed out holes through thick plastic with a hole-saw, though that was somewhat fraught process as the tool is really intended for thinner and slightly harder materials than an inch of soft thermoplastic (PVC)! I did that 'cos I wanted to keep the central slug as bound to come in handy one day…