I’m the opposite and hardly ever mark out these days. I tend to use an edge finder and DRO to position things on the mills and the drill press never really gets used to drill holes. If I do mark out it is usually just to work out the best placement in a casting which can be a case of placed to suit rather than place to measurement, even then I won’t punch but just eye up the drill over the mark or use a wobbler where it matters a bit more.
I would say 95% of the time I don’t even spot drill, I just go straight at it with a stub drill usually held in a collet unless I need a deeper hole in which case I spot and then use a standard jobber length drill. That applies to the manual mills as well as the CNC.
My drill bits of choice are Dormer A022 for the stubs and A002 for jobber length, these are “split point” and really don’t need spotting first.
I don’t seem to get problems with wandering, items drilled with a PCD Hole pattern tend to fit in any position and can be flipped over and still fit and that is with my smaller clearance hole. That is in a mix of steel, cast iron, brass, bronze & aluminium.