0.05 mm (0.002" ) run out does not sound too bad for a drill press, rather than a jig borer..
The chances are that your drills will not have absolutely equal length cutting edges, either, so will cut oversize!
If the chuck arbor is mounted in an adaptor, maybe that is the source of the run out? It should not be, BUT!
If you can remove the adaptor, is the MT section undamaged?
Is the MT in the quill undamaged? They are both possible sources of inaccuracy.
Any play in the bearings for the quill, that might be allowing the run out?
HTH
Howard