A few thoughts come to mind:
Surely, any misalignment that produces millimetres of wobble would break the tap, or at least the wobble of the tap would be visible as it cuts the thread?
Can we safely assume you have clocked the piece true in the chuck before drilling? Although if the turning, facing, drilling and tapping are done without removing the workpiece everything should be concentric anyway.
Have you tried taking the part off the lathe and tapping it by hand? That might prove whether the lathe is inducing the wobble.
I feel the drilling must be at the root of the problem, a tap can only follow the existing hole. If it were forced to deviate by millimetres it would only be cutting on one side of the hole, and would surely break, as above.