I would suggest that threaded holes in the mandrel would be the ideal means of jacking the chuck off. It should be a tight fit to maintain concentricity. The faceplate should be machined as a tight fit as well.
A better alternative is the bar in the chuck jaws and use it as a slide hammer, to pull the chuck off square (think here a suitable bolt, with nut, and a lump of suitable metal bored to slide on the bolt).
Also, best to refit your chuck(s) in the same position every time they are replaced, particularly for a self centering chuck, or a different concentrity error may be apparent on replacement.