I've had another (untested) thought on this that I think should work OK for some jobs and should be pretty straightforward to do on a vertical mill.
You could tweak the arrangement shown by offsetting the D piece a bit further away from the bore than in the pictures, so that it's more fully held around the circumference and able to be held by friction fit.
Martin.

