Both are your tapers with be non stick or release with a tap, don't over think it.
I can't think of a better way than how I achieved the required sleeve.
I had in stock a 30 int taper with a 1 inch spigot used for mounting a boring head
I chucked up a piece of bar and machined an internal taper to a blued fit to the 30 int taper.
Rough turned the outside to the approx size of a 1 3/8 taper.
chuck another billet and turn a spigot on the end to suit the hole in the 30 int tool.
without removing from the lathe, drill and tap the four mounting holes and a center tapped hole for securing the 30 tool.
secure the half made sleeve by the center taped bolt, and machine the outside to fit your mill, with the ability to remove it and replace it to check for a blued fit, without loosing concentricity.
Pete