Any milling solution will be made more difficult by swarf removal.
Turning the head of the mill sideways, having the shaft horizontal and starting at the bottom will allow swarf to drop away, as well as facilitating access for a blow gun to keep the cavity clean.
The shrink fit sleeve is a good suggestion, but we would need to know the diameter of the shaft. 20mm long in a 25mm diameter item is different to 20mm long in a 60mm diameter item.