To save time, you can buy from folk like ArcEuroTrade, Chronos etc. These will be double ended, and have spare holders, so that four different sizes of die can be used.
Often, a side arm rests against the four way toolpost to resist rotation forces.
Mine is commercial, but so old can't remember its origin, probably Chronos or Warco.
Made up a similar arbor to carry a shop made ER25 Collet holder for Taps. (Used both yesterday!)
In both cases, the holder has a little clearance on the arbor, so that it can "float" and follow the workpiece, instead of trying to do a little off centre machining of its own.
Howard