Here's an approach I haven't used but should give a very accurate result.
Drill and ream four oversize blank axle boxes.
One at a time, stick a (relatively small) round bar in the blanks and use bar across the top of the milling vice.
Use this set up with each block in turn to skim the TOP dimension of the boxes.
Now use a square on the top surface as a reference, to reset each box in turn at 90-degrees each way using the same bar and setting the cutter taking a cut at the appropriate height for the sides of the boxes. If set accurately the boxes will all be parallel with the hole central and sizes the same.
Now reset the cutter height and do the bottoms.
Finally groove or step the sides of the boxes as required.
I have found the bar in a hole method to give cuts to a very high level of repeatability from a hole.
This is also a very easy way to mill a round surface concentric with a hole, by taking multiple cuts.
Neil