Aim for your usual turning speed but adjust to keep the mill from bouncing off the table. A lot will depend on your boring head and which of the holes you have the tool in as to how out of balance it will be on any particular diameter.
I sometimes add a bit of bar to the horizontal hole to act as a counterbalance when using the outer vertical hole but just watch out as you won't see the spinning bit of bar.