One of the recent threads regarding ER collets for tool or work holding had the torque figure for correct tightening of ER nuts (or a link to it) as, IIRC, 100ft.lbs. I am surprised that with this in mind there are no spanner flats machined on the collet holder body in the link above to allow use of a large spanner for collet tightening and releasing. I know that some ER32 collet holders are supplied with suitable flats. Maybe it's the ones without flats that cause problems for some people with cutter movement during machining. I would be wary of trying to achieve this sort of torque with a tommy bar in a Ø6 hole or with a pin wrench. Insufficient tightening may lead to the unexpected movement of the milling cutter in the collet.
I know that milling in the lathe is not usually a high force, maximum metal removal rate, operation but any slipping of the cutter is clearly not welcome.
Martin