The eccentric flange inside the nut engages the external groove on the collet, and when the nut is slackened, extracts the collet from the 8 (half angle) major taper.
ER collets are very useful, as most have a range of 1mm (0.5mm in the smaller sizes, but not ER32) This allows them to be used for Imperial and metric tools /material without damaging the collet.
If you have a full set of collets, you should be able to cover from 3mm up to 20mm diameter
Unfortunately, unlike 5C, ER collets do not seem to be available for square or hexagon material.
Nevertheless, very useful for work or tool holding.
In Robin's case, I would wonder if the internal retraction flange had been distorted, and was causing the wobble?
Howard