I suspect that Michael is asking which style of collet to get, not whether a full set is required.
Personally I would say to get an ER32 set of collets with the holder to fit your, about to be purchased, mill. In addition I would suggest getting a myford screw on collet holder, be it direct fit or flange and back plate , do not bother with a Morse one for the lathe as this prevents using long lengths to fit through the head stock.
Again, given the choice, I would pick a collet holder with a separate back plate for the lathe because a. it allows you to, at some time in the future, fit it to another back plate for a larger machine and b, it allows you more scope to get the thing running truly by means of "bump true" and judicious use of tin foil packing. If you were to buy a direct mount holder and it were to be a off, the only way to correct it would be to re-cut the taper, which would then not align to the thread which in turn holds the closing taper. Re-cutting the taper accurately is perfectly possible but it might be outside the scope of some of the less well equipped machinists.
It would be worth reading Harold Hall's words of wisdom, posted a few weeks ago, about collets not closing, but don't let that put you off.
chriStephens
PS if DC1 is reading this, why does the spell checker not recognise "collet", seems a bit odd for an engineering site.