I'll second everyone else's advice: ER collets, each of which will compress down by 1mm (0.5mm for the smaller ones) combined with a bolt-on collet chuck, would serve you well. Very good for holding stock on the lathe, in addition to gripping milling tools.
I have a set of ER25s (the 25 refers to the o/d of the collet at its widest point; the capacity of the biggest ER25 collet is 16mm), with a bolt-on collet chuck for the lathe. My old miller has a Myford nose thread, so I bought a screw-on ER25 chuck which screws on, thereby avoiding the heavy whacks sometimes needed to free tooling from its internal MT2 taper. If you do buy a miller, R8 tapers are much easier to free off.
But as Joe says, expense can be considerably reduced by making up your own holder holder. Here, the milling cutter is only retained by a grubscrew so it would be as well to go for one with a flat on its shank, and to pull the tool forward as far as the grubscrew and flat will allow before finally tightening up. Otherwise, the tool might try to screw itself into the work and drag itself forward in the holder.
Andy