Aside from the challenge and enjoyment of constructing your own workshop machinery, you might find it easier to purchase a small milling machine that is already complete and works.
However, machine tools that have the words "robust", "rigid", and "precise" in their description (and reviews) do tend to cost a little more than those that also include the word "not". (The obvious exception to this is some of the machines made by Proxxon which while not cheap are perhaps better suited to the needs of those involved in model making and handicrafts rather than model engineering and horology.)
I have no hesitation in recommending the Sieg SX1LP which I purchased from Arc. Similar machines for a similar price are available from Chronos, Chester, Axminster, etc. Each is slightly different so it is worth comparing all of them to find the one that is right for you.
If a lack of funds or inherent tight-fistedness is an issue, you may like to consider a second-hand machine with the usual caveat that (where it is practical to do so) you should examine the machine in person before purchase and know what to look for. G&M Tools might be a good place to start your search and, if nothing else, may be a source of inspiration.