I think the main constraint on climb milling is how rigid the machine is. Climb milling puts more force on the machine which is OK provided the machine can handle it. The advantages of climb milling are that it removes more metal faster for a given amount of power and leaves a better finish.
You are unlikely to get good results climb milling on a lathe or a lathe/mill combo. They're just too wobbly. It's also a dubious proposition on a light hobby mill designed to keep cost and weight down. I have a WM18 – a large light hobby mill – and, like Ed, do most cuts conventionally before doing a moderate climb cut for finish. Although a new WM18 can climb mill, it complains on deep cuts. I can tell it's not happy!
Something like a Bridgeport is a better bet; the construction is altogether beefier. But I've seen criticism that they're not hefty enough for continuous production climb milling where machines remove metal as quickly as possible all day every day. They're also too big for my workshop!
Even if you own a heavy stiff mill, it has to be in good order and set-up correctly. Backlash isn't the main enemy, it's loose gibs and anything else that allows vibration. A worn machine may be hopeless.
All this tends to discourage climb milling in a home workshop. But it's a useful technique if your kit can stand it. As always it pays to experiment.
Dave