The answer depends on what material you are hoping to broach, and how deep. But mainly the material.
As you can imagine, broaching a keyway in a Delrin gear is going to be rather easier than a steel one. With Bronze, Brass and Aluminium and other light metals somewhere in between.
I guess that you are thinking of using the mill as a press (unpowered). The only difficulty I can foresee, if you are careful and don't try too much at one push, is holding the broach so that it cannot rotate as it is pushed down. This can be done by holding the guide firmly in place, rather than just putting it into the hole – but it does require some ingenuity in holding it properly.
Regards, Tim