You've had your question well answered Nige and have made the more than correct decision not to try and force a machine no where close to being designed for it into something it just can't do with any accuracy or surface finish. So just for future referance since the subject is sure to come up again.
Most if not all consumer grade light duty pillar / drill presses sold today don't even make a good job of drilling holes if your willing to run a few simple tests. Set up an indicators magnetic base on the rear column with the tip of the indicator at the middle or outside edge of the table then just start adding some pressure to the table with your thumb. Then visualise driving a 1/4", 3/8", or 1/2" drill through steel and how much added pressure the table sees plus the vise and work piece weight. Tramming the table to the spindle is almost a waste of time because of that variable table flexing.The usual very light wall column and head castings also flex under those pressures if your willing to set up an indicator that's not attached to the machine but can indicate the head movement while drilling with a larger diameter drill. Those mill / drills only resemble a pillar drill with an added X,Y table. Check there weights and there far heavier than adding that X,Y table would add on it's own. The column and head casting are much heavier and far more rigid. Better bearings that are designed for the side loads, a drawbar, and much more rigid and accurate fine feed on the Z axis. With all that they still don't make the greatest milling machine, but they can be made to work.I own a mill but still tried milling just once on a 180 lb. 16 speed floor model drill press that had the drill chuck secured with a center bolt into the end of the jacobs taper and a 35 lb X,Y table bolted down to see how well it might work. It was a waste of time with very poor ragged edges on aluminum even while reducing the cut to .005" depth. I'm extremely doubtful adding a proper end mill holder or collet chuck would have made any noticable improvment.
Youtube has more than a few videos by people saying it works and trying to prove it. Read the comments and the ones most in favor of it seem to know the least. The ones against it all seem to have a large amount of experience and understand the requirements and forces involved. Then watch the video and take note of the depth of cut, vibration and surface finish quality.Yes you probably could rebuild and work around the light duty components with something far better, and after all that time and money still have something that only works semi well compared to even an X2 sized mini mill. They do make industrial floor model drills with a built in X,Y table that can do light milling and even proper boring with a boring head. Check the weights and what they cost and both numbers approach what an off shore Bridgeport clone would cost and weigh. A 2200 lb Bridgeport because of it's design is actualy a very flexable machine if it's pushed harder than it should be. There used so much simply because of how versitile they are than there absolute rigidity. Trying to make a 100 – 200 lb. light duty pillar drill do milling is a great way to test your frustration and patience limits in my opinion.
I no longer even own any pillar / drill press for working with metal because of there limitations and poor accuracy. My knee mill will drill holes 10 times as well at least against any pillar drill I could afford.