Peter,
It is best not to use the full width of the cutter for reducing the thickness of a work piece – keep the cut just less than the radius of the cutter. Make sure you are not "climb" milling as this is not good practise on a lightly built mill – OK on a Bridgeport.
I do not have any of the problems you describe with my Tom Senior when using sharp cutters. Make sure the corners of your cutters are really sharp.
I always use the spindle speed that the machine is "happy" with and tend to ignore cutting speeds and feed quoted in various tables as they only apply to heavy duty industrial machines on production work not of small, light machines.
I use 3 flute 5% cobalt cutters wherever possible for milling and slotting applications.
Are you holding the work piece very firmly in a proper milling vise? Light drill press vises are not suitable for milling.
I hope this is of help.