What sort of mill do you have? If the table-top type mill/drill, it could be deflection in the vertical column, the spindle quill, the table/column mount, etc all adding to the amount of deflection of the cutter away from the work piece.
If the cutter is held in, for example a taper fit ER collet chuck, or the like, is the chuck taper a good fit in the spindle taper? If not, the cutter/chuck can be deflected away from the cut when side milling
Any play in the vertical column dovetails, etc…Side deflection would be more evident when milling on the side of material, or course, but any vertical play ( spindle bearings?) would have an effect when end milling as well.
And if the cutter is held in a collet chuck, ensure the collet is pulled up tight – if loose, the cutter could be pulled into or pushed away from the work.
My pennies worth..
Joe