Everything is lined up OK, and climb milling isn't the problem as the climb milled side has come out OK (assuming the diagram is right).
If you are using R8, then (1) they need to be done up very tight and (2) the cutter has to be the same size as the collet (unlike ER collets R8 won't work with undersize shanks).
You also need to be sure that both the head and the quill are locked in place.
I suggest marking the position of head, quill and tool, then taking another trial cut to see what is actually moving.
People are warning against climb milling, but it very much depends on how rigid you mill is and how tight your slides are and backlash is controlled. I climb mill on my wee X2.
There is actually more vibration with conventional milling when the mill is set up right, which could be the cause of the movement. How 'happy' did the mill sound and feel when you were taking the cuts?
Neil