Ah, a QCT, but a pretty badly battered combination.
The fact that cutting forces move it says that the clamping force between the Tool Holder and the Top Slide is insufficient.
Given the condition of the surfaces, that is not surprising, since the contact areas are greatly reduced.
Ideally, both surfaces should be ground or machined to clean up and get rid of the damage.
The least that I would try to do is to strip the Top Slide (Being Cast Iron ) and machine the surface flat to get rid of the dents. If the central bush is proud, that will also get rid of it standing proud.
If you don't have a mill. with some ingenuity, you could possibly mount the Top, Slide on a Faceplate, or possibly even in the 4 jaw, and machine it flat again. But it must be clocked level,as far as you can, given the deep indentations.
Maybe carefully gluing a shim to it,l to cover the many dents may help clocking?
Once clocked level, you take light cuts with a slow feed, with a HSS tool until the surface is acceptably clean.
Being cast iron, it will be a dirty job. Place a magnet beneath some newspaper under the chuck in the hope of collecting some of the swarf.
The Tool Holder may be hardened , in which case you have a long time to spend grinding it flat on emery on a surface plate.
Does that bush around the central post stand proud of the Top Slide, even by a thou or two? If it does, it may be bearing a lot of the clamping force applied by the handle.
If you do want, or cannot restore the surfaces, you will have to resort to bodge methods.
Even if you have to use a grinding point in a Dremel, it might be worth putting a chamfer around the hole in the tool holder, to prevent any interference.
Failing that, you could interpose some thin soft material (Copper, Brass or Aluminium shim ) between the Top Slide and the Tool Holder. Hopefully, being soft, when clamped it will be deformed to fill the many indentations in both. But having undamaged faces would be preferable.
In the past, someone has neglected and abused the machine, and you have to deal with that to have it behave as it should..
However you go about it, the centre height of the tool will, need to,be reset, either to take account of the Tool, Holder being lowered by the remachining processes, or raised by the shim interposed between the scored and dented surfaces.
Which method are going to adopt?
Hopefully, after all the hard work, you will have moved it from a total sow's ear nearer to being a silk purse.
Howard