Is the error consistent, i.e. every time you move 10mm you get the same 17.68 reading? Does the error vary depending on the starting point? If it’s consistent it would seem to rule out noise/earthing issues. Those would tend to give more random errors. Is the Error linear, i.e. if you move the cross slide only 5mm is the error halved?
When you say you have followed the recalibration instructions, have you checked that the encoder choice is correct? Have you looked at the linear compensation screens to make sure that no corruption has crept in.
The fact that it has never worked correctly points to either a faulty or mismatched encoder.
What happens if you swap over the channels so that the other channel sees that encoder. Do you get the same errors?
Edit – there is what appears to be a fairly detailed user manual here