With a 20 mm hole the ultimate object, there is plenty of material to be removed, so the 8 mm cored hole being 3 – 4 mm off centre won’t matter.
DON’T attempt to drill it. A drill will try to follow the off centre hole.
As already advised, Bore it. Even using a three jaw chuck, the resulting hole will be concentric within 0.1 mm probably.
Once the first 4 mm has been removed, with a boring tool, the hole will be 12 mm diameter, anyway, By that time, removing material from one side only, will be finished, and the bore will then be central, ready to be opened up to your desired 20 mm.
Howard