Dave,
Try mounting your face plate and clock it up. It might not need the skim. I would seriously consider boring the wheels also at the same setting.
This is my recent experience;
I have just machined a set of driving wheels (about 4 1/2" diameter) for a locomotive called Betty. I have previously learned not to rely on my 4 jaw chuck – not completely anyway.
You could also try this – clock the jaws up – mine were 0.002" out front to back!!!! And the chuck is a high quality version. The chuck body is fine but the jaws are slightly out.
The other error I had made with previous wheels is to bore the holes similar to your method. For whatever reason, the bores can wander (hard spots maybe?).
For Betty, I drilled the holes undersize then clocked the wheels up on my face plate (having checked the faceplate runout of course) and bored them to size with a stout boring tool. I never ever liked reamers – I now hate them.
I ended up machining the Betty wheels using methods that don't appear to be anywhere near the so called words and music.
Phil H