Not only might the tool deflect slightly downwards the work may try and climb over the tool. There seems to be some aversion to setting the tool slightly too high. Depending upon the offset and work diameter it may not matter in the slightest. If a tool is set 10 thou too high and the work is 1" diameter then the tool will be 0.0001" closer than if it was set dead on height. I can't see a tenth of thou having any affect at all on the cutting process.
I suspect that finish problems are mostly caused by the material, cheap tooling or incorrect grinding, and wrong speeds and feeds. Tool height is probably some way down the list.
Andrew