'… or is it a matter of tightening up gib strips etc?
If the tool deflects then use less cutting pressure and a sharper tool when doing those final cuts'
It's not that simple, and the question is about the final cut that may ruin the part, as the tool is withdrawn towards the tailstock.
The tool will always deflect: nothing has infinite rigidity. Even if you were to bore, back-off feed, retract, return tool to previous in-feed position, bore again, and repeat the sequence a few times (ie several 'spring cuts' ), you would in all probability find that, after all that, withdrawing the tool towards the tailstock, without backing-off the feed, would result in a light cut, ruining your day. The saddle guides, particularly of Myfords, are short, compared to their separation, and, whilst, as has been suggested, gib strip adjustment will help, it won't eliminate the phenomenon, particularly when the saddle's fixed guide has worn a bit.
The use of 'upside-down' boring tools is a standard technique, but it seems to be less well-known than it deserves to be. It has advantages – try it, if it's new to you.
Also, use boring tools of a suitable shape (ie with the 'corner' leading) because these will tend to deflect less and can face the end of a blind bore. GHT's writings explain it, IIRC.