I would just use the largest that I consider practical. We (mostly) don’t often make really large items on our machines – if we did, we might use a fixed steady to help matters?
The small pilot is there to both prevent the centre bottoming in the hole and also to contain some lubricant, for when running at the tailstock end with a solid centre (which is likely regarded as better practice) unless a good quality revolving centre is available.
It is likely that cutter interference with the centre may be a problem, so small tool-holders/cutters may need to be selected, extra length left for turning off later (as per Hopper, above), or half-centres used. Don't need any centre holes left unless the part is expected to be reconditioned in the future.