Rotating the block introduces the risk of slight lateral errors, albeit very small ones, potentially enough to set the drill a-wandering.
Taking only shallow bites with frequent clearing of the flutes will help keep the drill going straight.
Do the bolts need go all the way through the block?
Can you modify the design so that you can screw the bracket on from its own side, using tapped holes and set-screws or studs and nuts, that don’t need penetrate ever so deeply?
Do the fasteners also hold something on top, as if the block is just a big spacer? If so, can you use the same principle but when inverting the block to drill the opposing face, being careful to set off from the same datum faces?
Would it greatly matter if you do drill from both sides but perhaps 0.3mm over-diameter for about 30mm depth from both faces, so even if the holes have drifted slightly their centre portion is still very close to your intended centres?