When I worked at Ital in Sheffield we were part of the Clarkson group ) we made small diameter but long crankshaft drills ( minimum length 10x diameter ) that were purpose made for such tasks . In order to minimise wander , they had thicker webs , worm type flutes and modified points to reduce size and angle of chisel points . The most common problem we encountered were the drills not being driven fast enough and also the application of excessive feed , resulting in the drill wandering and sometimes even breaking within the work .
Assuming only standard HSS drills and limited facilities are available to OP , suggest using highest speed possible , the use of diligent thinning of the point to reduce chisel width and angle ( from 130° to around 115°) , copious use of paraffin to prevent the chips from binding to the lands ( since drilling aluminium) and also light , repeated pecking technique . Cumulative use of the above should minimise wander .
Hope the OP finds the above of constructive use .