I’d’ve thought it goes back to the movement of the the sun across the sky. If you’re forced to choose, you’d pick a familiar direction as the default, and move away from that by exception if there were a reason to do so.
Not sure why many British Imperial smallarms had barrels rifled with a left-hand twist, though.