While speed when using small drills helps it is not essential. Here is a photo of 1.2mm holes drilled 8mm deep in tungsten alloy:

These holes were drilled on a Bridgeport mill at 2500rpm with no issues.
When drilling tougher, or work hardening, materials with small drills (<2mm) I prefer to use carbide drills. These are not usually four facet ground so need a starter dimple. I mostly use a 2mm carbide spot drill.
If the OP is having trouble I suspect the issue is with the drill bit itself or the mill, not the speed per se.
I have an old PCB drilling machine which goes up to 10000 rpm but have never used it thus far. Of course such machines are now pretty much obsolete in industry. Commercial PCB fabrication uses drilling speeds in the hundreds of thousands of rpm.
Julie