If I'm going to use a masonry drill on hardened steel I first sharpen the carbide tip up on the grinder (they are usually a bit blunt straight out the box), then I use it on the backgear on the lathe, torque and pressure being more important than outright speed.
Take your time and let the tip cool off when it starts to get hot and use a bit of oil to help with the cutting, maintain a good solid constant pressure on the job while it's working, a 4mm will heat up pretty fast so you'll need to be patient but you should hear it cutting as it progresses
DO NOT use water to cool the tip, this will cause microscopic cracks to form on the carbide cutting edges, it needs to cool naturally
Higher speeds will simply burn the drillbit up when you do a really tuff job, it does sound like you have something in there like a bit of slag or a ball bearing
GL
Edited By Ady1 on 18/10/2016 14:52:42