What RPM are you running? What size pilot drills are you using? How sharp are you keeping your drill bits?
Stainless steel, including so-called "free machining" 303 are tricky to machine. They tend to work harden as you machine them, which then bluntens the tool, which then rubs on the stainless steel and makes it even harder.
But you should not have too much trouble drilling and boring those size holes if you take care. Don't run the lathe too fast. 500rpm for the larger diamter drill bit and the boring might be enough. Faster for centre drill and pilot drill/s.
Keep the feed up to the job. Don't slack off and let the drill bit or boring bar bit rub on the job. Keep the cutting edge cutting into the job without rubbing.
Keep your drill bits and boring bit sharp. Touch them up between each job if necessary.
And use the coolant pump if you have one.
You should be able to drill one hole in a matter of minutes, not hours. Ditto for boring it.