Having had similar problems drilling and boring washers, I have found it is often easier and faster to make a custom one with desired OD and ID on the end of a suitable piece of bar and part it off at desired thickness, instead of modifying an existing one. Stamped stainless is especially hard to hold to modify, the drill always snatches, and being stamped, the rollover (rounded top edge) and the fractured area of the OD make a very small contact area cut by the die for gripping.
Machining off bar works every time, no snatching and grabbing, or fancy collet making needed. (although the collets in Nick's photos are clever and look well made, and likely will work well.)
JD