Any abrasive is likely to remove material, but since we are unlikely to be working to hundredths of a thou, in a temperature and humidity controlled environment, it probably won't matter.
The object is to remove any rust standing above the hardened surface. Rust pits below the surface will not affect the dimension, although unsightly.
They may now be incapable of being wrung together, but should suffice for most purposes.
Just use the finest grade of Scotchbrite that you can get, which may be red, I think.
Howard