You don't need the rivits to be structural so no need for a full depth CSK. Kist a touch with a 60deg ctr drill will be enough or even a quick twist with a taper pin reamer if you have one that size. Cut the rivit shanks to be equal to the thickness of the plates and a couple or 3 blows to a Ctr punch will be enough to expand them into the holes.
The expanding rivit CSK head will have pushed the sheet outwards and given the distortion that you are getting.
Not sure how well this will come out but the lighter marks are single Ctr punch marks and you can just make out the outer 1/8" dia of the rivits on my 2" Fowler

Edited By JasonB on 28/02/2020 19:33:59