Rather than hiding the collets at the back of the drawer, if you have the dimensions for the holder, why not make your own chuck for the collets? We are supposed to be "Articifers in Metals" after all!
Being an ER enthusiast, have made various carriers for ER20, ER25, and ER32 collets. My carriers are soft, but they are unlikely to see so much use that wear will be a problem. For one specific purpose, even made three ER32 lookalike collets, but not as an addictive activity.
Chickened out on the nuts, and bought them!
And NO, you can't put ,say, an ER32 onto an ER25 holder, (or the other way round). For a start, the Clamp Nut won't accept the collet (understandably).
Just a suggestion
Howard