As it has 'started' falling off I assume it was OK once?
Take David's suggestion and check the fit, you can use a felt pen if you have no blue.
The fit can be spoilt as it spins free after a job pulls the chuck free.
If it's a B taper you can clean it up with a morse taper reamer. B10 is the small end of MT1 and B16 is the small end of MT2. It does work, I've done it with a B16 fit chuck, just a quick touch of the reamer by hand to remove any bruising.
Any raised burr on the adaptor can be carefully scraped or filed off, just take the minimum of metal away.
Neil