As said, your workshop, your rules.
Even when used for conventional work holding, lathe chucks aren’t immune to permanent damage if the user mistakenly uses an extension on the handle to over tighten the chuck beyond what the design and it’s material limits are. Because of the forces the chuck wasn’t really designed for, that’s not a method I’d want to try myself. Yes your gaining from the extra weight and the fly wheel effect as mentioned, and it’s worked so far. But your most likely losing more just due to the tool tip distance from the spindle bearings.
I bought a set of three different sized fly cutter bodies that weren’t very expensive and worked ok, or at least well enough when I properly sharpened the braised carbide cutting tips on a diamond wheel.
I then bought a commercial fly cutter that uses the replaceable RNMG-43 sized round carbide tips. In my opinion it produces a far better surface finish on almost any material I’ve tried, and is extremely cost effective because the tip can be slightly rotated each time it dulls or rarely might chip the edge. Unfortunately the one I bought doesn’t seem to be available anymore or I’d add a link. But if I was making a fly cutter body, tool tip shank and wanted to use carbide, it’s what I’d choose because it would be well worth the time and effort.
If it will help anyone here, mine has dimensions that might be a bit large for a lot of the more bench top sized mills, but could definitely be scaled down or even use the smallest diameter RNMG-32 tips for anyone with a small mill. Fwiw, mine has a .750″ tool shank, the body is rectangular 1.800″ wide X 2.500″ long, the tool slot offset to put the cutting edge on the tool center line, 1/2″ X 1/2″ tool tip holder X 5″ long. With it looks like 2-3 degrees of additional forward rake on the RNMG tool seat to make the cutting even more positive. And the usual 3 set screws to secure the tip holder at whatever radius it’s set to. I suspect that extra forward rake on the tip seat is why it’s design cuts so easily.