If it was me, I would use a polished, uncoated single-flute carbide cutter with an aggressive geometry designed specifically for machining aluminium. If possible, I would avoid using long small diameter cutters,
Speeds and feeds are pretty important. Spindle speed shold be basded on around 100metres/minute, so a 3mm cutter should be turning at somewhere around 10,000rpm. I'm guessing you are not running your spindle at that speed. A 10mm cutter should spin at around 3000rpm, which is achievable on many small mills, so that's a good go-to size for starters. Won't work for tight corners, of course, but ,as JasonB says, you may be able to deal with those by drilling, or by machining separately.
Rough out, leaving 0.2mm to finish. Climb cut the finishing cut. High speed, but a slower linear feed may help.
I have had decent results with a long series 2 flute cutter on 50mm deep finishing cuts, but it was a 10mm cutter, not a 3mm matchstick. Roughing at 3000rpm, and up to 500mm/min, taking cuts of 15, 15 and 20mm down the face, finisihing with a single climb cut pass at full depth, 3000rpm and 120mm/min. Material was grade 6082 (HE30).
Marcus