After some research online, I found Rotozip makes a 3pc collet set with 1/8, 5/32, and 1/4 ID collets. The 5/32 collet worked perfect with the 4mm shank end mills. The set also comes with a new but, but the info I read online said the rotozip nut will not fit correctly on the Dewalt. Haven't tried it, just used the nut that came with the DW660 and didn't have any problems.
Lowes and Home Depot stock them, and for a cheaper price than the amazon link below.
Maybe I got a poor quality one, but I cannot recommend these collet for any kind of precision cutting.