I wouldn't use aluminum alloy nuts or bolts anywhere where their purpose is critical, like on brakes, or a stem. For fender stays, bottle cages, maybe.
You can always go with Ti fasteners, which will be suitably strong for any application, but they'll cost you an arm and a leg.
I got a fairly good price on some bulk Ti fasteners, and used Ti bolts for my cantis, cranks and most of the miscellaneous M5 applications.
At the end of the day it's not going to save much, but using Ti for the largest bolts on the bike (like the brakes and cranks) will give you the best weight savings/cost benefit.
Anton