I sympathize coz I've had that problem too on occasion.
Invariably, it happened when using "styles... >> effects..." on large things like signs. Is that right? If not the rest may not help you...
Here's what I would do.
1 - Defragment (you can defragment can't you? ... 11GB isn't much room to turn-around in...)
2 - Delete the effect (if that's involved). Resize the object you're applying the effect to, at 20%. Apply the effect again. Resize by 500%. Then hope for the best.
3 - If still no good, do the effect in Photoshop and place in the composition.
4 - Upgrade to AI 10 or CS. AI9 was real SLOW.
Having said that, I've always wondered about AI effects. There's something in there that's out of whack. They are so slow and they get slower the bigger they are. I'd like to see them updated to match Photoshop's effects. That would be a real benefit.
good luck -- Mike
Make sure that Illustrator is using it's own plugins folder, and that you
have not pointed Illustrator to use Photoshop's Plugins - the Photoshop
plugins had to be re-written to allow live effects and attempting to use
Photoshop's own plugins will cause that error message.
Robin