I had this problem using Greasemonkey 0.9.12 on Firefox 7 on the Mac. I tried setting the editor using both the dialog and about:config. But I still couldn't edit files. The error console (Cmd-Shift-J) was showing an error in the file ~/Library/Application Support/Firefox/Profiles/7k43tcas.default/extensions/{e4a8a97b-f2ed-450b-b12d-ee082ba24781}/modules/util/openInEditor.js.
Examining the file, it was apparent it was trying to invoke
open -a appFile.path script.file.path
However, appFile was defined nowhere. So, I edited the file and changed appFile.path to 'TextWrangler'. Problem solved.
If you do it, you may want first to try the resulting open command in Terminal and make sure it does what you want. Substitute the name of your editor for appFile.path and the path to your script for script.file.path.