OK, answering my own post. Found the problem.
The box to choose a script lets you browse to any location, but will
shorten the path to something like ~/Library/Mail Scripts/script.scpt
even if you type it in manually like /Users/mmartel/Library/Mail
Scripts/script.scpt. This is obviously a limitation in Mail that it
can't find the script and just silently fails to run it. Will file a
radar.
The new preferred location for Mail scripts (not documented, akaik)
is /Library/Scripts/Mail Scripts but users don't have permission to
save to that location. Try putting your script in another location,
and it should work (i.e. /Users/Shared).
Thanks,
-Matt