I was able to track this down to using mathfontfamily="serif" in my
asciimathml, which sets the fontfamily attribute of the <mstyle> tag.
There must be some font on my system other than STIX which FireFox is
using when this attribute is set. Turns out IE+Mathplayer is doing
something screwy with this setting as well.
So immediate fix for me is to remove this attribute. However I really
want my math in a serif font even when embedded in sanserif text, so
need to try to work around that..