The problem is that within a container where display is "none", browsers do not compute the widths and heights of elements, and so MathJax can't tell how big the letters really are. In this case, it uses a div with visibility:hidden that it adds to the beginning of the page in order to typeset the formulas, hoping that the fonts and styles are similar enough that when they are placed into their correct location, they will still work. This works better in some browsers than other (with Internet Explorer being one that works the worst).
The way to handle this would be to NOT typeset the element with display:none until it is displayed (when the display is toggled via the button farther down the page).
Alternatively, you could call MathJax.Hub.Reprocess() on the div that has been revealed after the display:none has been changed. This should replace the math with re-processed output, correctly sized for the current location. This will cause an update of the layout, but it shouldn't be too bad, since the original math should be close to the correct size.
In MathJax version 2.0, you can use MathJax.Hub.Rerender() instead, which will avoid redoing the TeX to MathML conversion and just reproduce the output. Since v2.0 displays the equations in groups of several at a time (rather than one at a time), the update should occur in a single step, rather than equation-by-equation, for a smoother look.
Note that when I say "use MathJax.Hub.Reprocess()", you need to perform that via MathJax's queuing mechanism, as in
MathJax.Hub.Queue(["Reprocess",MathJax.Hub,revealedDiv]);
See
for details.
Davide