Your results are the expected result for using transform:scale() because transform does not change the bounding box of the element being transformed. So the space reserved for the transformed element is the same as the un-transformed element. That is why when you enlarge the math, it overflows onto the text that surrounds it. This is how transform is SUPPOSED to work. For example,
aaaaa
<span style="display:inline-block; background:red">
<div>XYZ</div>
</span>
bbbbb
<br><br>
aaaaa
<span style="display:inline-block; background:red">
<div style="transform:scale(2.0)">XYZ</div>
</span>
bbbbb
produces the following:
As you can see, the transformed XYZ extends outside its bounding box. This is as expected, and really is unrelated to MathJax, since that is just how transform works.
As Peter says, you should use the MathJax configuration to set the scaling that you want for the output renderer that you are using.
Davide