Again, a live example would help. Your message doesn't include enough information to tell what you have done.
The fact that you are getting this error means that MathJax has at least loaded, and (probably) that you have been able to make a call to it. The source of the problem is probably a call to
MathJax.Hub.Queue(["Typeset",MathJax.Hub]);
that is in the outer window but directed to the iframe. There reason that this may be a problem is that the array ["Typeset",MathJax.Hub] is created in the outset window, and so is not recognized as an array inside the iframe (which has its own Array object, and so "A instanceof Array" performed within the iframe will fail if A is an array created in the outer window).
One solution would be to use
w.MathJax.Hub.Queue(new w.Array("Typeset",w.MathJax.Hub));
where "w" is the window within the iframe (I assume you have already located that via the iframe's contentWindow).
Another would be to create a function IN THE IFRAME such as
function Typeset(node) {MathJax.Hub.Queue(["Typeset",MathJax.Hub,node])}
on have the code in the outer window call w.Typeset(node) rather than call MathJax directly.
Hope those help.
Davide