MathJax is focused on mathematical notation, so doesn't implement much in the way of text-mode material, and (by defualt) does not process macros in the limited text-mode support that it does have. Β See the documentation at
That is why you are seeing the code rather than having it processed. Β MathJax v3 does have a textmacros extension that allows you to process some macros in text mode:
but that doesn't do all macros, and would not handle your code even if you used it (e.g., \ensuremath is no implemented, and \hbox and \text are not supported when you are already in text mode, though I suppose those could be added).
Because MathJax doesn't do text mode, it is more lenient about what you can do in math mode, and doesn't require all the back-and-forth between math and text mode, and so you are allowed to use \Large in math mode (unlike LaTeX itself, where that is a text-mode macro). Β So your can just use
\operatorname*{\text{a}\Large\ast}
to get what would require all the boxing and text mode in actual LaTeX. Β Of course, this won't work if you put it back into LaTeX, so if you want something that works in both, you could use
\operatorname*{\mbox{a\Large$\ast$}}
which is less cumbersome than your original, but still gets the job done in MathJax and LaTeX. Β I would not go through \underset route that Bill suggested, as the \operatorname approach is more semantically meaningful, and also includes the larger asterisk (though you could add \Large to Bill's answer if you wanted).
Davide