Andrew:
There is not really enough information here to give you a definitive answer. Your CSS works for me in the simple test case that I used, so perhaps there is something more complicated going on in your page. We can't tell, since you haven't said anything about how you are calling MathJax, or you MathJax configuration, or what browser you are using or in what OS. Perhaps you can use the developer console to inspect one of the mjx-mtext elements and see if your rule is being applied to it or not. If not, then try removing the "not(:last-child)" and see if that changes the result. It may be that the nesting is no longer the same.
In any case, it is a bad idea to (and not a supposed use case) to use CSS that affects the size or spacing of MathJax's output. MathJax does not have access to such CSS changes, and so this will throw off MathJax's computations about the width of the elements in your expression. That can affect things like line breaking, centering of elements in under- and overset items (like limits on a summation, or over- and under-braces). Such CSS changes are likely to throw off MathJax's layout for such expressions. Also, if a user changes to SVG output, your changes won't be in effect.
Davide