There is a DITA OT opened issue to support MathML but it has not yet been done:
For PDF output you would need to add and install this extra DITA OT plugin:
The plugin would need to be slightly changed as it was not built for DITA OT 3.x, you would need to edit the XSLT stylesheet "org.dita-community.dita13.pdf\xsl\dita13Vocab2Pdf_template.xsl" and remove references to the "dita13base2fo.xsl", "hi-d2fo.xsl", "svg-d2fo.xsl", "xml-d2fo.xsl" because those XSLT fixes are already incorporated in the DITA OT.
Then you would need to get the Apache JEuclid libraries:
add two two JAR libraries to the folder "plugins\org.dita.pdf2.fop\lib" and modify the "
plugins\org.dita.pdf2.fop\plugin.xml" to add references to the two new JAR libraries.
Then run the DITA OT integrator "dita -install".
But there is a problem, DITA OT 3.1.2 updated the Batik library bundled in the "org.dita.pdf2.fop\lib" folder to
is no longer compatible with JEuclid.
So you would need to replace the Batik 1.10 with the equivalent Batik 1.9 libraries and update again the plugin.xml. (DITA OT 3.1.1 still came with the Batik 1.9 libraries so you can use the Batik 1.9 JAR libraries from it).