PDFs, at least per the spec used by dompdf, are fixed at 72ppi. Setting DOMPDF_DPI basically affects how dompdf translates coordinates/measurements from HTML to PDF. This really shouldn't matter for text, lines, or shapes since these are vector elements in the PDF. For graphics you should be able to use a large dimension and/or high resolution image; I recommend a large dimension image that is sized in the HTML independent of the actual resolution. I'll have to refresh my memory on this, but I don't believe images are resized when placed, though some formats may be set at 72dpi during processing.
I don't really know about phmagick. Feel free to post some examples of the PDF and image output for us to compare.