It is limited to 300 dpi because with a large image like mammography you can have an out of memory exception.
Increasing the dpi will only allow improving the quality of the annotations (text and graphics overlay are rasterized on a bigger image). If you have an image of 512x512 pix with 300 dpi, Weasis will increase the resolution by interpolation (bilinear or bicubic) depending on the size of the film, but this job of interpolation could also be done on the printer side. So if the original image is sent to the printer or a bigger image with interpolation, the result should be the same.
What is the selected value of "Magnification Type" in the DICOM Print dialog ? try CUBIC