In a gross simplification, PeptideProphet fits positive and negative
distributions to the overall search results distribution as part of
its processing to calculate probability scores for each
peptide-to-spectrum match. That checkbox you're referring to tells
PeptideProphet to use the decoy matches to model the negative
distribution because the decoy hits are explicitly known negatives.
That option is possibly helpful but not needed; in fact you do not
have to even need to do a decoy search to use PeptideProphet.
Mayu is a completely separate tool that calculates protein level false
discovery rates (as well as peptide level and peptide-to-spectrum
match level FDR). FWIW, I personally haven't used Mayu so I can't
help you with it but others here hopefully can.
But the real question on what tool(s) you should be using is based on
what questions you're asking of your data. Have you looked at
Peptide/ProteinProphet output and compared that to Mayu output to see
how useful the results from each are to you? What's your criteria for
better?