I've got an RSP1a but the available settings are the same. I've attached a screenshot of the gain settings I have at the moment. I've got the same jagged edges on some signals but others are a bit weak. I have to overdrive reception of one site to get the other three I also want to monitor. If I use auto mode I only get partial reception of that strongest site.
The Low Noise Amp (LNA) setting is wonky on the RSP1a so it's probably the same on the b model. Only the four rightmost ticks do anything because the LNA only has four user selectable settings. Starting fourth from the right is "off", followed by a "medium-high" setting, then "medium" setting, then finally "maximum". On the max LNA setting I get too much noise to decode anything even with the IF gain set to the minimum. Anything more than about halfway up the IF slider and I get the gain overload warning. The medium setting seems to work best for me even though I do have quite a bit of noise on the right hand side of the spectrum. If the LNA is off I barely get anything at all from just the strongest site on the waterfall, even with the IF gain maxed out.
My problems are two fold: my discone works better at lower elevations for decoding these P25 broadcasts but works best high on my mast for air band and other public safety listening, so I have to compromise a bit with the noise levels here. The other thing is that there's a massive LTE signal just off to the left of this visible spectrum. If I don't set a dummy frequency on an upper usable frequency, the edge of that LTE band will jump into view and overload the receiver.
It takes a bit of fiddling with the LNA and IF gain settings but you should be able to find a level that makes decoding more reliable while not overloading the receiver.