Hi Giancarlo,
I think I can explain what is happening.
As steepness gets larger and/or as right-shifted selectivity protects a large fraction of the SSB from F, the search for FMSY will go towards higher F. When I look at the warning.sso for the h=0.9 run, I see these two lines which are an indication of the problem:
Warning 1 : poor convergence in Fmsy, final dy/dy2= -0.284552
Warning 2 : Fmsy.mey is close to max allowed; check for convergence
Those warning lines do not appear in the h=0.8 run.
It was a good idea for you to compare the Fmsy estimate to the SPR/YPR profile. When I do that, I see a good match when h=0.8 and the profile has a well-defined maximum.
But with h=0.9, the profile is quite flat in the region F=2 to F=3. That is why the FMSY iterative search is reporting poor convergence.
The Fmsy search itself can be seen in forecast-report.sso. This is not a user-friendly report and not read into r4ss, but it does contain more detain on the benchmark iterative searches and then the forecast.
There you can see that SS3 searches over a fixed number (20) of iterations for MSY. Look under "find_Fmsy_to_maximize_dead_catch". It shows a reasonably negative curvature for the search, but the gradient at iteration 20 is not near zero, so SS3 reports poor convergence.
You'll also see that SS3 starts the FMSY search at the final F from the search that produces the target SSB/SSBzero, which you have set to 0.4. That F value is 0.61. So perhaps if you set the target SSB/SSBzero to a smaller value like 0.25, then the FMSY search will begin at a larger F and have a better chance of reaching convergence.
I hope this helps explain the results you are seeing.
Rick