I've found the issue -- I think it is a problem with how OpenVSP is feeding rotor data to VSPAERO. I think Justin may have already fixed this for the next version. I'll let him chime in.
The code to calculate FOM is
here.
Note that it refuses to calculate FOM if the thrust coefficient is negative. For one of your rotors, the thrust coefficient is always negative.
If you move your rotors to be non co-axial (but change nothing else in your setup), do you get the same result?
If your second rotor appears to be generating positive thrust -- but the coefficient is coming out negative, then we may have something wrong in transferring the file between OpenVSP and VSPAERO. Justin may have already fixed the issue.
However, if when moved, your second rotor switches from making negative thrust to positive thrust, then I think everything is behaving properly and you just have a lousy design for your second rotor.
The second rotor in a coaxial system needs a substantially different design from the first one. Particularly if the separation between the rotors is significant.
When you change RPM, are you still using the 'automatic' time step setup, or are you manually setting those things? You will need to adjust them for a change in RPM.
In fact, it would not surprise me if you need to take manual control of those settings for your coaxial rotor system. See if reducing dt and increasing the number of time steps helps with your rotors going chaotic.
Also, if these are in a hovering condition (no freestream axial flow) you might want to work with the hover ramp settings. Sometimes that can help to initialize the flow to blow the wake downstream as the solution develops.
In any case, if your second rotor is producing negative thrust, you are going to have an ugly wake. You need it producing positive thrust to blow the wake away.
Rob