Hi Ron,
I discussed this question with a former college yestereve.
The FidCompTraceDiff uses the Frobenius norm. It's not clear that this can be applied to states, but let's say it cannot.
Now if your system is not unitary, then it must be modelled as an open quantum system. This means that you need to consider the evolution of the density matrix using the master equation formalism. You will find a lot of literature on this. Most of qutip is for solving this kind of problem. You can find a control example of this in the notebooks:
There is another version in:
You will note however, that these example are for gate synthesis. So now the question is how do you compare evolved and target states. Well, my former college said he has been working on this recently, and it was not all that trivial. I guess as it is part of his current research, then he did not want to give it away as yet. So you may have to figure this one out yourself. There is nothing in qutip.control for it presently. We would be happy to receive a contribution :) It is also the part of my current research, so maybe I will add something soon. It is pretty easy to subclass a FidComp however. There are examples in my qtrl-tutorial repo for this.
All the best
Alex