SO,QN=0 MCSCF solver sometimes performs much better than the defaults

[Tibor Győri]

Dear Molpro community,

Here I would like to share an example where the default solvers for MCSCF and DF-MCSCF have much worse convergence than SO,QN=0.

1) MCSCF

If I understand correctly, MCSCF defaults to the WMK second-order solver with the L-BFGS acceleration activated. As seen in the attached outputs (default.out, default_fail.out), this results in chaotic convergence. Re-running the same input multiple times sometimes results in convergence after 20+ iterations, but sometimes convergence is not achieved within the hardcoded maximum of 40 iterations.

The run-to-run variability suggests that something is numerically unstable in L-BFGS, allowing the small non-deterministic noise of parallel execution to have a large impact. Either way, I think it can be said that WMK + L-BFGS does not work well in this specific case.

Disabling the L-BFGS acceleration with QN=0 solves the problem completely, and the WMK solver converges reliably in 6 iterations. (noQN.out)

2) DF-MCSCF

DF-MCSCF seems to default to the combined first- and second-order optimization method, SO-SCI. This struggles to converge even more than the default solver of non-DF MCSCF, and reproducibly does not converge in 40 iterations. (DF-default.out)

Enabling second order solvers with SO fixes this, and convergence is reliably achieved in 6 iterations. Curiously, unlike non-DF MCSCF,  L-BFGS works perfectly fine here and saves a little bit of time. (DF-SO.out, DF-SOnoQN.out)

Based on these results, it looks like the default solvers of MULTI can have poor convergence in some edge cases, in both DF and canonical computations.

The first step should of course be to verify that the wf, occ, etc. directives are correct, but perhaps trying SO,QN=0 should be a recommended course of action for resolving MCSCF convergence issues.

Best regards,

Tibor Győri

[hjwern...@gmail.com]

Dear Tibor,

I just came across your mail about the MCSCF convergence problems and took a look at it. Sorry for not replying earlier, but I hope that my reply will still be helpful.

Yes, I know that the CASSCF convergence is sometimes not reproducible, although in most cases the same correct minimum is found. This happens if the Hessian becomes nearly singular, which means that some orbital rotations are nearly redundant.  It is not surprising that QN has problems in such cases. Unfortunately, it is difficult to fix reliably, because one would have to remove singularities by excluding some orbital rotations, which would then unavoidably lead to small changes of the energy and possibly to small steps on potential energy surfaces.

In my view this is an intrinsic problem caused by the choice of the active space (and the number of states). My experience is: if the second-order CASSCF algorithm has problems to converge, very likely the active space is not well chosen.

In the current case the problem is quite obvious: the occupation numbers of several active orbitals are very close to 2, which means that rotations of these orbitals with closed-shell orbitals hardly change the energy, i.e. they are nearly redundant, leading to near-singularity of the Hessian. A closer look shows that only 4 active orbitals are relevant: the two pi orbitals of the OH radical and a pi/pi* pair of cytosine. All other occupied orbitals should be made closed-shell (inactive), leading to an active space of 5 electrons in 4 orbitals. Probably, the other pi/pi* orbitals of cytosine should also be included in the active space, but then OCC needs to be increased. I have not looked at this.

Another problem is that there are two nearly degenerate states, correlating asymptotically with the two degenerate ^2Pi states of the OH radical. Both states should be included in a state-averaged CASSCF in order to facilitate convergence and to get a smooth PES for all orientations of the OH radical.

In order to converge to the desired active space it is important to have a reasonable starting guess. In many cases this can be generated with AVAS, without the need of a preceding RHF (which can also converge to different local minima, depending on the starting guess). An example of how to generate the AVAS guess for the (5,4) active space is attached. With this, the second-order method converges very quickly with or without QN. Naturally, the first-order so-sci method needs more iterations, but overall it is faster than the second-order optimization. Interestingly, in this case QN helps a lot to reduce the number of iterations (from 57 to 15!)

We have made SO-SCI the default for DF-CASSCF, since the second-order method gets very expensive for larger molecules with many closed-shell orbitals. This has made it possible to run CASSCF calculations with thousands of basis functions and hundreds of electrons. On the other hand, the second-order method is the default for conventional CASSCF, which is typically used  for small molecules.

Best wishes
Joachim Werner

Attached are 2 tests with avas starting guess. One with the original geometry, and one with a reordered geometry so that the cytosine ring can be placed in the yz plane. 

Note that  the choice of atomic orbitals in AVAS depends on the orientation of the system.