Restarting TCE calculations

[Stephen Weitzner]

Hello,

I'm trying to run CCSD(T) calculations with TCE, but can't seem to restart the calculation properly. My input file is provided below. Also provided below is the header for the Extensible Many-Electron Theory Module. It looks like all of the TCE save / read options are set to False despite setting the parameters to true in the input. Does TCE restart work with 2EORB + 2EMET 15? If not, can anyone recommend a different set of TCE settings to enable restart options? 

Thank you, 

Stephen Weitzner

Materials Science Division, LLNL

###################################

start molecule

charge 0
scratch_dir ./scratch
permanent_dir ./permanent

memory stack 1200 mb heap 100 mb global 20000 mb

geometry units angstroms print xyz autosym noautoz
 ...

end

BASIS "ao basis" SPHERICAL PRINT
  * library  cc-pVTZ
END

scf
  singlet
  THRESH 1.0e-10
  TOL2E 1.0e-10
  maxiter 100
end

tce
  ccsd(t)
  freeze atomic
  2eorb
  2emet 15
  tilesize 15
  attilesize 40
  diis 5
end

set tce:nts T

set tce:read_integrals F
set tce:read_t  F
set tce:read_l  F
set tce:read_tr F

set tce:save_integrals T
set tce:save_t  T
set tce:save_l  T
set tce:save_tr T

set tce:save_interval 5

task tce energy

###################################

###################################

                  NWChem Extensible Many-Electron Theory Module

                   ---------------------------------------------

              ======================================================
                   This portion of the program was automatically
                  generated by a Tensor Contraction Engine (TCE).
                  The development of this portion of the program
                 and TCE was supported by US Department of Energy,
                Office of Science, Office of Basic Energy Science.
                      TCE is a product of Battelle and PNNL.
              Please cite: S.Hirata, J.Phys.Chem.A 107, 9887 (2003).
              ======================================================

            General Information
            -------------------
      Number of processors :    80
         Wavefunction type : Restricted Hartree-Fock
          No. of electrons :    74
           Alpha electrons :    37
            Beta electrons :    37
           No. of orbitals :  1626
            Alpha orbitals :   813
             Beta orbitals :   813
        Alpha frozen cores :    15
         Beta frozen cores :    15
     Alpha frozen virtuals :     0
      Beta frozen virtuals :     0
         Spin multiplicity : singlet
    Number of AO functions :   813
       Number of AO shells :   249
        Use of symmetry is : off
      Symmetry adaption is : off
         Schwarz screening : 0.10D-09

          Correlation Information
          -----------------------
          Calculation type : Coupled-cluster singles & doubles w/ perturbation
   Perturbative correction : (T)
            Max iterations :      100
        Residual threshold : 0.10D-06
     T(0) DIIS level shift : 0.00D+00
     L(0) DIIS level shift : 0.00D+00
     T(1) DIIS level shift : 0.00D+00
     L(1) DIIS level shift : 0.00D+00
     T(R) DIIS level shift : 0.00D+00
     T(I) DIIS level shift : 0.00D+00
   CC-T/L Amplitude update :  5-th order DIIS
                I/O scheme : Global Array Library
        L-threshold :  0.10D-06
        EOM-threshold :  0.10D-06
 no EOMCCSD initial starts read in
 TCE RESTART OPTIONS
 READ_INT:   F
 WRITE_INT:  F
 READ_TA:    F
 WRITE_TA:   F
 READ_XA:    F
 WRITE_XA:   F
 READ_IN3:   F
 WRITE_IN3:  F
 SLICE:      F
 D4D5:       F
 quantumc:   F
 qrotation:  F
 pwtce:      F
 qducc:      F
 nacto:                          0
 nrotv:                          0
 nrot:                           0

###################################

[Edoardo Aprà]

Could you try the following set directives

To write the integrals involved in the 4-index transformation

set tce:writeint t

To write CCSD amplitudes

set tce:writet t

To read the integrals involved in the 4-index transformation

set tce:readint t

To read CCSD amplitudes

set tce:readt t

[Stephen Weitzner]

And just to confirm, I found this thread here reporting a similar issue for NWChem v6.8: https://nwchemgit.github.io/Special_AWCforum/st/id2557/TCE_restart_on_BlueGene/Q.html, and it seems that the issue was resolved / a patch was made.

I am running into a similar issue using NWChem v7.0.0. I am currently testing this with NWChem v7.0.2.

On Thursday, September 15, 2022 at 10:36:02 AM UTC-7 Stephen Weitzner wrote:

[Stephen Weitzner]

Thanks, Edoardo. This is now enabling the read and write options (see below). How often are these quantities saved by default? Is there an updated set directive to change the save frequency? For some of the bigger systems I'd like to study, I would like to save this every iteration or two if possible.

Another question: I originally tried to follow the documentation here: https://nwchemgit.github.io/TCE.html. But the set directives you shared with me are different from what appears under the section TCE Restart Capability. Does this mean that the documentation for the set directives for TCE is outdated? If so, can you point me to a summary for the updated set directives for TCE?

Thanks again,

Stephen

############################

TCE RESTART OPTIONS
 READ_INT:   F

 WRITE_INT:  T
 READ_TA:    F
 WRITE_TA:   T

 READ_XA:    F
 WRITE_XA:   F
 READ_IN3:   F
 WRITE_IN3:  F
 SLICE:      F
 D4D5:       F
 quantumc:   F
 qrotation:  F
 pwtce:      F
 qducc:      F
 nacto:                          0
 nrotv:                          0
 nrot:                           0
############################

[Edoardo Aprà]

The quantities are only saved once: the integrals at the end of the 4-index transformation and the amplitudes at the end of the CCSD iterations.

You could use the following input file for the first run (to save integrals and amplitudes), where you can replace "maxiter 8" with the number of CCSD iterations that you can accomplish in the allocated walltime.

start molecule

...

ce
  ccsd(t)
  freeze atomic
  2eorb
  2emet 15
  tilesize 15
  attilesize 40
  diis 5

  maxiter 8

end

set tce:writeint t

set tce:writet t

task tce

For the subsequent runs, use the following (full input file here, please keep it this way to avoid re-converging the SCF).

restart molecule

set tce:writeint f

set tce:writet t

set tce:readint t

set tce:readt t

task tce

Yes, I agree that the information given here does not seem consistent with the user manual.

All the information about restart capabilities that has worked in my recent TCE CCSD runs is reported here.

[Stephen Weitzner]

This is extremely helpful. Thank you again for guiding me through this.

[Stephen Weitzner]

Hi Edoardo,

I tried the inputs like you suggested, but it seems like I'm still encountering some issues (details below).

When I run the first CCSD calculation (setting DIIS to 3 and maxiter to 5), the code states that it has reached maxiter which is expected. But when I go to restart the calculation it says that there is an error reading the Runtime Database. Does this mean that my first calculation is not exiting cleanly?

Steve

############ FIRST CCSD RUN #############################

CCSD iterations
 ---------------------------------------------------------
 Iter          Residuum       Correlation     Cpu    Wall
 ---------------------------------------------------------
NEW TASK SCHEDULING
CCSD_T1_NTS --- OK
CCSD_T2_NTS --- OK
    1   0.3037035355556  -1.3990220028087   511.1   265.9
    2   0.0717669712459  -1.4538963696273   512.4   265.7
    3   0.0236373000566  -1.4750935748311   512.8   268.3
 MICROCYCLE DIIS UPDATE:                     3                     3
    4   0.0145329172121  -1.4821975591820   516.1   266.6
    5   0.0055627893068  -1.4803457117598   517.2   267.2
  1 ga offset                3487 size_xx_perproc                3487mx    4
  2 ga offset                6974 size_xx_perproc                3487mx    4
  3 ga offset               10461 size_xx_perproc                3488mx    4
  0 ga offset                   0 size_xx_perproc                3487mx    4
 WRITE TENSOR
  filename: ./permanent/molecule.t1amp.000000
  unit nr:       79
  file size:           3487
  rec_mem (KB):     2048
  rec_size:         262144
  number of tasks:            1
  0 ga offset                   0 size_xx_perproc            67850090mx    4
 WRITE TENSOR
  filename: ./permanent/molecule.t2amp.000000
  unit nr:       80
  file size:       67850090
  rec_mem (KB):     2048
  rec_size:         262144
  number of tasks:          259
  1 ga offset            67850090 size_xx_perproc            67850090mx    4
  2 ga offset           135700180 size_xx_perproc            67850090mx    4
  3 ga offset           203550270 size_xx_perproc            67850093mx    4
 ------------------------------------------------------------------------
 ccsd_energy_loc: maxiter exceeded                   6
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line :
 ccsd_energy_loc: maxiter exceeded                   6
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line :
     0:
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 This type of error is most commonly associated with calculations not reaching
 convergence criteria
 ------------------------------------------------------------------------
 For more information see the NWChem manual at
 https://github.com/nwchemgit/nwchem/wiki

##########################################

############ RESTART CCSD RUN #############################

           Memory information
           ------------------

    heap     =   13107198 doubles =    100.0 Mbytes
    stack    =   13107195 doubles =    100.0 Mbytes
    global   =   26214400 doubles =    200.0 Mbytes (distinct from heap & stack)
    total    =   52428793 doubles =    400.0 Mbytes
    verify   = yes
    hardfail = no


           Directory information
           ---------------------

  0 permanent = .
  0 scratch   = .


 ------------------------------------------------------------------------
 start: rtdb_open old failed                    0
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line :
     0: task tce
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 An error occured in the Runtime Database
 ------------------------------------------------------------------------
 For more information see the NWChem manual at
 https://github.com/nwchemgit/nwchem/wiki


 For further details see manual section:

##########################################

[Edoardo Aprà]

What is the output of ls -l in the permanent_dir ?

https://nwchemgit.github.io/Start_Restart.html

[Stephen Weitzner]

Hi Edoarda,

It looks like there is not a problem in the end. I copied the restart file you suggested, but this didn't include the memory, scratch_dir, or permanent_dir directives.

Using the restart file below works as intended.

Thanks again for the help.

-Steve

##############################################

restart molecule

scratch_dir ./scratch
permanent_dir ./permanent

memory stack 1200 mb heap 100 mb global 12000 mb

set tce:writeint f
set tce:writet t
set tce:readint t
set tce:readt t

task tce

##############################################