CC calculations hangs up

[jstysz]

Dear Dirac Users,

my CC calculations crashed due to a not sufficient memory.

After increasing memory per node, it progresses farther

but now hangs for hours in the last step of CC, i.e. when perturbative treatment of triple excitations starts (corresponding output attached).

What can I do about it? Should I increase a memory farther or the 64-bit version of Dirac

is necessary?

Best regards, Jacek

[Ilias Miroslav, doc. RNDr., PhD.]

Hello Jacek,

looking into your output I see

Type VVVV :      2 643 582 485 integrals, what nears to max. 32-bit integer value,  2³² − 1 = 4,294,967,295 ( https://en.wikipedia.org/wiki/Integer_overflow#Origin )

I assume that perturbative triples do exceed 32bit range, because of their scaling of  O(N^7).

Please recompile your OpenMPI and  DIRAC with 64-bit integers, see also http://diracprogram.org/doc/master/installation/int64/mpi64.html and http://diracprogram.org/doc/master/installation/int64/faq.html

And concerning the memory, please check the count, http://diracprogram.org/doc/master/tutorials/cc_memory_count/count_cc_memory.html .

Miro

---

From: dirac...@googlegroups.com <dirac...@googlegroups.com> on behalf of jstysz <jst...@gmail.com>
Sent: Wednesday, December 1, 2021 12:01
To: dirac-users <dirac...@googlegroups.com>
Subject: [dirac-users] CC calculations hangs up

 

--
You received this message because you are subscribed to the Google Groups "dirac-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dirac-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dirac-users/4a80e5cd-e002-4c16-9248-d24f8f200739n%40googlegroups.com.

[jstysz]

Hi Miro,

thanks a lot. I will try to install 64-bit version of Dirac21.

Regards, Jacek

[Juan Jose Aucar]

Dear Jacek,

You can try a compilation with OpenMP or ThreadedMKL. In the manual you can also find something about a mixed MPI and threaded MKL compilation. That way you can save some RAM (and time). You will notice its advantages on the moltra module.

Juan José

To view this discussion on the web visit https://groups.google.com/d/msgid/dirac-users/4bd69493-c2b5-4cbe-9ec4-d4452304510fn%40googlegroups.com.

[Johann Pototschnig]

How do you know it hangs? Are the processors empty?

For large systems the triples take more time than the index transformation or CC.

I see in your output that the sorting of the integrals takes about 45 min, so I think the triples should take more than 270 min.

There are improvements you should first consider:

1) As already mentioned in earlier answers you should link your code to BLAS and LAPACK libraries, otherwise it uses a rather inefficient internal implementation, which is not recommended for any production runs.

2) Also you can play around with the distribution of MPI parallelization to OPENMP threads.

OPENMP threads uses shared memory, but has difficulties for parallelization over different nodes.

3) You could switch on printing (increase print level) to get an idea of the progress of triples. 

These things you can test for smaller systems to get an idea how much time you can expect the triples to take.

best,

Johann

[Johann Pototschnig]

Regarding the Crash and memory problem:

You are using 64 MPI threads, which means means that you have 64 copies of the active arrays.

This can use up a lot of memory, so more OpenMP threads and less MPI threads might help.

best, Johann

[jstysz]

Hello Johan and Juan José,

my job did not write over the night (more than 6 hrs) to any file in a scratch directory, so I assumed that it hangs.

Thank you very much for your comments and suggestions.

I will install Dirac21 with 64-bit integers and see if it helps.

Best regards, Jacek