FFTW_PLAN_TYPE PATIENT and EXHAUSTIVE not working

[Léon Luntadila Lufungula]

Dear all,

I've been running some AIMD calculations and am trying to speed up the calculations a bit by playing with the FFTW_PLAN_TYPE option. Unfortunately, only MEASURE and the default ESTIMATE are working. If I try to set it to PATIENT (as recommended for long AIMD runs) or EXHAUSTIVE, the calculation crashes almost immediately with the following error messages (see also attached files):

[PATIENT]

...

corrupted double-linked list

corrupted double-linked list (not small)

cp2k.popt: malloc.c:4106: _int_malloc: Assertion `(unsigned long) (size) >= (unsigned long) (nb)' failed.

 

Program received signal SIGABRT: Process abort signal.

...

[EXHAUSTIVE]

...

malloc_consolidate(): unaligned fastbin chunk detected

malloc_consolidate(): unaligned fastbin chunk detected

 

Program received signal SIGABRT: Process abort signal.

 

Backtrace for this error:

corrupted double-linked list

 

Program received signal SIGABRT: Process abort signal.

 

Backtrace for this error:

malloc_consolidate(): unaligned fastbin chunk detected

 

Program received signal SIGABRT: Process abort signal.

 

Backtrace for this error:

 

Program received signal SIGABRT: Process abort signal.

 

Backtrace for this error:

corrupted double-linked list

cp2k.popt: malloc.c:4106: _int_malloc: Assertion `(unsigned long) (size) >= (unsigned long) (nb)' failed.

...

I'm running CP2K/2023.2-foss-2022a as compiled with Easybuild by our HPC centre, but the same problems appear when I try the CP2K/2022.1-foss-2022a version. However, when I run it with the CP2K/7.1-intel-2020a version which is also available, both EXHAUSTIVE and PATIENT seem to be working properly... Is this something that can be solved in some way or will this require a different compilation of CP2K, possibly with the intel toolchain instead of the foss toolchain? 

Kind regards,

Léon

[Frederick Stein]

Dear Leon,

I do not know the root of the error and I cannot suggest a solution. The options themself are tested within our regtest suite and we do not find any issues there. So, it seems to be a more complicated problem either on FFTW site or on CP2K site.

I have two questions:

1. Did you check whether the FFT kernel actually needs an improvement? Check the runtime of the routines pw_transfer and those starting with fft_wrap_pw1pw2 (or similar).

2. Do you have a pdbg-version of CP2K available? If yes, can you run one of the failing tests with that one? It might also help to turn on the keywords TRACE and TRACE_MASTER in the GLOBAL section of your input files to identify the actual routine on CP2K site where the error occurs.

Regards,

Frederick

[Léon Luntadila Lufungula]

Dear Frederick,

Thanks for guiding me in solving this issue!

1) I have not yet checked if this is indeed the case but I'm quite new to AIMD calculations and the reference manual stated that this was recommended for long AIMD trajectories, so I assumed this would be a good option to enable. If you could show me how to check this, I would greatly appreciate the help! For the same calculation with the default FFTW_PLAN_TYPE ESTIMATE I get the following timings:

 -------------------------------------------------------------------------------
 -                                                                             -
 -                                T I M I N G                                  -
 -                                                                             -
 -------------------------------------------------------------------------------
 SUBROUTINE                       CALLS  ASD         SELF TIME        TOTAL TIME
                                MAXIMUM       AVERAGE  MAXIMUM  AVERAGE  MAXIMUM
 CP2K                                 1  1.0      0.0      0.0 259198.3 259201.0
 qs_mol_dyn_low                       1  2.0      0.5      0.6 259198.0 259200.7
 velocity_verlet                  21600  3.0      4.7      5.3 259117.8 259120.7
 qs_forces                        21601  4.0      6.7      8.3 259054.1 259057.9
 qs_energies                      21601  5.0      1.3      2.0 216944.6 216962.9
 scf_env_do_scf                   21601  6.0      0.7      1.0 197423.7 197426.9
 scf_env_do_scf_inner_loop       172819  7.0      2.9      3.7 147536.9 147542.8
 rebuild_ks_matrix               194420  8.7      0.7      0.7 113213.6 113417.8
 qs_ks_build_kohn_sham_matrix    194420  9.7     21.9     25.3 113212.9 113417.2
 qs_ks_update_qs_env             194420  8.0      1.5      1.8  85131.3  85315.6
 pw_transfer                    4104421 12.9    259.1    293.8  53272.8  54657.8
 sum_up_and_integrate            194420 10.7    158.9    210.4  54433.6  54518.3
 integrate_v_rspace              194420 11.7      6.8      8.9  54274.2  54375.6
 fft_wrap_pw1pw2                3715581 14.0     41.8     46.1  52788.9  54160.2
 fft_wrap_pw1pw2_200            2160221 15.4   5123.8   5554.3  50067.4  51727.8
 init_scf_loop                    21601  7.0      0.6      0.8  49795.6  49797.8
 qs_vxc_create                   194420 10.7      4.5      5.3  49235.6  49729.8
 xc_vxc_pw_create                194420 11.7    990.7   1608.7  49231.1  49725.4
 qs_rho_update_rho_low           194420  8.1      0.8      0.9  44519.7  44594.7
 calculate_rho_elec              194420  9.1    109.2    184.2  44518.9  44593.9
 fft3d_ps                       3715581 16.0  10122.9  14717.9  41927.7  43177.5
 dbcsr_multiply_generic         2981151 12.6    130.7    138.3  42145.8  42708.1
 prepare_preconditioner           21601  8.0      0.1      0.2  38552.7  38578.0
 grid_integrate_task_list        194420 12.7  34380.1  37401.8  34380.1  37401.8
 xc_pw_derive                   1166520 13.7      9.8     11.0  34752.0  36587.5
 make_preconditioner              21601  9.0      0.4      0.6  36451.6  36464.5
 qs_scf_new_mos                  172819  8.0      0.9      1.4  34340.9  34469.5
 qs_scf_loop_do_ot               172819  9.0      0.8      1.0  34340.0  34468.7
 make_full_all                    21601 10.0      4.6      5.3  34250.9  34260.8
 ot_scf_mini                     172819 10.0      4.0      4.7  31468.4  31598.1
 multiply_cannon                2981151 13.6    192.3    222.5  29228.9  30932.4
 mp_alltoall_z22v               3715581 18.0  27949.9  30346.5  27949.9  30346.5
 multiply_cannon_loop           2981151 14.6    215.7    231.9  27541.2  28832.2
 xc_pw_divergence                194420 12.7      4.0      4.7  26628.5  28220.2
 qs_ks_update_qs_env_forces       21601  5.0      0.1      0.2  28174.2  28195.0
 xc_rho_set_and_dset_create      194420 12.7    239.1    269.1  20595.1  28022.0
 ot_mini                         172819 11.0      1.0      1.4  23618.8  23759.5
 grid_collocate_task_list        194420 10.1  22590.9  23442.1  22590.9  23442.1
 rs_pw_transfer                 1987402 12.3     41.9     48.7  21225.8  22117.5
 yz_to_x                         993701 17.7    742.7   1149.4  19795.7  21777.5
 density_rs2pw                   194420 10.1      8.1      8.9  20317.1  21233.4
 mp_waitall_1                   ******* -4.3  17314.7  19114.1  17314.7  19114.1
 multiply_cannon_multrec        ******* 15.6  17135.6  18128.8  17149.2  18142.9
 cp_fm_diag_elpa                  64803 11.3      0.4      0.5  18118.8  18125.4
 cp_fm_diag_elpa_base             64803 12.3  17919.4  18033.2  18104.5  18104.8
 mp_waitany                     ******* 14.1  13555.6  14684.3  13555.6  14684.3
 xc_functional_eval              194420 13.7      2.0      2.6   8289.8  14473.2
 pbe_lda_eval                    194420 14.7   8287.8  14472.3   8287.8  14472.3

2) Unfortunately, no pdbg-version was compiled on my HPC center, but I did run a calculation with TRACE and TRACE_MASTER enabled. See attached files for the output.

Kind regards,

Léon

[Frederick Stein]

Dear Léon,

Regarding 1:

Did you optimize the accuracy-relevant parameters (CUTOFF, EPS_DEFAULT) to meet your demands regarding energies and gradients? If not, there is an how to concerning this topic (https://www.cp2k.org/howto:converging_cutoff). Choose the cutoff as high as necessary and as low as possible.

TLDR: From your timing report, I recommend you to employ the MEASURE mode, see below for a more lengthy explanation.

If you have optimized these parameters, check the timing report. The total CP2K runtime is given by the line with "CP2K" (ignoring warming up times of the cluster and of CP2K). Each line represents a routine within CP2K, from the name you may guess its purpose/task. Relevant to you are especially the last four columns. The last two columns indicate the average and maximum time spent in a given routine and its called routines. They give you an indication which functionality spends the most time. The fourth and third last columns indicate the the average and maximum time spent in the routine itself, i.e. without the routines called by the given routine. These two columns indicate in which compute kernel CP2K spends its most amount of time. If you check the routines with the largest self-time, you find the highest self-times of FFT-related routines, i.e. those containing 'fft' in their names, in case of fft3d_ps (10122.9 s, 3.9 % of CP2K) and fft_wrap_pw1pw2_200 (5123.8 s, 2.0 % of CP2K). Even if you were able to reduce the timing of these routines to zero, you would gain ca. 6 % of computation time, in practice it will be much less. So, I think that a further reduction would not be able to reduce the computational time significantly.

For completeness, the difference to the total runtime of the fft_wrap_pw1pw2 routine of ~50000 s is related to the respective communication routines which do not have 'fft' in their name. These are of course not affected by the FFTW planning times.

Regarding 2:

Checking the source code of the routine fft_get_scratch in src/pw/fft_tools.F indicates that the issue might be related to an issue on a lower level routine within CP2K (probably src/fft/fftw3_lib.F) or in FFTW3 directly. I did some local tests with one of the single-node tests and did not observe any issues in these modes. So, the issue will need more thorough investigations.

HTH

Frederick

[Léon Luntadila Lufungula]

Dear Frederick,

1) Thanks for the detailed explanation regarding the CP2K timings! I have optimized the parameters you mention for my static DFT calculations where I use CUTOFF=800, REL_CUTOFF=40, EPS_DEFAULT=1.0E-12 and EPS_SCF=1.0E-7, giving a difference of 1E-5 a.u. w.r.t. a reference calculation with CUTOFF=4000 and REL_CUTOFF=80. For my AIMD calculations I also tested the conservation of energy during an NVE calculation and got excellent conservation for CUTOFF=400, REL_CUTOFF=40, EPS_DEFAULT=1.0E-10 and EPS_SCF=1.0E-7, increasing EPS_DEFAULT to 1.0E-12 didn't seem to change much except for the time required to perform an MD step(7.5s vs. 8.2s). 

Fortunately, I have already tried FFTW_PLAN_TYPE MEASURE a couple of days ago when posting my initial question (see timings below), but this didn't give much of an improvement (21820 steps vs. 21590 steps). 

 -------------------------------------------------------------------------------
 -                                                                             -
 -                                T I M I N G                                  -
 -                                                                             -
 -------------------------------------------------------------------------------
 SUBROUTINE                       CALLS  ASD         SELF TIME        TOTAL TIME
                                MAXIMUM       AVERAGE  MAXIMUM  AVERAGE  MAXIMUM

 CP2K                                 1  1.0      0.0      0.0 259200.9 259202.7
 qs_mol_dyn_low                       1  2.0      0.5      0.6 259200.6 259202.4
 velocity_verlet                  21829  3.0      4.7      5.3 259116.7 259118.8
 qs_forces                        21830  4.0      6.7      7.9 259052.2 259054.9
 qs_energies                      21830  5.0      1.3      1.8 217472.4 217487.7
 scf_env_do_scf                   21830  6.0      0.7      1.0 198082.4 198085.3
 scf_env_do_scf_inner_loop       174656  7.0      3.0      3.7 147728.9 147733.7
 rebuild_ks_matrix               196486  8.7      0.6      0.7 112527.8 112729.8
 qs_ks_build_kohn_sham_matrix    196486  9.7     21.8     25.5 112527.2 112729.1
 qs_ks_update_qs_env             196486  8.0      1.4      1.7  85161.6  85342.5
 sum_up_and_integrate            196486 10.7    160.0    209.2  54201.8  54293.4
 pw_transfer                    4148036 12.9    263.2    304.5  52714.9  54167.5
 integrate_v_rspace              196486 11.7      7.0     10.2  54041.3  54133.1
 fft_wrap_pw1pw2                3755064 14.0     42.4     47.1  52224.1  53695.2
 fft_wrap_pw1pw2_200            2183176 15.4   5174.7   5626.2  49517.7  51266.7
 init_scf_loop                    21830  7.0      0.6      0.7  50261.2  50262.9
 qs_vxc_create                   196486 10.7      4.7      5.6  48643.3  49120.6
 xc_vxc_pw_create                196486 11.7   1000.6   1625.3  48638.7  49116.1
 qs_rho_update_rho_low           196486  8.1      0.8      0.9  44051.5  44126.2
 calculate_rho_elec              196486  9.1    110.3    185.9  44050.7  44125.4
 dbcsr_multiply_generic         3012817 12.6    132.8    140.6  42612.9  43191.8
 fft3d_ps                       3755064 16.0   9552.3  14496.2  41234.7  42520.7
 prepare_preconditioner           21830  8.0      0.1      0.2  38987.3  39009.8
 grid_integrate_task_list        196486 12.7  34732.0  37662.0  34732.0  37662.0
 make_preconditioner              21830  9.0      0.4      0.4  36853.2  36863.4
 xc_pw_derive                   1178916 13.7      9.7     11.2  34115.8  36015.3
 qs_scf_new_mos                  174656  8.0      0.9      1.3  34716.3  34849.9
 qs_scf_loop_do_ot               174656  9.0      0.8      1.0  34715.4  34849.1
 make_full_all                    21830 10.0      4.9      6.4  34635.3  34645.4
 ot_scf_mini                     174656 10.0      4.0      4.6  31757.0  31879.2
 multiply_cannon                3012817 13.6    193.3    227.5  29505.4  31072.7
 mp_alltoall_z22v               3755064 18.0  27816.9  30306.8  27816.9  30306.8
 multiply_cannon_loop           3012817 14.6    217.0    235.4  27788.4  29096.5
 xc_rho_set_and_dset_create      196486 12.7    241.7    271.6  20750.7  27765.4
 qs_ks_update_qs_env_forces       21830  5.0      0.1      0.2  27457.9  27479.8
 xc_pw_divergence                196486 12.7      4.0      5.2  25840.6  27415.2
 ot_mini                         174656 11.0      1.0      1.3  23884.9  24039.5
 grid_collocate_task_list        196486 10.1  22773.0  23579.2  22773.0  23579.2
 rs_pw_transfer                 2008520 12.3     42.1     49.3  20604.4  21564.4
 yz_to_x                        1004260 17.7    746.8   1231.3  19488.5  21530.0
 density_rs2pw                   196486 10.1      8.2      9.2  19637.2  20954.1
 mp_waitall_1                   ******* -4.1  17488.2  19251.7  17488.2  19251.7
 multiply_cannon_multrec        ******* 15.6  17302.7  18334.2  17316.4  18348.7
 cp_fm_diag_elpa                  65490 11.3      0.4      0.5  18289.1  18296.1
 cp_fm_diag_elpa_base             65490 12.3  18094.9  18211.7  18274.5  18274.9
 xc_functional_eval              196486 13.7      2.0      2.9   8365.0  14324.5
 pbe_lda_eval                    196486 14.7   8363.0  14323.4   8363.0  14323.4
 mp_waitany                     ******* 14.1  12907.9  14030.4  12907.9  14030.4
 build_core_hamiltonian_matrix_   21830  5.0      0.7      0.9  10163.4  12916.0
 potential_pw2rs                 196486 12.7     11.0     12.1  12811.3  12862.1
 apply_preconditioner_dbcsr      196486 13.0      0.4      0.6  12274.1  12495.7
 apply_all                       196486 14.0     13.8     14.5  12273.7  12495.3
 qs_ot_get_derivative            174656 12.0      1.0      1.1  12107.0  12231.2
 ot_diis_step                    174656 12.0      6.1      7.4  11670.0  11670.4
 multiply_cannon_metrocomm3     ******* 15.6     90.4    100.6   7449.1  11410.0
 init_scf_run                     21830  6.0      0.3      0.4  10764.8  10766.3
 scf_env_initial_rho_setup        21830  7.0      0.2      0.2  10764.4  10765.8
 cp_fm_cholesky_reduce            21830 11.0  10738.1  10744.9  10738.1  10744.9
 wfi_extrapolate                  21830  8.0      1.4      1.6  10707.6  10707.8
 mp_alltoall_d11v               4671757 13.6   8744.4  10633.3   8744.4  10633.3
 x_to_yz                        1178916 17.2   1200.0   1689.5   9900.2  10462.4
 make_m2s                       6025634 13.6    115.7    153.5   9932.7  10333.9
 rs_pw_transfer_RS2PW_200        218316 11.9    900.4   1148.9   8610.3   9435.7
 make_images                    6025634 14.6    244.1    262.0   8865.5   9141.7
 mp_sum_d                       3821027 12.6   5682.4   7874.9   5682.4   7874.9
 qs_energies_init_hamiltonians    21830  6.0      0.4      0.5   7800.9   7801.2
 rs_gather_matrices              196486 12.7    112.7    159.0   6365.5   7362.5
 make_images_data               6025634 15.6     59.0     83.4   6337.1   7169.9
 hybrid_alltoall_any            6396752 16.4     83.7    763.9   5601.4   7058.6
 qs_ot_get_derivative_taylor     174656 13.0      1.9      2.1   6380.8   6497.4
 build_core_ppnl_forces           21830  6.0   5341.1   6203.0   5341.1   6203.0
 multiply_cannon_metrocomm1     ******* 15.6     48.1     52.4   2392.5   5507.1

2) I just checked which version of FFTW was in the foss-2023a toolchain used by Easybuild to compile CP2K2023.2 and it is apparently the 3.3.10-version (see https://docs.easybuild.io/common-toolchains/). The other compilation on my HPC center was a 7.1 version of CP2K compiled with the intel-2020a toolchain, where FFTW_PLAN_TYPE PATIENT did work, but I can't seem to figure out which version of FFTW is used (see attached output of calculation). So I'm not quite sure if the thing that makes the other compilation work is the version of FFTW or the version of CP2K itself. I don't have experience with compiling CP2K either manually or through esybuild, so I wouldn't know how to test either hypothesis...

Kind regards,

Léon