Serious performance regression and potential LOCAL_AVERAGE bug in PLUMED 2.10 with GROMACS 2023.5

[张悦]

Dear PLUMED developers,

I am writing to report a significant performance issue and a potential bug when using PLUMED 2.10.0 with GROMACS 2023.5. I hope you can provide some guidance.

System and Setup

I am simulating an antifreeze protein system with an ice-water interface. The system contains:

• Ice crystal: 8,400 water molecules

• Solution: 36,950 water molecules (TIP4P/ICE model)

I am performing enhanced sampling on the 36,950 solution water molecules. The MD engine is GROMACS 2023.5. The PLUMED input script I am using is as follows:

AO: GROUP NDX_FILE=index2.ndx NDX_GROUP=OW_A
IO: GROUP NDX_FILE=index2.ndx NDX_GROUP=OW_I
WO: GROUP NDX_FILE=index2.ndx NDX_GROUP=OW_W (36,950 solution water molecules)

b: COM ATOMS=WO
q6: Q6 SPECIES=WO SWITCH={RATIONAL D_0=1.3 R_0=0.2}
la: LOCAL_AVERAGE SPECIES=q6 SWITCH={RATIONAL D_0=1.3 R_0=0.2} MEAN
s: AROUND DATA=la ATOM=b ZLOWER=-10 ZUPPER=20 SIGMA=0.01 MEAN

t1a: AVERAGE ARG=s.mean CLEAR=50
PRINT ARG=t1a FILE=colvar STRIDE=50

Performance Issue

• Pure GROMACS 2023.5 (1 GPU + 32 CPU cores): 4.9 ns/day

• GROMACS 2023.5 + PLUMED 2.9.1 (same hardware): 0.001 ns/day

This represents a ~4,900-fold slowdown, making the simulation practically infeasible. While I understand that PLUMED calculations can be expensive, this level of degradation seems extreme for a system of this size and is inconsistent with the typical expectation that PLUMED overhead should be small compared to force evaluation.

Potential Bug in PLUMED 2.10.0

I attempted to address the performance issue by upgrading to PLUMED 2.10.0, which I believe includes GPU-accelerated features. However, the simulation fails immediately with the following assertion error:

PLMD::symfunc::LocalAverage::LocalAverage(const PLMD::ActionOptions&)
+++ assertion failed: av && av->getNumberOfComponents()>0 && (av->copyOutput(0))->getRank()==2

This suggests a problem with the LOCAL_AVERAGE command, which is central to my CV calculation. The input script works without error in PLUMED 2.9.1, so this appears to be a regression introduced in version 2.10.0.

Questions

1. Is there a known solution for the extreme slowdown observed with PLUMED 2.9.1 for this type of LOCAL_AVERAGE and Q6 calculation? Could the performance be improved by adjusting the number of OpenMP threads or other GROMACS settings, as suggested in the documentation?

2. For the PLUMED 2.10.0 bug, is this a known issue with LOCAL_AVERAGE that will be fixed in an upcoming patch? Are there any workarounds I can implement in my input script to bypass it while still using the new version?

3. Given my goal is to perform this specific enhanced sampling simulation, do you recommend I stick with the older, working version (2.9.1) and focus on parallelization/optimization, or should I pursue a fix for the 2.10.0 regression? Any guidance on the recommended path forward would be greatly appreciated.

I am happy to provide any additional details, log files, or test inputs to help diagnose these issues. Thank you for your time and for developing this valuable software.

Sincerely,

Yue Zhang