error values in the target flux evolution

[Ibrahim Mohamed]

i was following the nacl tutorial and when i reached the w_ipa -ao command which produces the direct.h5 and assign.h5. when i downloaded them only from the remote cluster and opened direct.h5 using hdfview and double-clicked target flux evolution, it showed a table with values as ERROR.

What is the problem?

[Leung, Jeremy]

Hi Ibrahim,

It's impossible to debug just by looking at this.  Can you include your west.h5, west.cfg and direct.h5 to let us see what's going on?

-- JL

---
Jeremy M. G. Leung
PhD Candidate, Chemistry
Graduate Student Researcher, Chemistry (Chong Lab)
University of Pittsburgh | 219 Parkman Avenue, Pittsburgh, PA 15260
jml...@pitt.edu | [He, Him, His]

On Sep 2, 2023, at 2:40 PM, Ibrahim Mohamed <hollowic...@gmail.com> wrote:

i was following the nacl tutorial and when i reached the w_ipa -ao command which produces the direct.h5 and assign.h5. when i downloaded them only from the remote cluster and opened direct.h5 using hdfview and double-clicked target flux evolution, it showed a table with values as ERROR.

What is the problem?

<png.png>

--
You received this message because you are subscribed to the Google Groups "westpa-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to westpa-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/westpa-users/dc78e306-cdbd-4200-85e7-474181668ad3n%40googlegroups.com.
<png.png>

[Ibrahim Mohamed]

Thank you for your reply.

These are the files you asked for

[Jeremy Leung]

Hi Ibrahim,

Your west.h5 is corrupted. For example, iterations 1, 2, 35 are unreadable and the pcoord datasets are broken in other iterations. That likely caused the errors in the direct.h5.

The easiest way is probably to rerun this simulation.

-- JL

[Ibrahim Mohamed]

i re ran the simulation and used w_ipa -ao to perform the analysis, however, the problem is the same

[Leung, Jeremy]

Hi Ibrahim,

The west.h5 file is completely corrupted this time too. Would you be able to give us a list of all the commands (and files) you ran to generate this and what you've adjusted in the tutorial files?

Best,

Jeremy L.

---
Jeremy M. G. Leung
PhD Candidate, Chemistry
Graduate Student Researcher, Chemistry (Chong Lab)
University of Pittsburgh | 219 Parkman Avenue, Pittsburgh, PA 15260
jml...@pitt.edu | [He, Him, His]

To view this discussion on the web visit https://groups.google.com/d/msgid/westpa-users/564df1cf-72a5-470c-9a66-86bd3d607f8an%40googlegroups.com.
<west.h5><west.cfg><direct.h5>

[Ibrahim Mohamed]

i uploaded all the files before running the simulation as a compressed file

[Yang, Darian T]

Ibrahim,

 

Please send us some more info, e.g. describe the Python / conda env that you’re using, what version of westpa and gromacs?

Currently you can reproduce the corrupted h5 error, but if you could guide us on how we can reproduce it that would help.

 

When we validated the original tutorial files it worked fine, so tell us if you changed anything. If you’re not sure you could re-download the original files and run a diff.

 

Thanks,

Darian

To view this discussion on the web visit https://groups.google.com/d/msgid/westpa-users/78c517ba-90fd-48c8-a071-a9d8fa27a323n%40googlegroups.com.

[hollowic...@gmail.com]

Dear Darian,

I am using a remote server therefore i first activate the anaconda environment that contain the installed westpa (it was installed using pip not conda). the environmet itself was originally for another tool called gmx_MMPBSA which contained GROMACS 2021.3. i am using westpa version 2022.4.

attached are the installed libraries using conda and pip

in the previous email i sent a compressed file containing the files used for starting the run (after modifying the original tutorial). I think if the run was started using these files the error could be reproduced. 
Alternatively, could you please tell me what do you mean by "run a diff"?

Thanks

[Yang, Darian T]

Hi Ibrahim,

 

I was able to get our gmx tutorial to run after installing westpa and gromacs in a new conda environment:

https://github.com/westpa/westpa_tutorials/tree/main/additional_tutorials/basic_nacl_gmx

 

Try downloading it again and running it without changing the files as much. What I meant by diff is that you can use the “diff” command in the terminal to check the differences between files.

 

I did have to set my env.sh gromacs variables, and I updated the pcoord.py file to save as floats (which it looks like you did as well).

 

Using the same conda env, I ran your simulation file setup as well and I was able to get an uncorrupted h5 file out.

Note that you added some extra lines with absolute paths in the runseg.sh file that led to some ignorable errors in the log file.

 

Overall, I’d recommend try downloading the tutorial again and running it as is, then try changing one thing at a time to suit your needs and testing purposes. Since the NaCl system is lightweight, you could try setting it up locally to see if the remote server env could be causing any issues.

 

Best,

Darian

[hollowic...@gmail.com]

Thank you. 

I tried it on a workstation with gromacs installed from source and westpa installed using conda not pip and it worked well. Should I try to install it using conda on the remote cluster?

Sent from Yahoo Mail on Android

On Sat, 9 Sept 2023 at 1:23 am, Yang, Darian T

<DT...@pitt.edu> wrote:

[Yang, Darian T]

Ibrahim,

 

It shouldn’t matter for conda vs pip installing westpa, I pip installed westpa actually when I tested the tutorial, but into the new conda env which I also conda installed gromacs into.

 

Since this seems to be a cluster specific issue, usually we recommend reaching out to your computing resource staff.

From a guess, It could be a file i/o issue and maybe something could be changed with your submission script. Something is happening or being accessed too quickly/incorrectly on your cluster and the h5 file is not being populated correctly. You could also try making a new environment from scratch mimicking your local run before using the installed version already on the cluster (you may find out whether the issue is with the pre-installed software on the cluster or the job submission). If you’re using a work manager like zmq, the settings may need to be adjusted (see here).

 

-Darian

[hollowic...@gmail.com]

Dear Darian,

I tried to open the direct.h5 that was produced correctly on a workstation using hdfview on my PC but it showed the same error (all values are errors), although the same file worked with the hdfview installed on the workstation. However, when i tried to open it using h5py on WSL installed on the same PC, it showed the correct values.

Therefore i tried to run the simulation using my modified files on the cluster then tried to open the direct.h5 on my PC using either hdfview (this produced error) and h5py and this produced f['target_flux_evolution']['expected'] array values with shape of 50,2 (i ran 50 iterations)

i tried to plot the two columns separately and this is the image

I think the problem is with the hdfview not the installation of westpa on the cluster, right?

another question is what are the two columns in the expected values array?

[Yang, Darian T]

Hi Ibrahim,

 

I agree that it seems likely to come down to the hdfview, we have actually come into a few issues with hdfview before, but if you open using h5py and the data looks present and uncorrupted, I would assume that the h5 file is fine (this seems to be the case for you since you could plot the data).

 

In the f['target_flux_evolution']['expected'] array, there should be the flux for each of your labeled states. So here it may be the flux into state 0 and then state 1. This depends on how you set up the w_assign/w_direct or w_ipa calculation in west.cfg. You likely have the default NaCl setting where there is state 0 for ‘bound’ and state 1 for ‘unbound’. You should be seeing only flux into state 0 in the binding direction since these are the default recycling conditions. Although if you adjusted the target states or any boundaries it may look different. For reference, if I take the west.h5 for the NaCl turtorial and run a w_ipa -ao on it, I see only flux into the bound state (see attached direct.h5).