TypeError when trying to run w_ipa -ao with a MAB simulation (westpa 2.0)

[Rebecca Walters]

Hi, 

I am running a steady-state MAB simulation and trying to get some rate constants out. I have been following the Livecoms tutorial and see that I can use w_ipa -ao command to calculate rate constants (and other data).

However, when I try and run the w_ipa -ao command I get the following error: 

Welcome to w_ipa (WESTPA Interactive Python Analysis) v. 1.0B!

Run w.introduction for a more thorough introduction, or w.help to see a list of options.

Running analysis & loading files.

Reanalyzing file assign.h5 for scheme TEST.

Traceback (most recent call last):

  File "/mnt/storage/scratch/ft18983/miniconda3/envs/westpa-2.0/bin/w_ipa", line 33, in <module>

    sys.exit(load_entry_point('westpa', 'console_scripts', 'w_ipa')())

  File "/mnt/storage/scratch/ft18983/Neuraminidase/westpa/src/westpa/cli/tools/w_ipa.py", line 815, in entry_point                                                                  ]

    w.main()

  File "/mnt/storage/scratch/ft18983/Neuraminidase/westpa/src/westpa/tools/core.py", line 171, in main

    self.go()

  File "/mnt/storage/scratch/ft18983/Neuraminidase/westpa/src/westpa/cli/tools/w_ipa.py", line 733, in go

    self.analysis_structure()

  File "/mnt/storage/scratch/ft18983/Neuraminidase/westpa/src/westpa/cli/tools/w_ipa.py", line 324, in analysis_structure

    assign.go()

  File "/mnt/storage/scratch/ft18983/Neuraminidase/westpa/src/westpa/cli/tools/w_assign.py", line 574, in go

    assignments, trajlabels, pops, statelabels = self.assign_iteration(n_iter, nstates, nbins, state_map, last_labels)

  File "/mnt/storage/scratch/ft18983/Neuraminidase/westpa/src/westpa/cli/tools/w_assign.py", line 443, in assign_iteration

    assign_slice, traj_slice, slice_pops, lb, ub, state_slice = future.get_result(discard=True)

  File "/mnt/storage/scratch/ft18983/Neuraminidase/westpa/src/westpa/work_managers/core.py", line 334, in get_result

    raise self._exception.with_traceback(self._traceback)

TypeError: __traceback__ must be a traceback or None

Although I get this error, the ANALYSIS directory is successfully created, as well as the assign.h5 file - however, the direct.h5 file is not generated. 

I think the problem is to do with my west.cfg file (attached). In settings for the w_ipa section, I see that the required settings are: 'name (TEST), states, and bins'. I have defined the bin type to be RectilinearBinMapper (from the tutorial) however I realise that this will likely cause issues with the adaptive binning scheme. 

Is someone able to point me in the right direction? Is there a different bin type I need to define in my west.cfg? I am also attaching my adaptive.py script, in case that is useful. 

Please let me know if I can provide any more information to help solve the problem - thank you!

Rebecca 

[Rebecca Walters]

I don't think my adaptive.py attached correctly - I am re-attaching it here, apologies!

[Jeremy Leung]

Hi Rebecca,

The RectilinearBinMapper choice shouldn't matter. w_assign takes all the trajectories and place them in the new rectilinear bins generated. The older binning scheme (adaptive binning) is not considered. So this issue isn't related to the MAB scheme.

Looking at the error message, it's failing at the w_assign step before the data is return. The assign.h5 created, I assume, will be empty. I would suggest running each part of w_ipa (which is basically w_assign + w_direct) separately to check. You can do it a couple ways:

1) Look at the example in this link and using it for your simulation:

https://github.com/westpa/westpa/wiki/man:w_direct#Examples

2) Look at the following and run w_assign in debug mode to see what errors are produced.  

https://github.com/westpa/westpa/wiki/man:w_assign

If this is the same simulation data from the other help thread, I wouldn't be surprised it's the input data that is causing trouble.

Hope that helps,

Jeremy L.

--
Jeremy M. G. Leung
PhD Candidate, Chemistry
Graduate Student Researcher, Chemistry 

University of Pittsburgh | 219 Parkman Avenue, Pittsburgh, PA 15260
jml...@pitt.edu | [He, Him, His]

--
You received this message because you are subscribed to the Google Groups "westpa-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to westpa-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/westpa-users/631e63ff-676f-4610-849c-ce51845d9800n%40googlegroups.com.
<adaptive.py>

[Rebecca Walters]

Hi Jeremy, 

Thank you for your quick response. 

I have followed your advice in the other thread I made and updated my adaptive.py to include the targetstate=[value] and re-run my simulation - my bad!! I am including my updated adaptive.py, west.cfg, and tstate.file. 

However, I am still getting the same TypeError when I run w_ipa -ao... I followed your advice in this thread and got the following output:

1. Running w_assign --debug produced this output file (I am attatching it as 'w_assign_debug_output.txt'). I also got this error after I ran w_assign --debug:

(westpa-2.0) [ft18983@bc4login2 ChemDyME_NA_OSE]$ w_assign --debug > w_assign_debug_output.txt

Traceback (most recent call last):

  File "/mnt/storage/scratch/ft18983/miniconda3/envs/westpa-2.0/bin/w_assign", line 33, in <module>

    sys.exit(load_entry_point('westpa', 'console_scripts', 'w_assign')())

  File "/mnt/storage/scratch/ft18983/Neuraminidase/westpa/src/westpa/cli/tools/w_assign.py", line 593, in entry_point

    WAssign().main()

  File "/mnt/storage/scratch/ft18983/Neuraminidase/westpa/src/westpa/tools/core.py", line 171, in main

    self.go()

  File "/mnt/storage/scratch/ft18983/Neuraminidase/westpa/src/westpa/cli/tools/w_assign.py", line 589, in go

    h5io.stamp_iter_range(self.output_file[dsname], iter_start, iter_stop)

  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

  File "/mnt/storage/scratch/ft18983/miniconda3/envs/westpa-2.0/lib/python3.8/site-packages/h5py/_hl/group.py", line 288, in __getitem__

    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)

  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

  File "h5py/h5o.pyx", line 190, in h5py.h5o.open

KeyError: "Unable to open object (object 'statelabels' doesn't exist)"

2. I tried running w_direct on two different assign.h5 files - the one that was created when I tried to run w_ipa (which is in the ANALYSIS/TEST directory), and the one which I created when I ran w_assign directly from the main simulation directory (w_assign -o test_assign.h5). I am including the output of both just incase:

• Output from running w_direct all -a ANALYSIS/TEST/assign.h5 -W west.h5 -o direct.h5 --step-iter 1 (from the examples link you sent me) produced this output (I am attaching it as 'w_direct_output_ANALYSISTESTassign.txt')
• Output from running w_direct all -a assign.h5 -W west.h5 -o direct.h5 --step-iter 1 produced this output (attached as 'w_direct_output_assign.txt')

I hope that makes sense, are you able to help me understand the problem further?? Let me know if you require more information from me!

Kind regards, 

Rebecca 

[Jeremy Leung]

Hi Rebecca,

w_direct wouldn't work because the assign.h5 is not properly created. So let's debug w_assign first.

w_assign requires some inputs in order to work. I believe you're getting an error because you're solely running w_assign without any config. 

w_assign -W west.h5 --config-from-file --scheme TEST --serial --debug

-W specifies the h5 file you want to use. --config-from-file tells it to read from west.cfg (the default), looking at --scheme TEST. I'm assuming you're trying to assign states using the pcoord. Or else, you would have to specify a construct dataset function, which is also listed in the wiki.

If there aren't any errors, I would take a look to see the datasets listed here to see if they're properly created/populated (here are some resources if you're not familiar--i suggest just using the h5py module in ipython since you're working in the command line). If they're good, you can try the w_direct commands. Those look correct.

Hope that helps!

Jeremy L.

--
Jeremy M. G. Leung
PhD Candidate, Chemistry
Graduate Student Researcher, Chemistry 
University of Pittsburgh | 219 Parkman Avenue, Pittsburgh, PA 15260
jml...@pitt.edu | [He, Him, His]

To view this discussion on the web visit https://groups.google.com/d/msgid/westpa-users/ca31afef-37cf-4d0c-9327-f62eae68db8dn%40googlegroups.com.
<west.cfg><adaptive.py><w_assign_debug_output.txt><tstate.file><w_direct_output_assign.txt><w_direct_output_ANALYSISTESTassign.txt>

[Rebecca Walters]

Hi Jeremy,

Ah right that makes sense! You are correct in thinking I am trying to assign states using the pcoord.

I think I spotted another error in my scripts - in the ANALYSIS section of my west.cfg, the boundaries I set up were  boundaries: [0, 30.0, 70, 'inf'], when they should have been ['-inf', 30.0, 70, 'inf'] because my progress coordinate can go below 0 (and the mincap I set in adaptive.py was '-inf'). Now when I run  w_assign -W west.h5 --config-from-file --scheme TEST --serial --debug I don't get any errors. 

Thank you I will check the datasets to make sure everything is made correctly. If everything is fine I will try running w_direct. I will have limited access to my computer next week (vacation) but thank you for your help so far - I will let you know how I get on when I come back :)

Kind regards, 
Rebecca 

[Rebecca Walters]

Hi Jeremy, 

I hope you are well! I hope it is okay to come back to you with my westpa problem. 

I have had a look at my assign.h5 file that I have generated (by running the w_assign -W west.h5 --config-from-file --scheme TEST --serial command) and compared the datasets to what we expect to be created (following the link you sent in your last email). I am using HDFView 3.1.3 to open my h5 files. 

It appears that I have successfully created all the different datasets, although I don't know where to find the attributes nstates and nbins... I also have an additional file called statelabels (as well as the expected state_labels). 

Just to be clear, am I correct in thinking that e.g. in the labeled_populations dataset, what I expect to get out is the population of each of the analysis bins (that I assigned in my west.cfg) for each trajectory in each iteration? So, for example, my analysis bins are defined as ['-inf', 30.0, 70, 'inf'] in my west.cfg file, then I have 3 'states': ['-inf', 30.0], [30, 70], and [70, 'inf']. If I also request 100 iterations (also defined in my west.cfg) and a total of 4 trajectories per bin (defined in my adatpive.py, and I have a 1D progress coordinate). Therefore, I expect my labeled_populations dataset to be of the shape 100, 3, 4 i.e. 100 iterations x 3 possible states x 4 trajectories per bin. Is this correct??

If I am correct, I think my assign.h5 file looks okay, although I am attaching it just in case.

Now, when I run w_direct with the following command w_direct all -a ANALYSIS/TEST/assign.h5 -W west.h5 -o direct.h5 --step-iter 1, I get the following std output:

Operation:      Calculating Target Flux Evolution

fluxes into macrostates:                                                                                                                                                            ]

b'bound': mean=0.000000000000000e+00 CI=(0.000000000000000e+00, 0.000000000000000e+00) * tau^-1

b'unbound': mean=1.091473752739355e-09 CI=(0.000000000000000e+00, 3.274579618517711e-09) * tau^-1

fluxes from state to state:

b'bound' -> b'unbound': mean=1.091473752739355e-09 CI=(0.000000000000000e+00, 3.274262897918420e-09) * tau^-1

b'unbound' -> b'bound': mean=0.000000000000000e+00 CI=(0.000000000000000e+00, 0.000000000000000e+00) * tau^-1

rates from state to state:

b'bound' -> b'unbound': mean=1.091473753275574e-09 CI=(0.000000000000000e+00, 3.274262902743184e-09) * tau^-1

b'unbound' -> b'bound': mean=0.000000000000000e+00 CI=(0.000000000000000e+00, 0.000000000000000e+00) * tau^-1

Operation:      Calculating State Probability Evolution

average color probabilities:                                                                                                                                                        ]

b'bound': mean=9.999999995087192e-01 CI=(9.999999985263589e-01, 9.999999999999687e-01) * tau^-1

b'unbound': mean=4.912802268916663e-10 CI=(2.721817650152191e-14, 1.473694939711732e-09) * tau^-1

average state probabilities:

b'bound': mean=9.769687495876010e-01 CI=(9.619207654399207e-01, 9.920167337352814e-01) * tau^-1

b'unbound': mean=5.458884321251973e-11 CI=(2.969255618347843e-15, 1.637477862114684e-10) * tau^-1

and my direct.h5 file is successfully created. I checked whether I have all of the correct datasets in direct.h5 and I do. However, additional datasets were also created: avg_color_probs, avg_conditional_fluxes, avg_rates, avg_state_probs, avg_total_fluxes, color_prob_evolution, conditional_flux_evolution, rate_evolution, state_labels, state_pop_evolution, and target_flux_evolution.

Is this the same output as I would expect by just running w_ipa -ao alone, instead of w_assign and w_direct separately?? Can i just get the flux data from my direct.h5? Hope this makes sense and sorry for the long email!

Kind regards, 

Rebecca 

[Jeremy Leung]

Hi Rebecca, 

Glad everything worked out. I'll reply everything inline below.

Cheers,

Jeremy L.

--
Jeremy M. G. Leung
PhD Candidate, Chemistry
Graduate Student Researcher, Chemistry 
University of Pittsburgh | 219 Parkman Avenue, Pittsburgh, PA 15260
jml...@pitt.edu | [He, Him, His]

On 2 Sep 2021, at 11:05 AM, Rebecca Walters <rebecca...@live.com> wrote:

Hi Jeremy, 

I hope you are well! I hope it is okay to come back to you with my westpa problem. 

I have had a look at my assign.h5 file that I have generated (by running the w_assign -W west.h5 --config-from-file --scheme TEST --serial command) and compared the datasets to what we expect to be created (following the link you sent in your last email). I am using HDFView 3.1.3 to open my h5 files. 

It appears that I have successfully created all the different datasets, although I don't know where to find the attributes nstates and nbins... I also have an additional file called statelabels (as well as the expected state_labels). 

Since they are attributes, you'd have to see it in the attribute pane. (right click on assign.h5 > properties > attributes)

        Using h5py, if west is your h5 file, then it would be:

west.attrs['nbins']  and west.attrs['nstates']

Just to be clear, am I correct in thinking that e.g. in the labeled_populations dataset, what I expect to get out is the population of each of the analysis bins (that I assigned in my west.cfg) for each trajectory in each iteration? So, for example, my analysis bins are defined as ['-inf', 30.0, 70, 'inf'] in my west.cfg file, then I have 3 'states': ['-inf', 30.0], [30, 70], and [70, 'inf']. If I also request 100 iterations (also defined in my west.cfg) and a total of 4 trajectories per bin (defined in my adatpive.py, and I have a 1D progress coordinate). Therefore, I expect my labeled_populations dataset to be of the shape 100, 3, 4 i.e. 100 iterations x 3 possible states x 4 trajectories per bin. Is this correct??

Yes that is correct. The structure for all dataset are explained in our w_assign page on the wiki.

If I am correct, I think my assign.h5 file looks okay, although I am attaching it just in case. 

Structure-wise, It looks good. Though I'm having trouble reading some of the datasets in your files. I'm not sure if they were corrupted through email/saving or they are corrupted somehow during the process of creation. But if you're able to view them then that's all good! 

(It's possible you might have it open when you attached it. Only one process can open an h5 file at a time.)

Now, when I run w_direct with the following command w_direct all -a ANALYSIS/TEST/assign.h5 -W west.h5 -o direct.h5 --step-iter 1, I get the following std output:

Operation:      Calculating Target Flux Evolution

fluxes into macrostates:                                                                                                                                                            ]

b'bound': mean=0.000000000000000e+00 CI=(0.000000000000000e+00, 0.000000000000000e+00) * tau^-1

b'unbound': mean=1.091473752739355e-09 CI=(0.000000000000000e+00, 3.274579618517711e-09) * tau^-1

fluxes from state to state:

b'bound' -> b'unbound': mean=1.091473752739355e-09 CI=(0.000000000000000e+00, 3.274262897918420e-09) * tau^-1

b'unbound' -> b'bound': mean=0.000000000000000e+00 CI=(0.000000000000000e+00, 0.000000000000000e+00) * tau^-1

rates from state to state:

b'bound' -> b'unbound': mean=1.091473753275574e-09 CI=(0.000000000000000e+00, 3.274262902743184e-09) * tau^-1

b'unbound' -> b'bound': mean=0.000000000000000e+00 CI=(0.000000000000000e+00, 0.000000000000000e+00) * tau^-1

Operation:      Calculating State Probability Evolution

average color probabilities:                                                                                                                                                        ]

b'bound': mean=9.999999995087192e-01 CI=(9.999999985263589e-01, 9.999999999999687e-01) * tau^-1

b'unbound': mean=4.912802268916663e-10 CI=(2.721817650152191e-14, 1.473694939711732e-09) * tau^-1

average state probabilities:

b'bound': mean=9.769687495876010e-01 CI=(9.619207654399207e-01, 9.920167337352814e-01) * tau^-1

b'unbound': mean=5.458884321251973e-11 CI=(2.969255618347843e-15, 1.637477862114684e-10) * tau^-1

and my direct.h5 file is successfully created. I checked whether I have all of the correct datasets in direct.h5 and I do. However, additional datasets were also created: avg_color_probs, avg_conditional_fluxes, avg_rates, avg_state_probs, avg_total_fluxes, color_prob_evolution, conditional_flux_evolution, rate_evolution, state_labels, state_pop_evolution, and target_flux_evolution.

When you run `w_direct all`, you run w_direct kinetics, w_direct average and w_direct probs, so the extra data sets are those associated with the other modes. You can check what each one means in the w_direct page on our wiki.

Is this the same output as I would expect by just running w_ipa -ao alone, instead of w_assign and w_direct separately?? Can i just get the flux data from my direct.h5? Hope this makes sense and sorry for the long email!

Yes. It should more-or-less be same output, assuming your input settings are the same. w_ipa is a little less customizable, so there might be some differences (some datasets missing, etc). Feel free to get the data from the direct.h5. It's saved there for that purpose.

To view this discussion on the web visit https://groups.google.com/d/msgid/westpa-users/000efc3b-4a72-4dad-bd51-a88ce0c8dd83n%40googlegroups.com.
<direct.h5><assign.h5>