Re: Issue 2 in mdanalysis: reading Gromacs tpr files instead of psf

126 views
Skip to first unread message

mdana...@googlecode.com

unread,
Aug 2, 2012, 11:33:11 PM8/2/12
to mdnalys...@googlegroups.com

Comment #18 on issue 2 by jan...@gmail.com: reading Gromacs tpr files
instead of psf
http://code.google.com/p/mdanalysis/issues/detail?id=2

Any updates here? I'm considering giving this a look and would happily
refrain if the solution is behind the corner (or harness any leftovers;) )

Andy Somogyi

unread,
Aug 3, 2012, 3:02:11 AM8/3/12
to mdnalys...@googlegroups.com
I have a topo editor which reads, manipulates and writes top or psf
nearly ready, in my public git.

Would be really easy to add tpr reading also.

Still have a few issues to work on.

Actually the biggest issue I have is generalizing the build, i.e.
finding the gromacs lib and so forth. Also, the build for this module
will be cmake, as it seems to have the best support for finding
gromacs. Will using cmake be a problem for a mdanalysis extension
module??

The psf read , write functionality is built in, but top is a lot more
complicated, so I use the gromacs libs (it's their file format
anyway).

So, currently psf read , modify, write is perfect, still have bit left
to do with gromacs dihedrals, but should be publicly testable in 2 ~3
week (currently in the middle of a related project that I want to get
done next week).

Once this module is done, I need to write a bit of glue that stuffs
the universe using this module, just like it uses the existing psf
reader.

Sent from my iPad

On Aug 2, 2012, at 11:33 PM, "mdana...@googlecode.com"
> --
> You received this message because you are subscribed to the Google Groups "MDnalysis-devel" group.
> To post to this group, send email to mdnalys...@googlegroups.com.
> To unsubscribe from this group, send email to mdnalysis-dev...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Oliver Beckstein

unread,
Aug 3, 2012, 3:18:17 PM8/3/12
to mdnalys...@googlegroups.com
On 3 Aug, 2012, at 00:02, Andy Somogyi wrote:

> Actually the biggest issue I have is generalizing the build, i.e.
> finding the gromacs lib and so forth.

What is "and so forth" – is more needed than the appropriate -L/usr/local/gromacs/lib -lgmx -lmd -lpthreads ... strings?

> Also, the build for this module
> will be cmake, as it seems to have the best support for finding
> gromacs. Will using cmake be a problem for a mdanalysis extension
> module??

I don't know if cmake can be easily integrated in the build process. From a user's perspective, it is hard to beat

python setup.py install

or

pip install MDAnalysis

i.e. setuptools handling all the compiling automagically.

Has anyone experience with integrating other build tools in this process or would we have to switch to a non-setuptools based process?

It is also possible to use the setup.cfg file to pass paths, e.g. of the gromacs installation.

Finally, it shouldn't be too hard (famous last words...) to add logic to setup.py to find Gromacs (e.g. standard places and using environment variables) and then add the library paths to what's needed for building the extension.

I know that reinventing the wheel (coding library finding logic) is annoying but if at all possible, I'd favour the last approach as it would keep the installation simple for the end user.

Oliver


--
Oliver Beckstein * orbe...@gmx.net
skype: orbeckst * orbe...@gmail.com

Andy Somogyi

unread,
Aug 3, 2012, 3:32:25 PM8/3/12
to mdnalys...@googlegroups.com

On Aug 3, 2012, at 3:18 PM, Oliver Beckstein wrote:

> On 3 Aug, 2012, at 00:02, Andy Somogyi wrote:
>
>> Actually the biggest issue I have is generalizing the build, i.e.
>> finding the gromacs lib and so forth.
>
> What is "and so forth" – is more needed than the appropriate -L/usr/local/gromacs/lib -lgmx -lmd -lpthreads ... strings?
Gromacs may be installed anywhere, and I also need the gromacs include directory.


>
> I don't know if cmake can be easily integrated in the build process. From a user's perspective, it is hard to beat
>
> python setup.py install
Still does that, I have setup.py call cmake, although is very hackish at the moment, needs lots of cleanup. If a user built gromacs, then that also have cmake. If they don't have a gromacs include directory, than gromacs support needs to be disabled, this is accomplished via a HAVE_GROMACS preprocessor directive which is set by cmake.

>
> or
>
> pip install MDAnalysis
>
> i.e. setuptools handling all the compiling automagically.
>
> Has anyone experience with integrating other build tools in this process or would we have to switch to a non-setuptools based process?
>
> It is also possible to use the setup.cfg file to pass paths, e.g. of the gromacs installation.
>
> Finally, it shouldn't be too hard (famous last words...) to add logic to setup.py to find Gromacs (e.g. standard places and using environment variables) and then add the library paths to what's needed for building the extension.
>
> I know that reinventing the wheel (coding library finding logic) is annoying but if at all possible, I'd favour the last approach as it would keep the installation simple for the end user.
Might be an option, I'll take a look at how hard this would be to get working.

mdana...@googlecode.com

unread,
Aug 6, 2012, 10:53:12 PM8/6/12
to mdnalys...@googlegroups.com

Comment #19 on issue 2 by jan...@gmail.com: reading Gromacs tpr files
Here is a wiki article on the TPR Reader development, together with a few
lines of code to extract the basic header information from a file

http://code.google.com/p/mdanalysis/wiki/TPRReaderDevelopment

Any hands on board with this work are very welcome! The wiki article above
is a good starting point and shows that the task at hand is v easy. I'll
bake an e-cake for any brave participants ;)

More broadly, this reader should probably be advertised and expedited
outside of MDAnalysis since it might just be helpful to the MD community.


mdana...@googlecode.com

unread,
Nov 14, 2012, 3:00:45 PM11/14/12
to mdnalys...@googlegroups.com

Comment #20 on issue 2 by alfred53...@gmail.com: reading Gromacs tpr files
Hi, Oliver,

I picked up this issue again. Continued to from the header parser from Jan
(http://code.google.com/p/mdanalysis/wiki/TPRReaderDevelopment), I have
completed parsing the tpr for my particular version, which is 58. Next, I
will try to integrate it into the MDAnalysis package. Do you have any guide
to speed up my analyzing on the organization of MDAnalysis package. Once I
finish that. I may look back into the source code again and add support for
other versions of tpr files. You are very right, the format look changed
very frequently and the source code (mainly gromacs/gmxlib/tpxio.c)
contains a lot of (if..else..) statement for handling different versions.

mdana...@googlecode.com

unread,
Nov 14, 2012, 3:26:45 PM11/14/12
to mdnalys...@googlegroups.com

Comment #21 on issue 2 by orbeckst: reading Gromacs tpr files instead of psf
http://code.google.com/p/mdanalysis/issues/detail?id=2

Hi alfred532008,

The first step is to clone the MDAnalysis repository and create a feature
branch off the devel branch (see
https://code.google.com/p/mdanalysis/wiki/DevelopmentWorkflow ). Then work
on your new branch (e.g. named "TPRReader" or "Issue/2"). This will make
merging with the main code a lot easier.

I should also say that andy.somogyi has been working on a major rework of
the topology reader, including using the Gromacs libraries directly.
However, his code isn't quite ready yet so I'd be interested to see a
Python-only solution like yours, too.

In order to integrate a new topology reader you will need to add code to

1. MDAnalysis/topology, e.g. MDAnalysis/topology/TPRParser.py

- look at PSFParser.py, GROParsers.py for an example: you essentially
have one function parse(filename) that does all the work

- your code needs to read the topology file and build a universe from
atoms, which is represented as a dict structure[] with keys _atoms, _bonds,
_angles, _dihe, _impr. Only _atoms are absolutely required.

2. MDAnalysis/topology/__init__.py

Register your parser by adding it to __all__, the import statement, and
add the parse() function to the topology_parsers[] dict for key 'TPR'.

3. Write a test case and add a test tpr file, see
https://code.google.com/p/mdanalysis/wiki/UnitTests


If you have questions ask on the developer mailing list.

Oliver




Andy Somogyi

unread,
Nov 14, 2012, 11:51:58 PM11/14/12
to mdnalys...@googlegroups.com
Hi,

I have been working on the topo editor, but I've been pulled away on
another project that's consumed all my time for the past months.
However, looks like I will be able to get back to it in the next few
days.

Basically it's a set of py classes (extension library) written in c
that calls the gromacs lib functions to read and build a topology. It
can read tpr, top and pdb.

It uses the documented gromacs lib functions to do this.

Please be aware that tpr is an undocumented file format, its only
meant to be accessed through the gromacs public APIs. It is also a
very complicated format, there are some 30 different bonding types
alone.

The build process is not that bad, basically I've hacked setup.py to
look for the gromacs env vars and it figures out the lib and include
paths from these and passes them to the extension builder.

On Nov 14, 2012, at 3:26 PM, "mdana...@googlecode.com"

Zhuyi Xue

unread,
Nov 18, 2012, 10:54:30 PM11/18/12
to mdnalys...@googlegroups.com, codesite...@google.com, mdana...@googlecode.com
Hi, Oliver,

I have finished the steps you specified. You could browse the code at https://code.google.com/r/alfred532008-mdanalysis-tprreader/source/list?name=tprparser.

The tpr file in the testsuite/MDAnalysisTests/data happens to be exactly of version 58, which mean my code works for it properly.

I am interested in having a look at Andy's approach, as well.

Zhuyi

Oliver Beckstein

unread,
Nov 19, 2012, 1:25:18 PM11/19/12
to mdnalys...@googlegroups.com
Hi Zhuyi,

Thanks for yoru code. I pushed your additions as a new feature branch 'pyTPRparser' to the gcode repository.

Anyone who wants to play with the code should be able to do so with commands such as

git fetch origin
git checkout pyTPRparser

The integration with MDAnalysis and the test case looks ok. The main issue is that it only works for TPR file version 58. This seems a pretty severe limitation (which is of course due to the fact that the TPR API is undocumented).

Do you (or anyone else) intend to update the TPR reader to keep track of the Gromacs file format versions?

Oliver

On 18 Nov, 2012, at 20:54, Zhuyi Xue wrote:

> Hi, Oliver,
>
> I have finished the steps you specified. You could browse the code at https://code.google.com/r/alfred532008-mdanalysis-tprreader/source/list?name=tprparser.
>
> The tpr file in the testsuite/MDAnalysisTests/data happens to be exactly of version 58, which mean my code works for it properly.
>
> I am interested in having a look at Andy's approach, as well.
>
> Zhuyi

Zhuyi Xue

unread,
Nov 19, 2012, 2:54:00 PM11/19/12
to mdnalys...@googlegroups.com
Hi, Oliver,

Next, I may write one for version 73. I think it makes more sense to add version support upon request. The source code is kind of scary if you want to track down all versions. That's why I am interested in Andy's solution.

Thanks,
Zhuyi

mdana...@googlecode.com

unread,
Nov 24, 2012, 5:25:46 PM11/24/12
to mdnalys...@googlegroups.com

Comment #22 on issue 2 by orbeckst: reading Gromacs tpr files instead of psf
http://code.google.com/p/mdanalysis/issues/detail?id=2

alfred532008's TPR reader
https://code.google.com/r/alfred532008-mdanalysis-tprreader/source/list?name=tprparser
is available as the feature branch 'pyTPRparser'
https://code.google.com/p/mdanalysis/source/browse/package?name=pyTPRparser
in the repository

Anyone who wants to play with the code should be able to do so with
commands such as

git fetch origin
git checkout pyTPRparser

Current limitations:
* Only reads TPR file version 58 (Gromacs around release 4.5.3)

For more information see the developer thread
https://groups.google.com/d/topic/mdnalysis-devel/nMwUjAZR-iQ/discussion



mdana...@googlecode.com

unread,
Dec 1, 2012, 9:24:07 PM12/1/12
to mdnalys...@googlegroups.com

Comment #23 on issue 2 by jan...@gmail.com: reading Gromacs tpr files
Hi alfred532008, I'm still looking at this thread (monitoring only) and
your work here is absolutely excellent!

The version issue is a serious problem. It may be possible to use CTYPES
bindings here, to bind directly to gromacs/gmxlib/tpxio.c and use that c
code to read the tprs and then access the c data structure from python. For
an example, see 'grompy' by (Martin Hoefling, Roland Schulz)
http://orbeckst.github.com/GromacsWrapper/alternatives.html

If this C binding works the way I think it does (see grompy/tpxio.py), it
will make a lot of the code you wrote necessary, which is very sad, but
we'll navigate away for version issues relying on gromacs itself.

It is clear at this stage that absence of a python interface to gromacs (or
any MD code out there) is a major hurdle to some users.

mdana...@googlecode.com

unread,
Dec 1, 2012, 9:30:24 PM12/1/12
to mdnalys...@googlegroups.com

Comment #24 on issue 2 by jan...@gmail.com: reading Gromacs tpr files
Now seriously I'm reading the source, alfred532008, and this code is great.
I take my point back: we should keep your pure-python version no matter
what, because it gives us independence from the gromacs libraries (usage
case: a user without gromacs who wants to access a tpr file sent by a
collaborator).

mdana...@googlecode.com

unread,
Dec 2, 2012, 1:22:55 AM12/2/12
to mdnalys...@googlegroups.com

Comment #25 on issue 2 by alfred53...@gmail.com: reading Gromacs tpr files
Hi, Jan,

I have some updates, I am not familiar with the CTYPES you mentioned, but
feel it worth looking into it when I have time.

The good news is that I just find out that gromacs-4.0.x all have the same
tpx version, which is 58, and similarly gromacs-4.5.x all have tpx version
73. This is quite unexpected, I don't know why there is such a big jump.

I am almost done with version 73, and probably will push it next week.
Assuming very few people are using gromacs-3.x or even earlier versions
now, I think we are pretty much done with the tpr parsing, unless a lot of
people request for early versions. Of course, there is not bug-free
guarantee in the my code, even if it passes these simple unit test.

Zhuyi

mdana...@googlecode.com

unread,
Dec 2, 2012, 6:15:10 PM12/2/12
to mdnalys...@googlegroups.com

Comment #26 on issue 2 by jan...@gmail.com: reading Gromacs tpr files
Nothing concrete yet but i've summarized some ideas about wrapping gromacs
libraries in python
[https://code.google.com/p/mdanalysis/wiki/WrappingGromacsInPython]

Zhuyi Xue

unread,
Dec 3, 2012, 3:11:46 PM12/3/12
to mdnalys...@googlegroups.com
Hi, Oliver,


Now, it is able to parse tpr file generated by grompp in gromacs-4.0.x and 5.5.x.

Zhuyi

On Monday, November 19, 2012 1:25:25 PM UTC-5, Oliver Beckstein wrote:
Hi Zhuyi,
.

Oliver Beckstein

unread,
Dec 3, 2012, 3:51:54 PM12/3/12
to mdnalys...@googlegroups.com
Hi Zhuyi,

Thanks, I merged and pushed to the pyTPRparser branch.

Oliver


On 3 Dec, 2012, at 13:11, Zhuyi Xue wrote:

> Hi, Oliver,
>
> I have push my code for tpx version 73 to http://code.google.com/r/alfred532008-mdanalysis-tprreader/source/list?name=tprparser
>
> Now, it is able to parse tpr file generated by grompp in gromacs-4.0.x and 5.5.x.
>
> Zhuyi
>
> On Monday, November 19, 2012 1:25:25 PM UTC-5, Oliver Beckstein wrote:
> Hi Zhuyi,
> .
> Thanks for yoru code. I pushed your additions as a new feature branch 'pyTPRparser' to the gcode repository.
>
> Anyone who wants to play with the code should be able to do so with commands such as
>
> git fetch origin
> git checkout pyTPRparser
>
> The integration with MDAnalysis and the test case looks ok. The main issue is that it only works for TPR file version 58. This seems a pretty severe limitation (which is of course due to the fact that the TPR API is undocumented).
>
> Do you (or anyone else) intend to update the TPR reader to keep track of the Gromacs file format versions?
>
> Oliver
>
> On 18 Nov, 2012, at 20:54, Zhuyi Xue wrote:
>
> > Hi, Oliver,
> >
> > I have finished the steps you specified. You could browse the code at https://code.google.com/r/alfred532008-mdanalysis-tprreader/source/list?name=tprparser.
> >
> > The tpr file in the testsuite/MDAnalysisTests/data happens to be exactly of version 58, which mean my code works for it properly.
> >
> > I am interested in having a look at Andy's approach, as well.
> >
> > Zhuyi
>
> --
> Oliver Beckstein * orbe...@gmx.net
> skype: orbeckst * orbe...@gmail.com
>
>
> --
> You received this message because you are subscribed to the Google Groups "MDnalysis-devel" group.
> To post to this group, send email to mdnalys...@googlegroups.com.
> To unsubscribe from this group, send email to mdnalysis-dev...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msg/mdnalysis-devel/-/Kh6XfX0N2VwJ.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Zhuyi Xue

unread,
Jan 11, 2013, 11:57:07 AM1/11/13
to mdnalys...@googlegroups.com
Hi, Oliver,

I just fixed a bug found by one of my colleagues in the tpr_utils.py.

Thanks.
Zhuyi

Oliver Beckstein

unread,
Jan 11, 2013, 12:07:25 PM1/11/13
to mdnalys...@googlegroups.com
On 11 Jan, 2013, at 09:57, Zhuyi Xue wrote:

> I just fixed a bug found by one of my colleagues in the tpr_utils.py.
> shown in https://code.google.com/r/alfred532008-mdanalysis-tprreader/source/list?name=tprparser

Thanks, I pulled, merged, and pushed. The pyTPRparser branch on the MDAnalysis is fully up-to-date (it contains MDAnalysis 0.7.7 + Zhuyi's Python-based TPR reader).

Btw, feedback on the code would be welcome – does it work for you, are there problems, etc.

Oliver

mdana...@googlecode.com

unread,
Apr 17, 2013, 1:17:31 AM4/17/13
to mdnalys...@googlegroups.com
Updates:
Status: Started
Owner: alfred53...@gmail.com

Comment #27 on issue 2 by orbeckst: reading Gromacs tpr files instead of psf
http://code.google.com/p/mdanalysis/issues/detail?id=2

Hi Zhuyi,

Have you got updated code to read Gromacs 4.6.1 TPR files?

I would like to integrate your TPRParser into the upcoming 0.8 release. It
works pretty well, you have test cases, and there's no other Python code
available at the moment that can easily read TPR files. However, reading
modern TPR files would be rather important.

I did a little bit of work on your TPRParser, including adding docs
http://mdanalysis.googlecode.com/git-history/pyTPRparser/package/doc/html/documentation_pages/topology/TPRParser.html
and fixing a small bug (related to parsing of angles) and reorganized files
(I put your tpr_*.py files into a separate sub-module
MDAnalysis.topology.tpr). If you pull the changes from the 'pyTPRparser'
branch you will get all changes including the history (or see
https://code.google.com/p/mdanalysis/source/list?name=pyTPRparser ).

Oliver



--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

mdana...@googlecode.com

unread,
Apr 17, 2013, 8:42:07 PM4/17/13
to mdnalys...@googlegroups.com

Comment #28 on issue 2 by alfred53...@gmail.com: reading Gromacs tpr files
Hi, Oliver,

I took a look at the changes. Thanks for adding all the docs.

No, I haven't updated it for Gromacs 4.6.x yet. I am still interested in
writing the newer parser for modern tpr files, and would like to do it in
April or May. I am curious about when do you plan to release 0.8?

Zhuyi

Oliver Beckstein

unread,
Apr 17, 2013, 8:50:05 PM4/17/13
to mdnalys...@googlegroups.com
Hi Zhuyi,

On 17 Apr, 2013, at 17:42, mdana...@googlecode.com wrote:

>
> Comment #28 on issue 2 by alfred53...@gmail.com: reading Gromacs tpr files instead of psf
> http://code.google.com/p/mdanalysis/issues/detail?id=2
>
> Hi, Oliver,
>
> I took a look at the changes. Thanks for adding all the docs.
>
> No, I haven't updated it for Gromacs 4.6.x yet. I am still interested in writing the newer parser for modern tpr files, and would like to do it in April or May.

That would be great.

> I am curious about when do you plan to release 0.8?

There's no fixed date yet although in the past we have done a release roughly every 6 months.

Zhuyi Xue

unread,
May 26, 2013, 1:38:12 PM5/26/13
to mdnalys...@googlegroups.com
Hi, Oliver,

I have updated the tpr parser to support gromacs-4.6 and gromacs-4.6.1 which produce tpr files of version 83.

Zhuyi

mdana...@googlecode.com

unread,
May 29, 2013, 1:14:43 PM5/29/13
to mdnalys...@googlegroups.com
Updates:
Status: Fixed

Comment #29 on issue 2 by orbeckst: reading Gromacs tpr files instead of psf
http://code.google.com/p/mdanalysis/issues/detail?id=2

This issue was closed by revision 7131d6e8fe1b.

mdana...@googlecode.com

unread,
May 29, 2013, 1:28:06 PM5/29/13
to mdnalys...@googlegroups.com

Comment #30 on issue 2 by orbeckst: reading Gromacs tpr files instead of psf
http://code.google.com/p/mdanalysis/issues/detail?id=2

Hi Zhuyi,

excellent work on the TPRParser. I am currently merging it into the develop
branch for inclusion in 0.8. There are some issues to iron out with how
bonds are treated (introduced by Jan in relation to Issue 23) when the
Universe is built but it hopefully won't take too long.

Oliver

Zhuyi Xue

unread,
May 29, 2013, 2:21:31 PM5/29/13
to mdnalys...@googlegroups.com, codesite...@google.com, mdana...@googlecode.com
Hi, Oliver,

Sorry, just found a typo of function name in TPRParser.py. Have fixed it. Please merge again, or rebase.

Thanks,
Zhuyi

mdana...@googlecode.com

unread,
May 29, 2013, 8:06:14 PM5/29/13
to mdnalys...@googlegroups.com
Updates:
Status: Verified

Comment #31 on issue 2 by orbeckst: reading Gromacs tpr files instead of psf
http://code.google.com/p/mdanalysis/issues/detail?id=2

Hi Zhuyi and everyone else,

I merged the TPRparser into develop. I only had to make sure that bonds,
angles, etc are stored as a list of tuples. UnitTests pass.

I also updated the docs for topology parsers and outlined the definition of
the "structure" dict --- the data structure that we are currently using to
capture the topology. It is stored as Universe._psf. Note that this data
structure is subject to change and users should NOT rely on it.

Congratulations to Zhuyi on closing an Issue that was more than 5 years old
— this is the record for MDAnalysis.

Zhuyi Xue

unread,
May 31, 2013, 6:16:06 PM5/31/13
to mdnalys...@googlegroups.com, codesite...@google.com, mdana...@googlecode.com
Hi, Oliver,

Thank you for the update and merging. Looking forward to next release!

Zhuyi
Reply all
Reply to author
Forward
0 new messages