Configure location on the filesystem

[Jonathan Barnoud]

Hi,

I am looking at how I could use detreant and MDSynthesis in my workflow. I would like to have all detreant files for a given project in the same directory, regardless of where the script is run. This would allow me to build a Treant when I setup a simulation, to enrich the same Treant when I run analyses, and to use the Treant when I do other things, always by being in the mist relevant directory rather than in the directory where the Treants are stored.

Maybe I just missed it, but I did not find how to configure where to store, and where to read the Treants. From my tests, Treants are in the working directory.

Is there a way to tell a script (or a jupyter notebook) where to reand and write Treants?

Thank you,
Jonathan

[Max Linke]

To look for existing Treants you can use
`dtr.discover('~/simulations/')`. Storing any information in one of
these Treants will use the original folder the Treant was created.

Creating new Treants is trickier. You have to specify the absolute path
then. If you have an mda.Universe you can get the folder name where the
trajectory is stored and use that. Should be easy to automate with a
wrapper function.

But I guess David has more experience how to organize Treants.

best Max

>
> Thank you,
> Jonathan
>
> --
> You received this message because you are subscribed to the Google
> Groups "datreant" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to datreant+u...@googlegroups.com
> <mailto:datreant+u...@googlegroups.com>.
> To post to this group, send email to datr...@googlegroups.com
> <mailto:datr...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datreant/87d87b2c-e26d-4737-af47-2d400b0775bc%40googlegroups.com
> <https://groups.google.com/d/msgid/datreant/87d87b2c-e26d-4737-af47-2d400b0775bc%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Jonathan Barnoud]

 
Hi,

Thanks for the answer. I guessed creating the Treants would be the trickiest part. I really prefer keeping my simulation directories read-only, having MDAnalysis storing the offsets besides the trajectory already annoys me (even though I understand why it is most likely the best solution).

So far, I have a directory for the simulations ans a directory for the analyses. The two directories following roughly the same structure. Ideally I would like a third directory for all detreant related things.

Best,
Jonathan

[David Dotson]

Hi Jonathan,

In my experience, MDSynthesis Sims work best when they *are* the directory where all the data (setup files, trajectories, derived datasets) lives for a given simulation. From what I've read you have a bit of a different convention, which will still work (nothing will be broken).

Can you copy and paste some of the output of the shell command `tree` here for your data directory giving an idea as to how this is all laid out? That would help me to get a sense for what you're looking for.

David

To unsubscribe from this group and stop receiving emails from it, send an email to datreant+u...@googlegroups.com.
To post to this group, send email to datr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/datreant/2deae0de-8a0b-417b-8573-24fe524fef54%40googlegroups.com.

[Jonathan Barnoud]

Hi David,

I see how it is interesting to have the Sims being the directory with the trajectory. But I do not really like mixing simulations and analyses for various reasons. For instance, I often wipe out my analyses when I change my analysis scripts, in order to be sure all my simulations are analysed the same way; when I do this, I do not want to take the risk of deleting precious trajectory files.

Here is an excerpt of tree run on one of my project directories:

.

├── analysis

│   ├── BOL_102_295K_r1

│   │   ├── begin.txt

│   │   ├── box.xvg

│   │   ├── cholesterol_contact_fraction.xvg

│   │   ├── config.json

│   │   ├── figures

│   │   │   ├── box.pdf

│   │   │   └── [...]

│   │   └── [...]

│   ├── BOL_1198_295K_r1

│   │   ├── begin.txt

│   │   ├── box.xvg

│   │   ├── cholesterol_contact_fraction.xvg

│   │   ├── config.json

│   │   ├── figures

│   │   │   ├── box.pdf

│   │   │   └── [...]

│   │   └── [...]

│   ├── conc_ramp

│   │   ├── DCO_100_295K_r1

│   │   │   ├── begin.txt

│   │   │   ├── box.xvg

|   │   │   ├── cholesterol_contact_fraction.xvg

|   │   │   ├── config.json

|   │   │   ├── figures

|   │   │   │   ├── box.pdf

|   │   │   │   └── [...]

|   │   │   └── [...]

│   │   ├── DCO_300_295K_r1

│   │   │   ├── begin.txt

│   │   │   ├── box.xvg

|   │   │   ├── cholesterol_contact_fraction.xvg

|   │   │   ├── config.json

|   │   │   ├── figures

|   │   │   │   ├── box.pdf

|   │   │   │   └── [...]

|   │   │   └── [...]

|   |   └── [...]

│   ├── contact_extra.agr

│   ├── density_combine.agr

|   └── [...]

├── simulations

│   ├── BOL_102_295K_r1

│   │   ├── md.gro

│   │   ├── md.tpr

│   │   ├── md.xtc

│   │   ├── param_md.mdp

│   │   ├── topol.top

│   │   └── [...]

│   ├── BOL_1198_295K_r1

│   │   ├── md.gro

│   │   ├── md.tpr

│   │   ├── md.xtc

│   │   ├── param_md.mdp

│   │   ├── topol.top

│   │   └── [...]

│   ├── conc_ramp

│   │   ├── DCO_100_295K_r1

|   │   │   ├── md.gro

|   │   │   ├── md.tpr

|   │   │   ├── md.xtc

|   │   │   ├── param_md.mdp

|   │   │   ├── topol.top

|   │   │   └── [...]

│   │   ├── DCO_300_295K_r1

|   │   │   ├── md.gro

|   │   │   ├── md.tpr

|   │   │   ├── md.xtc

|   │   │   ├── param_md.mdp

|   │   │   ├── topol.top

|   │   │   └── [...]

|   |   └── [...]

|   └── [...]

├── ipython

│   ├── Create tables.ipynb

│   ├── Density profiles.ipynb

│   ├── summary.csv

│   ├── Untitled1.ipynb

│   └── [...]

├── scripts

│   ├── build.sh

│   ├── concat.sh

│   ├── populate_system_config.py

│   ├── Snakefile

│   └── [...]

└── [...]

The "simulations" and "analysis" directories follow the same architecture. "simulation" contains every files needed to reproduce the trajectories, "analysis" contains all the analysis output for the simulations. The analysis are run using Snakemake and scripts in the "scripts" directory. They are combined in jupyter notebooks in the "notebook" directory. 

Ideally, I would like to replace the "analysis" directory by a detreant tree. This would allow me to store the analyses as data fields or leafs in Sims, and to filter all of this in my notebooks. I expect that it would be a most efficient way to deal with metadata than the "config.json" file that I currently use.

Hopefully, the way I want to use MDSynthesis makes sense.

Best,

Jonathan