post-compression of gsd files

[Ali Malek]

Hi everyone,

I need to reduce the size of my gsd files, which are generated by HOOMD. The simplest solution I tried was as follows:

import gsd.hoomd

s = gsd.hoomd.open(name="new_small_file.gsd", mode="wb")
t = gsd.hoomd.open(name="old_big_file.gsd", mode="rb")

s.extend(t[::10])

which I hoped that the size of the new file is 1/10 of the old one. However, the size of the new one was not reduced that much (about 1/2).

Do you know any other way to do this?

Thanks,

Ali

[Joshua A. Anderson]

Ali,

Assuming your gsd file has many frames, then your code snippet should
reduce its size significantly. There is some overhead in the first
frame, so if the resulting gsd file only has a few frames it will be
much larger than 1/10 of the original.

Feel free to open an issue and attach the input gsd file so we can
reproduce the behavior and identify if it results from a bug:
https://github.com/glotzerlab/gsd/issues

----
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan
http://www-personal.umich.edu/~joaander/

On 2/26/19 6:38 AM, Ali Malek wrote:
> Hi everyone,
>
> I need to reduce the size of my gsd files, which are generated by HOOMD.
> The simplest solution I tried was as follows:
>
> |
> importgsd.hoomd
>
> s =gsd.hoomd.open(name="new_small_file.gsd",mode="wb")

> t =gsd.hoomd.open(name="old_big_file.gsd",mode="rb")

>
> s.extend(t[::10])
> |
>
> which I hoped that the size of the new file is 1/10 of the old one.
> However, the size of the new one was not reduced that much (about 1/2).
>
> Do you know any other way to do this?
>
> Thanks,
>
> Ali
>

> --
> You received this message because you are subscribed to the Google
> Groups "hoomd-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to hoomd-users...@googlegroups.com
> <mailto:hoomd-users...@googlegroups.com>.
> To post to this group, send email to hoomd...@googlegroups.com
> <mailto:hoomd...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/hoomd-users.
> For more options, visit https://groups.google.com/d/optout.

[Ali Malek]

Dear Josh,

I believe that what you said does not explain the problem, because even copying every slide using my code results in a gsd file much larger than the original one. If you still think that attaching my gsd file (about 5 GB) may help solving the issue let me know. 

> <mailto:hoomd-users+unsub...@googlegroups.com>.

[Joshua A. Anderson]

Ali,

I attempted to reproduce this issue locally and was unable to. I created
a trajectory with HOOMD, then ran your script. The trajectory file size
went from 2732k to 236k.

It may very well be a bug in gsd, but I need to be able to replicate the
problem to identify it. You do not need to share a 5GB file. Can you
instead produce a new file but stop the script that produces it after
100 frames? Or otherwise create a similar but smaller file for me to
test with?

---
Joshua A. Anderson, Ph.D.
Research Area Specialist, Chemical Engineering, University of Michigan
http://www-personal.umich.edu/~joaander/

> > an email to hoomd-users...@googlegroups.com <javascript:>
> > <mailto:hoomd-users...@googlegroups.com <javascript:>>.

> > To post to this group, send email to hoomd...@googlegroups.com

> <javascript:>
> > <mailto:hoomd...@googlegroups.com <javascript:>>.

> > Visit this group at https://groups.google.com/group/hoomd-users

> <https://groups.google.com/group/hoomd-users>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "hoomd-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to hoomd-users...@googlegroups.com

> <mailto:hoomd-users...@googlegroups.com>.