New WIP feature: jug pack

[Luis Pedro Coelho]

Hi everyone,

This is still a WIP and only available on the git version of jug (https://github.com/luispedro/jug), but it solves a problem that has plagued jug since the beginning: handling many small files. The redis backend was implemented initially just because of this problem, but it has its own issues. Now, this is a new way of doing it.

tl;dr: Run "jug pack" every once in a while and many small files will get aggregated into a single one.

Right now, the implementation is still somewhat simplistic, but it should work without any glitches. All functionality should work exactly the same.

Running jug pack while other processes are running _should_ work, but I would recommend against it.

I will probably use this internally for a while to battle-test it before making a release, but I appreciate bug reports. The internal representation may also change before the release, so maybe there may be breaking changes!

Best
Luis

Luis Pedro Coelho | Fudan University | http://luispedro.org
https://orcid.org/0000-0002-9280-7885