[ANN] rat, an extension to the tar archive, allowing random file access

[Máximo Cuadros]

Hi folks,

I want to share this small library:
rat is an extension to the classical tar archive, focused on allowing
constant-time random file access with linear memory consumption
increase. tape archive, was originally developed to write and read
streamed sources, making random access to the content very
inefficient.

Based on the benchmarks, we found that rat is 4x to 60x times faster
over SSD and HDD than the classic tar file, when reading a single file
from a tar archive.

https://github.com/mcuadros/go-rat

Any feedback is more than welcome!

Best regards

[Tamás Gulácsi]

Hi,

As I see you're replicated the functional ity of zip, without its standardness. What are the use cases?

[Máximo Cuadros]

To be honest I didn't know that zip has a feature for random access to
files on a zip, without read the whole archive.
Reading about it now, I see that my implementation is very similar to
the central directory. Thanks!

BTW We have like 10TB of tars so we cannot simply move to zip.

Best regards

[Gulácsi Tamás]

http://www.artpol-software.com/ZipArchive/KB/0610051629.aspx
Go supports Zip64 nicely, so 10TB is not an excuse :)

[Máximo Cuadros]

10TB of millions of tars, so I want move everything to zip. :D

[Brandon Philips]

We have been thinking about this problem in the application container spec[1] and the other issue is random access to the gzip compression of a tar.gz.

To that end we have two tools:

github.com/coreos/gzran
github.com/vbatts/tar-split

The idea is to have a secondary index alongside the tar.gz.

Brandon

[1] github.com/appc/spec

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[andrewc...@gmail.com]

How is this implemented? is the first entry an offset to a special index file or something?

[Brandon Philips]

The indexes would be external in our initial use case:
https://github.com/coreos/rkt/issues/544

[Gomgoru Koee]

Brandon, have you looked at using the ECMA 208 standard ( its an ISO standard as well)? It does most of what your trying to get done right now. And it's easily extended,  too.

There are a lot of defined standards that are being "Reinvented" because no one is checking for them...sigh.

[Brandon Philips]

There are tons of little known archive formats like ecma 208 or dar.

But we decided we  would rather tell the user just use tar, which has bindings and implementations everywhere, and we will do the fancy stuff out of band in tooling.

Thanks,

Brandon