archive/zip reader memory usage

760 views
Skip to first unread message

MA LIG

unread,
May 22, 2016, 1:56:52 PM5/22/16
to golang-nuts
Hi, does anyone know whether the "archive/zip" reader does (or can) operate without reading the entire file into memory? In particular, I would like to process a large zip file by just streaming the requested decompressed content to the calling process (or function) and using a limited, fixed-size internal buffer that never requires any heap usage (per concurrent execution) beyond a predetermined, fixed size.

Ian Lance Taylor

unread,
May 22, 2016, 2:15:06 PM5/22/16
to MA LIG, golang-nuts
The archive/zip Reader does not read the entire file into memory.
However, it requires an io.ReaderAt: it needs the ability to read data
from specific points in the input. So it can't read from an
unseekable stream. But it can produce a stream from a seekable file
using only a fixed amount of memory. I think the amount of memory
required will depend on the compression method.

Ian

krolaw

unread,
May 24, 2016, 5:57:02 PM5/24/16
to golang-nuts
I'd be curious to know if https://github.com/krolaw/zipstream suits your needs.

Thanks.
Reply all
Reply to author
Forward
0 new messages