Help with out of memory error when using gunzip

[Namu]

Hello there,

I'm trying to uncompress a gz file without touching the original file, and couldn't find a Go way to do it. So decided to use gunzip to do it : gunzip -c myfile.gz > myfile

I tried the following:
1.
cmd := exec.Command(GUNZIP, "-c", basePathGz, ">", copyFilePath)
output, err := cmd.CombinedOutput()

2.
err = exec.Command(GUNZIP, "-c", basePathGz, ">", copyFilePath).Run()

3.
gunZipCmd := fmt.Sprintf("%s %s %s %s %s", GUNZIP, "-c", basePathGz, ">", copyFilePath)
err = exec.Command("/bin/sh", "-c", gunZipCmd).Run()

The third option works. But the first two options give an "out of memory" error message -

"runtime: out of memory: cannot allocate 68719476736-byte block (68720525312 in use)
fatal error: out of memory"

Am I doing it wrong? Is there a better way to do this?

Thank you!

[Dave Cheney]

On Friday, 21 February 2014 11:04:02 UTC+11, Namu wrote:

Hello there,

I'm trying to uncompress a gz file without touching the original file, and couldn't find a Go way to do it. So decided to use gunzip to do it : gunzip -c myfile.gz > myfile

I tried the following:
1.
cmd := exec.Command(GUNZIP, "-c", basePathGz, ">", copyFilePath)
output, err := cmd.CombinedOutput()

The output redirection character ">" is not a command or an argument in this case, it's a filename ">". So, cmd.CombinedOutput() will buffer the entire contents of the decompressed file in memory

 

 

2.
err = exec.Command(GUNZIP, "-c", basePathGz, ">", copyFilePath).Run()

 

Not sure what is going on here. Run() shouldn't buffer the output, in fact it should throw it away. Can you please provide a runnable sample that shows the problem and a complete panic message from when you tru to rnu it.

3.
gunZipCmd := fmt.Sprintf("%s %s %s %s %s", GUNZIP, "-c", basePathGz, ">", copyFilePath)
err = exec.Command("/bin/sh", "-c", gunZipCmd).Run()

The third option works. But the first two options give an "out of memory" error message -

 

"runtime: out of memory: cannot allocate 68719476736-byte block (68720525312 in use)
fatal error: out of memory"

Am I doing it wrong? Is there a better way to do this?

cmd := exec.Command(GUNZIP, "-c", basePathGz)

out, err := os.Create(copyFilePath)

cmd.Stdout = out

cmd.Stderr = os.Stderr

cmd.Run()

 

Thank you!

[Seth W. Klein]

On Feb 20, 2014, at 7:04 PM, Namu wrote:

> I'm trying to uncompress a gz file without touching the original file, and couldn't find a Go way to do it. [....]

I assume by touching you mean modifying.

http://play.golang.org/p/4Sev7Lyfk8

package main

import (
"bytes"
"compress/gzip"
"io"
"os"
)

func main() {
/*
f, err := os.Open("foo.txt.gz")
if err != nil {
panic(err)
}
defer f.Close() // ignoring an error here
*/
f := bytes.NewBuffer([]byte{0x1f, 0x8b, 0x8, 0x8, 0x43, 0xb1, 0x6, 0x53, 0x0, 0x3, 0x66, 0x6f, 0x6f, 0x2e, 0x74, 0x78, 0x74, 0x0, 0x4b, 0xcb, 0xcf, 0xe7, 0x2, 0x0, 0xa8, 0x65, 0x32, 0x7e, 0x4, 0x0, 0x0, 0x0})

r, err := gzip.NewReader(f)
if err != nil {
panic(err)
}
defer r.Close() // ignoring an error here
var buf bytes.Buffer
_, err = io.Copy(&buf, r)
if err != nil {
panic(err)
}
os.Stdout.Write(buf.Bytes())
}

Cheers,
Seth W. Klein <s...@sethwklein.net>

[nko...@gmail.com]

The following is the complete panic message:

Please be patient while the base KVM image file is being extracted and copied.

runtime: out of memory: cannot allocate 68719476736-byte block (68720525312 in use)
fatal error: out of memory

goroutine 4 [running]:
runtime.throw(0x6968b7)
    /usr/local/go/src/pkg/runtime/panic.c:464 +0x69 fp=0x7f272ddd1cc8
runtime.mallocgc(0xffffffe00, 0x4e9b81, 0x1)
    /usr/local/go/src/pkg/runtime/malloc.goc:91 +0x4a0 fp=0x7f272ddd1d38
cnew(0x4e9b80, 0xffffffe00, 0xd200000001)
    /usr/local/go/src/pkg/runtime/malloc.goc:718 +0xc1 fp=0x7f272ddd1d58
runtime.cnewarray(0x4e9b80, 0xffffffe00)
    /usr/local/go/src/pkg/runtime/malloc.goc:731 +0x3a fp=0x7f272ddd1d78
makeslice1(0x4e2d60, 0xffffffe00, 0xffffffe00, 0x7f272ddd1dd8)
    /usr/local/go/src/pkg/runtime/slice.c:57 +0x4d fp=0x7f272ddd1d90
runtime.makeslice(0x4e2d60, 0xffffffe00, 0xffffffe00, 0xca10077000, 0xffffffe00, ...)
    /usr/local/go/src/pkg/runtime/slice.c:38 +0x98 fp=0x7f272ddd1dc0
bytes.makeSlice(0xffffffe00, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/bytes/buffer.go:191 +0x63 fp=0x7f272ddd1df8
bytes.(*Buffer).ReadFrom(0xc210049d90, 0x7f272df59308, 0xc210000280, 0x7fffffe00, 0x0, ...)
    /usr/local/go/src/pkg/bytes/buffer.go:163 +0xcb fp=0x7f272ddd1e90
io.Copy(0x7f272df593f0, 0xc210049d90, 0x7f272df59308, 0xc210000280, 0x0, ...)
    /usr/local/go/src/pkg/io/io.go:348 +0x124 fp=0x7f272ddd1f18
os/exec.func·003(0x4e3a60, 0xc2100494d0)
    /usr/local/go/src/pkg/os/exec/exec.go:208 +0x62 fp=0x7f272ddd1f68
os/exec.func·004(0xc210048ae0)
    /usr/local/go/src/pkg/os/exec/exec.go:277 +0x2c fp=0x7f272ddd1f98
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1394 fp=0x7f272ddd1fa0
created by os/exec.(*Cmd).Start
    /usr/local/go/src/pkg/os/exec/exec.go:278 +0x86b

goroutine 1 [syscall]:
syscall.Syscall6(0x3d, 0xa3c9, 0xc2100002f8, 0x0, 0xc210065360, ...)
    /usr/local/go/src/pkg/syscall/asm_linux_amd64.s:43 +0x5
syscall.wait4(0xa3c9, 0xc2100002f8, 0x0, 0xc210065360, 0x424d82, ...)
    /usr/local/go/src/pkg/syscall/zsyscall_linux_amd64.go:120 +0x7d
syscall.Wait4(0xa3c9, 0x7f272ddb9b0c, 0x0, 0xc210065360, 0x7f272ddb9b70, ...)
    /usr/local/go/src/pkg/syscall/syscall_linux.go:222 +0x6e
os.(*Process).wait(0xc210048cc0, 0x583590, 0x0, 0x0)
    /usr/local/go/src/pkg/os/exec_unix.go:22 +0xec
os.(*Process).Wait(0xc210048cc0, 0xc21000ab00, 0xc210048b40, 0x4238e9)
    /usr/local/go/src/pkg/os/doc.go:43 +0x27
os/exec.(*Cmd).Wait(0xc210037500, 0x0, 0x0)
    /usr/local/go/src/pkg/os/exec/exec.go:311 +0x199
os/exec.(*Cmd).Run(0xc210037500, 0xc210049e00, 0x7f272ddb9d68)
    /usr/local/go/src/pkg/os/exec/exec.go:233 +0x54
main.extractBaseImage(0x7fff7f98883d, 0x1b, 0xc2100000d0, 0x7, 0x0, ...)
    /data/src/Gosrc/src/lhndiag/kvm-installer/installer.go:814 +0x379
main.installVSA(0x7fff7f988827, 0x9, 0x7fff7f98883d, 0x1b, 0x0, ...)
    /data/src/Gosrc/src/lhndiag/kvm-installer/installer.go:856 +0x425
main.main()
    /data/src/Gosrc/src/lhndiag/kvm-installer/installer.go:1049 +0x2d3

goroutine 5 [runnable]:
syscall.Syscall(0x0, 0x8, 0xc21004d600, 0x200, 0x6, ...)
    /usr/local/go/src/pkg/syscall/asm_linux_amd64.s:38 +0x64
syscall.read(0x8, 0xc21004d600, 0x200, 0x200, 0x4e9b81, ...)
    /usr/local/go/src/pkg/syscall/zsyscall_linux_amd64.go:838 +0x72
syscall.Read(0x8, 0xc21004d600, 0x200, 0x200, 0x417fab, ...)
    /usr/local/go/src/pkg/syscall/syscall_unix.go:135 +0x5c
os.(*File).read(0xc2100002a8, 0xc21004d600, 0x200, 0x200, 0x4e2d60, ...)
    /usr/local/go/src/pkg/os/file_unix.go:180 +0x62
os.(*File).Read(0xc2100002a8, 0xc21004d600, 0x200, 0x200, 0x7f272df59468, ...)
    /usr/local/go/src/pkg/os/file.go:95 +0x98
bytes.(*Buffer).ReadFrom(0xc210049e00, 0x7f272df59308, 0xc2100002a8, 0x0, 0x0, ...)
    /usr/local/go/src/pkg/bytes/buffer.go:169 +0x1f7
io.Copy(0x7f272df593f0, 0xc210049e00, 0x7f272df59308, 0xc2100002a8, 0x0, ...)
    /usr/local/go/src/pkg/io/io.go:348 +0x124
os/exec.func·003(0x0, 0x0)
    /usr/local/go/src/pkg/os/exec/exec.go:208 +0x62
os/exec.func·004(0xc210048b20)
    /usr/local/go/src/pkg/os/exec/exec.go:277 +0x2c
created by os/exec.(*Cmd).Start
    /usr/local/go/src/pkg/os/exec/exec.go:278 +0x86b

goroutine 6 [finalizer wait]:
runtime.park(0x40fbd0, 0x698dc0, 0x697468)
    /usr/local/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
    /usr/local/go/src/pkg/runtime/mgc0.c:2276 +0x84
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1394

[nko...@gmail.com]

The gz file that I'm trying to uncompress is a raw disk image, and I couldn't get what I wanted from the approach mentioned above.

By not touching the file, I mean that I don't want the original gz file deleted as is the default behaviour of gunzip.

[Henrik Johansson]

Doesn't the example Seth showed work?

You might want to write directly to either stdout or a new file though if you have an entire image that can be pretty large I guess.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[nko...@gmail.com]

I did try what Seth suggested but I get an out of memory error there too. It must be the size of the gz file that I'm using...approx. 0.4 GB.

fatal error: runtime: out of memory

goroutine 1 [running]:
runtime.throw(0x5a75b7)
    /usr/local/go/src/pkg/runtime/panic.c:464 +0x69 fp=0x7ff43e35ab68
runtime.SysMap(0xc410100000, 0x200000000, 0x5b47d8)
    /usr/local/go/src/pkg/runtime/mem_linux.c:131 +0xfe fp=0x7ff43e35ab98
runtime.MHeap_SysAlloc(0x5be720, 0x200000000)
    /usr/local/go/src/pkg/runtime/malloc.goc:473 +0x10a fp=0x7ff43e35abd8
MHeap_Grow(0x5be720, 0x200000)
    /usr/local/go/src/pkg/runtime/mheap.c:241 +0x5d fp=0x7ff43e35ac18
MHeap_AllocLocked(0x5be720, 0x200000, 0x0)
    /usr/local/go/src/pkg/runtime/mheap.c:126 +0x305 fp=0x7ff43e35ac58
runtime.MHeap_Alloc(0x5be720, 0x200000, 0x100000000, 0x1)
    /usr/local/go/src/pkg/runtime/mheap.c:95 +0x7b fp=0x7ff43e35ac80
runtime.mallocgc(0x1fffffe00, 0x492561, 0x1)
    /usr/local/go/src/pkg/runtime/malloc.goc:89 +0x484 fp=0x7ff43e35acf0
cnew(0x492560, 0x1fffffe00, 0x1)
    /usr/local/go/src/pkg/runtime/malloc.goc:718 +0xc1 fp=0x7ff43e35ad10
runtime.cnewarray(0x492560, 0x1fffffe00)
    /usr/local/go/src/pkg/runtime/malloc.goc:731 +0x3a fp=0x7ff43e35ad30
makeslice1(0x48cf60, 0x1fffffe00, 0x1fffffe00, 0x7ff43e35ad90)
    /usr/local/go/src/pkg/runtime/slice.c:57 +0x4d fp=0x7ff43e35ad48
runtime.makeslice(0x48cf60, 0x1fffffe00, 0x1fffffe00, 0xc4100f8000, 0x1fffffe00, ...)
    /usr/local/go/src/pkg/runtime/slice.c:38 +0x98 fp=0x7ff43e35ad78
bytes.makeSlice(0x1fffffe00, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/bytes/buffer.go:191 +0x63 fp=0x7ff43e35adb0
bytes.(*Buffer).ReadFrom(0xc21005c000, 0x7ff43e4fa140, 0xc21004a000, 0xfffffe00, 0x0, ...)
    /usr/local/go/src/pkg/bytes/buffer.go:163 +0xcb fp=0x7ff43e35ae48
io.Copy(0x7ff43e4fa118, 0xc21005c000, 0x7ff43e4fa140, 0xc21004a000, 0x0, ...)
    /usr/local/go/src/pkg/io/io.go:348 +0x124 fp=0x7ff43e35aed0
main.main()
    /home/nkothapa/Desktop/test.go:24 +0x185 fp=0x7ff43e35af48
runtime.main()
    /usr/local/go/src/pkg/runtime/proc.c:220 +0x11f fp=0x7ff43e35afa0
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1394 fp=0x7ff43e35afa8

[Benjamin Measures]

On Friday, 21 February 2014 00:04:02 UTC, Namu wrote:

I'm trying to uncompress a gz file without touching the original file, and couldn't find a Go way to do it.

Package gzip implements reading and writing of gzip format compressed files, as specified in RFC 1952.

http://golang.org/pkg/compress/gzip/

[Gustavo Niemeyer]

Seth's example is also caching the whole thing in memory.

Instead, you can dump it straight to the output file:

reader, err := gz.NewReader(inputFile)
if err != nil { ... }
err := io.Copy(outputFile, reader)
if err != nil { ... }

Assuming inputFile and outputFile are *os.File values.

--

gustavo @ http://niemeyer.net

[nko...@gmail.com]

Dumping straight to the output file worked! Thank you very much. :)