weird problem: flate: corrupt input before offset 14, when loading very specific images

[Mark Ivey]

I've hit a weird problem saving & loading images and I'm hoping someone can help figure out what's going on. I have a small program that generates random images, saves them as .png files, then tries to load them again. When I run it, png.Decode (from the image/png package) complains about corrupt data. 

Example error:

panic: flate: corrupt input before offset 14

runtime.panic+0xac /Users/mivey/hg/go/src/pkg/runtime/proc.c:1060

runtime.panic(0x85588, 0xe)

main.loadImage+0x207 /Users/mivey/hg/image_bug/image_bug.go:47

main.loadImage(0xf8400002d8, 0xf800000006, 0x0, 0x0, 0xf84002f870, ...)

main.main+0x20c /Users/mivey/hg/image_bug/image_bug.go:22

main.main()

runtime.mainstart+0xf /Users/mivey/hg/go/src/pkg/runtime/amd64/asm.s:77

runtime.mainstart()

runtime.goexit /Users/mivey/hg/go/src/pkg/runtime/proc.c:178

runtime.goexit()

----- goroutine created by -----

_rt0_amd64+0x8e /Users/mivey/hg/go/src/pkg/runtime/amd64/asm.s:64

goroutine 6 [4]:

runtime.gosched+0x5c /Users/mivey/hg/go/src/pkg/runtime/proc.c:603

runtime.gosched()

runfinq+0x50 /Users/mivey/hg/go/src/pkg/runtime/mgc0.c:671

runfinq()

runtime.goexit /Users/mivey/hg/go/src/pkg/runtime/proc.c:178

runtime.goexit()

----- goroutine created by -----

runtime.gc /Users/mivey/hg/go/src/pkg/runtime/mgc0.c:547

Sometimes it complains on the first image, sometimes the second. I've seen it get as far as the 12th image before complaining. The offset mentioned in the error changes too (I've seen 13, 14, 15, 16).

The thing that I find very strange is the problem is very unstable. Small tweaks to the code make this error go away (or at least happen must less frequently). For example, if I change this:

    img.Pix[i].Y = uint8(rand.Int31n(5))

to this:

    img.Pix[i].Y = uint8(rand.Int31n(2))

it can do 150 images without problems. Changing to Int31n(50) also lets it get through 150 images without trouble.

Replacing the rand.Int31n(5) call with 0, 1, 2, 3, 4, or 5 makes the error go away.

Changing the dimensions of the image makes the problem go away.

Changing the loop from this:

    for i, _ := range img.Pix {...}

to this:

    for i := 0; i < 16; i++ {...}

makes the problem go away, though this still exhibits the problem:

    for i := 0; i < 17; i++ {..}

In short, I'm stumped. Any ideas?

    

Here's the full code of the program:

package main

import (

"fmt"

"image"

"image/png"

"os"

"rand"

"time"

)

func main() {

rand.Seed(time.Nanoseconds())

for i := 0; i < 150; i++ {

img := image.NewGray(4, 5)

for i, _ := range img.Pix {

img.Pix[i].Y = uint8(rand.Int31n(5))

}

filename := fmt.Sprintf("%d.png", i)

saveImage(img, filename)

loadImage(filename)

}

}

func saveImage(img image.Image, filename string) {

writer, err := os.Create(filename)

if err != nil {

panic(err)

}

defer writer.Close()

if err = png.Encode(writer, img); err != nil {

panic(err)

}

fmt.Println("Wrote ", filename)

}

func loadImage(filename string) image.Image {

fmt.Println("Reading ", filename)

reader, err := os.Open(filename)

if err != nil {

panic(err)

}

defer reader.Close()

img, err := png.Decode(reader)

if err != nil {

panic(err)

}

return img

}

[peterGo]

Mark,

> Sometimes it complains on the first image, sometimes the second. I've seen
> it get as far as the 12th image before complaining. The offset mentioned in
> the error changes too (I've seen 13, 14, 15, 16).

> rand.Seed(time.Nanoseconds())

You are getting unpredictable results because you asked for
unpredictable results. Comment out the above line of code to get a
deterministic set of pseudo-random numbers. Then you should
consistently fail at:

Wrote 0.png
Reading 0.png

panic: flate: corrupt input before offset 14

Peter

[Mark Ivey]

(I tried to send this reply on Friday but it never showed up on
http://groups.google.com so I'm trying to send it again)

On May 12, 11:25 pm, peterGo <go.peter...@gmail.com> wrote:
> Mark,
>
> > Sometimes it complains on the first image, sometimes the second. I've seen
> > it get as far as the 12th image before complaining. The offset mentioned in
> > the error changes too (I've seen 13, 14, 15, 16).
> > rand.Seed(time.Nanoseconds())
>
> You are getting unpredictable results because you asked for
> unpredictable results. Comment out the above line of code to get a
> deterministic set of pseudo-random numbers. Then you should
> consistently fail at:

You're right. The random numbers were confusing the issue. I hadn't
figured out a way to reproduce this problem without them, but of
course the answer was obvious: print out the numbers that get used.
So, I'd like to ask a simpler version of my original question:

This (revised) code gives an error when loading the image. I can't
figure out why. I don't see that I'm doing anything obviously wrong
when creating the image. Any idea what's going wrong?

package main

import (
"image"
"image/png"
"os"
)

func main() {
img := image.NewGray(4, 5)
img.Pix[0].Y = 4
img.Pix[1].Y = 3
img.Pix[2].Y = 0
img.Pix[3].Y = 1
img.Pix[4].Y = 4
img.Pix[5].Y = 0
img.Pix[6].Y = 2
img.Pix[7].Y = 0
img.Pix[8].Y = 4
img.Pix[9].Y = 3
img.Pix[10].Y = 2
img.Pix[11].Y = 4
img.Pix[12].Y = 4
img.Pix[13].Y = 1
img.Pix[14].Y = 0
img.Pix[15].Y = 2
img.Pix[16].Y = 4
img.Pix[17].Y = 4
img.Pix[18].Y = 1
img.Pix[19].Y = 1
saveImage(img, "foo.png")
loadImage("foo.png")

}

func saveImage(img image.Image, filename string) {
writer, err := os.Create(filename)
if err != nil {
panic(err)
}
defer writer.Close()
if err = png.Encode(writer, img); err != nil {
panic(err)
}
}

func loadImage(filename string) image.Image {

reader, err := os.Open(filename)
if err != nil {
panic(err)
}
defer reader.Close()
img, err := png.Decode(reader)
if err != nil {
panic(err)
}
return img
}







>

> Wrote  0.png
> Reading  0.png
> panic: flate: corrupt input before offset 14
>
> Peter
>

[gmallard]

It is somehow data realted.

Change just these two lines:

img.Pix[2].Y=254
img.Pix[3].Y=255

and try it. That works for me.

Sorry, I have no idea how .pngs are structured, so no real
ideas .......


On May 15, 5:28 pm, Mark Ivey <m...@countlessprojects.com> wrote:
> (I tried to send this reply on Friday but it never showed up onhttp://groups.google.comso I'm trying to send it again)

[andrey mirtchovski]

the culprit is this line in compress/flate/inflate.go:

if !f.h1.init(f.bits[0:nlit]) || !f.h2.init(f.bits[nlit:nlit+ndist]) {

the image you've created is compressed using dynamic huffman tables.
when uncompressing, the code fills in the first table (h1) but the
second table (h2) contains only one element: 0 (nlit=257, ndist=1).
h2's init code short-circuits when it sees only zeroes and you get the
error.

I can confirm that commenting out the check for max == 0 in init()
circumvents the particular issue. writing out an image thus read
results in an identical to the original PNG file.

to test comment out lines 98-100 of pkg/compress/flate/inflate.go and
try reading back the image which previously resulted in an error.

[Mark Ivey]

I can confirm this fixes the problem I was seeing. Thanks!

Should I open a bug report on this?

[andrey mirtchovski]

> Should I open a bug report on this?

I don't know enough about huffman coding to say what exactly is
causing the issue. It seems that the png image is correct as created
because it can be opened without visible warnings by photoshop, osx's
"Preview", plan9's png and the netpbm library. The issue only appears
with sufficiently small images (4x4, 4x5) -- anything smaller switches
to a static encoding table (flate type 1), anything bigger has enough
bits for the second table.

http://zlib.net/feldspar.html has more information, saying "if the
last elements of an alphabet are of 0 codelengths, they can and
probably should be left out"

I suspect that deflate could trim off that bit from the end and
inflate could be more accepting, but that's not an educated guess.

To answer your question: you can file an issue with "this png image is
accepted by everybody else without warning but fails with go's
image/png library" or you can use this program that exhibits the
behaviour without involving png:

---snip---
package main

import (
"compress/flate"
"os"
)

func main() {
b := []byte{0, 0, 1, 0, 4, 4, 2, 4, 1, 2, 0, 3, 4, 2, 1, 4, 0}

r, w, _ := os.Pipe()
fw := flate.NewWriter(w, -1)
fr := flate.NewReader(r)

n, err := fw.Write(b)

if err != nil {
panic(err)
}

fw.Close()
nn, err := fr.Read(b)

if err != nil {
panic(err)
}

if n != nn {
panic("different r/w sizes")
}
}

$ 6g t.go; 6l t.6; ./6.out
# debug messages from compress/flate
typ: 2
nlit, ndist+nlit: 257 258
panic: flate: corrupt input before offset 13

[andrey mirtchovski]

Just to see if it'll work, I ran a randomized 16-byte array through
deflate/inflate a million times. The run results in slightly more than
46000 values which would have been rejected by the unpatched inflate.
With the patch they all uncompress to the same array that was
compressed beforehand, so this patch seems to avoid corruption.

[Gustavo Niemeyer]

> To answer your question: you can file an issue with "this png image is
> accepted by everybody else without warning but fails with go's
> image/png library" or you can use this program that exhibits the
> behaviour without involving png:

The zlib bug seems more generic than this, and easy to reproduce with
simple textual data:

http://code.google.com/p/go/issues/detail?id=1833

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/blog
http://niemeyer.net/twitter

[andrey mirtchovski]

> The zlib bug seems more generic than this, and easy to reproduce with
> simple textual data:
>
> http://code.google.com/p/go/issues/detail?id=1833

Not a decoding problem then if python doesn't like the compressed
data. This sample decoder from zlib.net also chokes on output from
deflate:

http://www.zlib.net/zpipe.c

[Mark Ivey]

Thanks for all the help looking into this, and the workaround until a fix is in. Much appreciated.