ANN: ImageMagick/GraphicsMagick bindings

[Alberto García Hierro]

Hi,

We've just released our bindings for ImageMagick/GraphicsMagic. The former is used by default, but the latter can be enabled using the gm build tag. We debated for a few weeks which backend should be default, and we ended up going for IM because it's better at handling optimized GIFs (GM can't decode heavily optimized GIF images), although GM is noticeably faster. Depending on your use case, you might be better off using the GM backend.

In our performance tests, we've found that GM is around 120x faster than the image related modules in the standard library when it comes to decoding/resizing/cropping/encoding images.

Source code is available at https://github.com/rainycape/magick

Regards,

Alberto

[James Bardin]

Nice work.

There's a number of bindings floating around currently (another cgo version here, https://github.com/gographics/imagick), and we have our own we use internally for GraphicsMagick.

Did you by chance compare the gographics library?

As an aside, the developer on the gographics/imagick library had mentioned writing a Go version of the {Image|Graphics}Magick api. I think that would be great if some of the image library users got together to start work on a GoMagick library. https://groups.google.com/d/msg/golang-nuts/6qVpOGyDuHI/kpjrtX39aHgJ

A first step that I think would be interesting, is cgo bindings to the native image encoders, where you could `import _ "image/jpeg_cgo"` and get faster encode/decode performance with the stdlib.

[Alberto García Hierro]

 

On Wednesday, October 30, 2013 6:12:59 PM UTC+1, James Bardin wrote:

Nice work.

There's a number of bindings floating around currently (another cgo version here, https://github.com/gographics/imagick), and we have our own we use internally for GraphicsMagick.

Did you by chance compare the gographics library?

Thanks! I wasn't aware of any other bindings for IM nor GM. We've been using this internally since August 2012 and there were no public bindings at that time, so we wrote our own. From a quick look at go graphics, it seems they're wrapping the MagickWand library, while we're wrapping MagickCore, so it seems both packages are intended for different use cases.

 

As an aside, the developer on the gographics/imagick library had mentioned writing a Go version of the {Image|Graphics}Magick api. I think that would be great if some of the image library users got together to start work on a GoMagick library. https://groups.google.com/d/msg/golang-nuts/6qVpOGyDuHI/kpjrtX39aHgJ

I think that would be a terrible waste of time and the developer is severely underestimating the effort required. IM or GM are big projects, around 500K lines for IM and I guess GM is in the same ballpark. Furthermore, they rely on other C libraries for some other tasks, like decoding some formats. Porting everything from C to Go would require many many hours and, probably, you'll end up with a slower implementation than using cgo to call to the C libraries. The only benefit would be able to run on environments when using cgo/unsafe is not allowed (like GAE), but I personally have zero interest in those.

 

A first step that I think would be interesting, is cgo bindings to the native image encoders, where you could `import _ "image/jpeg_cgo"` and get faster encode/decode performance with the stdlib.

We already have written (and thrown away) decoders for png, jpeg and gif, with both encoding and decoding support. We initially wrote them to get better decoding/encoding support (image/gif didn't support encoding at the time, and all image/* packages failed to decode 20-30% of our sample set), but we found that even with libpng/libjpeg/libgif based decoders, Go's image package still sucked. Want to resize an image? Write your own algorithm! Need to work with animated GIFs? Not supported! If you need to do anything barely advanced with images, just forget about the packages in the standard library. You'll eventually hit a wall and will have to rewrite all your image handling code using a real image library.

[James Bardin]

On Wed, Oct 30, 2013 at 2:32 PM, Alberto García Hierro <alb...@garciahierro.com> wrote:

Thanks! I wasn't aware of any other bindings for IM nor GM. We've been using this internally since August 2012 and there were no public bindings at that time, so we wrote our own. From a quick look at go graphics, it seems they're wrapping the MagickWand library, while we're wrapping MagickCore, so it seems both packages are intended for different use cases.

 

Oh, I totally missed the wand vs core difference. I'll have to take a closer look at your code and see if it's something I could make use of. I know you prefer IM right now, but have you seen any problems with GM compatibility (I didn't think they had api feature parity any longer)? Also FYI, GM has had a number of fixes in the past year, and a lot if changes pending for the next release. We run a backported version with all the latest supporting libs to keep performance/reliability up.

 

I think that would be a terrible waste of time and the developer is severely underestimating the effort required. IM or GM are big projects, around 500K lines for IM and I guess GM is in the same ballpark. Furthermore, they rely on other C libraries for some other tasks, like decoding some formats. Porting everything from C to Go would require many many hours and, probably, you'll end up with a slower implementation than using cgo to call to the C libraries. The only benefit would be able to run on environments when using cgo/unsafe is not allowed (like GAE), but I personally have zero interest in those.

 

All good points :) I was hoping one could drop a lot of the legacy baggage in there too, and greatly simplify things, but yeah, it would be a huge project.

 

We already have written (and thrown away) decoders for png, jpeg and gif, with both encoding and decoding support. We initially wrote them to get better decoding/encoding support (image/gif didn't support encoding at the time, and all image/* packages failed to decode 20-30% of our sample set), but we found that even with libpng/libjpeg/libgif based decoders, Go's image package still sucked. Want to resize an image? Write your own algorithm! Need to work with animated GIFs? Not supported! If you need to do anything barely advanced with images, just forget about the packages in the standard library. You'll eventually hit a wall and will have to rewrite all your image handling code using a real image library.

Yes, there would need to be some improved stdlib image functions to make this really worthwhile, but I figured those would need to come down the pipline eventually to keep the stdlib image library relavant. It's really not good for very much as it stands right now.

[Nigel Tao]

Hi. I'm the author of Go's standard image library.

On Thu, Oct 31, 2013 at 3:29 AM, Alberto García Hierro
<alb...@garciahierro.com> wrote:
> In our performance tests, we've found that GM is around 120x faster than the
> image related modules in the standard library when it comes to
> decoding/resizing/cropping/encoding images.

120x surprises me. Do you have some example Go code and test data that
shows this?

One thing is that newcomers to Go's standard library often use the
generic At and Set methods in their inner loops, which are the easiest
to use but slower, and definitely not how I would e.g. do resizing or
cropping. For example,
https://groups.google.com/forum/#!topic/golang-nuts/f0X0I_JZP0I
describes a 3x speed-up by using the Pix field directly.

On Thu, Oct 31, 2013 at 5:32 AM, Alberto García Hierro
<alb...@garciahierro.com> wrote:
> The only benefit would be able to run on environments when using
> cgo/unsafe is not allowed (like GAE), but I personally have zero interest in
> those.

Another benefit is that decoding untrusted, third party images is far
less likely to result in remote code execution through buffer
overflows, as Go is a memory-safe language, but I don't know how
important that is to you.

> all image/* packages failed to decode 20-30% of our sample set),

20-30% also sounds surprisingly large to me. Can you send me some of
these failed images (off-list)? How many of these are 'optimized
GIFs'?

I looked at the 7 image files in
https://github.com/rainycape/magick/tree/master/test_data, and Go's
decoders only failed on optimized.gif, in the LZW code, and I note
that you said that GraphicsMagick also fails to decode this.

> Want to resize an image? Write your own algorithm!

There are many different resizing algorithms and approaches. I would
imagine that pure Go image resizing can and should be done by a "go
get"table third party library. Not everything has to be in the
standard library.

> Need to work with animated GIFs? Not supported!

Animated GIFs are supported, via gif.DecodeAll. It's true that
image.Decode will only give you the first frame, but the image
package's format.go is trivial to fork if you really need a catch-all
decoder that can give you Image or *GIF (or []Image).


At the end of the day, though, if binding ImageMagick works for you,
then don't let me stop you. :-)

[Alberto García Hierro]

On Thursday, October 31, 2013 1:14:57 AM UTC+1, Nigel Tao wrote:

Hi. I'm the author of Go's standard image library.

Hi Nigel. Don't take me wrong, it's great that Go has image related packages in the standard library, which are getting better by the day. I'm sure they work fine for a lot of people, and that number is only going to increase. However, right now, if you have advanced needs, like decoding every possible image that a user might throw at you or work with animated GIFs, Go's standard library is not there yet.

 

 

On Thu, Oct 31, 2013 at 3:29 AM, Alberto García Hierro
<alb...@garciahierro.com> wrote:
> In our performance tests, we've found that GM is around 120x faster than the
> image related modules in the standard library when it comes to
> decoding/resizing/cropping/encoding images.

120x surprises me. Do you have some example Go code and test data that
shows this? 

One thing is that newcomers to Go's standard library often use the
generic At and Set methods in their inner loops, which are the easiest
to use but slower, and definitely not how I would e.g. do resizing or
cropping. For example,
https://groups.google.com/forum/#!topic/golang-nuts/f0X0I_JZP0I
describes a 3x speed-up by using the Pix field directly.

One of the tests is even in the magick package. Using GM it is 129x times faster, while IM is 93x. For this benchmark we're using http://github.com/nfnt/resize, which seems to access the Pix field directly, although it calls an interface method in the inner loop.

fiam@ubuntu:~/gocode/src/magick$ go test -v -tags gm -run=TestDecode$ -bench=ResizePng 
=== RUN TestDecode 
--- PASS: TestDecode (0.25 seconds) 
magick_test.go:90: Using backend GraphicsMagick 
BenchmarkResizePng 20 72172794 ns/op 
BenchmarkResizePngNative 1 9352362699 ns/op

On Thu, Oct 31, 2013 at 5:32 AM, Alberto García Hierro
<alb...@garciahierro.com> wrote:
> The only benefit would be able to run on environments when using
> cgo/unsafe is not allowed (like GAE), but I personally have zero interest in
> those.

Another benefit is that decoding untrusted, third party images is far
less likely to result in remote code execution through buffer
overflows, as Go is a memory-safe language, but I don't know how
important that is to you.

Truth to be told, almost zero. If someone were to target us specifically, I'm sure there are 0-days in other services

running on our servers. Even our own code could have security holes, we're all humans after all. It is nice to close a potential attack vector, but when closing it comes with very significant tradeoffs and you still have a bunch of other vectors open, it's not worth it.

 

 

> all image/* packages failed to decode 20-30% of our sample set),

20-30% also sounds surprisingly large to me. Can you send me some of
these failed images (off-list)? How many of these are 'optimized
GIFs'?

Keep in mind that this was in August 2012 and I'm sure the situation has certainly improved. We tested against a list of file ids from our blob store and I can't find that list right now. I'll take a look at the history in the internal repository to see if I can find it.

 

I looked at the 7 image files in
https://github.com/rainycape/magick/tree/master/test_data, and Go's
decoders only failed on optimized.gif, in the LZW code, and I note
that you said that GraphicsMagick also fails to decode this.

It does not fail on that one, but on others more "optimized" IIRC.

 

> Want to resize an image? Write your own algorithm!

There are many different resizing algorithms and approaches. I would
imagine that pure Go image resizing can and should be done by a "go
get"table third party library. Not everything has to be in the
standard library.

I totally agree, resizing algorithms should be pluggable. However, I think the standard library should provide a basic framework for them, so users could write only the code for the interpolation.

e.g.

type Interpolator interface... or type Interpolator func... for better performance

func Resize(im Image, width, height int, inter Interpolator) (Image, error)

 

 

> Need to work with animated GIFs? Not supported!

Animated GIFs are supported, via gif.DecodeAll. It's true that
image.Decode will only give you the first frame, but the image
package's format.go is trivial to fork if you really need a catch-all
decoder that can give you Image or *GIF (or []Image).

Everything is trivial given enough time. The problem with gif.DecodeAll is that you have to either special case when image.Decode returns a GIF, to decode the images again (and, according to the code, all of them they were previously decoded, but thrown away, so you're doing 2x the required work) or either read the image bytes into a buffer, check the magic number for GIF89A and then pass a bytes.Reader to either image.Decode or gif.DecodeAll. Then if you want to do something useful with them you need to do the coalescing yourself, since gif.DecodeAll doesn't do that for you. With IM or GM you have a unified interface, where you can decode an image, resize it and then encode it using any supported format. You don't need to care about all the details unless you're encoding to a format which supports animation and want to only show the first frame (like e.g. when generating a thumbnail). And even in that case, you can just call im.Frame(0) and it will work on both animated and non-animated images.

At the end of the day, is a matter of engineering the better solution to your problem, and for us that was wrapping ImageMagick.

 

At the end of the day, though, if binding ImageMagick works for you,
then don't let me stop you. :-)

I won't, don't worry ;)

Regards,

Alberto 

[Nigel Tao]

> One of the tests is even in the magick package. Using GM it is 129x times
> faster, while IM is 93x. For this benchmark we're using
> http://github.com/nfnt/resize, which seems to access the Pix field directly,
> although it calls an interface method in the inner loop.

I skimmed the nfnt/resize package. I haven't really looked at it before.

Well, it reads the Pix field... but wraps that reading in a
converter.at interface method, so we're back to going through a
general-purpose interface. Filter.Interpolate is another interface
method (which IIUC then calls the kernel, a closure, multiple times).
All this indirection in the inner loop will kill you, regardless of
whether you read the underlying pixel data via At or Pix.

That package also converts everything to float32 for intermediate
computation, and does that conversion multiple times per pixel, which
also doesn't sound cheap to me.

Your comparison also used nfnt's Lanczos2 instead of Lanczos2Lut, and
avoiding the lookup table also means that the underlying math.Sin
trigonometric function is called *multiple times* for every pixel,
which is *super expensive*.

Anyway, long story short: I'm sure that Go can do better than 100x C.

>> > all image/* packages failed to decode 20-30% of our sample set),
>>
>> 20-30% also sounds surprisingly large to me. Can you send me some of
>> these failed images (off-list)? How many of these are 'optimized
>> GIFs'?
>
> Keep in mind that this was in August 2012

Oh, one image/jpeg package difference between Go 1.0 and Go 1.1 was
the ability to decode progressive JPEGs. That probably explains a lot
of that number.

[Nigel Tao]

On Thu, Oct 31, 2013 at 3:29 AM, Alberto García Hierro
<alb...@garciahierro.com> wrote:

> In our performance tests, we've found that GM is around 120x faster than the
> image related modules in the standard library when it comes to
> decoding/resizing/cropping/encoding images.
>
> Source code is available at https://github.com/rainycape/magick

Are there other performance tests (e.g. just decoding, cropping and/or
encoding, not resizing) that I can look at, other than
magick/bench_test.go's BenchmarkResizePng and
BenchmarkResizePngNative? As I explained in my previous post, I'd say
that magick/bench_test.go's 120x says more about the
github.com/nfnt/resize package and how its API guided the way you're
using it (e.g. not using trig look up tables), than it says about Go's
standard library.

[Nigel Tao]

On Thu, Oct 31, 2013 at 3:29 AM, Alberto García Hierro
<alb...@garciahierro.com> wrote:

> handling optimized GIFs

One more thing... when you say "optimized GIFs", do you know what
software is doing the "optimizing"?

[Dobrosław Żybort]

> Want to resize an image? Write your own algorithm!

There are many different resizing algorithms and approaches. I would
imagine that pure Go image resizing can and should be done by a "go
get"table third party library. Not everything has to be in the
standard library.

I totally agree, resizing algorithms should be pluggable. However, I think the standard library should provide a basic framework for them, so users could write only the code for the interpolation.

e.g.

type Interpolator interface... or type Interpolator func... for better performance

func Resize(im Image, width, height int, inter Interpolator) (Image, error)

 

I filled feature request:
https://code.google.com/p/go/issues/detail?id=6755

BR

 

[jonathan...@gmail.com]

Nigel, I was hoping for golang only solution to simple resizing jpeg/pngs but am not coming up with anything fast enough, do you have any suggestions on the following? My comparison is mainly libvips with jpeg which can do a 7kx5k image in half (decode, resize, and encode) in ~200 ms (bilinear) on my machine and ~500ms for bicubic. For the stdlib on 1.6beta2 I can only come up with 1.3s (decode, resize approxbilinear, encode) or 3.3s (decode, catmullrom, encode). This is a 6x difference or so. I could potentially do a few sizes from one source, which saves a decode or two. I could also use a mipmapy thing going from an intermediate image to the next size, however that first halving is what the numbers I gave was for. Any help would be much appreciated.

Thanks!

[Nigel Tao]

On Sat, Feb 13, 2016 at 11:58 AM, <jonathan...@gmail.com> wrote:
> Nigel, I was hoping for golang only solution to simple resizing jpeg/pngs
> but am not coming up with anything fast enough, do you have any suggestions
> on the following? My comparison is mainly libvips with jpeg which can do a
> 7kx5k image in half (decode, resize, and encode) in ~200 ms (bilinear) on my
> machine and ~500ms for bicubic. For the stdlib on 1.6beta2 I can only come
> up with 1.3s (decode, resize approxbilinear, encode) or 3.3s (decode,
> catmullrom, encode). This is a 6x difference or so. I could potentially do a
> few sizes from one source, which saves a decode or two. I could also use a
> mipmapy thing going from an intermediate image to the next size, however
> that first halving is what the numbers I gave was for. Any help would be
> much appreciated.

I don't have any quick wins for you, other than to try disabling
bounds checking. I forget exactly how to do this, but it's something
like passing "-gcflags=-B". Naturally, your program will be less safe
when disabling bounds checking, so I wouldn't use it if you're
decoding images from untrusted sources, such as hosting a web service
open to the internet at large.

As image manipulation is also compute-heavy, you could also try the
SSA branch of the standard Go compiler, or the GCCGo compiler, but I
haven't tried either of those recently, so I can't remember what the
exact recipes to do so are.

I do know that the image/jpeg encode code hasn't been optimized much
at all, and there may be some quick wins there, but that's for the Go
1.7 release cycle or later.

I presume you're also using golang.org/x/image/draw, which has had
some optimization work but not a lot. Once again, there could be some
more wins there, if somebody's willing to do the sweat.

There's a longer term idea to make golang.org/x/image/draw's code
generator generate C-like unsafe Go code (using unsafe.Pointer) or
even asm code, instead of safe (bounds-checked) Go code, but that'd be
a lot of work, it'd be experimental, and you'd have to be very careful
to keep things safe when handling potentially malicious images.

When you say things like decode + resize + encode takes 1.3s, which of
those three takes the longest? Knowing that would help prioritize
where any optimization efforts should look.

If your input is a JPEG and you're doing an exact 1:2 minification and
you're happy with a box filter (as opposed to approx bi-linear or
Catmull-Rom), you could possibly be really clever and do the reduction
in terms of JPEG's underlying 8x8 blocks, kind of the way you can do a
lossless 90-degree JPEG rotation without a full blown
decode-and-encode. Or at least cop the JPEG decode-and-encode but do
the scaling in YCbCr space via custom code (exact 1:2 minification
with a box filter is relatively straightforward) instead of converting
YCbCr to and from RGBA as the golang.org/x/image/draw package needs
to. But by really clever, I mean really tricky.