golang.org/x/image/draw Copy, Scale and Transform

[Nigel Tao]

The image/draw package in the standard library has these functions:

----
func Draw(dst Image, r image.Rectangle, src image.Image, sp image.Point, op Op)

func DrawMask(dst Image, r image.Rectangle, src image.Image, sp
image.Point, mask image.Image, mp image.Point, op Op)
----

These functions are covered by the Go 1 backwards compatibility guarantee.

The golang.org/x/image/draw package is intended to be a drop-in
replacement for image/draw, and its API is a superset of the standard
image/draw's API. The additional API currently looks like:

----
func Scale(dst Image, dr image.Rectangle, src image.Image, sr
image.Rectangle, q Interpolator)

type Interpolator interface {
NewScaler(dw, dh, sw, sh int32) Scaler
}

type Scaler interface {
Scale(dst Image, dp image.Point, src image.Image, sp image.Point)
}

var NearestNeighbor = Interpolator(etc)
----

I would like to expand that API to allow for affine transformations,
including arbitrary rotations, as well as for clipping and masking.
The aim is, together with a Bezier curve rasterizer and a font
library, to be able to rasterize something similar to PDF 1.4's
graphics model. I propose for golang.org/x/image/draw to have three
key functions:

----
func Copy(dst Image, dp image.Point, src image.Image, sr
image.Rectangle, opts *Options)

func Scale(dst Image, dr image.Rectangle, src image.Image, sr
image.Rectangle, q Interpolator, opts *Options)

func Transform(dst Image, m *f64.Mat3, src image.Image, sr
image.Rectangle, q Interpolator, opts *Options)
----

(I might also add Rotate90CW, Rotate180 and Rotate90CCW in the future,
with the same signature as Copy, but that's not part of my immediate
plans).

All three functions map the sr part of src onto the destination. All
three signatures are the same, except for the second argument (the map
from the src co-ordinate space to the dst co-ordinate space) and Copy
doesn't have an Interpolator, since it has a 1:1 relationship between
dst and src pixels.

For Copy, the map is a "dp image.Point" that denotes where the src
image's src.Min maps to. This is similar to how the dr and sp
arguments work for the existing Draw function, except rectangleness
has moved from dst to src, to be consistent with Scale and Transform.

For Scale, both dst and src are integer-quantized axis-aligned
rectangles, so the map is a "dr image.Rectangle" that denotes where
the src image's sr maps to.

For Transform, m is an arbitrary 2D affine transformation, and
dst-space co-ordinates equals m times src-space co-ordinates. It is an
*f64.Mat3, or if a specialized affine 3x2 matrix type exists, an
*f64.Aff3. I'll start a separate discussion thread about whether to
have an Aff3 type separate from Mat3.

A nil *Options means to use the default Options, which is equivalent
to a non-nil Options with zero-valued fields. Options is:

----
type Options struct {
// A zero Op means to use the Over operator.
Op Op
// Masks limit what parts of the dst image are drawn to and
// what parts of the src image are drawn from.
//
// The DstMask is otherwise known as a clip mask, and its
// pixels map 1:1 to dst pixels. DstMaskP in DstMask space
// corresponds to image.Point{X:0, Y:0} in dst space. For
// example, when limiting re-painting to a 'dirty rectangle',
// use that rectangle as the DstMask.
//
// The SrcMask's pixels map 1:1 to src pixels. SrcMaskP
// in SrcMask space corresponds to image.Point{X:0, Y:0}
// in src space. For example, when drawing font glyphs in
// a uniform color c (of type *image.Uniform), use c as the
// src, and use the glyph atlas image and the per-glyph
// offset as SrcMask and SrcMaskP.
DstMask image.Image
DstMaskP image.Point
SrcMask image.Image
SrcMaskP image.Point
}
----

The Scale and Interpolator types would then become:

----
type Interpolator interface {
NewScaler(dw, dh, sw, sh int32) Scaler
Transform(dst Image, m *f64.Mat3, src image.Image, sr
image.Rectangle, opts *Options)
}

type Scaler interface {
Scale(dst Image, dp image.Point, src image.Image, sp image.Point,
opts *Options)
}
----

I made a hand-wavy remark about using rectangles as masks. This would
be literally possible if image.Rectangle implemented image.Image. I'll
start a separate discussion thread about whether to do so.

golang-dev, WDYT?

[David Crawshaw]

SGTM.

Are you keeping Interpolator out of *Options because it does not apply
to Copy? Otherwise it seems like a good for the Options struct.

[Brad Fitzpatrick]

I was about to reply with the same as David. If you have Options already, it seems you should use it to make the signature and call sites more readable. You could make Copy return an error if Interpolator is set (or simply ignore it).

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nigel Tao]

On Tue, Mar 3, 2015 at 1:00 AM, David Crawshaw <craw...@golang.org> wrote:
> Are you keeping Interpolator out of *Options because it does not apply
> to Copy? Otherwise it seems like a good for the Options struct.

I kept Interpolator out of Options because it doesn't apply to Copy or
Scale (the method, not the function), since Interpolator.NewScaler
returns something already bound to an Interpolator.

But on second thoughts, I have a better design: remove the Scale and
Transform functions, and only have Scale and Transform methods:

----
type Interpolator interface {
Scaler
Transformer
}

type Scaler interface {
Scale(dst, dr, src, sr, opts)
}

type Transfomer interface {
Transform(dst, m, src, sr, opts)
}
----

Previously:
draw.Scale(dst, dr, src, sr, draw.NearestNeighbor, opts)
would become
draw.NearestNeighbor.Scale(dst, dr, src, sr, opts)

Kernel, which implements Interpolator, would have a NewScaler function
that reduces redundant computation (and garbage) when scaling multiple
images (all of the same dst and src sizes). But NearestNeighbor (a
non-kernel interpolator) would no longer have a NewScaler method.

In any case, there doesn't seem to be any major objections, so I'll
start sending out CLs.

[anl...@gmail.com]

I would like to expand that API to allow for affine transformations,
including arbitrary rotations, as well as for clipping and masking.
The aim is, together with a Bezier curve rasterizer and a font
library, to be able to rasterize something similar to PDF 1.4's
graphics model. I propose for golang.org/x/image/draw to have three
key functions:

good luck without generics in go ;)

meanwhile we'll prepare for RotatorCliperMaskerBezierer interface

[Nigel Tao]

On Mon, Mar 2, 2015 at 4:16 PM, Nigel Tao <nige...@golang.org> wrote:
> I would like to expand that API to allow for affine transformations,
> including arbitrary rotations, as well as for clipping and masking.

The new "package draw" is feature-complete and welcomes all testers.

go get golang.org/x/image/draw

[Andrew Gerrand]

You're gonna need to write another blog post :)

[Egon Elbre]

Is the Options struct really necessary?

e.g. there is also a possibility of using something like:

func Copy(Masked(dst, mask), Clip(drect, dclip), Masked(src, mask), Clip(srect, sclip))

or:

func Copy(Clipped(dst, dclip), dp, Clipped(src, sclip), sp)

Clipped would return a struct that exposes the same interface as Image, but ignores everything outside the clip region.

+ Egon

[Egon Elbre]

One thing that I don't notice anywhere in the x/image packages is gamma correction. Also, I'm not quite sure what should be the correct approach to dealing with it. Maybe a separate package with: gamma.FromLinear(image, 2.1), gamma.ToLinear(image, 2.1)?

http://www.4p8.com/eric.brasseur/gamma.html

+ Egon

[Ian Davis]

Great job! I've been watching the CLs with interest. I'm going to
feature this package heavily in the next chapter of the book I'm
writing.

Ian

[Nigel Tao]

On Wed, May 6, 2015 at 7:03 PM, Egon Elbre <egon...@gmail.com> wrote:
> Is the Options struct really necessary?

The Options struct gives us the ability to add features in the future
without breaking existing code.

For example, gamma correction, which I acknowledge is something that
the Go image packages don't handle (or even have a good proposal) yet.
:-)

[Egon Elbre]

The gamma correction needs to be done on image level basis so it can be done similarly to the previously shown examples. Although, I think converting images to linear scale after loading and after manipulating them convert them back to sRGB is probably the best way. At least I'm not aware of any such problems.

Regarding options, the most I've seen are specifying the clipping rect and masking images and blending operation. Everything else usually was either image property or kernel property. Currently it feels like it's future proofing something that is unlikely to change, at the cost of making API more complicated.

+ Egon

[Nigel Tao]

On Thu, May 7, 2015 at 5:07 PM, Egon Elbre <egon...@gmail.com> wrote:
> Regarding options, the most I've seen are specifying the clipping rect and
> masking images and blending operation. Everything else usually was either
> image property or kernel property. Currently it feels like it's future
> proofing something that is unlikely to change, at the cost of making API
> more complicated.

There's already one potential TODO in scale.go that is a smooth vs
sharp edges option, for when the source rectangle doesn't map to
integer-quantized destination pixel boundaries. I don't think that's
really a dst image, src image or kernel property.

There's also the possibility of providing a callback so that UIs can
display a progress bar for long-running operations, or cancel. Again,
that's neither an image or kernel property.

[Egon Elbre]

On Friday, 8 May 2015 02:29:01 UTC+3, Nigel Tao wrote:

On Thu, May 7, 2015 at 5:07 PM, Egon Elbre <egon...@gmail.com> wrote:
> Regarding options, the most I've seen are specifying the clipping rect and
> masking images and blending operation. Everything else usually was either
> image property or kernel property. Currently it feels like it's future
> proofing something that is unlikely to change, at the cost of making API
> more complicated.

There's already one potential TODO in scale.go that is a smooth vs
sharp edges option, for when the source rectangle doesn't map to
integer-quantized destination pixel boundaries. I don't think that's
really a dst image, src image or kernel property.

That's one that I really don't know where to place.

There's also the possibility of providing a callback so that UIs can
display a progress bar for long-running operations, or cancel. Again,
that's neither an image or kernel property.

Here I would use something like:

type Process interface {

Step() (ok bool)

Err() error

Abort() error

}

for process.Step() { }

I find it easier to control/manage rather than fiddling with callbacks.

Anyways, I really don't have any other thoughts.

And just be clear, my ideas/questions aren't an objection, mainly alternate design ideas.

+ Egon

[roger peppe]

Is there a plan to provide convenient operations on the f64, f32 Aff* and Mat*
types (rotate, translate, combine, etc) somewhere?

[Nigel Tao]

On Sat, May 9, 2015 at 7:27 PM, roger peppe <rogp...@gmail.com> wrote:
> Is there a plan to provide convenient operations on the f64, f32 Aff* and Mat*
> types (rotate, translate, combine, etc) somewhere?

I'd like to see such methods, but we haven't reached consensus on what
the API should be. For example, should methods return the result as a
value or modify the receiver? Are rotations specified in radians,
degrees or something else? Are positive rotations clockwise or
counter-clockwise, and does it depend on whether the Y axis points up
or down? Etc.

There was a previous discussion at
https://groups.google.com/forum/#!topic/golang-dev/9sYm9-2Bc2w but I
don't think we came to a conclusion.

[Nigel Tao]

Just a random thought... the Transform method currently takes an
affine matrix to convert from src coordinate space to dst coordinate
space:

type Transformer interface {
Transform(dst Image, m *f64.Aff3, src image.Image, sr
image.Rectangle, op Op, opts *Options)
}

An alternative to an Aff3 is to take three corners of the dst
parallelogram, such as what the top-left, top-right and bottom-right
of the src rectangle map to.

Either way specifies six degrees of freedom, but the three corners API
might be easier to visualize, and is a nice progression from the Copy
and Scale methods, which take a destination point (i.e. top-left) and
destination rectangle (i.e. top-left and bottom-right) for their 2 and
4 degrees of freedom.

I haven't made a decision either way, but if anyone has arguments for
one or the other, I'm listening.

[roger peppe]

On 18 May 2015 at 00:55, Nigel Tao <nige...@golang.org> wrote:
> Just a random thought... the Transform method currently takes an
> affine matrix to convert from src coordinate space to dst coordinate
> space:
>
> type Transformer interface {
> Transform(dst Image, m *f64.Aff3, src image.Image, sr
> image.Rectangle, op Op, opts *Options)
> }
>
> An alternative to an Aff3 is to take three corners of the dst
> parallelogram, such as what the top-left, top-right and bottom-right
> of the src rectangle map to.

This seems nice, but the advantage of a matrix is that
you can easily combine an arbitrary number of transformations
into a single transformation, something that may be harder to do
with the three corners API.

An alternative might be to provide a method to convert a
(Rectangle, three corners) tuple into an Aff3, providing
the convenience of one approach with the flexibility of
the other.

cheers,
rog.