Resize image to specific size

[Andrei Zh]

There's pretty cool package "Images" by Tim Holy (kudos, Tim!), but it seems to leak one standard and pretty helpful function - resizing image to specific size. There's "restrict" method (probably imported from Grid.jl), but it only reduces image size by 2 at each dimension. Are there any other options that work with arbitrary images sizes?

[Tim Holy]

On Tuesday, July 29, 2014 02:30:55 PM Andrei Zh wrote:
> There's pretty cool package "Images" by Tim Holy (kudos, Tim!),

...and Ron, Kevin, Lucas, etc ...

> but it
> seems to leak one standard and pretty helpful function - resizing image to
> specific size.
> There's "restrict" method (probably imported from Grid.jl),
> but it only reduces image size by 2 at each dimension. Are there any other
> options that work with arbitrary images sizes?

Yeah, it's a pretty obvious omission, but to be truthful it's never come up
for me so I've never implemented it. I will get to it eventually, but in the
meantime if you need it soon it might be worth considering putting something
together yourself. If you use Grid to do it (+ smoothing with imfilter or
imfilter_gaussian if you're downsizing and don't want aliasing), it shouldn't
be more than 20 lines or so. But there are other approaches, see
http://stackoverflow.com/questions/384991/what-is-the-best-image-downscaling-algorithm-quality-wise
which might involve more effort but also yield higher quality/speed.

BTW, the version of restrict in Images is actually considerably faster than
the one in Grid. Sometime I plan to implement what should be an even faster
version and put it back into Grid, but I haven't gotten around to it.

--Tim

[Andrei Zh]

...and Ron, Kevin, Lucas, etc ...

Surely, I'm just referring to someone I see on the mailing list very often :)

Yeah, it's a pretty obvious omission, but to be truthful it's never come up
for me so I've never implemented it. I will get to it eventually, but in the
meantime if you need it soon it might be worth considering putting something
together yourself. If you use Grid to do it (+ smoothing with imfilter or
imfilter_gaussian if you're downsizing and don't want aliasing), it shouldn't
be more than 20 lines or so. But there are other approaches, see
http://stackoverflow.com/questions/384991/what-is-the-best-image-downscaling-algorithm-quality-wise
which might involve more effort but also yield higher quality/speed.

I'm not so concerned about high quality, so I think I will dive into your code and maybe come up with a general solution.

 

[Andrei]

For those looking for the same thing, below is my simple implementation. It uses bilinear interpolation and works with 2D arrays of Real-s and Integer-s (tested on Uint8 matrices mostly). Usage example:

 using Images

 using ImageView
 im = imread("...")
 dat = convert(Array, im)   # say, dat is a Matrix of size (256, 256)

 new_dat = imresize(dat, 128, 128)

 view(new_dat)

Resizing from (256, 256) to (128, 128) takes ~1.5ms on my machine, while resizing same (256, 256) image to (512, 512) takes as long as ~60ms. So it's definitely not high-performance implementation, but it does its work and handles most frequent use cases.

---------------------------

# interpolate point at (x, y) from 4 nearby pixel values
function interp_bilinear{T <: Union(Real, Integer)}(dat::Array{T, 2},
                                                    x::Float64, y::Float64,
                                                    x1, x2, y1, y2)
    q11 = dat[y1, x1]        
    q12 = dat[y2, x1]
    q21 = dat[y1, x2]
    q22 = dat[y2, x2]
    if x1 != x2
        r1 = (x2 - x) / (x2 - x1) * q11 + (x - x1) / (x2 - x1) * q21
        r2 = (x2 - x) / (x2 - x1) * q12 + (x - x1) / (x2 - x1) * q22
    else
        # special case of x1 == x2, no interpolation needed
        r1 = q11
        r2 = q12
    end
    if y1 != y2
        r = (y - y1) / (y2 - y1) * r1 + (y2 - y) / (y2 - y1) * r2
    else
        # special case of y1 == y2, no interpolation needed
        r = r1
    end
    if typeof(r) <: Real
        r = round(r)
    end
    r = convert(T, r)
    return r
end


function imresize{T <: Union(Real, Integer)}(dat::Array{T, 2},
                                             new_size::(Int, Int))
    new_dat = similar(dat, new_size)
    h, w = size(dat)
    new_h, new_w = new_size
    for new_j=1:new_w, new_i=1:new_h
        # coordinates in original image
        x = new_j * w / new_w
        y = new_i * h / new_h
        # coordinates of 4 points to interpolate from
        x1, x2 = max(1, floor(x)), min(w, ceil(x))
        y1, y2 = max(1, floor(y)), min(h, ceil(y))
        new_dat[new_i, new_j] = interp_bilinear(dat, x, y, x1, x2, y1, y2)
    end
    return new_dat
end


function imresize{T <: Union(Real, Integer)}(dat::Array{T, 2},
                                             new_size...)
    return imresize(dat, new_size)
end

---------------

[Tim Holy]

Neat!

The easiest way to share this with others would be to submit a pull request to
Images using Pkg.submit("Images"). There are a couple of things that could be
done pretty easily to generalize this, and we could punt on anything hard---
better to have an implementation that works for many types. We could discuss
them if you decide to move forward.

Best,
--Tim

[Andreas Lobinger]

Hello colleague,

i might be wrong, but you are already using Cairo in Images. You could do all the bitmap resizing/interpolation stuff by painting on a scaled CairoSurface. This is even available in different interpolation schemes
http://www.cairographics.org/manual/cairo-cairo-pattern-t.html#cairo-filter-t

Wishing a happy day,
      Andreas

[Tim Holy]

Cairo is only used for ImageView.

--Tim

[Andrei]

@Andreas: As fas as I understand, Cairo only supports resizing when displaying, while I was more interested in interpolating data matrices (I use image as a source for machine learning algorithms and resizing as a way to enlarge/reduce feature space). Please, let me know if there's a way to extract interpolated data from Cairo surface.

[Andrei]

@Tim: generalizing code and contributing it back to Images.jl was my initial intent, but I found it much harder than I expected initially. I'm reading "Core" section [1] from Julia Images Guide right now, and it seems to address most issues I faced. Are there any other docs I should read before proceeding?

[1]: https://github.com/timholy/Images.jl/blob/master/doc/core.md

 

[Tim Holy]

Anything in doc/ could be of interest, but that covers the main ideas.

But I'm happy to help give tips if you submit the original as a pull request.
There's a balance between being able to handle anything that anyone throws at
you and getting something that works for you; if nothing else, one can check
whether the supplied image satisfies your expectations and throw an error if
not.

Best,
--Tim

[Andreas Lobinger]

Hello colleague,

you can do everything Cairo internally without painting to screen (actually the part in Cairo that accesses Screen devices is the smaller one).

So you can setup CairoImageSurfaces on data array (so something like
d = [8,4; 2,1]
d0 = reinterpret(Uint32,d);
cs = CairoImageSurface(d0,Cairo.FORMAT_ARGB32);
(written from remembering, i do not have julia available on this computer...)
Then you find your data in cs.data. The same you can do with a target surface and you can use the cs Surface via CairoPattern(cs) as source to paint. And then scaling applies.

When i'm back home, i can post an (working) example.

Wishing a happy day,
       Andreas

[Andrei]

@Andreas: could you please describe it in a bit more detail? I was able to create CairoImageSurface, but not sure how it corresponds to CairoPatterns() and drawing/resizing? (I'm pretty much to Cairo and probably looking to a wrong documentation).

[Andreas Lobinger]

An example looks like this:

using Cairo

# prepapration, an image of 2x2 pixels, an target of 8x8 pixels

d = [8 4; 2 1];
d0 = reinterpret(Uint32,d);
cs0 = Cairo.CairoImageSurface(d0,0); # 0 as FORMAT_ARGB32
show(cs0.data)
print('\n')

d2 = zeros(Uint32,8,8);
cs2 = Cairo.CairoImageSurface(d2,0);

p = Cairo.CairoPattern(cs0);

# just scale by using FILTER_NEAREST

pattern_set_filter(p,Cairo.FILTER_NEAREST);
c = Cairo.CairoContext(cs2);
scale(c,4,4);
set_source(c,p);
paint(c);
show(cs2.data)
print('\n')

# now interpolate

d3 = zeros(Uint32,8,8);
cs3 = Cairo.CairoImageSurface(d3,0);

pattern_set_filter(p,Cairo.FILTER_GOOD);
c = Cairo.CairoContext(cs3);
scale(c,4,4);
set_source(c,p);
paint(c);
show(cs3.data)
print('\n')

and gives on my command line:

julia> include("ca.jl")
Uint32[8 2
       4 1]
Uint32[8 8 8 8 2 2 2 2
       8 8 8 8 2 2 2 2
       8 8 8 8 2 2 2 2
       8 8 8 8 2 2 2 2
       4 4 4 4 1 1 1 1
       4 4 4 4 1 1 1 1
       4 4 4 4 1 1 1 1
       4 4 4 4 1 1 1 1]
Uint32[3 4 4 3 2 1 1 0
       4 6 6 5 3 2 1 1
       4 6 6 5 3 2 1 1
       4 5 5 4 3 2 1 1
       3 4 4 3 2 1 1 0
       2 3 4 3 2 1 0 0
       2 3 3 2 1 1 0 0
       1 2 2 1 1 0 0 0]

But this is just how to handle the data. For interpreting the pixels correctly you need to read little bit about the colormodel and pixel formats.

[Andrei]

@Andreas: thanks and sorry for late reply.

Using your code, I created testable function that works almost 10x times faster then my naive implementation! Right now, though, it rotates image 90 degrees to the left for some reason I don't really understand, but I believe it's fixable.

Here's a code for (not fully correct) function:

function imresize_cairo(dat::Array{Uint32, 2}, new_size::(Int, Int))
    cs = CairoImageSurface(dat, 0)
    new_dat = zeros(Uint32, new_size)
    new_cs = CairoImageSurface(new_dat, 0)
    pat = CairoPattern(cs)
    pattern_set_filter(pat, Cairo.FILTER_BILINEAR)
    c = CairoContext(new_cs)

    h, w = size(dat)
    new_h, new_w = new_size

    scale(c, new_h / h, new_w / w)
    set_source(c, pat)
    paint(c)
    return new_cs.data
end

I'm pretty satisfied with this version (or what it should turn into) both - because of its speed and possibility to use different interpolation schemas. Though, one disadvantage of using it in public package like Images.jl is that it brings additional dependency, and we all know how annoying these dependencies may be sometimes. We could implement 2 versions - one with Cairo and fast and another simple and slow - and then load appropriate function via macros. But since simple version doesn't support interpolation other than bilinear, we will have to restrict Cairo version to this type too to keep same interface. Or we can just 2 different functions, but it may be pretty confusing for new users.

Some opinions on a better approach would be helpful here.

[Tim Holy]

I bet we can meet or beat Cairo; I've never found Cairo to be terribly fast.
But of course one doesn't know until one tries.

Given that the main issue is really interpolation, it (and some of the
refactoring that's currently under discussion) seems relevant. There's some
work brewing on a new version of Grid, and being able to do this performantly
and easily can become a goal there. Grid already has more interpolation
options than Cairo does.

--Tim

[Andrei]

I've implemented Grid-based version, but it seems much slower than both - Cairo and mine. Am I using Grid the wrong way? Or does Grid just take care of more use cases?

-----

function imresize_grid{T <: Float64}(dat::Array{T, 2}, new_size::(Int, Int))
    new_dat = similar(dat, new_size)

    h, w = size(dat)
    new_h, new_w = new_size

    grid = InterpGrid(dat, BCnil, InterpLinear)

    for new_j=1:new_w, new_i=1:new_h
       # coordinates in original image  

        i = new_i * h / new_h
        j = new_j * w / new_w
        new_dat[new_i, new_j] = grid[i, j]
    end
    return new_dat
end

----

And here are performance tests (imf64 is an Array{Float64, 2}, while imu32 is the same image converted to Array{Uint32, 2}):

 julia> @time for i=1:1000 imresize(imf64, (128, 128)) end
 elapsed time: 0.697520586 seconds (131184000 bytes allocated, 15.41% gc time)

 julia> @time for i=1:1000 imresize_cairo(imu32, (128, 128)) end
 elapsed time: 0.288527903 seconds (393616000 bytes allocated, 24.78% gc time)

 julia> @time for i=1:1000 imresize_grid(imf64, (128, 128)) end
 elapsed time: 4.759103481 seconds (1443800000 bytes allocated, 14.29% gc time)

[Tim Holy]

Now, it looks like you're doing it right. I expected this, see
https://github.com/timholy/Grid.jl/pull/38. This is part of what I meant by
"refactoring" :). However, for image interpolation even further savings beyond
that pull request are possible: for example, you only need one call to floor
per pixel, because you can nest the index operations.

Here's a reasonably well-optimized prototype (a flexible implementation will
take longer to write, but should not cost performance). Note there may be some
additional optimizations possible.

First, the results:
julia> include("/tmp/resize.jl");
Cairo:
elapsed time: 0.098905264 seconds (53896192 bytes allocated, 38.25% gc time)
Julia:
elapsed time: 0.034537582 seconds (6340816 bytes allocated)

Julia is ~3 times faster.

Here's the code (I just copied your Cairo implementation):


using Images, Cairo

function imresize_julia!(resized, original)
scale1 = (size(original,1)-1)/(size(resized,1)-0.999f0)
scale2 = (size(original,2)-1)/(size(resized,2)-0.999f0)
for jr = 0:size(resized,2)-1
jo = scale2*jr
ijo = itrunc(jo)
fjo = jo - oftype(jo, ijo)
@inbounds for ir = 0:size(resized,1)-1
io = scale1*ir
iio = itrunc(io)
fio = io - oftype(io, iio)
tmp = (1-fio)*((1-fjo)*original[iio+1,ijo+1] +
fjo*original[iio+1,ijo+2])
+ fio*((1-fjo)*original[iio+2,ijo+1] +
fjo*original[iio+2,ijo+2])
resized[ir+1,jr+1] = convertsafely(eltype(resized), tmp)
end
end
resized
end
imresize_julia(original, new_size) = imresize_julia!(similar(original,
new_size), original)
convertsafely{T<:FloatingPoint}(::Type{T}, val) = convert(T, val)
convertsafely{T<:Integer}(::Type{T}, val::Integer) = convert(T, val)
convertsafely{T<:Integer}(::Type{T}, val::FloatingPoint) = itrunc(T,
val+oftype(val, 0.5))

function imresize_cairo(dat::Array{Uint32, 2}, new_size::(Int, Int))
cs = CairoImageSurface(dat, 0)
new_dat = zeros(Uint32, new_size)
new_cs = CairoImageSurface(new_dat, 0)
pat = CairoPattern(cs)
pattern_set_filter(pat, Cairo.FILTER_BILINEAR)
c = CairoContext(new_cs)
h, w = size(dat)
new_h, new_w = new_size
scale(c, new_h / h, new_w / w)
set_source(c, pat)
paint(c)
return new_cs.data
end

img = rand(0x00:0xff, 774, 512)
new_size = (3096, 2048)
imresize_cairo(convert(Array{Uint32}, img), new_size)
println("Cairo:")
@time imresize_cairo(convert(Array{Uint32}, img), new_size)
imresize_julia(img, new_size)
println("Julia:")
@time imresize_julia(img, new_size)



More advantages of doing this in Julia rather than through Cairo:
- No binary dependencies (as you stated earlier)
- Cairo is basically 8-bit, which is inadequate for many applications. Cairo's
Uint32s are actually for encoding color, and internally they get treated like
4 Uint8s---you can't get 16-bit dynamic range, for example.
- For Cairo you'd have to convert images with different datatypes into Uint32.
Measuring the performance of Cairo should include the cost of conversion. With
Julia, we can write versions that work for any input. For example, try a
Float32 image, you'll see the julia version is even faster that what I showed
above.
- We can separately implement algorithms for grayscale and color. Part of the
reason this is three times faster than Cairo's is that Cairo is basically
doing three times the work.


--Tim

[Andrei]

Wow, what a clever implementation! I'm really impressed how several simple optimizations (like inlining pixel interpolation code and moving common computations outside the loop) gave ~20 times faster implementation than mine (and ~3 times faster than Cairo)!

[Chiyuan Zhang]

Hi all,

I don't know anything about image resizing and interpolation. But I was just trying to run the code mentioned above because I need some image resizing tool in Julia. However, when saving the resized image by the procedure described above, the result does not look correct. Here is the code I'm using to test:

using Images

convertsafely{T}(::Type{T}, val) = convert(T, val)

img = imread("/tmp/foo.png")

img2 = imresize_julia(data(img), (300, 300))

imwrite(img2, "/tmp/bar.png")

Best,

Chiyuan

[Tim Holy]

I think it's just acting weird because email clients wrap lines.

With a bit of reluctance, I just threw this into Images. (You're doing cool
stuff with Mocha, and I'd hate to slow you down!) You'll have to do
Pkg.checkout("Images") to get the latest master. It's also not exported, so
use it as Images.imresize(img, newsize). When it gets a little more polished,
then I'll export it.

This will not antialias---if you're making images smaller, you'd be better off
calling restrict or imfilter_gaussian before calling imresize.

--Tim

> > On Fri, Aug 8, 2014 at 4:11 AM, Tim Holy <tim....@gmail.com <javascript:>>

[Phelipe Wesley]

This function is implemented in package Images ?

[Andrei]

@Phelipe: yes, but as Tim noticed earlier, you need to get latest dev version and use qualified name to import it:

julia> Pkg.checkout("Images")

...

julia> using Images

julia> methods(Images.imresize!)
# 1 method for generic function "imresize!":
imresize!(resized,original) at /home/slipslop/.julia/v0.3/Images/src/algorithms.jl:1018

julia> methods(Images.imresize)
# 1 method for generic function "imresize":
imresize(original,new_size) at /home/slipslop/.julia/v0.3/Images/src/algorithms.jl:1039

[Phelipe Wesley]

Ok, thank you! I will try to use again .