GSoC - Masking in GUI
Fahim Mannan
You probably know by now that I'll be working on an easy-to-use mask
editor for Hugin as part of GSoC 2008. I'll be using the following wiki
page to document the project and the blog to provide weekly updates.
wiki: http://wiki.panotools.org/SoC_2008_Masking_in_GUI
blog: http://maskingingui.blogspot.com/
Just to summarize the main features -
I plan to provide an editor where the users can
1. draw few rough brushstrokes to define masks (masks will be defined at
pixel level)
2. fine tune the boundary of the masks (the way it would work is
probably easier to understand from the following video:
http://research.microsoft.com/~jiansun/videos/LazySnapping_Tiny.wmv)
3. use features like zooming in/out, undo/redo, etc to assist in editing
4. automatically segment sequence of images (the initial solution may
not be very general)
The users will be able to perform these editing tasks from the preview
window, because there the images are already aligned and its easy to see
the overlapped regions.
I'd be interested to know about different requirements for creating
masks and also to have some dataset that are especially difficult.
Thanks,
Fahim
Rich
> Hello everyone,
>
> You probably know by now that I'll be working on an easy-to-use mask
> editor for Hugin as part of GSoC 2008. I'll be using the following wiki
> page to document the project and the blog to provide weekly updates.
>
> wiki: http://wiki.panotools.org/SoC_2008_Masking_in_GUI
> blog: http://maskingingui.blogspot.com/
>
> Just to summarize the main features -
> I plan to provide an editor where the users can
> 1. draw few rough brushstrokes to define masks (masks will be defined at
> pixel level)
> 2. fine tune the boundary of the masks (the way it would work is
> probably easier to understand from the following video:
> http://research.microsoft.com/~jiansun/videos/LazySnapping_Tiny.wmv)
> 3. use features like zooming in/out, undo/redo, etc to assist in editing
> 4. automatically segment sequence of images (the initial solution may
> not be very general)
>
> The users will be able to perform these editing tasks from the preview
> window, because there the images are already aligned and its easy to see
> the overlapped regions.
i was just wondering, would this project require close collaboration
with opengl preview implementation or not ?
will zooming use opengl ?
> I'd be interested to know about different requirements for creating
> masks and also to have some dataset that are especially difficult.
>
> Thanks,
> Fahim
--
Rich
Fahim Mannan
Rich wrote:
>> The users will be able to perform these editing tasks from the preview
>> window, because there the images are already aligned and its easy to see
>> the overlapped regions.
>>
>
> i was just wondering, would this project require close collaboration
> with opengl preview implementation or not ?
> will zooming use opengl ?
>
>
underlying implementation independent. So, when the OpenGL
implementation is done it should be possible to adapt only the GUI
related part (and it probably shouldn't be too difficult). But at the
moment, some of the GUI functionalities like zooming will be done
without OpenGL.
-Fahim
Klaus
Every now and then I find myself revisiting a stitch project, and I
find it quite annoying in case I had to edit a mask in between hugin
remapping and enblend blending, as it means either to save the big
TIFF file as well, or editing the mask again.
Ideally the mask would be saved into the pto file. Personally I think
saving a polygon outline (or a Bezier curve) usually requires less
Bytes than a pixel-describing trace, given that I would not like the
pto file to bloat.
Good luck with your project
Klaus
James Legg
I assume for masking user interface, you will want to use quite high
resolution images. However my preview will only be using small scaled
down versions at first. We will need the OpenGL preview to load higher
resolution images when you zoom in before we get masking in an OpenGL
preview.
I'm not changing the existing preview, so when you (Fahim) add a zoom
into it we should just try to get the user interface of zoom controls
consistent between the previews.
James
Fahim Mannan
Yes, storing the mask is a must feature. I'll add it to the feature
list. It can be stored in a pto file as you've suggested. Also I think
it should be possible to recalculate the mask (at pixel level) from a
rough polygonal outline. This will avoid dealing with curves but I've to
see how it performs in terms of speed and accuracy. As for storing a
mask at pixel level, I think if only the alpha channel is stored as a
separate compressed file (pto file will point to it) it should be very
small.And then it can be combined using Bruno's enblend-mask tool and
fed to enblend.
> Good luck with your project
>
>
Thanks:)
> Klaus
>
-Fahim
Fahim Mannan
James Legg wrote:
> 2008/5/13 Fahim Mannan <fma...@gmail.com>:
>
>> Hi,
>>
>> Rich wrote:
>> >> The users will be able to perform these editing tasks from the preview
>> >> window, because there the images are already aligned and its easy to see
>> >> the overlapped regions.
>> >>
>> >
>> > i was just wondering, would this project require close collaboration
>> > with opengl preview implementation or not ?
>> > will zooming use opengl ?
>> >
>> >
>> Good point...I'm planning to keep the GUI related part and the
>> underlying implementation independent. So, when the OpenGL
>> implementation is done it should be possible to adapt only the GUI
>> related part (and it probably shouldn't be too difficult). But at the
>> moment, some of the GUI functionalities like zooming will be done
>> without OpenGL.
>>
>> -Fahim
>>
>
> I assume for masking user interface, you will want to use quite high
> resolution images. However my preview will only be using small scaled
> down versions at first. We will need the OpenGL preview to load higher
> resolution images when you zoom in before we get masking in an OpenGL
> preview.
>
>
Yes, for masking, the zoomed in portion should have good resolution.
> I'm not changing the existing preview, so when you (Fahim) add a zoom
> into it we should just try to get the user interface of zoom controls
> consistent between the previews.
>
>
I agree...we should try to make the controls consistent once they are
implemented.
-Fahim
Bruno Postle
To give you an idea, a simple indexed TIFF bitmap mask is just a few
kB for a 20 megapixel panorama, so storing these on disk and
applying them using enblend-mask is quite efficient in terms of
space.
enblend-mask itself isn't very efficient as it has to merge each
image and mask file into a temporary TIFF file before passing it to
enblend - It also doesn't understand about cropped_TIFF which is a
major drawback.
autotrace can convert one of these bitmap masks to vector/bezier
format very accurately and quite fast, so that is one possibility.
You don't have to reinvent the wheel, if another opensource tool
does the job then you should just add it as a dependency.
If you are generating the masks inside hugin, then you should
definitely look at applying them at the nona rendering stage - This
should even speed things up as nona then doesn't have to render
masked-out areas.
As Klaus says, vector/bezier masks are quite small enough to fit in
the .pto file, here is a very complex one generated by Inkscape to
give you an idea:
<clipPath
clipPathUnits="userSpaceOnUse"
id="clipPath1">
<path
style="fill:#00ff00;fill-opacity:0.25;fill-rule:evenodd;stroke:#00ff00;stroke-width:1px;stroke-opacity:1"
d="M 4080,638 C 4069.78,658.545 4057.42,678.173 4047.33,698.911 C 4029.05,736.463 4013.4,775.639 3999.46,814.985 C 3958.45,930.756 3934.64,1052.48 3916.04,1173.58 C 3885.38,1373.11 3872.44,1574.11 3864.2,1775.67 C 3859.46,1891.73 3859.61,2007.88 3860.85,2124 C 3861.29,2165.37 3862.9,2206.65 3864.56,2248 C 3865.22,2264.38 3864.43,2281.87 3867,2298 C 3956.98,2334.92 4044.23,2378.91 4134,2416.58 C 4318.61,2494.04 4505.73,2565.4 4698,2621.58 C 5330.09,2806.26 6008.93,2840.06 6654,2707 L 6654,1070 C 6593.26,1049.45 6533.95,1023.23 6474,1000.42 C 6357.74,956.2 6240.72,914.116 6123,874.003 C 5810.98,767.687 5489.01,687.287 5162,644.282 C 4922.08,612.73 4677.95,598.451 4436,609.039 C 4355.57,612.559 4275.2,617.408 4195,625.17 C 4157.06,628.841 4118.06,637.216 4080,638 M 0,1070 L 0,2707 L 181,2662 C 180.401,2654.77 183.25,2648.99 184.356,2642 C 186.732,2626.98 189.793,2612 192.573,2597 C 200.097,2556.4 204.868,2514.98 209.525,2473.96 C 218.331,2396.41 222.39,2318.13 225.63,2240.18 C 227.059,2205.81 224.791,2170.1 229,2136 C 178.325,2027.67 170.079,1897.71 187.272,1781 C 191.533,1752.07 197.592,1723.18 205.424,1695 C 209.34,1680.91 217.581,1664.8 218.637,1650.29 C 219.726,1635.32 216.705,1619.01 215.755,1604.01 C 213.463,1567.8 211.667,1531.49 207.892,1495.41 C 201.115,1430.63 196.503,1365.75 188.844,1301 C 182.274,1245.45 172.155,1190.4 165,1135 L 116,1115.81 L 0,1070 z "
id="path1" />
</clipPath>
Again there are probably all sort of libraries available for
reading/writing and editing this sort of information.
--
Bruno
James Legg
Hi,
Is it not better to store the original strokes the user gave to
calculate the mask in the pto file and regenerate the mask from that,
or is this too slow? (Pardon my ignorance...)
James
Daniel M German
Fahim> Yes, storing the mask is a must feature. I'll add it to the feature
Fahim> list. It can be stored in a pto file as you've suggested. Also I think
Fahim> it should be possible to recalculate the mask (at pixel level) from a
Hi Fahim,
I forgot to mention to you that it would be nice to be able to provide
the mask from an external source (probably as a png or jpg). Frequently
we modify the mask in the final image, and find we need to reprocess the
panorama. In this case there might be some mapping needed, but don't
worry about how that is done, just assume the user can say: "here is a
mask, please use it".
--dmg
--
Daniel M. German
http://turingmachine.org/
http://silvernegative.com/
dmg (at) uvic (dot) ca
replace (at) with @ and (dot) with .
Fahim Mannan
This is possible. I think if its implemented properly then it shouldn't
be slow. Also doing this would be useful if the user wants the modify
the mask later.
-Fahim
Fahim Mannan
Daniel M German wrote:
> Fahim> Yes, storing the mask is a must feature. I'll add it to the feature
> Fahim> list. It can be stored in a pto file as you've suggested. Also I think
> Fahim> it should be possible to recalculate the mask (at pixel level) from a
>
> Hi Fahim,
>
> I forgot to mention to you that it would be nice to be able to provide
> the mask from an external source (probably as a png or jpg). Frequently
> we modify the mask in the final image, and find we need to reprocess the
> panorama. In this case there might be some mapping needed, but don't
> worry about how that is done, just assume the user can say: "here is a
> mask, please use it".
>
>
>
ok, I'll keep that in mind when doing the design.
-Fahim
Fahim Mannan
> autotrace can convert one of these bitmap masks to vector/bezier
> format very accurately and quite fast, so that is one possibility.
> You don't have to reinvent the wheel, if another opensource tool
> does the job then you should just add it as a dependency.
>
>
there's already tool for doing something its better to use it.
> If you are generating the masks inside hugin, then you should
> definitely look at applying them at the nona rendering stage - This
> should even speed things up as nona then doesn't have to render
> masked-out areas.
>
>
doing any design.
> As Klaus says, vector/bezier masks are quite small enough to fit in
> the .pto file, here is a very complex one generated by Inkscape to
> give you an idea:
>
>
and also see if there are any available libraries for doing that. But
I'll keep it (ie. understanding the exact mechanism and looking for
existing libraries) as a low priority task for now and focus on having a
design that allows some flexibility.
-Fahim
Tom Sharpless
Easy pre-blend masking is what spherical panographers need most
urgently! If this project works it will make hugin the premier tool
for that.
I agree that the ability to save masks as separate files and apply
them to other images has great potential value. So I, too, would urge
you to adopt the text mode Bezier spline as a mask import/export
format, even if you do not use vector outlines on screen. It is a
very widely understood form and the basis of many successful products
going back to PostScript and the original Apple QuickDraw. Even
Photoshop can import and export it.
I personally find it much easier to create masks with on-screen
Bezier vectors than with bitmap painting tools, so I hope you will at
least consider that as an editing mode as well. There is plenty of
good code for this, though I can't recommend any package from personal
experience. The Inkscape source is one place to look, but it is
pretty complex. I suspect there are small simple C packages, too,
since this is a fairly old technology. Any PostScript interpreter
will have code for rasterizing Bezier splines, and I believe there is
a Gnu package that can go the other way.
Anyhow, thanks much for undertaking this.
Good Luck, Tom
Fahim Mannan
> Hooorraaay Fahim!!
>
> Easy pre-blend masking is what spherical panographers need most
> urgently! If this project works it will make hugin the premier tool
> for that.
>
>
> I agree that the ability to save masks as separate files and apply
> them to other images has great potential value. So I, too, would urge
> you to adopt the text mode Bezier spline as a mask import/export
> format, even if you do not use vector outlines on screen. It is a
> very widely understood form and the basis of many successful products
> going back to PostScript and the original Apple QuickDraw. Even
> Photoshop can import and export it.
>
>
> I personally find it much easier to create masks with on-screen
> Bezier vectors than with bitmap painting tools, so I hope you will at
> least consider that as an editing mode as well. There is plenty of
> good code for this, though I can't recommend any package from personal
> experience. The Inkscape source is one place to look, but it is
> pretty complex. I suspect there are small simple C packages, too,
> since this is a fairly old technology. Any PostScript interpreter
> will have code for rasterizing Bezier splines, and I believe there is
> a Gnu package that can go the other way.
>
>
come in handy for some cases. I'll definitely look at Inkscape and also
if someone knows of any similar library for doing this that would be great.
-Fahim
Bruno Postle
>
>Just to summarize the main features -
>I plan to provide an editor where the users can
>1. draw few rough brushstrokes to define masks (masks will be defined at
>pixel level)
>2. fine tune the boundary of the masks (the way it would work is
>probably easier to understand from the following video:
>http://research.microsoft.com/~jiansun/videos/LazySnapping_Tiny.wmv)
Note that for the purposes of image stitching, enblend picks a seam
that is as far away from edges as possible - This means that your
mask doesn't need to be very accurate, so long as there _is_ a mask
enblend will steer away from it.
So by masking images before blending, you are doing two things:
1. Creating hints for the enblend seam placement.
2. Excluding pixels from contributing colour to the final blended
image - A few off-colour pixels some distance from the seam won't
cause any problems.
The only situation where a careful pixel accurate mask makes sense
is when it is needed to avoid having areas with no contributing
input images.
--
Bruno
Yuval Levy
> I'm planning to keep the GUI related part and the
> underlying implementation independent.
very wise - the least thing you want is a dependency on another project
(and it is the same for James).
> when the OpenGL implementation is done it should be possible to adapt
> only the GUI related part (and it probably shouldn't be too difficult).
that's good forward thinking.
Yuv
Daniel M German
Hi Yuv, Fahim,
Yuval> Fahim Mannan wrote:
>> I'm planning to keep the GUI related part and the
>> underlying implementation independent.
Yuval> very wise - the least thing you want is a dependency on another project
Yuval> (and it is the same for James).
>> when the OpenGL implementation is done it should be possible to adapt
> only the GUI related part (and it probably shouldn't be too difficult).
Yuval> that's good forward thinking.
I am thinking aloud here, so bear with me.
What if hugin was able to talk to other apps via some mechanism for its
preview? This would allow for a more extendable preview system that does
not depend directly on hugin. It could allow for innovation by not
having to hack directly into hugin to extend it.
What do you think about it? Pablo, what is your view on this? What if
hugin exported the parameters via a simple mechanism and allow others to
provide previews? the next step would be how to feed any change back to
hugin.
For the masks this will be enough: let hugin use a mask if available;
the mask editor will provide the masks in the same directory as the
project. The crucial part is that the mask editor knows the projection
parameters for each image, so it can remap it before the editing starts.
Another method is to create a method to support previewing plug-ins.
I think one of the challenges with hugin today is that it is difficult
to get started into hacking it. I personally do not understand a lot of
its code (particularly because I am very weak on templates).
Again, this is just thinking aloud,
BTW, Fahim, I am not sure if you are already aware of the following:
once the person edits the mask, it is possible that the image is
remapped again with different parameters. In that case the mask has to
be updated to reflect such changes. Keep this in mind. A special case of
this is the user wanting to modify the mask in the original image.
--dmg
--
Yuval Levy
> I am thinking aloud here, so bear with me.
interesting thinking. spinning it further - it's no longer a preview,
it's the document itself, just visualized differently.
see it in action - somewhere after minute 16 in this podcast:
<https://admin.connectpro.acrobat.com/_a791863308/p40652716/>
no, I don't think we have the 15.000 man-hours budget that they have,
but yes, I think Daniel's idea is worth being fleshed out.
Yuv
Yuval Levy
> The only situation where a careful pixel accurate mask makes sense
> is when it is needed to avoid having areas with no contributing
> input images.
another situation that I find often is when making panos with a lot of
people. I will want to have one person in the foreground and one in the
background, and will need pixel accurate masking for that. Currently, I
do it in photoshop on the layered, warped images.
Yuv
Daniel M German
Yuval> Bruno Postle wrote:
>> The only situation where a careful pixel accurate mask makes sense
>> is when it is needed to avoid having areas with no contributing
>> input images.
Yuval> another situation that I find often is when making panos with a lot of
Yuval> people. I will want to have one person in the foreground and one in the
Yuval> background, and will need pixel accurate masking for that. Currently, I
Yuval> do it in photoshop on the layered, warped images.
This is the situation where I think it will be useful to be able to go
back. Now that you finetuned the mask inside photoshop/gimp, feed it
back to Hugin so it can reuse it if you change the projection, the
images or the parameters used to created (lens parms, angle, fov,
etc).
--dmg
Yuval>
Jim Watters
> it's no longer a preview,
> it's the document itself, just visualized differently.
>
> see it in action - somewhere after minute 16 in this podcast:
> <https://admin.connectpro.acrobat.com/_a791863308/p40652716/>
>
together. I wished Adobe got it in the previous version. Corel Painter
could edit a transformed view for about seven years. Although just
rotation.
Looks like their stitcher has a problem with the zenith. :) We wont be
able to compete with editing a finished panorama. They don't show
manipulating an unstitched pano ( only a single layer), so it might not
be possible. If they fix their zenith problem there won't be a place
for PTAdjust any more.
I think Panotools, Hugin and associated tools has driven a bunch of the
development going into Photoshop.
I hope one day to have a 3D viewer to create all aspects of the panorama
within Hugin.
--
Jim Watters
Yahoo ID: j1vvy ymsgr:sendIM?j1vvy
jwatters @ photocreations . ca
http://photocreations.ca
Bruno Postle
>
>another situation that I find often is when making panos with a lot of
>people. I will want to have one person in the foreground and one in the
>background, and will need pixel accurate masking for that. Currently, I
>do it in photoshop on the layered, warped images.
Except that enblend isn't any use for this kind of montage work, it
really requires the two image areas being blended to be of the same
subject. If the two images are of different subjects then the
feathering gets very ugly.
So you are still going to have to do this 'overlapping people' work
in an image editor.
Ultimately this SoC project can only really be an interface to
exclude pixels from the (en)blending process.
--
Bruno
Daniel M German
Bruno> On Thu 15-May-2008 at 16:30 -0400, Yuval Levy wrote:
>>
>> another situation that I find often is when making panos with a lot of
>> people. I will want to have one person in the foreground and one in the
>> background, and will need pixel accurate masking for that. Currently, I
>> do it in photoshop on the layered, warped images.
Bruno> Except that enblend isn't any use for this kind of montage work, it
Bruno> really requires the two image areas being blended to be of the same
Bruno> subject. If the two images are of different subjects then the
Bruno> feathering gets very ugly.
Bruno> So you are still going to have to do this 'overlapping people' work
Bruno> in an image editor.
I agree. Enblend does not help much in this situations, but not
everybody uses enblend in all situations.
Also, one could specify that a given region should be "added" as a final
layer after enblend has produced its output (for example, to fix this
ugly feathering.
--dmg
prokoudine
> blog:http://maskingingui.blogspot.com/
I'd love to see this blog added to the planet ;-) Bruno? ;-)
By the way, if you think we should have Fahim's blog aggregated by
graphicsplanet.org, I can sort this out.
Alexandre
Klaus
I have been musing about brush strokes and polygon traces (or Bezier
traces). Clicking and dragging on the polygon traces would move
handles, create new ones. When brushing areas, starting away from a
polygon trace, this action would push away the polygon trace once the
brush is approaching the polygon trace. Somewhat like a rope on the
floor that one shifts by brushing into it. Broad brush strokes would
give a coarse outline, that could be refined by finer action or
smaller brush.
Another thing. Fahim, it is probably not what you are going to work
on, but it is masks, so I want to throw it into the general
discussion. You all know that enblend has an optimising seam algorithm
which usually works great. But sometimes tweaking the seam would
improve it. In such cases I am dreaming of the following work flow: 1)
run hugin + enblend, saving *all* masks and *all* visualisation images
2) review the seam visualisation*s* for potential problems 3) load the
blended image and the masks with problems into gimp and edit the
masks; masks and blended image need to be of *same* size, and
uncropped until gimp understands cropped tiff files 4) rerun enblend
with *all* masks loaded from file.
Cheers
Klaus
Bruno Postle
>On 13 май, 09:31, Fahim Mannan <fman...@gmail.com> wrote:
>
>> blog:http://maskingingui.blogspot.com/
>I'd love to see this blog added to the planet ;-) Bruno? ;-)
Sorry, not sure what you mean? Definitely it would be good on the
graphicsplanet feed:
>By the way, if you think we should have Fahim's blog aggregated by
>graphicsplanet.org, I can sort this out.
--
Bruno
Bruno Postle
>In such cases I am dreaming of the following work flow:
>1) run hugin + enblend, saving *all* masks and *all* visualisation
>images
We need a tool to assemble the various layers produced as input for
enblend and combine them with the enblend output itself into a
multilayer image that can be opened in the gimp.
See doc/batch-processing/Makefile.psd.mk for how this might work.
There used to be a tiff2xcf tool that did something similar, also
the nona TIFF_multilayer output contains crop offsets and opens fine
in the gimp - So it ought to be possible using TIFF entirely.
>2) review the seam visualisation*s* for potential problems
This is difficult, the enblend seam path is calculated for a pair of
images at a time, adjusting a path modifies the input for the next
pair of images.
>3) load the blended image and the masks with problems into gimp and
>edit the masks; masks and blended image need to be of *same* size,
>and uncropped until gimp understands cropped tiff files 4) rerun
>enblend with *all* masks loaded from file.
A gimp enblend plugin that merged two layers at a time would help,
especially if you could use the active gimp path as the blend path.
--
Bruno
Fahim Mannan
Daniel M German wrote:
> BTW, Fahim, I am not sure if you are already aware of the following:
> once the person edits the mask, it is possible that the image is
> remapped again with different parameters. In that case the mask has to
> be updated to reflect such changes. Keep this in mind. A special case of
> this is the user wanting to modify the mask in the original image.
>
>
>
user input and recompute the mask. Another option is to have a spline
representation and remap that.
-Fahim
Klaus
> On Fri 16-May-2008 at 11:09 -0700, Klaus wrote:
>
> >In such cases I am dreaming of the following work flow:
> >1) run hugin + enblend, saving *all* masks and *all* visualisation
> >images
>
> We need a tool to assemble the various layers produced as input for
> enblend and combine them with the enblend output itself into a
> multilayer image that can be opened in the gimp.
would benefit from some editing, and the --visualize would tell you
where the problems might be.
> See doc/batch-processing/Makefile.psd.mk for how this might work.
>
> There used to be a tiff2xcf tool that did something similar, also
> the nona TIFF_multilayer output contains crop offsets and opens fine
> in the gimp - So it ought to be possible using TIFF entirely.
>
> >2) review the seam visualisation*s* for potential problems
>
> This is difficult, the enblend seam path is calculated for a pair of
> images at a time, adjusting a path modifies the input for the next
> pair of images.
blend seam finder would not be evoked in the rerun as all the blend
masks would be loaded from file.
> >3) load the blended image and the masks with problems into gimp and
> >edit the masks; masks and blended image need to be of *same* size,
> >and uncropped until gimp understands cropped tiff files 4) rerun
> >enblend with *all* masks loaded from file.
>
> A gimp enblend plugin that merged two layers at a time would help,
> especially if you could use the active gimp path as the blend path.
useful for a high-quality cut-and-paste.
Although I would reckon that most optimised seams would be ok for
usage, so step 3 would once affect a few masks, and the blending step
(4) would require less interaction, allowing for a coffee break while
enblend does a dozen blending steps.
Cheers
Klaus