You probably know by now that I'll be working on an easy-to-use mask
editor for Hugin as part of GSoC 2008. I'll be using the following wiki
page to document the project and the blog to provide weekly updates.
wiki: http://wiki.panotools.org/SoC_2008_Masking_in_GUI
blog: http://maskingingui.blogspot.com/
Just to summarize the main features -
I plan to provide an editor where the users can
1. draw few rough brushstrokes to define masks (masks will be defined at
pixel level)
2. fine tune the boundary of the masks (the way it would work is
probably easier to understand from the following video:
http://research.microsoft.com/~jiansun/videos/LazySnapping_Tiny.wmv)
3. use features like zooming in/out, undo/redo, etc to assist in editing
4. automatically segment sequence of images (the initial solution may
not be very general)
The users will be able to perform these editing tasks from the preview
window, because there the images are already aligned and its easy to see
the overlapped regions.
I'd be interested to know about different requirements for creating
masks and also to have some dataset that are especially difficult.
Thanks,
Fahim
i was just wondering, would this project require close collaboration
with opengl preview implementation or not ?
will zooming use opengl ?
> I'd be interested to know about different requirements for creating
> masks and also to have some dataset that are especially difficult.
>
> Thanks,
> Fahim
--
Rich
-Fahim
I assume for masking user interface, you will want to use quite high
resolution images. However my preview will only be using small scaled
down versions at first. We will need the OpenGL preview to load higher
resolution images when you zoom in before we get masking in an OpenGL
preview.
I'm not changing the existing preview, so when you (Fahim) add a zoom
into it we should just try to get the user interface of zoom controls
consistent between the previews.
James
Yes, storing the mask is a must feature. I'll add it to the feature
list. It can be stored in a pto file as you've suggested. Also I think
it should be possible to recalculate the mask (at pixel level) from a
rough polygonal outline. This will avoid dealing with curves but I've to
see how it performs in terms of speed and accuracy. As for storing a
mask at pixel level, I think if only the alpha channel is stored as a
separate compressed file (pto file will point to it) it should be very
small.And then it can be combined using Bruno's enblend-mask tool and
fed to enblend.
> Good luck with your project
>
>
Thanks:)
> Klaus
>
-Fahim
James Legg wrote:
> 2008/5/13 Fahim Mannan <fma...@gmail.com>:
>
>> Hi,
>>
>> Rich wrote:
>> >> The users will be able to perform these editing tasks from the preview
>> >> window, because there the images are already aligned and its easy to see
>> >> the overlapped regions.
>> >>
>> >
>> > i was just wondering, would this project require close collaboration
>> > with opengl preview implementation or not ?
>> > will zooming use opengl ?
>> >
>> >
>> Good point...I'm planning to keep the GUI related part and the
>> underlying implementation independent. So, when the OpenGL
>> implementation is done it should be possible to adapt only the GUI
>> related part (and it probably shouldn't be too difficult). But at the
>> moment, some of the GUI functionalities like zooming will be done
>> without OpenGL.
>>
>> -Fahim
>>
>
> I assume for masking user interface, you will want to use quite high
> resolution images. However my preview will only be using small scaled
> down versions at first. We will need the OpenGL preview to load higher
> resolution images when you zoom in before we get masking in an OpenGL
> preview.
>
>
Yes, for masking, the zoomed in portion should have good resolution.
> I'm not changing the existing preview, so when you (Fahim) add a zoom
> into it we should just try to get the user interface of zoom controls
> consistent between the previews.
>
>
I agree...we should try to make the controls consistent once they are
implemented.
-Fahim
To give you an idea, a simple indexed TIFF bitmap mask is just a few
kB for a 20 megapixel panorama, so storing these on disk and
applying them using enblend-mask is quite efficient in terms of
space.
enblend-mask itself isn't very efficient as it has to merge each
image and mask file into a temporary TIFF file before passing it to
enblend - It also doesn't understand about cropped_TIFF which is a
major drawback.
autotrace can convert one of these bitmap masks to vector/bezier
format very accurately and quite fast, so that is one possibility.
You don't have to reinvent the wheel, if another opensource tool
does the job then you should just add it as a dependency.
If you are generating the masks inside hugin, then you should
definitely look at applying them at the nona rendering stage - This
should even speed things up as nona then doesn't have to render
masked-out areas.
As Klaus says, vector/bezier masks are quite small enough to fit in
the .pto file, here is a very complex one generated by Inkscape to
give you an idea:
<clipPath
clipPathUnits="userSpaceOnUse"
id="clipPath1">
<path
style="fill:#00ff00;fill-opacity:0.25;fill-rule:evenodd;stroke:#00ff00;stroke-width:1px;stroke-opacity:1"
d="M 4080,638 C 4069.78,658.545 4057.42,678.173 4047.33,698.911 C 4029.05,736.463 4013.4,775.639 3999.46,814.985 C 3958.45,930.756 3934.64,1052.48 3916.04,1173.58 C 3885.38,1373.11 3872.44,1574.11 3864.2,1775.67 C 3859.46,1891.73 3859.61,2007.88 3860.85,2124 C 3861.29,2165.37 3862.9,2206.65 3864.56,2248 C 3865.22,2264.38 3864.43,2281.87 3867,2298 C 3956.98,2334.92 4044.23,2378.91 4134,2416.58 C 4318.61,2494.04 4505.73,2565.4 4698,2621.58 C 5330.09,2806.26 6008.93,2840.06 6654,2707 L 6654,1070 C 6593.26,1049.45 6533.95,1023.23 6474,1000.42 C 6357.74,956.2 6240.72,914.116 6123,874.003 C 5810.98,767.687 5489.01,687.287 5162,644.282 C 4922.08,612.73 4677.95,598.451 4436,609.039 C 4355.57,612.559 4275.2,617.408 4195,625.17 C 4157.06,628.841 4118.06,637.216 4080,638 M 0,1070 L 0,2707 L 181,2662 C 180.401,2654.77 183.25,2648.99 184.356,2642 C 186.732,2626.98 189.793,2612 192.573,2597 C 200.097,2556.4 204.868,2514.98 209.525,2473.96 C 218.331,2396.41 222.39,2318.13 225.63,2240.18 C 227.059,2205.81 224.791,2170.1 229,2136 C 178.325,2027.67 170.079,1897.71 187.272,1781 C 191.533,1752.07 197.592,1723.18 205.424,1695 C 209.34,1680.91 217.581,1664.8 218.637,1650.29 C 219.726,1635.32 216.705,1619.01 215.755,1604.01 C 213.463,1567.8 211.667,1531.49 207.892,1495.41 C 201.115,1430.63 196.503,1365.75 188.844,1301 C 182.274,1245.45 172.155,1190.4 165,1135 L 116,1115.81 L 0,1070 z "
id="path1" />
</clipPath>
Again there are probably all sort of libraries available for
reading/writing and editing this sort of information.
--
Bruno
Hi,
Is it not better to store the original strokes the user gave to
calculate the mask in the pto file and regenerate the mask from that,
or is this too slow? (Pardon my ignorance...)
James
Hi Fahim,
I forgot to mention to you that it would be nice to be able to provide
the mask from an external source (probably as a png or jpg). Frequently
we modify the mask in the final image, and find we need to reprocess the
panorama. In this case there might be some mapping needed, but don't
worry about how that is done, just assume the user can say: "here is a
mask, please use it".
--dmg
--
Daniel M. German
http://turingmachine.org/
http://silvernegative.com/
dmg (at) uvic (dot) ca
replace (at) with @ and (dot) with .
This is possible. I think if its implemented properly then it shouldn't
be slow. Also doing this would be useful if the user wants the modify
the mask later.
-Fahim
Daniel M German wrote:
> Fahim> Yes, storing the mask is a must feature. I'll add it to the feature
> Fahim> list. It can be stored in a pto file as you've suggested. Also I think
> Fahim> it should be possible to recalculate the mask (at pixel level) from a
>
> Hi Fahim,
>
> I forgot to mention to you that it would be nice to be able to provide
> the mask from an external source (probably as a png or jpg). Frequently
> we modify the mask in the final image, and find we need to reprocess the
> panorama. In this case there might be some mapping needed, but don't
> worry about how that is done, just assume the user can say: "here is a
> mask, please use it".
>
>
>
ok, I'll keep that in mind when doing the design.
-Fahim
-Fahim
-Fahim
>2. fine tune the boundary of the masks (the way it would work is
>probably easier to understand from the following video:
>http://research.microsoft.com/~jiansun/videos/LazySnapping_Tiny.wmv)
Note that for the purposes of image stitching, enblend picks a seam
that is as far away from edges as possible - This means that your
mask doesn't need to be very accurate, so long as there _is_ a mask
enblend will steer away from it.
So by masking images before blending, you are doing two things:
1. Creating hints for the enblend seam placement.
2. Excluding pixels from contributing colour to the final blended
image - A few off-colour pixels some distance from the seam won't
cause any problems.
The only situation where a careful pixel accurate mask makes sense
is when it is needed to avoid having areas with no contributing
input images.
--
Bruno
very wise - the least thing you want is a dependency on another project
(and it is the same for James).
> when the OpenGL implementation is done it should be possible to adapt
> only the GUI related part (and it probably shouldn't be too difficult).
that's good forward thinking.
Yuv
Yuval> Fahim Mannan wrote:
>> I'm planning to keep the GUI related part and the
>> underlying implementation independent.
Yuval> very wise - the least thing you want is a dependency on another project
Yuval> (and it is the same for James).
>> when the OpenGL implementation is done it should be possible to adapt
> only the GUI related part (and it probably shouldn't be too difficult).
Yuval> that's good forward thinking.
I am thinking aloud here, so bear with me.
What if hugin was able to talk to other apps via some mechanism for its
preview? This would allow for a more extendable preview system that does
not depend directly on hugin. It could allow for innovation by not
having to hack directly into hugin to extend it.
What do you think about it? Pablo, what is your view on this? What if
hugin exported the parameters via a simple mechanism and allow others to
provide previews? the next step would be how to feed any change back to
hugin.
For the masks this will be enough: let hugin use a mask if available;
the mask editor will provide the masks in the same directory as the
project. The crucial part is that the mask editor knows the projection
parameters for each image, so it can remap it before the editing starts.
Another method is to create a method to support previewing plug-ins.
I think one of the challenges with hugin today is that it is difficult
to get started into hacking it. I personally do not understand a lot of
its code (particularly because I am very weak on templates).
Again, this is just thinking aloud,
BTW, Fahim, I am not sure if you are already aware of the following:
once the person edits the mask, it is possible that the image is
remapped again with different parameters. In that case the mask has to
be updated to reflect such changes. Keep this in mind. A special case of
this is the user wanting to modify the mask in the original image.
--dmg
--
interesting thinking. spinning it further - it's no longer a preview,
it's the document itself, just visualized differently.
see it in action - somewhere after minute 16 in this podcast:
<https://admin.connectpro.acrobat.com/_a791863308/p40652716/>
no, I don't think we have the 15.000 man-hours budget that they have,
but yes, I think Daniel's idea is worth being fleshed out.
Yuv
another situation that I find often is when making panos with a lot of
people. I will want to have one person in the foreground and one in the
background, and will need pixel accurate masking for that. Currently, I
do it in photoshop on the layered, warped images.
Yuv
Yuval> Bruno Postle wrote:
>> The only situation where a careful pixel accurate mask makes sense
>> is when it is needed to avoid having areas with no contributing
>> input images.
Yuval> another situation that I find often is when making panos with a lot of
Yuval> people. I will want to have one person in the foreground and one in the
Yuval> background, and will need pixel accurate masking for that. Currently, I
Yuval> do it in photoshop on the layered, warped images.
This is the situation where I think it will be useful to be able to go
back. Now that you finetuned the mask inside photoshop/gimp, feed it
back to Hugin so it can reuse it if you change the projection, the
images or the parameters used to created (lens parms, angle, fov,
etc).
--dmg
Yuval>
Looks like their stitcher has a problem with the zenith. :) We wont be
able to compete with editing a finished panorama. They don't show
manipulating an unstitched pano ( only a single layer), so it might not
be possible. If they fix their zenith problem there won't be a place
for PTAdjust any more.
I think Panotools, Hugin and associated tools has driven a bunch of the
development going into Photoshop.
I hope one day to have a 3D viewer to create all aspects of the panorama
within Hugin.
--
Jim Watters
Yahoo ID: j1vvy ymsgr:sendIM?j1vvy
jwatters @ photocreations . ca
http://photocreations.ca
Except that enblend isn't any use for this kind of montage work, it
really requires the two image areas being blended to be of the same
subject. If the two images are of different subjects then the
feathering gets very ugly.
So you are still going to have to do this 'overlapping people' work
in an image editor.
Ultimately this SoC project can only really be an interface to
exclude pixels from the (en)blending process.
--
Bruno
Bruno> On Thu 15-May-2008 at 16:30 -0400, Yuval Levy wrote:
>>
>> another situation that I find often is when making panos with a lot of
>> people. I will want to have one person in the foreground and one in the
>> background, and will need pixel accurate masking for that. Currently, I
>> do it in photoshop on the layered, warped images.
Bruno> Except that enblend isn't any use for this kind of montage work, it
Bruno> really requires the two image areas being blended to be of the same
Bruno> subject. If the two images are of different subjects then the
Bruno> feathering gets very ugly.
Bruno> So you are still going to have to do this 'overlapping people' work
Bruno> in an image editor.
I agree. Enblend does not help much in this situations, but not
everybody uses enblend in all situations.
Also, one could specify that a given region should be "added" as a final
layer after enblend has produced its output (for example, to fix this
ugly feathering.
--dmg
Sorry, not sure what you mean? Definitely it would be good on the
graphicsplanet feed:
>By the way, if you think we should have Fahim's blog aggregated by
>graphicsplanet.org, I can sort this out.
--
Bruno
>In such cases I am dreaming of the following work flow:
>1) run hugin + enblend, saving *all* masks and *all* visualisation
>images
We need a tool to assemble the various layers produced as input for
enblend and combine them with the enblend output itself into a
multilayer image that can be opened in the gimp.
See doc/batch-processing/Makefile.psd.mk for how this might work.
There used to be a tiff2xcf tool that did something similar, also
the nona TIFF_multilayer output contains crop offsets and opens fine
in the gimp - So it ought to be possible using TIFF entirely.
>2) review the seam visualisation*s* for potential problems
This is difficult, the enblend seam path is calculated for a pair of
images at a time, adjusting a path modifies the input for the next
pair of images.
>3) load the blended image and the masks with problems into gimp and
>edit the masks; masks and blended image need to be of *same* size,
>and uncropped until gimp understands cropped tiff files 4) rerun
>enblend with *all* masks loaded from file.
A gimp enblend plugin that merged two layers at a time would help,
especially if you could use the active gimp path as the blend path.
--
Bruno
-Fahim