GSoC - OpenGL Preview
James Legg
As you may or may not be aware, I'll be adding a new preview mode to
Hugin as a Google Summer of Code project, mentored by Pablo. It will
be far faster than the current preview, although not as accurate. This
will allow the preview to become "interactive". You will be able to
switch between this and the current previewer whenever you want.
For more information you can try the following web links:
The official abstracts for this year's Hugin/Panotools GSoC projects
are found here:
http://code.google.com/soc/2008/pano/about.html
I've started a new Wiki page:
http://wiki.panotools.org/SoC_2008_Project_OpenGL_Preview
Also the project proposal is still available here:
http://wiki.panotools.org/SoC_2008_student_proposals
Of course you can also ask me questions in this thread. Also in this
thread, I would like some input!
What extra features would you request for the new preview system?
We should be able to update the preview a few times each second for
changes to the panorama's projection, the image positions, and other
geometric parameters.
So, for example, we could drag the image to recentre it, instead of
clicking and waiting for a redraw. It has also been suggested I allow
the user to drag sets of images around on the preview. I think it
would be nice if there was an option to identify which image numbers
the images under the mouse pointer had, and show their outlines.
So what what would you like to have?
-James
Rich
> Hello,
>
> As you may or may not be aware, I'll be adding a new preview mode to
> Hugin as a Google Summer of Code project, mentored by Pablo. It will
> be far faster than the current preview, although not as accurate. This
> will allow the preview to become "interactive". You will be able to
> switch between this and the current previewer whenever you want.
hooray :)
...
> What extra features would you request for the new preview system?
>
> We should be able to update the preview a few times each second for
> changes to the panorama's projection, the image positions, and other
> geometric parameters.
>
> So, for example, we could drag the image to recentre it, instead of
> clicking and waiting for a redraw. It has also been suggested I allow
> the user to drag sets of images around on the preview. I think it
> would be nice if there was an option to identify which image numbers
> the images under the mouse pointer had, and show their outlines.
>
> So what what would you like to have?
some time ago i wrote several wishlist items for hugin, including
preview. found some of them.
1. "it would be nice if preview window would display image number when
mouse is moved at images (if several images overlap at that area, all
corresponding image numbers would be shown)."
Date: Mon, 13 Jun 2005 10:29:49 +0300
so that finally could become true =)
2. "a number and border for each image could be enabled with a simple
checkbox to ease finding the source image"
Date: Wed, 01 Nov 2006 09:20:54 +0200
this one might be a bit problematic to do nicely ui-wise, but it
basically expands on the first idea to not only show image borders
(configurable), but also superimpose transparent image numbers on them.
also, hovering over particular image control button could hilight it n
the preview.
3. one other, larger thing i remember...
integrated preview/control points view.
something where we can arrange images in a preview mode, then zoom in on
image pairs/groups to manage control points.
this would help a lot in relatively large multirow panos, where finding
the correct image is PAIN...
it would also allow positioning image pairs/groups more intuitively
instead of one besides the other.
now, i understand that this is outside of the project scope and probably
very, very hard to get the interface right (zooming probably should
update image quality as well !), but maybe your implementation can
happen with the idea in mind, so that future changes in that area would
be easier :)
> -James
--
Rich
Klaus
If you want to change the GUI behaviour from mouse clicking to mouse
dragging, or add a new behaviour on mouse dragging, then the left
mouse button should move the image (not introducing any local roll,
whatever one chooses for the definition) and the right mouse button
roll (around the centre).
Visualising a chosen image is important. Not only optional frame
highlighting, also somehow being able to move an image in the stack to
the front, to the back, to see how it actually contributes. Maybe some
ring front -> back -> gone -> outline -> normal in sequence -> ...
I can see myself doing a rough aligning with the quick interactive
version, then switching to fine to tweak it.
Handling the crop coordinates (Stitcher window) should be better,
mouse-moveable, the GUI on OSX does not show the outside area shaded.
Less important, maybe some magnetic behaviour when shift-moving near
the maximum image coverage (although that may need some more detailed
maths to get the coordinates right). Would save cropping post-
processing.
Foresee some editing with this window as a background.
- Define paths to crop images before blending.
- Manage (autogenerated) Control Points, i.e. mass removal from the
sky or water
Be aware of mission creep, and good luck
Klaus
J. Schneider
> So, for example, we could drag the image to recentre it, instead of
> clicking and waiting for a redraw.
move with vertical constraint, i.e. without changing the horizon.
> It has also been suggested I allow
> the user to drag sets of images around on the preview. I think it
> would be nice if there was an option to identify which image numbers
> the images under the mouse pointer had, and show their outlines.
Yes!
regards
Joachim
J. Schneider
> now, i understand that this is outside of the project scope and probably
> very, very hard to get the interface right (zooming probably should
> update image quality as well !), but maybe your implementation can
> happen with the idea in mind, so that future changes in that area would
> be easier :)
The word zooming reminds me of this: For a long thin panorama it would
be helpful to be able to zoom in because a single overlap becomes
smaller the longer the panorama gets. So zooming in to fit panorama
height into screen hight would help a lot.
regards
Joachim
James Legg
2008/5/7 J. Schneider <j-sc...@gmx.de>:
I will make zoom feature, although it's highly likely I'll use a low
image quality for all images all the time during the project. So,
until the image quality changes dynamically, I think I will just limit
it to a few zoom options, say fit panorama, fit panorama height, fit
panorama width, and fit to cropped area.
I think a "mass control point eraser" for areas with movement and
parallax errors, and a integrated control point editor are good ideas,
but I wouldn't try it all this summer. Also I think, on projects with
many control points, trying to manipulate them all at once might be
slow without sorting them into quad tree covering the whole panorama
or similar first.
I could add a feature that shows a pair of markers for each control
point, with a line between them showing the error distance. This will
show you where the worst control points are, but you'll have to use
the control point tab to remove them. I could make double clicking
somewhere covered by exactly two images, give you the control point
tab with those images selected.
I think I should stay out of editing control points directly in the
preview for this project. However, I will bear in mind that control
point tools and dynamic image quality will be added after summer, and
perhaps even do that myself.
Limiting movement to keep the horizon the same when dragging will be
fairly trivial. Editing the crop in the preview is part of the plan.
Snapping the crop is a good idea, it will require some more processing
to get exactly right, but I will consider coding it.
James
Bruno Postle
> I could make double clicking somewhere covered by exactly two
> images, give you the control point tab with those images selected.
That is a great idea, this would be very useful.
--
Bruno
Klaus
On 7 May, 16:34, "James Legg" <lankyle...@gmail.com> wrote:
> Limiting movement to keep the horizon the same when dragging will be
> fairly trivial. Editing the crop in the preview is part of the plan.
for instance:
- click: change yaw and pitch
- SHIFT-click: only change yaw
- ALT-click: only change pitch
> Snapping the crop is a good idea, it will require some more processing
> to get exactly right, but I will consider coding it.
moves by a pixel,
SHIFT-up-key snaps the active crop line to the next suitable
alignment) and that there are several potential positions one might
want to snap to. It could be all image data, intersections of the
perpendicular crop lines with image boundary, no black area at all. I
imagine most uses (but not all) would be to select a rectangle that is
fully filled with image data.
Cheers
Klaus
Erik Krause
> I will make zoom feature, although it's highly likely I'll use a low
> image quality for all images all the time during the project. So,
> until the image quality changes dynamically, I think I will just limit
> it to a few zoom options, say fit panorama, fit panorama height, fit
> panorama width, and fit to cropped area.
I suspect you'll store a smaller preview of the source images. This
could be f.e. 400 or 600 pixels (long side). With the above options
you miss to properly zoom into mosaics, hence both the the pano width
and height might be far smaller than the available preview
dimensions. A good zoom limit would be the approximat pixel
resolution of the preview images. If the preview image size could be
an option, the user could adjust the zoom in to his computers
capabilities. You'll need an efficient disk swap of course...
> I think a "mass control point eraser" for areas with movement and
> parallax errors, and a integrated control point editor are good ideas,
> but I wouldn't try it all this summer. Also I think, on projects with
> many control points, trying to manipulate them all at once might be
> slow without sorting them into quad tree covering the whole panorama
> or similar first.
I don't think, a pano previewer should deal with control points.
There are nice statistical solutions to delete bad controlpoints,
like f.e. Fulvios APClean, which pretty good help to eliminate those
controlpoints on unreliable objects.
> I could add a feature that shows a pair of markers for each control
> point, with a line between them showing the error distance. This will
> show you where the worst control points are, but you'll have to use
> the control point tab to remove them. I could make double clicking
> somewhere covered by exactly two images, give you the control point
> tab with those images selected.
This would be a nice addition, but I wouldn't focus on it.
> Limiting movement to keep the horizon the same when dragging will be
> fairly trivial. Editing the crop in the preview is part of the plan.
> Snapping the crop is a good idea, it will require some more processing
> to get exactly right, but I will consider coding it.
imagemagick -trim option does a nice automatic crop. It removes any
edges that are exactly the same color as the corner pixels...
best regards
Erik Krause
http://www.erik-krause.de
Erik Krause
> > Limiting movement to keep the horizon the same when dragging will be
> > fairly trivial. Editing the crop in the preview is part of the plan.
>
> Although that is GUI-related, maybe write up what actions should do,
> for instance:
>
> - click: change yaw and pitch
> - SHIFT-click: only change yaw
> - ALT-click: only change pitch
Why not use the photoshop logic: depress shift key while moving
limits the movement either to vertical or to horizontal dependent
whether the mouse is moved more vertical or more horizontal....
Klaus
> imagemagick -trim option does a nice automatic crop. It removes any
> edges that are exactly the same color as the corner pixels...
I still feel that the user should have some say in choosing the crop.
Imagine a stitch with all four corners black. It is an artistic choice
how to crop. Landscape, portrait, something in between? For instance,
choose vertical crop lines manually, then snap the horizontal crop
lines.
On preview change when dragging the mouse: the O3 group is non-abelian
(rotations do not commute). Shall the rotation depent on the actual
path being drawn, or only on the start point of mouse button being
pressed and held plus the current mouse location?
Same question worded differently: does one introduce image roll when
dragging the mouse along a closed loop?
Cheers
Klaus
Yuval Levy
> So what what would you like to have?
hi James. I'm coming a bit late to the party, but here is what I would like.
VISION: the OpenGL preview is the hub from where all functionalities are
linked and quickly accessible.
LIMITATION: GPU/CPU power forces some compromises for the time being.
RESULTS: this is how I would like to work in hugin.
I start with the preview window. It is empty. It just displays a grid on
a black background.
There are three levels of interaction:
- panorama
- images
- control points
PANORAMA:
In panorama mode I can interactively frame / straighten the panorama. I
can set the field of view by moving two sliders and it all appears in
real time. I can click somewhere and drag my sphere around, like in
Google Earth. I can change projection, and I have access to
straightening functionalities.
I can add images by opening them from the menu or dragging them from the
desktop. They are immediately displayed in panospace, on the grid.
I can select a picture by clicking on it with the left mouse button, and
it highlights its borders. If there is more than one picture under my
mouse pointer the pointer indicates it to me by changing shape, and I
can rotate among the selected pictures with the space bar.
IMAGES:
When the picture I want to work on is highlighted, I can drag it around
in pano space, and I can roll it with the right-left arrow keys.
When I use the right mouse key, a context menu appear that gives me
access to a form with the image parameters (current "images tab"); to
the camera and lens parameters; to the crop; to the control points list;
all filtered for the specific image.
When I press the enter key, I enter into control point mode.
CONTROL POINTS:
In control points mode, all CPs of the selected image are displayed, as
well as their counterparts on the neighboring images.
Since zooming to 1:1 is not yet an option (GPU/CPU/RAM limits), a double
click on a CP calls up the CP editor tab with the selected pair, for
editing.
Everything is visual, but hovering over a CP can call up an information
pop-up with the distance and other statistically relevant information
currently displayed in the CP tab.
ADDITIONAL DIMENSIONS:
besides the above three levels of "zoom", I would like to be able to
switch to different displays mode. for example when I am working on an
HDR pano, I'd like the ability to increase and decrease "brightness",
selecting which range of the spectrum is visible. the already existing
subtractive view of the overlap should be available too.
I hope this helps.
Yuv
Fahim Mannan, mentored by Daniel M German
Pablo d'Angelo
On 7 Mai, 10:06, "James Legg" <lankyle...@gmail.com> wrote:
> Hello,
I like the testcase, but could you also do a similar experiment with a
single, ordinary image placed at the zenith?
As you initially suggested, the "Remapping Vertices" does indeed looks
nicer and introduces less geometric distortions, and is probably the
way to go, especially if you find a good way to handle the zenith and
nadir.
> Also the project proposal is still available here:http://wiki.panotools.org/SoC_2008_student_proposals
>
> Of course you can also ask me questions in this thread. Also in this
> thread, I would like some input!
>
> What extra features would you request for the new preview system?
list of features you'd like to implement during this Summer of Code?
ciao
Pablo
James Legg
>
> Hi James,
>
> On 7 Mai, 10:06, "James Legg" <lankyle...@gmail.com> wrote:
> > Hello,
>
> >
> > I've started a new Wiki page:http://wiki.panotools.org/SoC_2008_Project_OpenGL_Preview
>
> I like the testcase, but could you also do a similar experiment with a
> single, ordinary image placed at the zenith?
I've just done so, and added it to the Wiki.
>
> As you initially suggested, the "Remapping Vertices" does indeed looks
> nicer and introduces less geometric distortions, and is probably the
> way to go, especially if you find a good way to handle the zenith and
> nadir.
>
I think the worst thing in mapping using texture coordinates came from
a lack negative coordinates. Two of the edges are really distorted
(the ones on the negative side). Nona seems to give only unsigned 16
bit integers for the coordinate images, and I need the coordinates
that are just off all sides of the input image, so having the negative
points clamped to 0 messed it up.
If you ignore the faces around those two edges, the image was similar
in accuracy to remapping with vertices, however remapping with
vertices does make the curves prettier, so I guess I should look for a
good solution for the images crossing the zenith and nadir using it.
:-)
> So after you got some feedback on the desired features, do you have a
> list of features you'd like to implement during this Summer of Code?
Here's the features I would like to implement this summer:
- Images used as textures, with and without photometric correction
(option in the preview window).
- Texture size set automatically to try and get the total size of
textures roughly constant (except for when there is more texture
memory than is needed for all the input images), keeping the texture
size of each image relative to its field of view.
- Simple real time exposure and white balance correction for the
textures when not using photometric correction, just by scaling the
colour components of a low dynamic range texture.
- Support for all reprojections accessible via the GUI.
- Mesh resolution choosen automatically to get the total number of
faces roughly constant, and roughly proportional to the space they
cover.
- An 'Identify Images' mode, that shows where the image is when you
hover over its visibility button, drawing it at the top of the stack;
and what the image numbers are when you hove over the images, complete
with highlighted borders of the images. I'll adjust the details of
this one if I get a better suggestion.
- Drag to recentre the panorama (optionally constrained in one direction)
- Drag to roll the panorama
- A dragable cropping rectangle
- Realtime update when adjusting field of view
- View with one image subtracted from the others
- Preliminary support for zooming
- (hopefully) a mode where you can drag images to rearrange them.
- (hopefully) double click where two images overlap to open them up in
the control point tab.
- (possibly - I'm building a preview not a control point editor!) Show
the control points.
- and anything else that can be agreed upon here (and that I have time for!).
James
breic
extremely slow preview window.
I don't think adding control points to this window makes much sense,
since it will just become slower. To me, dragging the images is also
not important. However, it would be great to be able to show the
numbers of all images, either centered or in the upper-left-hand
corner of the image. It would also be helpful to see the connectivity
graph. Two images should be connected with an edge if they overlap in
the projection (or at least if they share a control point), and the
color/thickness/dashing of the edge should indicate the quality of the
match between that pair (many/few/no control points, high/low quality
of control points). Clicking on or around an edge would take you to
the correct tab.
On May 13, 7:42 am, "James Legg" <lankyle...@gmail.com> wrote:
> 2008/5/12 Pablo d'Angelo <pablo.dang...@web.de>:
James Legg
>
> This is a great idea. Probably the biggest problem with Hugin is the
> extremely slow preview window.
>
> I don't think adding control points to this window makes much sense,
> since it will just become slower. To me, dragging the images is also
> not important.
I think the dragging mode may be useful for cases where automatic
aligning failed, although it is not a priority.
If I do implement a dragging mode, from a usability perspective would
it be good to limit dragging to unconnected sets of images? Hugin
already tells you the unconnected components in the assistant tab. It
would be easier to connect these components if you could align them
independently, then drag each component to about the right position,
then jump to the control point tab for each major overlap and connect
them. After that, all aligning should be done with the optimiser, so
I don't think it is necessary to drag images once they are connected
by control points.
> However, it would be great to be able to show the
> numbers of all images, either centered or in the upper-left-hand
> corner of the image. It would also be helpful to see the connectivity
> graph. Two images should be connected with an edge if they overlap in
> the projection (or at least if they share a control point), and the
> color/thickness/dashing of the edge should indicate the quality of the
> match between that pair (many/few/no control points, high/low quality
> of control points). Clicking on or around an edge would take you to
> the correct tab.
A connectivity graph may be useful, although I'm not sure about how I
should intuitively and unobtrusively mark the edges with something
that shows how good the control points are and how many of them there
are.
I think perhaps a function that simply shades overlapping areas with
no control points (or maybe less than 4 of them) would help without
providing a confusing amount of information. Where your control points
are fine, there will be no change in the display; and the biggest
overlaps without control points would stand out, which is great since
the bigger the overlap the more important having control points across
it is.
James
Yuval Levy
> If I do implement a dragging mode, from a usability perspective would
> it be good to limit dragging to unconnected sets of images?
* default: drag the whole panorama
* with a modifier down: drag all images that are connected to the image
being dragged
* with another modifier key down: drag only the individual image
Yuv
Bruno Postle
>
>If I do implement a dragging mode, from a usability perspective would
>it be good to limit dragging to unconnected sets of images? Hugin
>already tells you the unconnected components in the assistant tab.
It would be useful to limit dragging so that each connected set is
moved as a group, then:
* if all your images are connected together, the entire set gets
dragged around, equivalent to panning around the panorama.
* if there are no connections, each image can be dragged
independently.
>A connectivity graph may be useful, although I'm not sure about how I
>should intuitively and unobtrusively mark the edges with something
>that shows how good the control points are and how many of them there
>are.
Some experiments here:
http://flickr.com/photos/36383814@N00/2321777799/
http://flickr.com/photos/sbprzd/2324271815/
I think this is something for a further project, in fact it could be
entirely orthogonal to the preview.
>I think perhaps a function that simply shades overlapping areas with
>no control points (or maybe less than 4 of them) would help without
>providing a confusing amount of information.
Yes, having overlapping photos with no control points is a real
problem - and identifying these image pairs is currently very
difficult.
--
Bruno
Pablo dAngelo
>2008/5/15 breic <ben.re...@gmail.com>:
>
>
>>This is a great idea. Probably the biggest problem with Hugin is the
>>extremely slow preview window.
>>
>>I don't think adding control points to this window makes much sense,
>>since it will just become slower. To me, dragging the images is also
>>not important.
>>
>>
>
>I think the dragging mode may be useful for cases where automatic
>aligning failed, although it is not a priority.
>
>
I'm not quite sure why people want that feature either, the only area
where it would help is probably with hard to match areas in the sky etc.
For this the precision of the preview would be accectable. Another
option would be to roughly sort out the overlap and then running a
control point generator only on the actually neighbouring images (for
panoramas with many images that sorting phase also becomes tiresome,
though...)
>>However, it would be great to be able to show the
>>numbers of all images, either centered or in the upper-left-hand
>>corner of the image. It would also be helpful to see the connectivity
>>graph. Two images should be connected with an edge if they overlap in
>>the projection (or at least if they share a control point), and the
>>color/thickness/dashing of the edge should indicate the quality of the
>>match between that pair (many/few/no control points, high/low quality
>>of control points). Clicking on or around an edge would take you to
>>the correct tab.
>>
>>
>
>A connectivity graph may be useful, although I'm not sure about how I
>should intuitively and unobtrusively mark the edges with something
>that shows how good the control points are and how many of them there
>are.
>
>
Autopano Pro has such a graph with some information about the control
points overlayed on the panorama. Such a display should work fine with
ordinary panoramas, but will probably fail with stacked (HDR) panoramas.
(I haven't tried Autopano Pro with such panoramas yet, so I don't know
if it works differently there or not).
>I think perhaps a function that simply shades overlapping areas with
>no control points (or maybe less than 4 of them) would help without
>providing a confusing amount of information. Where your control points
>are fine, there will be no change in the display; and the biggest
>overlaps without control points would stand out, which is great since
>the bigger the overlap the more important having control points across
>it is.
>
>
Sounds very reasonable as well, although there might be problems with
overlaps between multiple images (at least when selecting images etc).
Actually, the "difference" mode in the hugin preview provides a roughly
similar information, but does lack interactivity for editing such as
opening the control point editor for the overlap in question.
ciao
Pablo
Yuval Levy
> James Legg schrieb:
>
>> 2008/5/15 breic <ben.re...@gmail.com>:
>>
>>
>>> This is a great idea. Probably the biggest problem with Hugin is the
>>> extremely slow preview window.
>>>
>>> I don't think adding control points to this window makes much sense,
>>> since it will just become slower. To me, dragging the images is also
>>> not important.
>>>
>>>
>> I think the dragging mode may be useful for cases where automatic
>> aligning failed, although it is not a priority.
>>
>>
>
> I'm not quite sure why people want that feature either, the only area
> where it would help is probably with hard to match areas in the sky etc.
it depends on what this window is used for.
if the window is used for *preview only*, then I only need dragging of
the full spherical panorama.
if the window is used for *editing* too, then I would like the ability
to select a single image or a group of images, that I would like to drag
along latitude/longitude, rotate, and even zoom.
With that ability, I would have used hugin for a project to which I
contributed some panoramas that is scheduled to go live next week. The
challenge was to embed historical pictures in a full spherical panorama.
We ended up with much less potential inserts than initially planned for
a number of reasons. And for those inserts left I ended up working in
PTgui with the (fast) preview on one screen and the image parameters on
the other screen, fiddling with the image parameters until the insert
was almost right, and doing the final touch ups in Photoshop. Sure, I
could do it with hugin and the OpenGL Preview once available, but better
than keying in the individual parameters would be for me if I could
drag,rotate,zoom in the OpenGL Preview window, without changing focus
and keys all the time.
and I still think that in the long term (beyond a GSoC project) the
window should become the hub from which all the process steps are
accessed - in which case control points are necessary in there (and I
don't think they should slow things significantly, like hotspots do not
slow down things significantly when viewing a panorama).
I found that with my typical project, pre-positioning the images before
running the optimizer can make a difference between success and failure,
and having access to pre-position the images on the preview window is
more intuitive and easy to use than keying in y/p/r values. Using a
template is surely more efficient, but the template has to be created
initially, and even with templates, sometimes there is the odd insert
that call for a little bit of dragging.
Yuv