Improvements to the fast preview window
Darko Makreshanski
I have some ideas which I would like to present and get feedback of you
for a project. It basically includes the "Zooming for Fast Preview" and
other improvements to "Fast Preview"
My ideas are mostly concerned for users that are just beginning with
hugin, and users with not much knowledge of map projections and panorama
creation.
* one goal is to attract more users by giving a fancy and more
user-friendly interface to the preview.
* another is to give some features, that might be useful to most of
you as well
Basically currently the fast preview is a openGL version of the old
preview with some features and some drawbacks (the texture quality), and
I believe that OpenGL can provide much more user-friendliness to the
preview.
Here are the problems to the preview that I thing currently are the
biggest problem for beginners:
- it's completely unintuitive that in the move/drag section, users
actually rotate the panorama. The problem of this is that the 3D
rotation space commands are represented in a 2D plane, with the vertical
axis for two dimensions.
- when there are several groups of images, users can handle these
independently, however this is also unintuitive, since for a user to
handle a specific group he must click/drag the group, and this combined
with the previous problem is a pain.
- currently with this preview to get a larger overview of the
panorama, you must change the field of view or the projection mode.
- people with no knowledge of the types of projections, are only
left to experiment with the projections and look at the result without
actually having a good visualization of the distortions that are occurring
Also something I didn't like is that in many cases I couldn't spot areas
where there was no image data, because the background was black and
blended with parts of the image.
What I propose is the following:
First the current mode of preview will remain as it is (with some
improvements) and will be called a '2D Projection' mode where the user
would just use to modify the final result.
Apart from this mode I also propose two additional modes (3D panosphere,
and combination of 3D panosphere and 2D projection):
First, some features that would be included in both 'Projection' and
'Panosphere' mode:
- the user could zoom in/out and image resolutions would be
dynamically increased for more detail. (the idea on the wiki)
- better manipulation of image groups where user could select a
group or the whole panorama and adjust accordingly
- adjustable and very distinctive background for both modes, to spot
the areas without image data
- a interleaving colorful grid will be displayed to examine distortion
1. A '3D Panosphere' mode.
- I read the 'Next GUI' discussion, and I noticed there were some
thoughts on this already.
In this mode the user would basically look into a 3D sphere mapped
with the images, with option to look at the sphere either from inside or
outside.
The purpose of the sphere mode is that it is the basic representation of
what the panorama actually is, and I believe it is the most intuitive
representation.
The benefits of this mode:
- primarily to distinguish between looking at the output and looking
at an overview of the panorama.
- the most intuitive and most exact preview of the panorama (in
terms of distortions)
- in here the 3D rotation adjustments would actually make sense and
would be intuitive.
- the layout submode in this mode would also make a lot more sense
- a very intuitive and eye-candy preview for new users
Some of the features that would be included in panosphere mode:
- a look at the panosphere either from outside or inside (all
features available in both modes)
* from the inside, the viewpoint would be fixed to the center
of the sphere and adjustable would be rotation of the camera and field
of view (zoom in/out)
* from the outside, the viewpoint would move around a larger
virtual sphere, and would be faced always to the center of the sphere
(also adjustable FOV)
- the camera adjustment would be done with the mouse or
keyboard (for mouse drag to rotate, mouse wheel to zoom in/out)
- a layout mode (same as it is currently) with small images and
their connections
- a set of 3 interactive circles drawn around the sphere, which
could be dragged with the mouse to rotate the whole panorama or a group
of images (as it is done in 3dsmax) (also shown in [1])
- a possibility to choose a central point of projection on the sphere
2. Combination of 3D panosphere and 2D projection:
This mode is mostly for people to test the projections and understand
them easily, or to work with the panosphere and see the results directly
on the projection
- it will contain both the panosphere and the projection, which can
be either:
* the panosphere and the projection in separate windows
* the panosphere and the projection in one window (one opengl
scene)
- both the panosphere and the projection will have an interleaving
colorful grid which will correspond to each other (basically the grid of
the sphere would be projected as well)
* the purpose of this would be to have an overview of the
distortions on the projections and to easily understand all projections
- ofcourse by rotating the panosphere the projection changes as well.
about the two submodes:
1. panosphere and projection in separate windows (or in same window
but in different canvases)
* this is more useful, and I believe will be useful to
everybody, even experienced people.
2. panosphere and projection in same window
* this might be used to animate the projections, which would be
awesome and very attractive, however not very useful
* I have done a simple scene with a basic projection in opengl [1]
* just a note: in that scene, the sphere's grid doesn't
have interpolated color as the projection
So that's pretty much my idea. It might be too much or not enough for a
gsoc project. Next, I would have to look more into the code of the
previewer, and make a decent estimation of the time needed. In any case
I would really like to see all of that included in the previewer.
I am really eager to see the responses for this, and whether you would
like to see this in Hugin.
Best regards,
Darko
James Legg
These have been asked for before, so they would be welcome additions.
> - adjustable and very distinctive background for both modes, to spot
> the areas without image data
This could be useful.
> - a interleaving colorful grid will be displayed to examine distortion
That might help people understand the projections, if they are aware
what the grid means.
> 1. A '3D Panosphere' mode.
> - I read the 'Next GUI' discussion, and I noticed there were some
> thoughts on this already.
>
> In this mode the user would basically look into a 3D sphere mapped
> with the images, with option to look at the sphere either from inside or
> outside.
>
> The purpose of the sphere mode is that it is the basic representation of
> what the panorama actually is, and I believe it is the most intuitive
> representation.
>
> The benefits of this mode:
> - primarily to distinguish between looking at the output and looking
> at an overview of the panorama.
> - the most intuitive and most exact preview of the panorama (in
> terms of distortions)
I wouldn't say it was more exact. You will still be mapping a spherical
image to a 2D display. It is also not the most intuitive way for linear
panoramas.
> - in here the 3D rotation adjustments would actually make sense and
> would be intuitive.
> - the layout submode in this mode would also make a lot more sense
> - a very intuitive and eye-candy preview for new users
>
> Some of the features that would be included in panosphere mode:
> - a look at the panosphere either from outside or inside (all
> features available in both modes)
> * from the inside, the viewpoint would be fixed to the center
> of the sphere and adjustable would be rotation of the camera and field
> of view (zoom in/out)
Is this the same as using rectilinear projection?
> * from the outside, the viewpoint would move around a larger
> virtual sphere, and would be faced always to the center of the sphere
> (also adjustable FOV)
Is this similar to orthographic projection? However there is a bug with
orthographic projection in the fast preview: it doesn't clip the images
on the back of the sphere.
I don't think using a small field of view in this projection will be any
less confusing. I think when looking at the sphere from the outside, the
whole sphere should always be visible, and any gap in the images should
show a differently shaded version of the images on the other side.
> - the camera adjustment would be done with the mouse or
> keyboard (for mouse drag to rotate, mouse wheel to zoom in/out)
> - a layout mode (same as it is currently) with small images and
> their connections
> - a set of 3 interactive circles drawn around the sphere, which
> could be dragged with the mouse to rotate the whole panorama or a group
> of images (as it is done in 3dsmax) (also shown in [1])
> - a possibility to choose a central point of projection on the sphere
How will the user interface for this work when inside the sphere? How
would you rotate the camera and rotate some images?
> 2. Combination of 3D panosphere and 2D projection:
> This mode is mostly for people to test the projections and understand
> them easily, or to work with the panosphere and see the results directly
> on the projection
> - it will contain both the panosphere and the projection, which can
> be either:
> * the panosphere and the projection in separate windows
> * the panosphere and the projection in one window (one opengl
> scene)
> - both the panosphere and the projection will have an interleaving
> colorful grid which will correspond to each other (basically the grid of
> the sphere would be projected as well)
> * the purpose of this would be to have an overview of the
> distortions on the projections and to easily understand all projections
> - ofcourse by rotating the panosphere the projection changes as well.
I think the best way to understand most projections is with a good
diagram. The panosphere view could include some of the properties of the
projection in a diagram. For example, something like [1] could be used
for cylindrical projections. For diagrams to make sense, the view must
be from outside the panosphere; and it must be at an angle to the centre
of the projection used, since the shapes will not be seen head on.
Unfortunately this could be a lot of work, as each projection would
require its own diagram drawing code (and some projections would still
look weird). I think just covering rectilinear, equiangular, and
cylindrical projections would be enough to cover the main use cases
though.
>
> about the two submodes:
> 1. panosphere and projection in separate windows (or in same window
> but in different canvases)
> * this is more useful, and I believe will be useful to
> everybody, even experienced people.
> 2. panosphere and projection in same window
> * this might be used to animate the projections, which would be
> awesome and very attractive, however not very useful
If by animate you mean show the panorama being unwrapped from the sphere
to form the output image, this could be useful. It should help
understanding of the projections.
It would be easiest to do this with an already stitched panorama.
Perhaps you could take a preview of the projected panorama, copy it into
an OpenGL texture and animate that?
> * I have done a simple scene with a basic projection in opengl [1]
> * just a note: in that scene, the sphere's grid doesn't
> have interpolated color as the projection
>
>
> So that's pretty much my idea. It might be too much or not enough for a
> gsoc project. Next, I would have to look more into the code of the
> previewer, and make a decent estimation of the time needed. In any case
> I would really like to see all of that included in the previewer.
There are some ideas that are unrelated to the rest, for example the
customizable display of transparent areas. After you've got a good
discussion here, pick what you think are the most important parts for
your project plan.
> I am really eager to see the responses for this, and whether you would
> like to see this in Hugin.
An extension (maybe a bit much for your summer of code project) would be
to use the XYZ properties in 3D space instead of just the spherical
projection. To visualize this well you would need to be able to move the
camera around as well as rotate it.
-James
Darko Makreshanski
Thanks a lot for your reply.
First, just let me clarify something for everyone, which I should have
done in the previous mail.
The 3D panosphere mode would have very little in common with the current
projection mode. So it will not use the current projection techniques to
display the result (rectilinear for inside and orthographic for outside
look) but rather create a 3D sphere in OpenGL, and then texturized 3D
meshes for each image which would lie on the panosphere. So it will not
use the 2D "GL_PROJECTION" mode, but the 3D "GL_MODELVIEW". This means
that the remappers would need to be refactored or reimplemented to
return a 3D texturized mesh.
Thus, this mode wouldn't represent the output, but would represent an
overview of the panorama in 3D space, so the two modes could be also
named as "Panorama overview" and "Output preview"
the other comments are below:
James Legg wrote:
> On Sat, 2010-03-20 at 17:15 +0100, Darko Makreshanski wrote:
>
>> - a interleaving colorful grid will be displayed to examine distortion
>>
>
> That might help people understand the projections, if they are aware
> what the grid means.
>
>
My idea was that this same grid would rendered on the panosphere as
well, and also the grids would be correspond to each other. This means
that at each area of the projection the color of the grid in that area
would be the same as the color of the grid in the corresponding area on
the panosphere. This I believe would automatically explain the grid on
the projection, and will help understand the projection.
Also another way of providing means for understanding the projection
would be to draw lines from points on the sphere's grid to the
corresponding points on the projection's grid. But I like the other
method better, because it's easier to handle with colors, rather than
lines. And also in this method the projection and the panosphere would
have to be rendered in one window.
>> 1. A '3D Panosphere' mode.
>> - I read the 'Next GUI' discussion, and I noticed there were some
>> thoughts on this already.
>>
>> In this mode the user would basically look into a 3D sphere mapped
>> with the images, with option to look at the sphere either from inside or
>> outside.
>>
>> The purpose of the sphere mode is that it is the basic representation of
>> what the panorama actually is, and I believe it is the most intuitive
>> representation.
>>
>> The benefits of this mode:
>> - primarily to distinguish between looking at the output and looking
>> at an overview of the panorama.
>> - the most intuitive and most exact preview of the panorama (in
>> terms of distortions)
>>
>
> I wouldn't say it was more exact. You will still be mapping a spherical
> image to a 2D display. It is also not the most intuitive way for linear
> panoramas.
>
Yes, you are right. My point was that as you can freely move around the
sphere, people would not percept this as a projection of a sphere, but
rather than a presentation of a 3D sphere.
You are right about the linear panoramas.
But I believe the linear panoramas are a totally different type of
panorama, and they are represented by a plane.
In that case, as some changes are required for processing, this would be
a totally different mode of creating panoramas (I believe this was Dev
Ghosh's GSOC project last year, but I don't know in which state it is now)
>
>> - in here the 3D rotation adjustments would actually make sense and
>> would be intuitive.
>> - the layout submode in this mode would also make a lot more sense
>> - a very intuitive and eye-candy preview for new users
>>
>> Some of the features that would be included in panosphere mode:
>> - a look at the panosphere either from outside or inside (all
>> features available in both modes)
>> * from the inside, the viewpoint would be fixed to the center
>> of the sphere and adjustable would be rotation of the camera and field
>> of view (zoom in/out)
>>
>
> Is this the same as using rectilinear projection?
>
yes, basically would give similar view
>
>> * from the outside, the viewpoint would move around a larger
>> virtual sphere, and would be faced always to the center of the sphere
>> (also adjustable FOV)
>>
>
> Is this similar to orthographic projection? However there is a bug with
> orthographic projection in the fast preview: it doesn't clip the images
> on the back of the sphere.
>
Yes, it basically will look like the orthographic projection. As I have
explained above, the system wouldn't use existing projections, because
it will model the sphere in 3D
> I don't think using a small field of view in this projection will be any
> less confusing. I think when looking at the sphere from the outside, the
> whole sphere should always be visible, and any gap in the images should
> show a differently shaded version of the images on the other side.
>
yes, you are right, the point of the FOV changing is about zooming
in/out and checking on some details on the images
>
>> - the camera adjustment would be done with the mouse or
>> keyboard (for mouse drag to rotate, mouse wheel to zoom in/out)
>> - a layout mode (same as it is currently) with small images and
>> their connections
>> - a set of 3 interactive circles drawn around the sphere, which
>> could be dragged with the mouse to rotate the whole panorama or a group
>> of images (as it is done in 3dsmax) (also shown in [1])
>> - a possibility to choose a central point of projection on the sphere
>>
>
> How will the user interface for this work when inside the sphere? How
> would you rotate the camera and rotate some images?
>
Well, you could display the same set of 3 circles in front of the user,
just smaller in size (compared to the circles on the outside). Of course
these circles will change orientation themselves as the user rotates the
camera around.
In any case, I think the inside look is not very useful for providing
overview as the outside view, but it shouldn't be a big problem to
implement it if we have the outside view.
I was also thinking of adding a 4th circle (also for outside view) when
a group of images is selected. This 4th circle would have the normal
that passes through its center also pass through the center of gravity
of the group selected and will also rotate the group by the axis that
passes through the center of gravity of the group.
>
>> 2. Combination of 3D panosphere and 2D projection:
>> This mode is mostly for people to test the projections and understand
>> them easily, or to work with the panosphere and see the results directly
>> on the projection
>> - it will contain both the panosphere and the projection, which can
>> be either:
>> * the panosphere and the projection in separate windows
>> * the panosphere and the projection in one window (one opengl
>> scene)
>> - both the panosphere and the projection will have an interleaving
>> colorful grid which will correspond to each other (basically the grid of
>> the sphere would be projected as well)
>> * the purpose of this would be to have an overview of the
>> distortions on the projections and to easily understand all projections
>> - ofcourse by rotating the panosphere the projection changes as well.
>>
>
> I think the best way to understand most projections is with a good
> diagram. The panosphere view could include some of the properties of the
> projection in a diagram. For example, something like [1] could be used
> for cylindrical projections. For diagrams to make sense, the view must
> be from outside the panosphere; and it must be at an angle to the centre
> of the projection used, since the shapes will not be seen head on.
>
Yes, this was basically the idea with the corresponding grids on the
projection and the sphere
> Unfortunately this could be a lot of work, as each projection would
> require its own diagram drawing code (and some projections would still
> look weird). I think just covering rectilinear, equiangular, and
> cylindrical projections would be enough to cover the main use cases
> though.
>
Why is this so? I was just thinking of projecting the grid of the sphere
the same way all the other projections are generated.
>
>>
>> about the two submodes:
>> 1. panosphere and projection in separate windows (or in same window
>> but in different canvases)
>> * this is more useful, and I believe will be useful to
>> everybody, even experienced people.
>> 2. panosphere and projection in same window
>> * this might be used to animate the projections, which would be
>> awesome and very attractive, however not very useful
>>
>
> If by animate you mean show the panorama being unwrapped from the sphere
> to form the output image, this could be useful. It should help
> understanding of the projections.
>
> It would be easiest to do this with an already stitched panorama.
> Perhaps you could take a preview of the projected panorama, copy it into
> an OpenGL texture and animate that?
>
yes, kind of. It doesn't have to be stitched, but it would definitely be
easier. Maybe, this would be combined with some implementation of a
'Postview' mode as it was discussed in the GUI discussion.
>
>> * I have done a simple scene with a basic projection in opengl [1]
>> * just a note: in that scene, the sphere's grid doesn't
>> have interpolated color as the projection
>>
>>
>> So that's pretty much my idea. It might be too much or not enough for a
>> gsoc project. Next, I would have to look more into the code of the
>> previewer, and make a decent estimation of the time needed. In any case
>> I would really like to see all of that included in the previewer.
>>
>
> There are some ideas that are unrelated to the rest, for example the
> customizable display of transparent areas. After you've got a good
> discussion here, pick what you think are the most important parts for
> your project plan.
>> I am really eager to see the responses for this, and whether you would
>> like to see this in Hugin.
>>
>
> An extension (maybe a bit much for your summer of code project) would be
> to use the XYZ properties in 3D space instead of just the spherical
> projection. To visualize this well you would need to be able to move the
> camera around as well as rotate it.
>
Yes, I didn't take into consideration that hugin can also estimate
translation.
All I can think of are two models. That is the sphere and plane model.
A generalized model which would show irregular cloud-shaped model of the
panorama is currently a science-fiction for me :)
The closest thing I can think of is just a layout mode in which images
wouldn't originate from the same point as in the panosphere and, the
origins of each image (the apparent camera position) would be rendered
as well, e.g. in a form of a camera :)
About the translation, I actually have no idea how it currently affects
the projections. I will need to look more into the code, since I
couldn't understand the effects of it on the preview.
James Legg
> The 3D panosphere mode would have very little in common with the current
> projection mode. So it will not use the current projection techniques to
> display the result (rectilinear for inside and orthographic for outside
> look) but rather create a 3D sphere in OpenGL, and then texturized 3D
> meshes for each image which would lie on the panosphere. So it will not
> use the 2D "GL_PROJECTION" mode, but the 3D "GL_MODELVIEW".
GL_PROJECTION and GL_MODELVIEW are enumerations used for glMatrixMode,
which sets the matrix the other matrix manipulation functions use.
The projection matrix and the model-view matrix are used simultaneously
to draw anything, 2D or 3D.
> This means
> that the remappers would need to be refactored or reimplemented to
> return a 3D texturized mesh.
You should set up an equirectangular projection that is 360 degrees by
180 degrees, then either:
* Make a variation of MeshManager::MeshInfo::CompileList() [1]
that uses the 2D coordinates as angles in spherical coordinates,
then converts it the spherical coordinates to 3D Cartesian ones,
or
* Make something remapper-like that produces the 3D mesh directly.
Since spherical coordinates from an equirectangular projection
do not exhibit weird behaviour at poles or seams, this should
work without duplicating too much effort. It could be more
efficient than going several layers of the fast preview. This
would also mean you have more control of the layout mode.
> You are right about the linear panoramas.
> But I believe the linear panoramas are a totally different type of
> panorama, and they are represented by a plane.
> In that case, as some changes are required for processing, this would be
> a totally different mode of creating panoramas (I believe this was Dev
> Ghosh's GSOC project last year, but I don't know in which state it is now)
It has been merged into Panotools trunk. It provides the X, Y, and Z
variables in Hugin trunk. All variables are used together so we can get
lens correction and perspective correction in linear panoramas for free.
The only difference in processing linear panoramas and spherical ones is
which variables you optimize (X, Y, Z instead of yaw, pitch, roll).
> > I think the best way to understand most projections is with a good
> > diagram. The panosphere view could include some of the properties of the
> > projection in a diagram. For example, something like [1] could be used
> > for cylindrical projections. For diagrams to make sense, the view must
> > be from outside the panosphere; and it must be at an angle to the centre
> > of the projection used, since the shapes will not be seen head on.
> >
> Yes, this was basically the idea with the corresponding grids on the
> projection and the sphere
> > Unfortunately this could be a lot of work, as each projection would
> > require its own diagram drawing code (and some projections would still
> > look weird). I think just covering rectilinear, equiangular, and
> > cylindrical projections would be enough to cover the main use cases
> > though.
> >
> Why is this so? I was just thinking of projecting the grid of the sphere
> the same way all the other projections are generated.
I was thinking about visualising the mathematics of the projection:
* Rectilinear projects the sphere onto a plane, so it would be
nice if I could see a plane for the panorama's field of view on
the diagram. The way the plane changes size when adjusting the
field of view would make it clear that a field of view of 180
degrees or more is impossible.
* Cylindrical projection projects the sphere onto a cylinder which
is then unrolled, so I would like to see the part of cylinder
used for the panorama's field of view drawn around it. Similarly
to the rectilinear projection, I would see that the vertical
field of view cannot be near 180 degrees because the cylinder
would get too tall; but I can wrap around the whole sphere
horizontally.
* Equirectangular projection could just shade the field of view on
the sphere directly. The curved top will help identify where the
distortions come from, and it would be obvious I can get the
whole sphere by increasing the field of view in both directions.
* Biplane works like a rectilinear projection, but should show two
planes instead of one.
* Triplane is similar, but with three planes.
I don't know a good way of representing every projection. I think
several would end up with a generic cylinder diagram or nothing special
at all.
> > An extension (maybe a bit much for your summer of code project) would be
> > to use the XYZ properties in 3D space instead of just the spherical
> > projection. To visualize this well you would need to be able to move the
> > camera around as well as rotate it.
> >
> Yes, I didn't take into consideration that hugin can also estimate
> translation.
>
> All I can think of are two models. That is the sphere and plane model.
> A generalized model which would show irregular cloud-shaped model of the
> panorama is currently a science-fiction for me :)
I think restricting to sphere or plane models is enough. Technically
panoramas could work in a mixture of the two models, but picking one or
the other should be good enough for display purposes.
> The closest thing I can think of is just a layout mode in which images
> wouldn't originate from the same point as in the panosphere and, the
> origins of each image (the apparent camera position) would be rendered
> as well, e.g. in a form of a camera :)
I think it is possible to draw the images in one sphere, but mark an
image's camera position in the Outside Panosphere View's identify mode.
Drawing all the cameras at once will be confusing since the images
should overlap lots.
> About the translation, I actually have no idea how it currently affects
> the projections. I will need to look more into the code, since I
> couldn't understand the effects of it on the preview.
I don't really understand what it does either!
-James
Darko Makreshanski
> On Sun, 2010-03-21 at 22:44 +0100, Darko Makreshanski wrote:
>
>
>> This means
>> that the remappers would need to be refactored or reimplemented to
>> return a 3D texturized mesh.
>>
>
> You should set up an equirectangular projection that is 360 degrees by
> 180 degrees, then either:
> * Make a variation of MeshManager::MeshInfo::CompileList() [1]
> that uses the 2D coordinates as angles in spherical coordinates,
> then converts it the spherical coordinates to 3D Cartesian ones,
> or
> * Make something remapper-like that produces the 3D mesh directly.
> Since spherical coordinates from an equirectangular projection
> do not exhibit weird behaviour at poles or seams, this should
> work without duplicating too much effort. It could be more
> efficient than going several layers of the fast preview. This
> would also mean you have more control of the layout mode.
>
Basically I was thinking of projecting all of the images separately,
each with its center as the center of projection. Then to convert each
projection into a 3D mesh, and finally join the meshes based on the yaw,
pitch, roll parameters. This would be to avoid any problems with the
poles, however I am not sure if it would be necessary.
animation part of describing the projections.
This approach would require rendering the projection and panosphere in
the same opengl scene (canvas).
What I was thinking of, apart from this, is also to provide a connection
between the overview and the projection while they are in separate
canvases. The alignment of the canvases would be adjustable in the sense
that the user can choose to see either one at a time or both aligned
either vertically or horizontally. This I believe should be the default
mode of preview of the panorama.
I mentioned the grid as the solution to this connection. The grid would
be perfect because it will clearly show the distortions on the
projection, and with the interleaving colors it will also provide
uniqueness to every part on the grid.
>>> An extension (maybe a bit much for your summer of code project) would be
>>> to use the XYZ properties in 3D space instead of just the spherical
>>> projection. To visualize this well you would need to be able to move the
>>> camera around as well as rotate it.
>>>
>>>
>> Yes, I didn't take into consideration that hugin can also estimate
>> translation.
>>
>> All I can think of are two models. That is the sphere and plane model.
>> A generalized model which would show irregular cloud-shaped model of the
>> panorama is currently a science-fiction for me :)
>>
>
> I think restricting to sphere or plane models is enough. Technically
> panoramas could work in a mixture of the two models, but picking one or
> the other should be good enough for display purposes.
>
Yes, just the problem remains of how to represent yaw and pitch in a
planar model, and x,y,z translation in the sphere model. About the
sphere model I proposed something below.
>> About the translation, I actually have no idea how it currently affects
>> the projections. I will need to look more into the code, since I
>> couldn't understand the effects of it on the preview.
>>
>
> I don't really understand what it does either!
>
>
The only logical explanation I could think of, which is consistent with
the result in the preview, is that the images with non zero x,y,z
parameters are represented on a sphere with a shifted center, however
the projection is still done from the original center.
This actually could also be represented in the overview, in which case
the overview would no longer be a sphere but rather images projected on
spheres with different centers.
The grid that I was talking about would also be suitable to explain the
effects of the x,y,z parameters as well.
This might be tricky to explain, but I'll try:
A grid 'B' is a grid that needs to be displayed on an image with some
x,y,z parameters that is projected on a sphere 'B' with a shifted center
of x,y,z.
This grid 'B' would actually be a mapping of another grid 'A'. The grid
'A' is a grid which lies on the sphere 'A' with its center at zero x,y,z.
The grid 'A' would actually represent the portion of the sphere 'A' that
covers the image as it is seen from zero x,y,z (center of sphere 'A')
If my model for what happens with the x,y,z parameters is right, then
this proposed solution I think would be perfect to visualize an overview
which would explain the effects of x,y,z as well.
> -James
>
> [1]
> http://hugin.svn.sourceforge.net/viewvc/hugin/hugin/trunk/src/hugin1/hugin/MeshManager.cpp?revision=5076&view=markup#l_176
>
Cheers,
Darko
James Legg
> Yes, the equirectangular would be most suitable to convert to a 3D mesh.
> Basically I was thinking of projecting all of the images separately,
> each with its center as the center of projection. Then to convert each
> projection into a 3D mesh, and finally join the meshes based on the yaw,
> pitch, roll parameters. This would be to avoid any problems with the
> poles, however I am not sure if it would be necessary.
Have you thought about how the z-order of the images would work in the
outside sphere view?
The overlapping images in Hugin eventually get merged with enblend. In
the previews they are just drawn over the top of each other for speed.
In a 3D sphere, the images will still overlap. If you continue to draw
them over the top of each other, bits of the images on the back half of
the sphere may be drawn over the front of the sphere.
I don't think you could use OpenGL's depth buffer and depth test
directly, because the overlapping images would z-fight.
-James
Darko Makreshanski
I was so far assuming to use the depth buffer because it is most
appropriate for 3D scenes.
In this case if we don't use the depth buffer, then we will have to some
how avoid drawing the back of the faces.
But in this case the problem is also that if we use the thing that I
explained about shifting the spheres for the translation, then the scene
becomes a mess.
For example some image may have a combination of translation and
position such that it will displayed inside or outside the central
sphere. In some cases images might cross each other.
As for the depth buffer, the problem with the z-fight I believe can not
be solved, because of the existence of translations. If we have
projections on spheres with different centers, then z-fight is
inevitable, and the only thing we can do is fiddle with the resolution
of the meshes.
Lukáš Jirkovský
> result in the preview, is that the images with non zero x,y,z parameters are
> represented on a sphere with a shifted center, however the projection is
> still done from the original center.
> This actually could also be represented in the overview, in which case the
> overview would no longer be a sphere but rather images projected on spheres
> with different centers.
If I undestood it correctly it's like you just explained for XYZ. It
should create several panospheres with shifted center. The projection
camera is moved to the point from which images has good overlap. The
problem (which affects mainly end user) is that there is only one
camera position for which the overlap is good. In other words – if you
create panorama using XYZ you can't move camera at all (there may be
some case in which it doesn't matter).
The Tr parameters Dev Ghosh proposed does it in a bit different way.
It creates one panosphere but the images are not placed directly on
the sphere as a tangential plane but rather image is a bit rotated (so
it's not more tangent plane). I've no idea how Tr could affect
projections and the preview window
Lukas
T. Modes
> should create several panospheres with shifted center. The projection
> camera is moved to the point from which images has good overlap. The
> problem (which affects mainly end user) is that there is only one
> camera position for which the overlap is good. In other words – if you
> create panorama using XYZ you can't move camera at all (there may be
> some case in which it doesn't matter).
The Tr[xyz] parameter reproject the image out of the center of the
panosphere using one plane (not a sphere). See Pablos mail on panotool-
dev-list http://article.gmane.org/gmane.comp.graphics.panotools.devel/1701
The image with non-zero Trxyz parmeters is projected on one plane and
then from this plane reprojected to the panosphere. So it is
understandable that this only works for panos with hfov <180°.
Thomas
Darko Makreshanski
>> If I undestood it correctly it's like you just explained for XYZ. It
>> should create several panospheres with shifted center. The projection
>> camera is moved to the point from which images has good overlap. The
>> problem (which affects mainly end user) is that there is only one
>> create panorama using XYZ you can't move camera at all (there may be
>> some case in which it doesn't matter).
>>
>
> The Tr[xyz] parameter reproject the image out of the center of the
> panosphere using one plane (not a sphere). See Pablos mail on panotool-
> dev-list http://article.gmane.org/gmane.comp.graphics.panotools.devel/1701
> The image with non-zero Trxyz parmeters is projected on one plane and
> then from this plane reprojected to the panosphere. So it is
>
> Thomas
>
>
Thank you, this makes more sense, as it converts to planar and then back
to spherical model.
So now actually for each image that has nonzero Tr[xyz] parameter, the
projection mode is automatically switched to planar.
Also in this case, adjusting rotation for groups in the fast preview
should also rotate the position of the shifted origins, since now any
rotation breaks the correspondence.
Darko
Lukáš Jirkovský
> panosphere using one plane (not a sphere). See Pablos mail on panotool-
> dev-list http://article.gmane.org/gmane.comp.graphics.panotools.devel/1701
> The image with non-zero Trxyz parmeters is projected on one plane and
> then from this plane reprojected to the panosphere. So it is
> understandable that this only works for panos with hfov <180°.
>
> Thomas
>
I read this some time ago and it's really well written. What I posted
was how I understand it works in 3D space of panosphere. Apparently I
was wrong, because I thought the reprojection to a different plane is
done in one step with reprojection to the panosphere.
Have a nice day,
Lukas
Darko Makreshanski
> T. Modes wrote:
>>
>> The Tr[xyz] parameter reproject the image out of the center of the
>> panosphere using one plane (not a sphere). See Pablos mail on panotool-
>> dev-list
>> http://article.gmane.org/gmane.comp.graphics.panotools.devel/1701
>> The image with non-zero Trxyz parmeters is projected on one plane and
>> then from this plane reprojected to the panosphere. So it is
>> understandable that this only works for panos with hfov <180�.
>>
>> Thomas
>>
> should also rotate the position of the shifted origins, since now any
> rotation breaks the correspondence.
>
with changing orientation (except roll) in linear panorama mode. This
means that in this mode, yaw and pitch should only be left to the
optimizer. On the other hand translation would be very convenient to
adjust in the fast preview for a linear panorama.
Thus, it seems that there is a need of some kind of switch between a
linear mode and normal mode.
Darko
Darko Makreshanski
the problem is now simplified.
Also, as I understood now how the Tr parameters affect the result, I
believe there should be two modes for the overview: A normal one just
showing the spherical model, and a linear mode where instead of just the
sphere, the plane would be rendered as well.
The linear mode would also change some stuff in the projection canvas as
well. For example, limit hfov to 180, disable yaw, pitch adjustments,
and enable translation adjustments.
Darko
James Legg
I've thought of an elegant solution.
Back-face culling can hide one side of the sphere. You only need to
check the vertex order is the same for all faces, and enable
GL_CULL_FACE.
So you can just leave depth testing turned off and draw the images over
the top of each other with no worries.
You can switch which side is culled with glCullFace, so if you want you
can even draw the back of the sphere, blend a circle over the sphere,
and then draw the front of the sphere over the top. This will get both
sides visible, ordered correctly, and the backside can be darkened (or
something else) to distinguish it.
This takes two passes to show both sides, but is better than trying to
clip meshes all the time.
-James
Darko Makreshanski
with some initial ideas how to tackle the problem for zooming in the
preview.
1. Meshes:
Upon each remapping of a mesh, the remappers (or something else) would
also create a bounding box for each mesh which will be used for fast
approximation of whether the mesh is inside the viewing area of the
preview or not.
On zooming event, the meshes that are inside the viewing area will be
assigned a dirty state if level of detail of the mesh needs to be updated.
Also these boxes can be used to completely disable drawing some meshes
even when zooming is not taking place. This could be just some preview tool.
2. Textures:
For the textures, I propose that the current scheme of deciding the size
of textures remains the same, and additional texture memory is
"reserved" for creating textures of larger resolutions in the zooming
scenario.
Also this allocation of extra textures might be done in parallel to
increase responsiveness of the zooming action.
Also, to increase the efficiency of this system, I propose to refactor
the current system a bit and instead of having one display_list/texture
per image we divide every image into smaller parts and handle these
parts separately, each with its own bounding box, display list and texture.
Best,
Darko
Darko Makreshanski
The only constriction is that everything would have to be rendered as a
mesh, and lines and grids would have to be avoided. So, the outside
circles that I proposed and the grid on the sphere would be meshes, but
this is not a problem.
Darko
Yuval Levy
> this mode wouldn't represent the output, but would represent an
> overview of the panorama in 3D space
if you want to represent the panorama in 3D space properly, a Photosynth-like
approach is better than the panosphere.
Each picture is like a pyramid, with the base of the pyramid being the
camera's sensor and the tip of the pyramid being the theoretical no-parallax
point behind it.
The old assumption was that the tip of those pyramids lie at the center of the
panosphere and the bases on the panosphere itself with the perpendicular
through the center of the basis passing through the tip / no-parallax-point.
This assumption has been relaxed with the TrX, TrY, TrZ parameters.
Yuv
Darko Makreshanski
> On March 21, 2010 05:44:49 pm Darko Makreshanski wrote:
>
>> this mode wouldn't represent the output, but would represent an
>> overview of the panorama in 3D space
>>
>
> if you want to represent the panorama in 3D space properly, a Photosynth-like
> approach is better than the panosphere.
>
>
browse through a panorama, but rather present the results of the
optimizer in a more convenient and intuitive way for beginners. Thus,
the final goal with the panosphere would be to visualize and let users
interact with the intermediate step of the panorama, that is the
panosphere. This system would contain both the spherical model without
Tr parameters and the mosaic model with Tr parameters. In the mosaic
model the hypothetical plane would also be rendered. Also some other
features, as I have explained in previous mails, will be included to
increase the user experience.
I'm actually still not sure what is the general opinion about this
feature, since apart from James, no one else has commented on it.
Best,
Darko
Yuval Levy
> This system would contain both the spherical model without
> Tr parameters and the mosaic model with Tr parameters. In the mosaic
> model the hypothetical plane would also be rendered.
sorry, my confusion. I mixed up Tr parameters with Tx,Ty,Tz,Ts (tilt and
scale) and my dream to move around the "pyramids" in real 3D space free of any
constraint, photosynth like.
> I'm actually still not sure what is the general opinion about this
> feature, since apart from James, no one else has commented on it.
James is the expert on the subject and I guess he will likely want to mentor
the project.
I don't know the "general opinion". For as much as my personal opinion counts,
you got me interested. I like your ideas and I think that your project has the
potential to make Hugin even more fun to interact with. I find it a little bit
difficult to "connect the dots" and imagine an overall impression of how Hugin
will feel like after your project - too many fragments in too many posts. I
guess you'll summarize the concept/plan in your application.
The danger is to conceptualize too much and implement too little. If I was you
I would get the vetting exercise out of the way sooner than later. Patch the
fast preview, maybe implement one of the low hanging fruits among your wealth
of interesting ideas.
Then focus. Put together a structured, organic plan. Maybe with a mockup. I
suspect that the time restriction of GSoC will force you to be selective. What
is important in your concept and what is nice to have? How can you make a
lasting impact in 12 weeks? The rest can follow.
Good luck
Yuv
Bruno Postle
> I'm actually still not sure what is the general opinion about this feature,
> since apart from James, no one else has commented on it.
Some thoughts from me, though there are more ideas already in this
thread than can be implemented in one project:
Fancy effects are a good thing, the more fun Hugin is to use the
better. So I really like the idea of animating the transitions between
projections.
Another trick that would help comprehension would be to animate the
transition to layout mode, though this would be harder to implement.
Overlaying a standard latitude/longitude grid over the preview would
also help comprehension, I would show this with mouse-over in the
Projection mode and during animated transitions (we need less buttons
and options in Hugin not more).
I'm a bit confused about the overview and zoom suggestions, I think
they are orthogonal ideas.
I like the idea of a view of a globe showing the entire sphere around
the camera, however this would need to show the following information:
* A grid that matched the latitude/longitude grid used elsewhere.
* The area with coverage by photos.
* The outline of the current panorama canvas rectangle.
* The outline of the current crop rectangle.
With all this info, I think remapping the photos is superfluous and
would confuse the clarity of the feature as a diagram. Also if it was
a simple line drawing it could be overlaid onto the panorama in
Projection and Drag mode (again using mouse-over for visibility).
I'm not sure what the value of zooming is, it won't help users see the
quality of alignment - As Hugin simply draws all photos on top of each
other, the actual seam placement is done by enblend during stitching.
What would be useful would be a standard-ish immersive panorama
viewer, where panning around didn't change the panorama itself and
where the wheel mouse zoomed in and out just like QuickTimeVR. This
would be another tab in the preview.
You have identified that rotating groups in Drag mode is a bit strange
when the group is off-centre, the rotation point could be the 'centre
of gravity' of the group rather than the centre of the panorama.
The current Drag mode where yaw and pitch are changed actually makes
no sense when there are any XYZ mosaic camera transformations in the
project - As the optimiser will simply put the panorama back where it
was.
This doesn't need to be user configurable: when there are XYZ
parameters in the project, Drag mode should switch to an XY
transformation instead of a rotation.
Note that we still have some unresolved issues with the fast preview:
Layout mode currently doesn't know about XYZ mosaic parameters, as a
result it draws the photos in crazy positions.
The 'old' preview still isn't integrated, if the remaining
functionality can't be reproduced with OpenGL then the 'old' preview
needs to be moved to another tab in the fast preview at some point.
We have a nice distinction between 'modes' that are switched by
changing tabs and 'actions' that are widgets within those tabs, but
there is a Panorama tab which is a mixture of tools that don't fit
anywhere else, e.g. the 'show points' mode could be another tab, but
doesn't have enough functionality to justify it (It would be nice to
somehow integrate the floating Control Points table with 'show
points').
--
Bruno
Darko Makreshanski
> Fancy effects are a good thing, the more fun Hugin is to use the
> better. So I really like the idea of animating the transitions between
> projections.
>
sphere to explain the nature of projections.
The transitions between projections actually would also be cool, however
currently I don't really have an idea how I would do this based on the
current code.
> Another trick that would help comprehension would be to animate the
> transition to layout mode, though this would be harder to implement.
>
easier, then it could be easily simplified.
> Overlaying a standard latitude/longitude grid over the preview would
> also help comprehension, I would show this with mouse-over in the
> Projection mode and during animated transitions (we need less buttons
> and options in Hugin not more).
>
would be displayed both in the overview on the sphere and in the
projection. Also the grid would contain interleaving colors, so for each
position there would be a unique color to create a more exact
correspondence between the overview grid and the projection grid.
> I'm a bit confused about the overview and zoom suggestions, I think
> they are orthogonal ideas.
>
time needed for both projects, so I might end up writing one proposal
for both or two separate proposals for each topic.
> I like the idea of a view of a globe showing the entire sphere around
> the camera, however this would need to show the following information:
>
> * A grid that matched the latitude/longitude grid used elsewhere.
>
> * The area with coverage by photos.
>
> * The outline of the current panorama canvas rectangle.
>
> * The outline of the current crop rectangle.
>
> With all this info, I think remapping the photos is superfluous and
> would confuse the clarity of the feature as a diagram. Also if it was
> a simple line drawing it could be overlaid onto the panorama in
> Projection and Drag mode (again using mouse-over for visibility).
>
photos as well.
> What would be useful would be a standard-ish immersive panorama
> viewer, where panning around didn't change the panorama itself and
> where the wheel mouse zoomed in and out just like QuickTimeVR. This
> would be another tab in the preview.
>
the camera in the center of the panosphere.
> You have identified that rotating groups in Drag mode is a bit strange
> when the group is off-centre, the rotation point could be the 'centre
> of gravity' of the group rather than the centre of the panorama.
>
> The current Drag mode where yaw and pitch are changed actually makes
> no sense when there are any XYZ mosaic camera transformations in the
> project - As the optimiser will simply put the panorama back where it
> was.
>
> This doesn't need to be user configurable: when there are XYZ
> parameters in the project, Drag mode should switch to an XY
> transformation instead of a rotation.
>
> Note that we still have some unresolved issues with the fast preview:
>
> Layout mode currently doesn't know about XYZ mosaic parameters, as a
> result it draws the photos in crazy positions.
>
> The 'old' preview still isn't integrated, if the remaining
> functionality can't be reproduced with OpenGL then the 'old' preview
> needs to be moved to another tab in the fast preview at some point.
>
> We have a nice distinction between 'modes' that are switched by
> changing tabs and 'actions' that are widgets within those tabs, but
> there is a Panorama tab which is a mixture of tools that don't fit
> anywhere else, e.g. the 'show points' mode could be another tab, but
> doesn't have enough functionality to justify it (It would be nice to
> somehow integrate the floating Control Points table with 'show
> points').
>
>
and a normal tab. Then in the normal tab you would have options to
choose a tool or for example enable 'indentify' mode etc. In this way I
could combine for example using the drag tool and have the control
points displayed.
Thanks a lot for your thoughts.
Best,
Darko
T. Modes
> In general I think that there should be only two tabs, the layout tab,
> and a normal tab. Then in the normal tab you would have options to
> choose a tool or for example enable 'indentify' mode etc. In this way I
> could combine for example using the drag tool and have the control
> points displayed.
Please don't. The version before 2010.0 had one toolbar (without tabs)
with all buttons. But the place was not enough to show all buttons. So
after long discussions we switch to the tabbed design and the release
2010.0 is the first one with this new approach. Now some controls
could moved into the tab and so increase the preview area. Also some
long wished features could so integrated (e.g non-modal numeric
transform of pano). Nearly all tabs are full with no or only little
space for more buttons. So it is impossible to move all functions into
one view (you can only reduce the functions or decrease the preview
area).
When it only the control point display, this could be changed that the
cp are shown also in drag and crop mode. But can somebody explain me a
workflow which works better with this display?
Thomas
Bruno Postle
> Bruno Postle wrote:
>> I like the idea of a view of a globe showing the entire sphere around
>> the camera
>> With all this info, I think remapping the photos is superfluous and
>> would confuse the clarity of the feature as a diagram. Also if it was
>> a simple line drawing it could be overlaid onto the panorama in
>> Projection and Drag mode (again using mouse-over for visibility).
> Actually once we have the setup it wouldn't be too difficult to add the
> photos as well.
Ok, I guess where I'm coming from is that the orthographic view is
perhaps the worst way of viewing a panoramic scene possible. Not only
does it turn everything inside-out, but the edges are extremely
distorted and you only get to see half of the scene at any one time.
I'll attach a diagram showing the alternative of how this view could
work. The globe represents the entire scene, the irregular blue patch
represents the area with coverage from photos, the red rectangle is
the current panorama 'canvas' and the white rectangle is the part of
the canvas that is cropped. This doesn't need to be interactive or
spin-able, you could map the actual photos to this, but I'm not sure
it would make it any clearer.
>> We have a nice distinction between 'modes' that are switched by
>> changing tabs and 'actions' that are widgets within those tabs, but
>> there is a Panorama tab which is a mixture of tools that don't fit
>> anywhere else, e.g. the 'show points' mode could be another tab, but
>> doesn't have enough functionality to justify it
> In general I think that there should be only two tabs, the layout tab, and a
> normal tab. Then in the normal tab you would have options to choose a tool
> or for example enable 'indentify' mode etc. In this way I could combine for
> example using the drag tool and have the control points displayed.
Hugin is already overly complex, and users don't appreciate the
possibilities of infinite combinations of options. Just because we can
show the control points in Drag mode, it doesn't mean we should - I
would rather enhance Show Points into a separate mode that also
highlights areas of the panorama that have overlaps, and then
integrate it properly with the Control Points table somehow.
While James was developing the layout mode it was possible to combine
Drag with the Layout mode (which I thought was very cool at the time),
but it had no practical purpose so James removed it and Hugin is
better as a result.
--
Bruno
Lukáš Jirkovský
> possibilities of infinite combinations of options. Just because we can
> show the control points in Drag mode, it doesn't mean we should - I
> would rather enhance Show Points into a separate mode that also
> highlights areas of the panorama that have overlaps, and then
> integrate it properly with the Control Points table somehow.
This reminds me of the thing someone said. It was something like: "The
software is perfect when you can't remove any code without losing
functionality".
Anyway, I'm not very keen of idea adding another tab to the Fast
Preview. I thing even the current number is "just too much". Rather
I'd like to see the current look (how the panorama is showed…)
replaced altogether if this proves to be an improvement.
Lukas
Darko Makreshanski
> On 30 March 2010 23:03, Darko Makreshanski <dmakre...@gmail.com> wrote:
>
>> Bruno Postle wrote:
>>
>>> like the idea of a view of a globe showing the entire sphere around
>>> the camera
>>>
>>> With all this info, I think remapping the photos is superfluous and
>>> would confuse the clarity of the feature as a diagram. Also if it was
>>> a simple line drawing it could be overlaid onto the panorama in
>>> Projection and Drag mode (again using mouse-over for visibility).
>>>
>> Actually once we have the setup it wouldn't be too difficult to add the
>> photos as well.
>>
>
> Ok, I guess where I'm coming from is that the orthographic view is
> perhaps the worst way of viewing a panoramic scene possible. Not only
> does it turn everything inside-out, but the edges are extremely
> distorted and you only get to see half of the scene at any one time.
>
the current orthographic projection, because back-faces wouldn't be
rendered, so this problem of turning everything inside-out would be
eliminated.
> I'll attach a diagram showing the alternative of how this view could
> work. The globe represents the entire scene, the irregular blue patch
> represents the area with coverage from photos, the red rectangle is
> the current panorama 'canvas' and the white rectangle is the part of
> the canvas that is cropped. This doesn't need to be interactive or
> spin-able, you could map the actual photos to this, but I'm not sure
> it would make it any clearer.
>
rendering of the images could just be like a faded addition to what you
have shown just so to get a rough explanation of which parts of the
silhouette on the panosphere correspond to which part of the projection.
This ofcourse could be adjustable.
I actually think that there should be some preferences window for the
fast preview, to allow the users to adjust everything in the fast
preview according to their needs. These may be adjustments for both user
interface and performance.
>
>>> We have a nice distinction between 'modes' that are switched by
>>> changing tabs and 'actions' that are widgets within those tabs, but
>>> there is a Panorama tab which is a mixture of tools that don't fit
>>> anywhere else, e.g. the 'show points' mode could be another tab, but
>>> doesn't have enough functionality to justify it
>>>
>> In general I think that there should be only two tabs, the layout tab, and a
>> normal tab. Then in the normal tab you would have options to choose a tool
>> or for example enable 'indentify' mode etc. In this way I could combine for
>> example using the drag tool and have the control points displayed.
>>
>
> Hugin is already overly complex, and users don't appreciate the
> possibilities of infinite combinations of options. Just because we can
> show the control points in Drag mode, it doesn't mean we should - I
> would rather enhance Show Points into a separate mode that also
> highlights areas of the panorama that have overlaps, and then
> integrate it properly with the Control Points table somehow.
>
>
Yes, you may be right. My idea was that the layout tab is the only one
that distinguishes from the rest in terms of what is shown in the
glCanvas, and all the others are just switching between tools.
Best,
Darko
Yuval Levy
> Hugin is already overly complex, and users don't appreciate the
> possibilities of infinite combinations of options. Just because we can
> show the control points in Drag mode, it doesn't mean we should
+1
for the occasional user less is better than more.
also for the expert user, information overload does not help.
always weight the cost of adding information in term of distracting clutter on
the user's screen vs. the added benefit of that extra information at this
stage.
Yuv
