Projective Reconstructions and Update

[altruis...@gmail.com]

Hi Johannes,

thank you for sharing this wonderful library!

I applied colmap recently to drone video data. Since the drone has a fixed focal length I checked "Single camera for all images" in the "Feature Extraction" dialogue. Unfortunately, the reconstruction keeps sticking in a local minimum. However, if I deselect "Single camera for all images" colmap creates a reasonable reconstruction.

Now, it is a known approach to compute a reconstruction without any constraints (i.e. projective reconstructions) first and later update the reconstruction by introducing additional constraints (e.g. same calibration parameters).

If I create a reconstruction without "Single camera for all images", then checking "Single camera for all images" and running bundle adjustment, the resulting reconstruction contains still individual parameters for each camera. Is this a bug? If not, do you intend to add the corresponding functionality?

Cheers,
Sebastian

[Johannes Schönberger]

Hi Sebastian,

Find my answers inline below.

> thank you for sharing this wonderful library!

Thanks, much appreciated.

> I applied colmap recently to drone video data. Since the drone has a fixed focal length I checked "Single camera for all images" in the "Feature Extraction" dialogue. Unfortunately, the reconstruction keeps sticking in a local minimum. However, if I deselect "Single camera for all images" colmap creates a reasonable reconstruction.

What camera model are you using? If you have a prior calibration of your distortion coefficients, you could try to fix them during the reconstruction to improve the robustness, see https://colmap.github.io/faq.html#fix-intrinsics.

> Now, it is a known approach to compute a reconstruction without any constraints (i.e. projective reconstructions) first and later update the reconstruction by introducing additional constraints (e.g. same calibration parameters).
>
> If I create a reconstruction without "Single camera for all images", then checking "Single camera for all images" and running bundle adjustment, the resulting reconstruction contains still individual parameters for each camera. Is this a bug? If not, do you intend to add the corresponding functionality?

The "single camera" option in the feature extraction initializes the camera_id field in the database for imported images accordingly. All later reconstruction will use that setting. You can change the relationship between cameras and images manually afterwards using the "Database Management" tool, see the "Processing" menu dialog. Here you can add new cameras and share intrinsics between arbitrary groups of cameras, see https://colmap.github.io/faq.html#share-intrinsics.

If that doesn't help, you could share the database and/or images with me and I could have a look.

Let me know if you have any other questions.

Cheers,
Johannes

>
> Cheers,
> Sebastian
>
> --
> You received this message because you are subscribed to the Google Groups "COLMAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to colmap+un...@googlegroups.com.
> To post to this group, send email to col...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/colmap/fe321deb-2fd8-4bd4-b3bc-bb58bed46111%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[altruis...@gmail.com]

Hi Johannes,

thank you for the fast response!

What camera model are you using? If you have a prior calibration of your distortion coefficients, you could try to fix them during the reconstruction to improve the robustness, see https://colmap.github.io/faq.html#fix-intrinsics.

I've used the Simple Camera Model and have (atm) no prior calibration parameters.

The "single camera" option in the feature extraction initializes the camera_id field in the database for imported images accordingly. All later reconstruction will use that setting. You can change the relationship between cameras and images manually afterwards using the "Database Management" tool, see the "Processing" menu dialog. Here you can add new cameras and share intrinsics between arbitrary groups of cameras, see https://colmap.github.io/faq.html#share-intrinsics.

I've already played with the data management tool (without the desired effect). However, I experience some strange results here.
My test dataset contains 250 images. I computed a reconstruction without "Single camera for all images".  The database management tool lists for each image a different camera. However, all cameras share the SAME intrinsics, i.e. the "params" column in the database management tool contains for each entry the following "2304.000000, 960.000000, 540.000000, 0.000000".

However, if I export the reconstruction to disc, the "camera.txt" file contains entries like "1 SIMPLE_RADIAL 1920 1080 1250 960 540 -0.00449804", "2 SIMPLE_RADIAL 1920 1080 1250.45 960 540 -0.00489968", etc. I get a similar result if I export the reconstruction as ".nvm" file.

Note: The focal length does not match, i.e. 2304 vs. 1250

Going one step further, and adding a new camera (e.g. something like "1 SIMPLE_RADIAL 1920 1080 1250 960 540 -0.00449804) with the database management tool, setting the id for all images to the id of the recently added camera, clicking "save", and applying bundle adjustment does not result in sharing the intrinsics for this reconstruction. (If I export the reconstruction again, I see still cameras with different intrinsic camera parameters, e.g. "250 SIMPLE_RADIAL 1920 1080 1147.19 960 540 -0.00244725", "249 SIMPLE_RADIAL 1920 1080 1147.54 960 540 -0.00265687") I think the bundle adjustment was applied without using the new camera information.

Am I missing something here?

Bests,
Sebastian

p.s. If changing the camera model after performing feature extraction has no effect, it would be nice to grey the functionality out, so that the user is aware that a change of this value has no effect

[Johannes Schönberger]

> I've already played with the data management tool (without the desired effect). However, I experience some strange results here.
> My test dataset contains 250 images. I computed a reconstruction without "Single camera for all images". The database management tool lists for each image a different camera. However, all cameras share the SAME intrinsics, i.e. the "params" column in the database management tool contains for each entry the following "2304.000000, 960.000000, 540.000000, 0.000000".

They have the same intrinsic parameters in the database, since they were automatically extract from the images. But as long as the camera_id column of the images does not have the same ID, they do not "share" the same intrinsics in the SfM optimization. If the camera_id of the images is different, the camera intrinsics are simply initialized with whatever is in the database. So, after the reconstruction, the parameters of each camera might be different from each other.

> However, if I export the reconstruction to disc, the "camera.txt" file contains entries like "1 SIMPLE_RADIAL 1920 1080 1250 960 540 -0.00449804", "2 SIMPLE_RADIAL 1920 1080 1250.45 960 540 -0.00489968", etc. I get a similar result if I export the reconstruction as ".nvm" file.

See above. The cameras.txt will only contain a single camera, if all images have the same camera_id.

> Note: The focal length does not match, i.e. 2304 vs. 1250

Probably because the initial guess was bad. I guess the images do not have EXIF information?

> Going one step further, and adding a new camera (e.g. something like "1 SIMPLE_RADIAL 1920 1080 1250 960 540 -0.00449804) with the database management tool, setting the id for all images to the id of the recently added camera, clicking "save", and applying bundle adjustment does not result in sharing the intrinsics for this reconstruction. (If I export the reconstruction again, I see still cameras with different intrinsic camera parameters, e.g. "250 SIMPLE_RADIAL 1920 1080 1147.19 960 540 -0.00244725", "249 SIMPLE_RADIAL 1920 1080 1147.54 960 540 -0.00265687") I think the bundle adjustment was applied without using the new camera information.

The manual bundle adjustment step simply takes the reconstruction as is. In order to use the new camera settings from the database, you have to restart the reconstruction again.

> p.s. If changing the camera model after performing feature extraction has no effect, it would be nice to grey the functionality out, so that the user is aware that a change of this value has no effect

Hm, since you can extract images multiple times, there is no way to know when the user is done with extracting images. I will try to clarify the option though.

Thanks for the feedback, let me know if you have other questions.

Cheers,
Johannes

>
>
> --
> You received this message because you are subscribed to the Google Groups "COLMAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to colmap+un...@googlegroups.com.
> To post to this group, send email to col...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/colmap/078cec2a-7c15-4044-a92f-e17df838f2d6%40googlegroups.com.

[altruis...@gmail.com]

Hi Johannes,

I've just realized where my confusion came from :) . After finishing the SfM computation, I expected the database management tool to show the current intrinsic parameters of each camera.
But the cameras still show the values after the extraction step / the initial values of the SfM computation.
Thus, the focal length is set max(1920,1080) x 1.2 = 2304, since the images do not contain EXIF information (as you already guessed ;) )

Hm, since you can extract images multiple times, there is no way to know when the user is done with extracting images. I will try to clarify the option though.

I think this will make the software more transparent! Thank you! One solution could be to delay the selection of the cameras (i.e. add another step later in the pipeline)

The manual bundle adjustment step simply takes the reconstruction as is. In order to use the new camera settings from the database, you have to restart the reconstruction again.

What a pity. If I have to restart the reconstruction, I can not apply the "projective reconstructions + update" approach. I' ll check if another camera model will solve my "local minimum" issue.
If not, I guess I ll try to implement it by myself. If it works, are you interested in a pull request?


Thank you!

Cheers,
Sebastian

[Johannes Schönberger]

> On Sep 15, 2016, at 1:50 PM, altruis...@gmail.com wrote:
>
> Hi Johannes,
>
> I've just realized where my confusion came from :) . After finishing the SfM computation, I expected the database management tool to show the current intrinsic parameters of each camera.
> But the cameras still show the values after the extraction step / the initial values of the SfM computation.
> Thus, the focal length is set max(1920,1080) x 1.2 = 2304, since the images do not contain EXIF information (as you already guessed ;) )
>
> Hm, since you can extract images multiple times, there is no way to know when the user is done with extracting images. I will try to clarify the option though.
>
> I think this will make the software more transparent! Thank you! One solution could be to delay the selection of the cameras (i.e. add another step later in the pipeline)

Hm, this step is optimized for large image collections to only read each image once from disk... splitting it into two steps would make it clearer but also slower :-(

> The manual bundle adjustment step simply takes the reconstruction as is. In order to use the new camera settings from the database, you have to restart the reconstruction again.
>
> What a pity. If I have to restart the reconstruction, I can not apply the "projective reconstructions + update" approach. I' ll check if another camera model will solve my "local minimum" issue.
> If not, I guess I ll try to implement it by myself. If it works, are you interested in a pull request?

In general, yes, of course. It depends a little, since projective + update would require a lot of changes across the entire pipeline.

Would you be able to share the database file (+ optionally the images), then I could have a look. You could share the data privately with me, the email address is in the copyright notice of the code. I am sure there is some parameter tweak that could make the reconstruction work better.

Cheers,
Johannes

>
>
> Thank you!
>
> Cheers,
> Sebastian

>
> --
> You received this message because you are subscribed to the Google Groups "COLMAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to colmap+un...@googlegroups.com.
> To post to this group, send email to col...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/colmap/915c7222-f2e1-45ae-a90b-9a4258439754%40googlegroups.com.

[altruis...@gmail.com]

Hi Johannes,

I have to apologize, since I can not recreate all the results I have observed before. Because the reconstruction using a single camera took really long I worked on different subsets. On one of the subsets I observed the described behaviour.

Finally, applying colmap with "single camera for all images" on all 250 images created first a model with 14 images. Since this model contained only 14 images and due to the other reconstructions (containing only partial models) I had seen before, I expected the second model would contain also only another subset. Eventually, almost 1,5 hours later the second model contained the complete set of images. Sorry!

When comparing both reconstructions I realized that there is a dramatic difference in computation time:
Multiple Cameras: 58 min
Single Camera: 1h 50 min

So "Multiple Cameras" + update is probably much faster than "Single Camera". I don't know if this is a general effect. Do you know if the "single camera" approach is supposed to take that much longer? If so, projective reconstructions + update could speed up this use case a lot.

Thank you for your time!

Cheers,
Sebastian

p.s. In case you are still interested in the the images, the corresponding db files and reconstructions you can access them here:
https://dl.dropboxusercontent.com/u/13686414/colmap_background.tar.bz2

[Johannes Schönberger]

If you use SIMPLE_RADIAL and do NOT share intrinsics, COLMAP automatically switches to PBA bundle adjustment as opposed to Ceres. The former is a very optimized bundle adjustment library and much faster.

> --
> You received this message because you are subscribed to the Google Groups "COLMAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to colmap+un...@googlegroups.com.
> To post to this group, send email to col...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/colmap/e323b732-3dca-401a-b18e-36acc6ad5e18%40googlegroups.com.