Question on DlsPnp input

[djc...@uw.edu]

We're attempting to use the DlsPnp function and are encountering some difficulties regarding the expected input. For reference, we are using pixel coordinates of points from a fisheye photo of a dome. Documentation states that we should be passing in "normalized image rays", but we are unclear on what this means. We've passed in pixel coordinates in different formats but consistently get either no solutions or very inaccurate solutions. What specifically does DslPnp expect as input for this parameter? If it is just pixel coordinates, should they be normalized beforehand, as some of the documentation states? If so, how should they be normalized / what normalization function should be used? Every time we've passed in non-normalized pixel coordinates, the function returns no solutions. Any help you can give us would be greatly appreciated!

[Chris Sweeney]

Hi,

Admittedly, it is a bit of a confusing name. "Normalized" image rays in this instance are what you get when you take pixel coordinates and remove the effect of camera intrinsics. In reality, this would be more clear as a unit-norm ray instead of the 2d "normalized" coordinate. So to get the normalized coordinate that DlsPnP expects you can:

given a pixel (x, y), remove the effects of intrinsics:   x' = (x - px) / f,   y' = (y - px) / f

or 

given a unit-norm ray (x, y, z) that corresponds to the pixel coordinate of interest: x' = x / z,   y' = y / z

Hope this helps!

On Tue, Nov 6, 2018 at 1:55 PM <djc...@uw.edu> wrote:

We're attempting to use the DlsPnp function and are encountering some difficulties regarding the expected input. For reference, we are using pixel coordinates of points from a fisheye photo of a dome. Documentation states that we should be passing in "normalized image rays", but we are unclear on what this means. We've passed in pixel coordinates in different formats but consistently get either no solutions or very inaccurate solutions. What specifically does DslPnp expect as input for this parameter? If it is just pixel coordinates, should they be normalized beforehand, as some of the documentation states? If so, how should they be normalized / what normalization function should be used? Every time we've passed in non-normalized pixel coordinates, the function returns no solutions. Any help you can give us would be greatly appreciated!

--
You received this message because you are subscribed to the Google Groups "Theia Vision Library" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theia-vision-lib...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[djc...@uw.edu]

Hi Chris, thanks for your quick reply! That did indeed help. Unfortunately, we've run into a different issue now that we're attempting to use the calibrate_camera_intrinsics application to get our camera intrinsics.

We're using a 185-degree FOV camera to get 4608x3288 images and our command to run the application looks like this:

./calibrate_camera_intrinsics --images=home/user/Pictures/in/*.png --matching_working_directory=/home/user/Pictures/out/ --camera_model=FOV --logtostderr --num_calibration_iterations=3


The output we're getting looks like this:
histogram =
[2 - 3) = 370
[3 - 4) = 269
[4 - 5) = 150
[5 - 6) = 102
[6 - 7) = 64
[7 - 8) = 45
[8 - 9) = 30
[9 - 10) = 38
[10 - 15) = 106
[15 - 20) = 46
[20 - 25) = 17
[25 - 50) = 0
I1108 11:24:52.176913 6082 fov_camera_model.cc:161] Camera model type: FOV
Focal length (pixels): 5529.6
Principal Point (px, py) = (2304, 1644)
Aspect Ratio: 1
RadialDistortion: 0.75

Using these intrinsics, our output from PnP is pretty drastically off from what we expect, so we're assuming we're using the calibration application incorrectly. We know at least some of our results are wrong from calibration; for example, the actual focal length of our camera is 3786, but even when we manually use the correct focal length in our PnP with the rest of the calculated intrinsics, our PnP results are wrong. If you can help us clear up a couple of things, we'd really appreciate it!

- Is our command to run the application structured correctly? We had some difficulty building the command such that we got any output, and we're still not 100% sure it's correct.

- What should our input images to the calibration application be like? We've tried a number of different approaches, all of which yielded *exactly* the same output from the intrinsic calibrator. Our last attempt was a series of ~25 photos that slowly swept across a room, with a lot of overlap between photos. We've also tried some shots of a checkerboard grid.

- We're also unclear what the histogram represents, so if you could shed some light on that we'd be grateful.

Our team has gone through Theia's documentation pretty thoroughly, but if these answers are readily available somewhere, please point us there and we'll use that! Thanks again for the help.

-

[Chris Sweeney]

For calibration, it is helpful if you have a set of images from a diverse set of viewpoints (varying position and rotation). Also note that while it is possible to reconstruct from planar areas, it's actually not too ideal for this specific calibration procedure which is simply running SfM multiple times and bootstrapping the intrinsics from the previous run into the next run.

The histogram is a histogram of track lengths -- it looks like you have a nice set of longer track lengths so you problem should be relatively constrained.

Why do you need to use PnP? Do you get the expected solution if you use P3P?

[djc...@uw.edu]

OK, we've tried supplying input images like you've described as well, different positions and rotations, etc. We still got exactly the same output, which leads us to believe something is going wrong.

By setting the intrinsics_to_optimize flag for focal length, we were able to get a focal length correct for our camera. However, even with that, our PnP results were still incorrect, so we're assuming our radial distortion, at least, is off. Additionally, the aspect ratio being returned by the calibrator is definitely wrong -- the actual aspect ratio for our input images is 1.4. One of our devs suspects that the intrinsics results we're getting are 'idealized' values and not actually tied to our camera / the images we're inputting.

We're currently using PnP because it sounds like it will produce more accurate results, and we need a relatively high degree of accuracy for our project. However, we did try P3P and got bad results there too -- presumably because something is wrong with our intrinsics. If we get functioning results from the intrinsics calibrator, we could potentially use P3P rather than PnP.

[Chris Sweeney]

Would you be able to share some sample data with me over email? If it helps, I was a UW postdoc the past two years in the GRAIL lab :)

[djc...@uw.edu]

Hey Chris, I'd be happy to share some sample data with you! However, before I do that, one of my developers got me some an error log of what he's seeing when he runs the calibration app. We're getting the same Eigen failure *a lot* of times. I'm now wondering if there's an issue with our version of Eigen; we initially installed Eigen via Aptititude, which I believe is on version 3.3.4. The latest release of Eigen on GitHub, though, is 3.3.5. I've since tried to build it from source, rebuild Ceres, and then rebuild Theia, but I'm now hitting a compilation error in Theia, so I can't confirm if this fixes the problem or not. In any case, the Eigen failure we're seeing is:

W1108 16:53:34.403436 32086 levenberg_marquardt_strategy.cc:114] Linear solver failure. Failed to compute a step: Eigen failure. Unable to perform dense Cholesky factorization.

We're seeing 10 to 20 of these per image.

Additionally, I now see we're failing two of the tests from 'make test' -- 6 and 16. I don't know if these are related to our problems or not; as far as we know, we strictly just want to calibrate our camera, get the intrinsics, and use those for our P3P / PnP as needed. So, if these failures are irrelevant, feel free to ignore! If they do pertain to the problem, here are the error logs:

Running the cascade hashing feature matcher test by itself, I get this:

I1109 10:24:56.294071 1958 test_main.cc:48] Running tests with logs enabled.
[==========] Running 5 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 5 tests from CascadeHashingFeatureMatcherTest
[ RUN ] CascadeHashingFeatureMatcherTest.NoOptionsInCore
I1109 10:24:56.294692 1958 fisher_vector_extractor.cc:116] Training GMM for Fisher Vector extractin with 20 features sampled from 20 total features.
[ OK ] CascadeHashingFeatureMatcherTest.NoOptionsInCore (1 ms)
[ RUN ] CascadeHashingFeatureMatcherTest.RatioTestInCore
I1109 10:24:56.295593 1958 fisher_vector_extractor.cc:116] Training GMM for Fisher Vector extractin with 3 features sampled from 3 total features.
[ OK ] CascadeHashingFeatureMatcherTest.RatioTestInCore (0 ms)
[ RUN ] CascadeHashingFeatureMatcherTest.NoOptionsOutOfCore
I1109 10:24:56.296512 1958 fisher_vector_extractor.cc:116] Training GMM for Fisher Vector extractin with 20 features sampled from 20 total features.
[ OK ] CascadeHashingFeatureMatcherTest.NoOptionsOutOfCore (1 ms)
[ RUN ] CascadeHashingFeatureMatcherTest.RatioTestOutOfCore
I1109 10:24:56.297212 1958 fisher_vector_extractor.cc:116] Training GMM for Fisher Vector extractin with 3 features sampled from 3 total features.
[ OK ] CascadeHashingFeatureMatcherTest.RatioTestOutOfCore (1 ms)
[ RUN ] CascadeHashingFeatureMatcherTest.NoDescriptorsInCore
F1109 10:24:56.297806 1958 feature_matcher.cc:157] Check failed: descriptors.size() > 0 (0 vs. 0)
*** Check failure stack trace: ***
@ 0x7fb03b3ea0cd google::LogMessage::Fail()
@ 0x7fb03b3ebf33 google::LogMessage::SendToLog()
@ 0x7fb03b3e9c28 google::LogMessage::Flush()
@ 0x7fb03b3ec999 google::LogMessageFatal::~LogMessageFatal()
@ 0x55f39b8cca3e (unknown)
@ 0x55f39b8c00ac (unknown)
@ 0x55f39b8906f1 (unknown)
@ 0x55f39b8be3ca (unknown)
@ 0x55f39b8b406d (unknown)
@ 0x55f39b8b467d (unknown)
@ 0x55f39b8b48d5 (unknown)
@ 0x55f39b8b5b8f (unknown)
@ 0x55f39b8b5e92 (unknown)
@ 0x55f39b88806c (unknown)
@ 0x7fb039a3f1c1 __libc_start_main
@ 0x55f39b88e71a (unknown)
@ (nil) (unknown)
Aborted (core dumped)

Which looks related to GLog?

Running the normalized graph cut test, the first two tests fail and the third runs fine:

I1109 10:24:26.285338 1941 test_main.cc:48] Running tests with logs enabled.
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from NormalizedGraphCut
[ RUN ] NormalizedGraphCut.SimpleGraph
unknown file: Failure
C++ exception with description "ncv must satisfy nev < ncv <= n, n is the size of matrix" thrown in the test body.
[ FAILED ] NormalizedGraphCut.SimpleGraph (0 ms)
[ RUN ] NormalizedGraphCut.SimpleGraph1
unknown file: Failure
C++ exception with description "ncv must satisfy nev < ncv <= n, n is the size of matrix" thrown in the test body.
[ FAILED ] NormalizedGraphCut.SimpleGraph1 (0 ms)

(the third runs normally after that).

Sorry to hit you with all these questions; we're very new to Theia, but we're really hoping to get it to work because it would save us a lot of time and trouble. Thanks so much for all your support! Let me know if you'd still like me to email you some sample data.