Re: Using Theia for relative pose and translation estimation

[Chris Sweeney]

Hi Rajvi,

Thanks for the note. It would be great if we can keep Theia discussions on the email list so everyone can benefit :)

First, the "compute_two_view_geometry" module will compute feature matches AND two view geometry (hence the name of the module). So the matches file it writes contains what you want.

If you use that module, you an then read back the matches and two-view geometry with this function. The ImagePairMatch objects contain the two view geometry and matches for a particular pair of image.

However, if you have your own matches already and want to compute geometry, then I'd recommend the VerifyTwoViewMatches function. This function takes in feature matches and outputs the relative pose and inlier indices of the input matches (i.e. which features were geometrically verified as matches).

Hope that helps!

Chris

On Wed, Sep 23, 2015 at 12:44 PM rajvi shah <rajvi....@gmail.com> wrote:

Dear Chris,

I am Rajvi, a PhD candidate at IIIT Hyderabad, India working in SfM area.

I recently came across your SfM library -- Theia.
It is an excellent resource, I wish I had known about it before.
Thank you very much for making it open.

It would be immensely useful to the community.

I am trying to use the rotation averaging + 1DSFM code of Kyle Wilson to reconstruct my datasets.

I was hoping to use Theia to generate pairwise rotations and translations from precomputed pairwise matches. I could see the solvers for relative pose but could not fully understand the  interface.

I looked at the "compute_two_view_geometry" application code but that only seems to write the matches. I have the matches for image pairs and need to estimate relative geometry (rotations and translation). What would be a good starting point for me to use Theia for that?

I would be thankful for any help I can get.

Thank you for your time.

Regards,

Rajvi

[Chris Sweeney]

By the way, Rajvi, Kyle Wilson's 1DSfM filter and the nonlinear translations solver are both already implemented in Theia so you could just use Theia for the whole pipeline :)

[Rajvi Shah]

Hi Chris,

Thank you for the quick informative reply.
As per your suggestions I had a look at the VerifyTwoViewMatches(...) function.
It seems apt for my purpose. I did this,

1. Read list of images and keyfiles

2. Read all correspondences

3. Extract exif info and populate CameraIntrinsicsPrior structures for all images.

I have the following questions regarding how I proceed further,

1. How do I supply the image correspondence coordinates? Image centered, normalized by intrinsics, or as they are -- between (0,0) and (W,H)?

2. Since the exif reader in some cases does not find the focal length information, how do I proceed with filling the CameraIntrinsicsPrior structure for such cases?

Also, thank you for pointing out the 1D_sfm implementation in theia.

It would indeed be very useful for me. :)

I had a look at the build_1dsfm_reconstruction.cc file in applications directory.

What is expected as input by this input argument "1dsfm_dataset_directory"?

Pairwise reconstructions or just images? It appears to be an end-to-end pipeline.

What would be the best possible way for me to initialize this with my own feature correspondence file (match-graph)?

Thank you very much for your help.

Regards,

Rajvi

[Chris Sweeney]

Hi Rajvi,

Happy to help. Much of this information is located on the website and within code comments in the files I linked to you. My replies are inline.

On Thu, Sep 24, 2015 at 12:39 PM Rajvi Shah <rajvi....@gmail.com> wrote:

Hi Chris,

Thank you for the quick informative reply.
As per your suggestions I had a look at the VerifyTwoViewMatches(...) function.
It seems apt for my purpose. I did this,

1. Read list of images and keyfiles

2. Read all correspondences

3. Extract exif info and populate CameraIntrinsicsPrior structures for all images.

I have the following questions regarding how I proceed further,

1. How do I supply the image correspondence coordinates? Image centered, normalized by intrinsics, or as they are -- between (0,0) and (W,H)?

They should be the standard pixel locations. If available, the camera focal intrinsics from EXIF will be used to normalize these coordinates internally for estimating the essential matrix (then relative pose). Otherwise, the fundamental matrix (i.e. uncalibrated relative pose) will be used so that intrinsics are not needed.

 

2. Since the exif reader in some cases does not find the focal length information, how do I proceed with filling the CameraIntrinsicsPrior structure for such cases?

You simply pass in an empty/default CameraIntrinsicsPrior object. By default, this object is initialized to indicate that no camera intrinsic information is available.

 

Also, thank you for pointing out the 1D_sfm implementation in theia.

It would indeed be very useful for me. :)

I had a look at the build_1dsfm_reconstruction.cc file in applications directory.

What is expected as input by this input argument "1dsfm_dataset_directory"?

Pairwise reconstructions or just images? It appears to be an end-to-end pipeline.

build_1dsfm_reconstructions operates on the 1dsfm datasets. So it takes in the matches and relative pose information from that dataset as input and computes a reconstruction.

Alternatively, build_reconstructions.cc is more flexible and will utilize input images or matches files.

 

What would be the best possible way for me to initialize this with my own feature correspondence file (match-graph)?

build_reconstructions.cc is the way to go. Using your own application, write out a match file then just pass it to build_reconstructions.cc

The simplest way to write a match file is to populate the relevant data fields then call this function to write the matches out.

 

--
You received this message because you are subscribed to the Google Groups "Theia Vision Library" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theia-vision-lib...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rajvi Shah]

Hi Chris,

Thanks for the detailed reply. You are right, the comments are pretty detailed, my bad, I didn't look in the right .cc files before.
I am now able to estimate the two-view geometry with inliers and write these out to a binary file using the functions you pointed at.

As per your suggestion, I ran the build_reconstruction application to run rotation averaging + 1dsfm pipeline with precomputed viewgraph file as input .
<SomePath>/build_reconstruction --matches_file=data/view_graph.bin --output_reconstruction=output/output.theia --log_dir=output/log/

It produced the following error,
F0925 18:57:33.972544  8991 reconstruction_builder.cc:269] Check failed: view_id1 != kInvalidViewId (4294967295 vs. 4294967295) Tried to add a view with the name /data/rajvi/Rome16KData/pantheon_interior/1605684920.jpg to the view graph but does not exist in the reconstruction.

The problematic part is this part in function: reconstruction.cc : bool ReconstructionBuilder::AddTwoViewMatch
const ViewId view_id1 = reconstruction_->ViewIdFromName(image1);
const ViewId view_id2 = reconstruction_->ViewIdFromName(image2);

I found that  std::unordered_map<std::string, ViewId> view_name_to_id_ stores only the file names and not the full paths, while image1, image2 stored the full paths.
I am not sure if this is a desired behaviour but I changing the view-graph output to store only the filenames and not full paths, that solved this problem for me.

The program now throws an error here. 
E0925 22:53:32.338779 21490 estimate_rotations_robust.cc:250] Failed to solve the least squares system.

The relevant part of the log file is attached. Can you please guide me how do I go about fixing this?
Please let me know if I should supply more information or the view-graph file.

Thank you for your time.

Regards,
Rajvi
 

On Thursday, 24 September 2015 01:23:37 UTC+5:30, Chris Sweeney wrote:

[Chris Sweeney]

Hi Rajvi,

You are correct on the filename/filepath issue. I should probably write a comment in the matches writer file to indicate it should only be the filename... or maybe you could submit a patch with those comments? :)

Could you send me the matches file? That would be great to help me debug.

Chris

[rajvi shah]

Hi Chris,

I sent you a pull request with the added comment.

Thank you for offering to help. You can find the view graph file here
https://github.com/rajvishah/theia-pairwise-geometry/tree/master/pairwise-estimator/data

Again, thanks.

Regards,

Rajvi

[Chris Sweeney]

Hi Rajvi,

In the TwoViewInfos you are populating for your matches, how are you obtaining the values for focal_length_1 and focal_length_2? It seems to be that focal_length_1 is always VERY badly initialized:

Some examples from the matches file you've provided:

Focal lengths 1,2 = 1.39956, 2944.16

Focal lengths 1,2 = 1.15356, 1948.87

Focal lengths 1,2 = 0.786712, 2240.23

and so on...

Chris

[rajvi shah]

Dear Chris,

Thank you for taking time to reply and pointing out the focal length issue.

I use ExtractEXIFMetaData function to find focal length. If that fails, I read it from a list of focal lengths estimated by other method (Bundler estimated focal length for testing). If both fails, I assign an empty CameraIntrinsicsPrior object to that image as you suggested previously. Here is the relevant code block for your reference.

    // Read EXIF data find focal length
    printf("\nReading exif data %d/%d", i, numKeys);
    CameraIntrinsicsPrior cPrior, dummyPrior;
    bool status = exReader.ExtractEXIFMetadata(imgList.getImageName(i), &cPrior);
    if(status) {
      printf("\nRead EXIF data:");
      if(cPrior.focal_length.is_set) {
        calibrationFlag[i] = true;
        focalLengths[i] = cPrior.focal_length.value;
        camPriors.push_back(cPrior);
      } else if(precompFocals[i] != 0) {
        calibrationFlag[i] = true;
        focalLengths[i] = precompFocals[i];
        camPriors.push_back(cPrior);
      } else {
        camPriors.push_back(dummyPrior);
      }
    } else {
      camPriors.push_back(dummyPrior);
    }
  }

 
The focal length supplied to VerifyTwoViewMatches is always either 0.000 (uninitialized object) or a valid focal length in pixels.
I printed the values of focal length in CameraIntrinsicsPrior object and TwoViewInfo object after verification.

    VerifyTwoViewMatchesOptions options;
    bool status = VerifyTwoViewMatches(options,
        camPriors[img1], camPriors[img2],
        featCorrs,&currPairInfo,&inliers);

    if(status) {
      printf("\nSuccessfully Verified Matches");
      printf("\nFocal Length 1 = %f, Focal Length 2 = %f", camPriors[img1].focal_length.value,
camPriors[img2].focal_length.value);
      printf("\nFocal Length 1 = %f, Focal Length 2 = %f", currPairInfo.focal_length_1, currPairInfo.focal_length_2);

      for(int inl_id=0; inl_id < inliers.size(); inl_id++) {
        inlFeatCorrs.push_back( featCorrs[inliers[inl_id]] );
      }

      ImagePairMatch imPair;
      imPair.image1_index = img1;
      imPair.image2_index = img2;
      imPair.twoview_info = currPairInfo;
      imPair.correspondences = inlFeatCorrs;
      matches.push_back(imPair);
    }
  }

  bool status = WriteMatchesAndGeometry("view_graph.bin", view_names, camPriors, matches);
  if(!status) {
    printf("\nSuccessfully written view-graph file");
  }

The result is something like this,

Successfully Verified Matches
Focal Length 1 = 2582.260870, Focal Length 2 = 2804.869565  <--- #CameraIntrinsicsPrior
Focal Length 1 = 2582.260870, Focal Length 2 = 2804.869565  <--- #TwoViewInfo
Successfully Verified Matches
Focal Length 1 = 0.000000, Focal Length 2 = 2804.869565
Focal Length 1 = 2781.766951, Focal Length 2 = 1.043372
Successfully Verified Matches
Focal Length 1 = 0.000000, Focal Length 2 = 2804.869565
Focal Length 1 = 1425.175805, Focal Length 2 = 0.387760
Successfully Verified Matches
Focal Length 1 = 2727.334083, Focal Length 2 = 2804.869565
Focal Length 1 = 2727.334083, Focal Length 2 = 2804.869565
Successfully Verified Matches
Focal Length 1 = 0.000000, Focal Length 2 = 2804.869565
Focal Length 1 = 1140.065631, Focal Length 2 = 1.071549

So, I understand that when any one of the two focal lengths is un-initialized, two view geometry verify function probably estimates and assigns different focal lengths to the cameras which doesn't come out very well. Should I discard all uncalibrated images from the list then?

On a side note, I meanwhile wrote the two view information generated by theia (same 'view-graph.bin' file as above), and the other information as needed by Kyle's 1D SFM and tried to run that. It runs without an error but I haven't verified the output yet. I realize now that it would also be erroneous due to the focal length issue.

Please suggest me the best way to handle uncalibrated images.

Please inform me if you need more information.

Thank you very much for your time.

Regards,
Rajvi

[Chris Sweeney]

Hi Rajvi,

Thanks for the very clear explanation. I think you are correct in diagnosing that uncalibrated images are giving your dataset fits with Theia.

The issue of calibration is one that plagues a lot of SfM, especially global SfM. I have written a patch to extend the create_calibration_file_from_exif.cc that can be found here. You'll see that a new command line option --initialize_uncalibrated_images_with_median_viewing_angle  will initialize images that do not have EXIF with a guess for the focal length based on a median viewing angle. This is the same strategy that VisualSfM uses, and it seems to work well.

Let me know if this patch is useful, and I can merge it into the main repo.

Chris

[Chris Sweeney]

By the way, the issue of estimating relative pose when one image has calibration and the other does not has been on my radar for a while (see this issue). If you want to take a crack at writing the 6pt solver that fixes this issue I would gladly welcome it :)

[rajvi shah]

Dear Chris,

The calibration patch you sent was very useful. The reconstruction came out well (see attached).

There is a small issue with the way the input flag in the file is defined.
FLAG_initialize_uncalibrated_images_with_median_viewing_angle
Setting this to false initializes the uncalibrated images with median angle, which I feel is non-intuitive.

I definitely would like to take a look at the solver you mentioned and give it a try.

I am a bit occupied right now with many experiments to perform for a paper, but I would be able to take that up after a while.

Theia has been very useful for my experiments, so it would make me happy to contribute.

Thanks for all the help. :)

Regards,

Rajvi

[Chris Sweeney]

Hi Rajvi,

I agree that the median viewing angle is not intuitive. However, given no other priors there is not a very good way of estimating focal lengths unfortunately. I have a paper at ICCV that will get good focal lengths estimates for uncalibrated images, but only if you have really really good input data. 

I'm very glad you're finding it useful. I have tried to prioritize development of Theia with the main goal of being useful for other researchers, engineers, and enthusiasts. Luckily, the performance is pretty good too ;)

If you need help tuning parameters to obtain better reconstructions for your (CVPR?) submission, please let me know and I'll gladly help. Usually increasing the number of retriangulations for global SfM helps. Also, please do not forget to cite the Theia manual in your paper:

@manual{theia-manual,

        Author = {Chris Sweeney},

        Title = {Theia Multiview Geometry Library: Tutorial \& Reference},

        Organization = {University of California Santa Barbara.}

}

Best,

Chris

[rajvi shah]

Hi Chris,

I am able to obtain reconstructions using build_reconstruction application but facing a problem with conversion to bundler format.
Since, most of my code deals with reconstructions of bundler's .out format I need to convert theia output reconstruction to bundler's .out format.
I found the WriteBundlerFiles routine in export_reconstruction_to_pmvs application.
I modified the application to only output the bundle file. I basically commented the following two lines.

 //const int num_cameras = WriteCamerasToPMVS(reconstruction);
 //WritePMVSOptions(FLAGS_pmvs_working_directory, num_cameras);


To test that the conversion works, I did the following.
1) converted a theia reconstruction to bundle file using export_reconstruction_to_pmvs
2) conveted the bundle file back to theia's binary format using convert_bundle_file
3) computed the reconstruction statistics for original theia file and converted file using compute_reconstruction_statistics

The re-projection statistics don't match.

#------------ Original Reconstruction -----------#
Num reprojections behind camera: 0                                                                   
Mean reprojection error = 0.717685                                                                    
Median reprojection_error = 0.416321

#------------ Converted Reconstruction -----------#
Num reprojections behind camera: 1
Mean reprojection error = 1210.75
Median reprojection_error = 972.753

Am I doing something wrong?
Should I use some other way to convert theia reconstruction to bundler format?

Also, should I start this as a new thread as it is not related to the first issue?

Regards,
Rajvi

[Chris Sweeney]

[moved to a new thread for clarity]

Hi Rajvi,

Thanks for bringing this to my attention. This is most certainly a bug (or several). I believe I have made the fix in this change. I tried your experiment (converting back and forth and checking the reconstruction stats) and get consistent results for Theia and Bundler files.

I have also added a new utility file convert_theia_reconstruction_to_bundler_file.cc I think it should do exactly what you're looking for :)

Let me know if this solves the problem or if you still have issues.

Chris

[rajvi shah]

Hi Chris,

Thank you for fixing the bug so quickly. The reconstructions have similar re-projection errors as you mention.

For my work, I need to merge new tracks to bundler tracks.

In my codes, I do this based on ( View Index, Feature Index ) pairs.

Since theia does not keep feature indices and does not write it to bundler, I did an inverse search from x,y coordinates in tracks to centered feature sets ( 0,0 at the center of the image) .

I have normalized my feature sets to match bundler's representation,

x' = x - w/2

y' = h/2 - y

I couldn't find any features in the corresponding feature files using this reverse search.

Looking at write_bundler_files.cc : function WriteBundleFile() also seems to be applying the same transformation to the features.

      const Eigen::Vector2d adjusted_feature(
          feature->x() - view->Camera().PrincipalPointX(),
          -(feature->y() - view->Camera().PrincipalPointY()));

 

In an effort to figure out the problem I printed values of features before and after transformation as well as the value of PrinciplePoint()

The x and y values of PrinciplePoint doesn't match W/2 and H/2. They are off by some pixels. One example,

Half Width          : 1824           Half Height         : 1368
Principle Point X : 1816.29      Principle Point Y : 1311.75

I verified that the CameraIntrinsicsPriors objects I pass for view graph creation have the right values.

Is there some other transformation that is applied to W/2 and H/2 to get different values for principle point?

Is there a way for me to prevent that?

 

Regards,

Rajvi

[Chris Sweeney]

There seems to be a few things going on here. The main issue is that Bundler assumes the principal point to be *exactly* at the center of the image. This is not the case for all cameras (e.g., the cameras in the Strecha MVS dataset) and is why you're seeing a difference.

As a hack-ish fix for your purposes, you can change adjusted_feature in write_bundler_files.cc to subtract w/2 and h/2 instead of the actual principal point. I think this should fix your problem.

[rajvi shah]

I thought of that. But would that not create inaccuracies with re-projection during later stages?

As in, theia would have estimated camera parameters with 3D-2D correspondences where 2D features have different normalization (with accurate principal point).

If I use these estimated camera parameters and apply a different normalization to feature sets would that not create a problem?

Regards,

Rajvi

[Chris Sweeney]

Yes, you are probably right. Though for many cases the principal point is very close to the center so it may not be horribly wrong.

I would suggest one of two solutions:

1) [Preferred choice] Output a file that specifies the map of adjusted_features to the features with w/2 and h/2 subtracted. Then you can perform a lookup on that mapping.

2) You mentioned before that you are using create_calibration_file_from_exif.cc. This file should set the principal point to be at the center of the image. Are you keeping camera intrinsics constant during SfM estimation? If you keep them constant then you should have a direct agreement between bundler features and adjusted features

[Chris Sweeney]

By the way, Theia has a hash function implementation for std::pair types. So if you read in the map from 1) you can fill a 

std::unordered_map<std::pair<double, double>, std::pair<double, double> > 

by including //theia/util/hash.h

[rajvi shah]

Hi Chris,

Thanks for the tips. I will look into these options.

I am using create_calibration_from_exif.
So option 2 seems the easiest way, but I assume keeping intrinsics constant would probably also not optimize focal lengths, which may not be ideal.

I will try both options and see which one works better.

Also thanks for the tip on hash function. :)

Regards,

Rajvi

[rajvi shah]

Hi Chris,

On a second look there is still a discrepancy between re-projection errors, not as severe as before though. But something is off.
For the same exercise of converting theia to bundler and back and measuring reprojection errors, this is what I get,

# Original Stats

Num reprojections behind camera: 0

Mean reprojection error = 1.31752
Median reprojection_error = 1.09254

# Stats after conversion
Num reprojections behind camera: 6
Mean reprojection error = 6.86311
Median reprojection_error = 1.10031

Ideally shouldn't the two be very similar?

This probably isn't a big problem but I thought I would bring it to your notice.

I have attached the original theia reconstruction file and a script that lists the commands I used. 

Regards,

Rajvi

[Chris Sweeney]

Hi Rajvi,

Thanks for sending your files, that makes diagnosing the problem 100x simpler for me. A tool that you can use here is the compare_reconstructions application. When I run that, I see the comparison below. To be honest, I think it may just be floating point errors? Let me do a quick test to confirm this.

I1006 10:13:39.422787 20833 compare_reconstructions.cc:198] Number of cameras:

Reconstruction 1: 464

Reconstruction 2: 464

Number of Common cameras: 464

I1006 10:13:39.422876 20833 compare_reconstructions.cc:205] Number of 3d points:

Reconstruction 1: 13587

Reconstruction 2: 13587

I1006 10:13:39.436990 20833 compare_reconstructions.cc:116] Rotation difference when aligning orientations:

Mean = 0.021759

Median = 0.021759

Histogram:

< 1.000000 = 464

[1.000000 - 2.000000) = 0 

[2.000000 - 5.000000) = 0 

[5.000000 - 10.000000) = 0 

[10.000000 - 15.000000) = 0 

[15.000000 - 20.000000) = 0 

[20.000000 - 45.000000) = 0 

I1006 10:13:39.438375 20833 compare_reconstructions.cc:159] Pose error:

Rotation Error:

Mean = 0.000001 

Median = 0.000001

Histogram:

< 1.000000 = 464

[1.000000 - 2.000000) = 0 

[2.000000 - 5.000000) = 0 

[5.000000 - 10.000000) = 0 

[10.000000 - 15.000000) = 0 

[15.000000 - 20.000000) = 0 

[20.000000 - 45.000000) = 0 

Position Error:

Mean = 0.024774 

Median = 0.006287

Histogram:

< 1.000000 = 464

[1.000000 - 5.000000) = 0 

[5.000000 - 10.000000) = 0 

[10.000000 - 50.000000) = 0 

[50.000000 - 100.000000) = 0 

[100.000000 - 1000.000000) = 0 

I1006 10:13:39.438477 20833 compare_reconstructions.cc:165] Focal length errors: 

Mean = 0.000000

Median = 0.000000

Histogram:

< 0.010000 = 464

[0.010000 - 0.050000) = 0 

[0.050000 - 0.200000) = 0 

[0.200000 - 0.500000) = 0 

[0.500000 - 1.000000) = 0 

[1.000000 - 10.000000) = 0 

[10.000000 - 100.000000) = 0 

[Chris Sweeney]

Yes, indeed it was a floating point type of error (see this patch). Sync to the latest repo to integrate that patch. When I run your experiment and compare_reconstructions I get zero error now!!

Thanks much for the notes and as always let me know how I can best help if you have more issues.

Chris

[rajvi shah]

Hi Chris,

Thanks for the update again. I have run into a new problem now. 

I had previously used compare_reconstructions application to compare various reconstruction outputs.

For that I first used to convert bundle files to theia reconstructions using convert_bundle_file application and then use compare reconstructions for comparing the two reconstructions. This used to work in an earlier version of the code, before cereal library based read write was introduced. But now I receive segmentation fault when using this application.

The segmentation fault occurs in EvaluateRotations functions in compare_reconstructions.cc. 

In the for loop that populates rotation1 and rotation2 vectors. 

Here

 rotations1.push_back(

        reconstruction1.View(view_id1)->Camera().GetOrientationAsAngleAxis());

    rotations2.push_back(

        reconstruction2.View(view_id2)->Camera().GetOrientationAsAngleAxis());

I have attached one example set of reconstructions, theia files, original bundle and list files (list file is the same for both). 

Can you please look into it? Thanks again. 

​

 compare_reconstructions_test.tar.gz

​

Regards,

Rajvi

 

[Chris Sweeney]

Hi Rajvi,

It's taken me a while to understand why this is happening. What I believe is going on is that in read/write_reconstructions.cc the method is exiting before Cereal finishes reading/writing the file. For writing the file this leaves and incomplete representation, which is why we get an error when reading the file back. I think I have a fix for it (still running some experiments to be sure) and I will let you know once it is merged to the latest repo.

Also, you'll notice in the HEAD of the repo that today I made some major refactors for the global estimation pipeline. For end-users, this should only affect a few flags in the flagfile (see this commit). Only a few flags were changed, but several more were added.

Chris

[Chris Sweeney]

Ok here's another update. I believe the reconstruction file you created using Theia (i.e. the coarse one) is mal-formed. Not your fault, but probably is a result on the bug I mentioned. Here's what I have done:

I converted the bundler files to pantheon_gt. Below are the comparisons I ran,

(note the order  of command lines for these)

pantheon_gt vs pantheon_theia

• Compare reconstructions  reconstruction1=pantheon_gt reconstruction2=pantheon_theia  -------> Works consistently (and is verified to be 100% the same)
  
• Compare reconstructions  reconstruction1=pantheon_theia reconstruction2=pantheon_gt  -------> Works consistently (and is verified to be 100% the same)
  

pantheon_gt vs pantheon_coarse

• Compare reconstructions  reconstruction1=pantheon_gt reconstruction2=pantheon_coarse  -------> Fails
  
• Compare reconstructions  reconstruction1=pantheon_coarse reconstruction2=pantheon_gt  -------> Works consistently (???)

pantheon_coarse vs pantheon_theia

• Compare reconstructions  reconstruction1=pantheon_coarse reconstruction2=pantheon_theia  -------> Works consistently (???)
  
• Compare reconstructions  reconstruction1=pantheon_theia reconstruction2=pantheon_coarse  -------> Fails
  

It's very puzzling that any time pantheon_coarse is the 2nd reconstruction, it fails. Not sure what's up with that. But when I run compute_reconstruction_statistics on pantheon_coarse, it has some wild reprojection errors, so I definitely think the file was unfortunately mal-formed.

I've submitted this patch. Hopefully that will solve your problem for the future, but I don't think there's much we can do now to recover your coarse file since it seems that the file write was cut off before it was finished. Sorry :(

[Pierre Moulon]

Hi,

I notice that you never close the stream  sweeneychris/TheiaSfM / src/theia/io/write_matches.cc

That perhaps the reason why you need the {}

Because the stream has not been flushed ;-)

Regards/Cordialement,

Pierre M

[rajvi shah]

​​Hi Chris, Pierre,

Thanks for the update on the the writer functions.
I am not really sure though that the source of the problem is incomplete write.

I feel I was not very clear about the data files I sent you.

Both gt and coarse reconstructions are generated by bundler and not theia.
I have the original .out files (also in the tarball), so I didn't really lose anything. :)

Here is what I did,

0. I want to compare two reconstructions (for now both are generated by bundler and in bundler's format)

1. Convert bundler_output1 (coarse) to theia_reconstruction1
2. Convert bundler_output2 (ground-truth) to theia_reconstruction2

3. Compare theia_reconstruction1 and theia_reconstruction2

I have attached original bundler outputs and a small script that replicates this experiment.

Now, this used to work (didn't produce segfault at least) with older version (pre-cereal version).

Diagnosing it further, I've found the problem originates here.

  const std::vector<std::string> common_view_names =
      theia::FindCommonViewsByName(*reconstruction1, *reconstruction2);
 

The vector common_view_names is always the size of the first reconstruction!!
I found this in debug mode and also in the log file right before the program crashes.
So when the first reconstruction is coarse and has fewer cameras than the second reconstruction, all of which are common to the second reconstruction, it works without a problem.

When the second reconstruction is coarse, the number of actually estimated cameras is less than the common_view_names and hence the segfault during EstimateRotations.

I think the reason for this could be this,
Bundler writes all cameras to output, even the ones which are not registered as "0 0 0".

The list file essentially has num_images entries, including the cameras that are not registered.

In earlier version of reconstruction_writer (now deprecated), only the estimated cameras were written.

The newer version removes these views before writing the reconstruction but the file names somehow still must be getting written somehow.
I could not figure out this part though.

I have attached the new tar file with both bundle reconstructions (coarse and gt), list file (common for both) and the script I used for the experiment.

​​​

 compare_reconstructions_test_new.tar.gz

​

The problem of crazy reprojection errors is something else altogether.
I get the same reprojection errors using both old and new versions of the theia code.

The reconstructions are generated by bundler so the errors ideally shouldn't be so high.

The median error is pretty normal on the other hand and the coarse reconstruction looks very neat.

This makes me think that it would have to do with one or two or a small number of outlier points than theia.
How these points survived incremental bundle adjustment is not clear to me though but that's a theia problem.

Sorry for a lengthy post. Thanks again.

Regards,

Rajvi

[Chris Sweeney]

Rajvi,

You are 100% correct, and this is a great diagnosis. In Reconstruction::RemoveView I was not removing the view name from the internal unordered_map. See this patch, which includes a unit test to cover this scenario properly. I have also refactored the write_reconstructions file to only write out the estimated views and points. I verified that both patches work on your dataset, so I hope this fixes your issues!

Chris

[rajvi shah]

Hi Chris,

Thanks for the changes! 

The new version works fine, tested on several files. 

Regards,

Rajvi