Issues with COLMAP running on clusters

[zl...@cornell.edu]

Hello,

I am trying to run COLMAP in my lab's clusters without display but after I compiled the code there were mainly two problems:

(1) For one of the clusters, it could run feature extraction in CPU mode, but when I run feature matcher, it will issues "multiply descriptor: an illegal memory access was encountered" continuously (but the program sometimes didn't crash but matching speed is super fast which is werid). However, when I run SfM using these feature matchers' result, I found these matches were definitely wrong. Therefore, do you know the reason it will issues "multiply descriptor: an illegal memory access was encounter"? Is it related to some problems of gpu architecture?

(2) For another cluster, when I run any executable such as feature extractor, they all directly crash giving me error "Illegal instruction (core dumped)". 

Two clusters are similar, both's CUDA version is 7.5, output of nvidia-smi is below: 

| NVIDIA-SMI 361.42     Driver Version: 361.42         |                       

|-------------------------------+----------------------+----------------------+

| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |

| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |

|===============================+======================+======================|

|   0  GeForce GTX TIT...  Off  | 0000:02:00.0     Off |                  N/A |

| 22%   42C    P0    73W / 250W |     23MiB / 12287MiB |      0%      Default |

+-------------------------------+----------------------+----------------------+

I would appreciate it if someone can help me figure our the problem

Thank you,

Zhengqi Li

[Johannes Schönberger]

Hi,

Can you post the command that you used to run the *_matcher?

Did you see: https://colmap.github.io/cli.html#example and the last paragraph in that section?

Best,
Johannes

> --
> You received this message because you are subscribed to the Google Groups "COLMAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to colmap+un...@googlegroups.com.
> To post to this group, send email to col...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/colmap/397b130d-5b0e-47f0-adb1-09bafdd8d2dd%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[zl...@cornell.edu]

Hi,

Thank you for quick reply. My first two commands are as follow:

$./feature_extractor --General.database_path $PROJECT_PATH/database.db --General.image_path $PROJECT_PATH --use_gpu false

$./vocab_tree_matcher --MatchOptions.gpu_index 0 --VocabTreeMatchOptions.vocab_tree_path /phoenix/S2/zl548/vocab_tree-262144.bin --General.database_path $PROJECT_PATH/database.db

Then when I run the matcher, part of output of my terminal is:

Matching image [189/297]MultiplyDescriptor: an illegal memory access was encountered

MultiplyDescriptor: an illegal memory access was encountered

MultiplyDescriptor: an illegal memory access was encountered

MultiplyDescriptor: an illegal memory access was encountered

MultiplyDescriptor: an illegal memory access was encountered

MultiplyDescriptor: an illegal memory access was encountered

......

Actually, the code didn't crash, but I don't think the matches results are correct.

Thank you,

Zhengqi Li

[zl...@cornell.edu]

By the way, when I try exhaustive matching on another dataset, it crashed with following output:

Matching block [1/27, 1/27]MultiplyDescriptor: an illegal memory access was encountered

*** Aborted at 1484844693 (unix time) try "date -d @1484844693" if you are using GNU date ***

MultiplyDescriptor: an illegal memory access was encountered

MultiplyDescriptor: an illegal memory access was encountered

PC: @           0x5cd1d4 colmap::TwoViewGeometry::EstimateCalibrated()

*** SIGSEGV (@0x7f5e240a6e80) received by PID 48737 (TID 0x7f5e617fa700) from PID 604663424; stack trace: ***

    @     0x7f5ed9868330 (unknown)

    @           0x5cd1d4 colmap::TwoViewGeometry::EstimateCalibrated()

MultiplyDescriptor: an illegal memory access was encountered

MultiplyDescriptor: an illegal memory access was encountered

    @           0x5117a5 colmap::SiftFeatureMatcher::VerifyImagePair()

    @           0x5254ee std::_Function_handler<>::_M_invoke()

    @           0x548cb9 _ZNSt5_BindIFSt8functionIFvN6colmap18SiftFeatureMatcher25GeometricVerificationDataERKNS1_16SiftMatchOptionsEPNS1_15TwoViewGeometryEEES3_S4_S8_EE6__callIvJEJLm0ELm1ELm2EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE

    @           0x548d68 std::_Function_handler<>::_M_invoke()

    @           0x522ede std::_Function_handler<>::_M_invoke()

    @           0x5234fe std::__future_base::_State_base::_M_do_set()

MultiplyDescriptor: an illegal memory access was encountered

    @     0x7f5ed9865a80 __GI___pthread_once

    @           0x53ba9f std::__future_base::_Task_state<>::_M_run()

    @           0x5276f5 _ZNSt17_Function_handlerIFvvEZN6colmap10ThreadPool7AddTaskIRSt8functionIFvNS1_18SiftFeatureMatcher25GeometricVerificationDataERKNS1_16SiftMatchOptionsEPNS1_15TwoViewGeometryEEEJRS6_S9_SB_EEESt6futureINSt9result_ofIFT_DpT0_EE4typeEEOSI_DpOSJ_EUlvE_E9_M_invokeERKSt9_Any_data

    @           0x59e023 colmap::ThreadPool::WorkerFunc()

    @     0x7f5ed6913a60 (unknown)

    @     0x7f5ed9860184 start_thread

    @     0x7f5ed607b37d (unknown)

    @                0x0 (unknown)

Segmentation fault (core dumped)

The command I used is:

./src/exe/exhaustive_matcher --General.database_path /phoenix/S2/zl548/colmap_image/0040/db.db --MatchOptions.gpu_index 0

In addition, I used a database I run on my own laptop where I finished feature extraction and matching and I have no problem with SfM part using CUDA in the cluster so I don't think the reason comes from CUDA.

Thanks