CMTK plugin error & Docker

George Yerou

unread,

Jun 12, 2020, 5:57:53 AM6/12/20

to nat-user

Dear Dr Jefferis,

Following my previous issues trying to set up NBLAST locally, Docker on linux worked well although I had to use devtools::install_github("aoles/shinyURL") to get it to work.

I am currently having some issues with CMTK. Upon using CMTK, the the commands folder generated contains a .sh file. I open, copy, paste and run the commands in terminal, but receive error messages in the output.

george@george-VirtualBox:~$ cd "/home/george/Documents/expt/yourregistrationfolder"
george@george-VirtualBox:~/Documents/expt/yourregistrationfolder$ "/home/george/Fiji.app/bin/cmtk/munger" -b "/home/george/Fiji.app/bin/cmtk" -a -w -r 010203 -X 26 -C 8 -G 80 -R 4 -A '--accuracy 0.4' -W '--accuracy 0.4' -T 2 -s "refbrain/JRC2018_U.nrrd" images
hostname = george-VirtualBox; short hostname = george-VirtualBox
JOB ID =
Root directory is /home/george/Documents/expt/yourregistrationfolder
Changing to root directory: /home/george/Documents/expt/yourregistrationfolder
Setting image root to: images
Aff:brain1_01 /home/george/Fiji.app/bin/cmtk/registration: error while loading shared libraries: libdcmdata.so.4: cannot open shared object file: No such file or directory
MYEXEC: CMD = /home/george/Fiji.app/bin/cmtk/registration -i -v --dofs 6 --dofs 9 --accuracy 0.4 -o Registration/affine/JRC2018_brain1_01_9dof.list refbrain/JRC2018_U.nrrd images/brain1_01.nrrd exited with value 127
Aff:brain1_01 /home/george/Fiji.app/bin/cmtk/registration: error while loading shared libraries: libdcmdata.so.4: cannot open shared object file: No such file or directory
MYEXEC: CMD = /home/george/Fiji.app/bin/cmtk/registration -i -v --dofs 6 --dofs 9 --accuracy 0.4 -o Registration/affine/JRC2018_brain1_01_9dof.list refbrain/JRC2018_U.nrrd images/brain1_01.nrrd exited with value 127

Ran 0 initial affine registrations of which 0 failed
Ran 2 affine registrations of which 2 failed
Ran 0 warp registrations
Reformatted 0 images

From the output it seems to think that Fiji.app/bin/cmtk/registration does not exist - however I have checked and it does.

Someone who has used CMTK previously (on mac) tried rerunning the contents of their commands file (which previously worked) and they informed that they were receiving the same errors - which lead me to wonder whether something has changed?

Any help would be massively appreciated!

Best wishes,

George.

Greg Jefferis

unread,

Jun 15, 2020, 2:58:08 PM6/15/20

to George Yerou, nat-user

Dear George,

Great that you managed to get up and running with Docker.

As for this problem, you actually have 2 versions of CMTK now. One that I compiled from source when making the docker image. That is in /usr/local and another that was downloaded by the Fiji CMTK plugin. The CMTK error that you see below is related to a missing library that the CMTK binaries downloaded by the Fiji CMTK plugin expect to be installed (libdcmdata is related to reading DICOM images).

You should be able to get your scripts to by editing them in a text editor, replacing

-b "/home/george/Fiji.app/bin/cmtk"

with

-b "/usr/local/lib/cmtk/bin"

You can find further details about the error on the CMTK mailing list [1,2], but essentially, it Is necessary to compile CMTK from scratch on recent versions of ubuntu.

Two improvements that I hope to make would be to provide nightly binaries of cmtk built on modern ubuntu version and make the Fiji CMTK use an existing CMTK installation if It can find one [3]. Of course user contributions are very welcome in this area.

All the best,

Greg.

[1] https://www.nitrc.org/forum/forum.php?thread_id=10675&forum_id=857

[2] https://www.nitrc.org/forum/message.php?msg_id=21800

[3] https://github.com/jefferis/fiji-cmtk-gui/issues/6

George Yerou

unread,

Jun 18, 2020, 2:42:07 PM6/18/20

to nat-user

Dear Dr Jefferis,

Thank you very much for your response! I think I have managed to get it working by compiling CMTK from source using [2] - although I had to rename the folder called "cmtk 3.3.1" to "core" to get the script to work. The script is currently running.

My aim is to register images to the jrc2018 template brain, convert the reformatted image to colour depth MIP and use a mask search to find a neuron of interest in flyEM database so it can be further studied in neuprint. The database I intend to conduct the mask search against is the EM dataset found at https://www.janelia.org/open-science/color-depth-mip - which contains colour depth mips of flyEM neurons. I was wondering whether you think this is a good approach to take or whether you would suggest different ways of finding a neuron of interest in the EM dataset.

If you are interested I have outlined some of the issues I encountered with regards to your suggestion of using -b "/usr/local/lib/cmtk/bin". Specifically, I received the error:

"/home/george/Fiji.app/bin/cmtk/munger" -b "/usr/local/lib/cmtk/bin" -a -w -r 010203 -X 26 -C 8 -G 80 -R 4 -A '--accuracy 0.4' -W '--accuracy 0.4' -T 1 -s "refbrain/JRC2018_U.nrrd" images

hostname = george-VirtualBox; short hostname = george-VirtualBox

Can't access binary directory /usr/local/lib/cmtk/bin at /home/george/Fiji.app/bin/cmtk/munger line 121.

Having looked I found that usr/local/lib/cmtk/bin was under /home/cmtk not /usr/local however when I modified the script to reflect this, it returned same error as before.

Thank you for your time, it is really appreciated.

Best Wishes,

George.

Greg Jefferis

unread,

Jun 19, 2020, 2:34:13 PM6/19/20

to nat-user

Dear George,

On Thursday, June 18, 2020 at 7:42:07 PM UTC+1, George Yerou wrote:

Thank you very much for your response! I think I have managed to get it working by compiling CMTK from source using [2]

cmtk should already be installed in the Docker image at

/usr/local/lib/cmtk/bin

You can check this by doing

ls /usr/local/lib/cmtk/bin

in the docker command line [1-3]. If it isn't there perhaps you didn't pull the latest docker image, although this was current as of a month ago. You could try doing

docker images

and reporting what you find.

- although I had to rename the folder called "cmtk 3.3.1" to "core" to get the script to work. The script is currently running.

Yes I see that's correct. If you have a moment you could correct Torsten's howto on the CMTK mailing list. Nobody else pointed it out, but his instructions were correct if you svn / git clone the repository but not if you download the tgz.

My aim is to register images to the jrc2018 template brain, convert the reformatted image to colour depth MIP and use a mask search to find a neuron of interest in flyEM database so it can be further studied in neuprint. The database I intend to conduct the mask search against is the EM dataset found at https://www.janelia.org/open-science/color-depth-mip - which contains colour depth mips of flyEM neurons. I was wondering whether you think this is a good approach to take or whether you would suggest different ways of finding a neuron of interest in the EM dataset.

This should work, however if your intent is to search for EM neurons, there are other ways including NBLASTing the hemibrain neurons, or jumping to a location in the hemibrain dataset. It depends a bit what your query object is. A neuron? A GAL4 line? If it is something less sparse like a GAL4 line then the MIP search is a good option.

If you are interested I have outlined some of the issues I encountered with regards to your suggestion of using -b "/usr/local/lib/cmtk/bin". Specifically, I received the error:

"/home/george/Fiji.app/bin/cmtk/munger" -b "/usr/local/lib/cmtk/bin" -a -w -r 010203 -X 26 -C 8 -G 80 -R 4 -A '--accuracy 0.4' -W '--accuracy 0.4' -T 1 -s "refbrain/JRC2018_U.nrrd" images
hostname = george-VirtualBox; short hostname = george-VirtualBox
Can't access binary directory /usr/local/lib/cmtk/bin at /home/george/Fiji.app/bin/cmtk/munger line 121.

Having looked I found that usr/local/lib/cmtk/bin was under /home/cmtk not /usr/local however when I modified the script to reflect this, it returned same error as before.

So I am puzzled by this as the docker image should include cmtk at that location as mentioned above. In terms of why you might have needed to change the command line a second time, this is because Torsten's howto did not suggest doing

make install

at the end, which would have resulted in the tools being installed to

/usr/local/lib/cmtk/bin

instead of just being left in the original build location in your home folder.

I think we really need to figure out why cmtk is not in the correct build location. I have tried on my docker image and I can run munger with the

Thank you for your time, it is really appreciated.

[1]

(base) Gregs-MBP-2:r-natverse jefferis$ docker image ls

REPOSITORY TAG IMAGE ID CREATED SIZE

natverse/shiny-natverse latest f2ce9cc40882 4 weeks ago 3.43GB

rocker/shiny-verse latest 2dc98c900e0b 4 weeks ago 2.03GB

virtualflybrain/nblast-on-the-fly latest 097a6e39c75d 4 weeks ago 2.9GB

natverse/r-natverse <none> bb94d8d6350c 4 weeks ago 3.66GB

natverse/r-natverse latest 705e66ff670b 4 weeks ago 3.66GB

virtualflybrain/nblast-on-the-fly 3.6.3 5cb15356ed26 4 weeks ago 2.9GB

<none> <none> c972df413975 4 weeks ago 3.47GB

natverse/r-natverse-java latest 0a381d7b76ea 4 weeks ago 3.59GB

<none> <none> c64a156f68af 4 weeks ago 3.57GB

<none> <none> 6e606ddff39a 4 weeks ago 1.31GB

<none> <none> 5ddfdb28997f 4 weeks ago 2.22GB

<none> <none> 78069cc3165b 4 weeks ago 910MB

<none> <none> f378a671f797 4 weeks ago 2.63GB

<none> <none> 6a3b3cf8c567 4 weeks ago 2.73GB

<none> <none> e5cd6df3e212 4 weeks ago 2.63GB

rocker/tidyverse latest ba7955185ce8 4 weeks ago 2.31GB

rocker/rstudio latest bbe56e7a74af 4 weeks ago 1.69GB

rocker/r-ver latest 54dd779d5f43 4 weeks ago 809MB

r-base latest 3aad1ffccc53 3 months ago 679MB

[2] run a container starting the bash shell

docker run --rm -ti -v $"HOME":/home/rstudio --user rstudio bb94d8d6350c

rstudio@d272e87b5373:/$ ls /usr/local/lib/cmtk/bin/

asegment cmtk_locking.sh dcm2image fibxform fit_spline_xform groupwise_reformat jidb mcaffine overlap reorient stream_pixels vol2csv xform2dfield

average_affine concat_affine describe film geomatch groupwise_warp levelset mcwarp probe sba streamxform volume_injection xform2itk

average_images convert_warp destripe filter glm hausdorff lmsba mk_adni_phantom pxsearch sbai sympl volume_reconstruction xform2scalar

average_labels convertx dof2mat fit_affine_dfield gmm histogram lsba mk_analyze_hdr reformatx sequence symplx vtkxform

avg_adm correct_dwi_distortion dwi_mask_bad_slices fit_affine_xform gregxform imagemath lvote mk_nifti_hdr registration similarity ttest warp

cmtk_functions.sh correct_dwi_distortion_and_motion epiunwarp fit_affine_xform_landmarks groupwise_affine interleaved_bad_slices make_initial_affine mk_phantom_3d registrationx split unsplit warp2ps

cmtk_locking_procmail.sh correct_nex_motion fib2image fit_spline_dfield groupwise_init iterative_shape_averaging mat2dof mrbias regress statistics unwarp_image_phantom warpx

[3]

[4]

George Yerou

unread,

Jun 27, 2020, 3:42:56 PM6/27/20

to nat-user

Dear Dr Jefferis,

Many thanks for your time, I have addressed some of the topics from our earlier discussion below.

On Friday, June 19, 2020 at 7:34:13 PM UTC+1, Greg Jefferis wrote:

Dear George,

On Thursday, June 18, 2020 at 7:42:07 PM UTC+1, George Yerou wrote:
Thank you very much for your response! I think I have managed to get it working by compiling CMTK from source using [2]

My aim is to register images to the jrc2018 template brain, convert the reformatted image to colour depth MIP and use a mask search to find a neuron of interest in flyEM database so it can be further studied in neuprint. The database I intend to conduct the mask search against is the EM dataset found at https://www.janelia.org/open-science/color-depth-mip - which contains colour depth mips of flyEM neurons. I was wondering whether you think this is a good approach to take or whether you would suggest different ways of finding a neuron of interest in the EM dataset.

This should work, however if your intent is to search for EM neurons, there are other ways including NBLASTing the hemibrain neurons, or jumping to a location in the hemibrain dataset. It depends a bit what your query object is. A neuron? A GAL4 line? If it is something less sparse like a GAL4 line then the MIP search is a good option.

- although I had to rename the folder called "cmtk 3.3.1" to "core" to get the script to work. The script is currently running.

Yes I see that's correct. If you have a moment you could correct Torsten's howto on the CMTK mailing list. Nobody else pointed it out, but his instructions were correct if you svn / git clone the repository but not if you download the tgz.

I have amended this on the cmtk users site.

`

Yes, my intent is to search for EM neurons - ie I have a confocal image of my neuron and I would like to find it in neuprintexplorer so I can query its connections. You mention that I could NBLASTing with hemibrain neurons directly, but I can't quite find a way to do so. The elmr package looked promising but its functions (eg fetchdp_fafb and nblast_fafb) are more geared towards finding an EM tracing's counterpart in the flycircuit database and not the reverse, requiring a catmaid skeleton ids which are not listed for neurons in neuprintexplorer. The EM hemi brain data can be downloaded from Google Bucket and gsutil gs://hemibrain-release/neuprint/hemibrain_v1.0, would it be possible to NBLAST against this data instead of the flycircuit data to obtain neuron ids that could be further investigated in neuprint explorer?

If you are interested I have outlined some of the issues I encountered with regards to your suggestion of using -b "/usr/local/lib/cmtk/bin". Specifically, I received the error:

"/home/george/Fiji.app/bin/cmtk/munger" -b "/usr/local/lib/cmtk/bin" -a -w -r 010203 -X 26 -C 8 -G 80 -R 4 -A '--accuracy 0.4' -W '--accuracy 0.4' -T 1 -s "refbrain/JRC2018_U.nrrd" images
hostname = george-VirtualBox; short hostname = george-VirtualBox
Can't access binary directory /usr/local/lib/cmtk/bin at /home/george/Fiji.app/bin/cmtk/munger line 121.

Having looked I found that usr/local/lib/cmtk/bin was under /home/cmtk not /usr/local however when I modified the script to reflect this, it returned same error as before.

So I am puzzled by this as the docker image should include cmtk at that location as mentioned above. In terms of why you might have needed to change the command line a second time, this is because Torsten's howto did not suggest doing

The cmtk, that I think belongs to the docker image, resides at "/usr/lib/cmtk/bin" instead of '/usr/local/lib/cmtk/bin', if that is any help.

Greg Jefferis

unread,

Jul 6, 2020, 3:05:14 PM7/6/20

to gj.y...@gmail.com, nat-user

Dear George,

This is all as I was expecting. It looks like you have not been running your commands using the docker image in the past.

Imagine you have a .command or .sh file generated by the CMTK GUI located here

/home/george/registration/myscript.sh

You would do:

docker run --rm -ti -v "$HOME":/home/rstudio --user rstudio natverse/r-natverse bash

And then

sh ~/registration/myscript.sh

And it should start to run.

Note that this part of the docker command

-v "$HOME":/home/rstudio

Ensures that docker can see your home folder on your hard disk

All the best,

Greg.

On 6 Jul 2020, at 17:57, <gj.y...@gmail.com> <gj.y...@gmail.com> wrote:

Dear Dr Jefferis,
Here is the output:
<image002.jpg>
Please let me know if you would like me to try anything else.

Best wishes,
George.

From: Greg Jefferis <jeff...@gmail.com>
Sent: 06 July 2020 17:46
To: George Yerou <gj.y...@gmail.com>
Subject: Re: [nat-user] CMTK plugin error & Docker

# run this in your terminal
docker run --rm -ti -v "$HOME":/home/rstudio --user rstudio natverse/r-natverse bash

# then run this after the docker command line starts
ls /usr/local/lib/cmtk/bin/
ls /usr/lib/cmtk/bin
exit

On 6 Jul 2020, at 17:43, <gj.y...@gmail.com> <gj.y...@gmail.com> wrote:

<image002.jpg>

Reply all

Reply to author

Forward