Grappa Reconstruction Error

[joseph valentine]

Hey everyone,

I have a fresh install of gadgetron on ubuntu 16.04 and was hoping to use it to help with some parallel imaging reconstructions using GRAPPA.

here is my gadgetron_info output:

Gadgetron Version Info

  -- Version            : 3.14.1

  -- Git SHA1           : 37a70616d6335111ec846f37f02643bf98eaed14

  -- System Memory size : 48292 MB

  -- Python Support     : YES

  -- Matlab Support     : NO

  -- CUDA Support       : YES (-gencode arch=compute_30,code=sm_30;-gencode arch=compute_35,code=sm_35;-gencode arch=compute_50,code=sm_50;-gencode arch=compute_52,code=sm_52)

    * Unable to get device count

I used the siemens_to_ismrmrd package to convert some 3D 7T data and i have been trying to use the Generic_Cartessian_Grappa.xml chain to do the reconstructions

unfortunately i keep running into a few errors about halfway through the recon that have been keeping it from working properly:

2016-11-05 19:40:49 ERROR [hoNDArray_linalg.cpp:1330] info==0

2016-11-05 19:40:49 ERROR [hoNDArray_linalg.cpp:1334] Errors in posv(hoNDArray<T>& A, hoNDArray<T>& b) ...

2016-11-05 19:40:49 ERROR [hoNDArray_linalg.cpp:1927] posv failed in SolveLinearSystem_Tikhonov(... ) ...

(i can post the entire debug printout if thats helpful)

I have had a really hard time tracking down the cause and i havent seen any posts about this problem on here so i thought i might ask; Does anyone know what might be causing this error?

Thanks again for all your help!

[Michael Hansen]

We will likely need a dataset in order debug this. Can you provide one?

Also, full logs would be nice and are you using MKL? If so, which version?

This email was written on a mobile device. Please excuse brevity and/or typos.

_____________________________
From: joseph valentine
Sent: Saturday, November 5, 2016 21:52
Subject: Grappa Reconstruction Error
To: Gadgetron <gadg...@googlegroups.com>

--
You received this message because you are subscribed to the Google Groups "Gadgetron" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gadgetron+...@googlegroups.com.
To post to this group, send email to gadg...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gadgetron/001eab91-c8a7-4422-8cbd-4e313dc099c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Xue, Hui (NIH/NHLBI) [E]]

Hi, thank you for the message. Could you upload one dataset which error happened?

We can try to debug it.

Regards

Hui Xue
________________________________________
From: joseph valentine [josephva...@gmail.com]
Sent: Saturday, November 05, 2016 9:52 PM
To: Gadgetron
Subject: Grappa Reconstruction Error

--
You received this message because you are subscribed to the Google Groups "Gadgetron" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gadgetron+...@googlegroups.com<mailto:gadgetron+...@googlegroups.com>.
To post to this group, send email to gadg...@googlegroups.com<mailto:gadg...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/gadgetron/001eab91-c8a7-4422-8cbd-4e313dc099c2%40googlegroups.com<https://groups.google.com/d/msgid/gadgetron/001eab91-c8a7-4422-8cbd-4e313dc099c2%40googlegroups.com?utm_medium=email&utm_source=footer>.

[joseph valentine]

Thank you so much for your help Michael and Hue,

I included the full debug log and the dataset.h file (i can include the orignial dat file too if that is helpful).

here is the link for the files: 

https://drive.google.com/drive/folders/0B8AzCaEXugCKZ0wtdUlVRkxjc0E?usp=sharing

i am not using MKL

[Hui Xue]

Hi,

I downloaded the attachment. They are log files and output h5 files.

Could you please upload the raw data for this case? So I can try to reproduce the problem.

regards

Hui Xue

hui...@nih.gov

在 2016年11月5日星期六 UTC-4下午9:52:09，joseph valentine写道：

[Yishi Wang]

Hi joseph,

I saw that you enabled CUDA support. Which cuda and cula version did you install? It seems that there are currently two cula libraries, i.e. cula dense and cula sparse. Which one did you use?

Thanks

Yishi

在 2016年11月6日星期日 UTC+8上午9:52:09，joseph valentine写道：

[Michael Hansen]

Just FYI, CUDA and CULA are not the same thing. CULA is not needed anymore to compile the Gadgetron, but you may need it for your own stuff of course. 

[Yishi Wang]

Thanks, Michael. 

Another thing I found was when using a new Nvidia card such as GTX1060, cuda 8.0 must be used to compile. The nvidia-cuda-toolbox in Ubuntu source will not work since it's 7.5.

Regards

Yishi

在 2017年1月6日星期五 UTC+8下午10:27:44，Michael Hansen写道：

[joseph valentine]

Thank you Michael, Yishi and Hue,

My cuda version is v7.5.17 so perhaps that is the problem. I will try recompiling with cuda updated to v8 and see if that helps. I have also uploaded the raw data set in the drive folder that i posted a link to earlier.

[Will Clarke]

I've also encountered this error on one of the machines I'm running Gadgetron on. 3D flash images, from a Siemens 7T, produce this error on an ubuntu (16.04) virtual machine, but do not on an ubuntu (16.04) server which we have connected to the scanner for online processing. 

I've checked that the error is independent of which machine the data was converted to ismrmrd format on.

The Gadgetron Version info is as follows:
VM (error)

  -- Version            : 3.14.1

  -- Git SHA1           : 799ef265a27b6aa5a0ed9816559878b56ae43cac

  -- System Memory size : 7983 MB

  -- Python Support     : YES

  -- Matlab Support     : NO

  -- CUDA Support       : YES (-gencode arch=compute_30,code=sm_30;-gencode arch=compute_35,code=sm_35;-gencode arch=compute_50,code=sm_50;-gencode arch=compute_52,code=sm_52)

NVIDIA: no NVIDIA devices found

    * Unable to get device count

Online Server (no error)

   -- Version            : 3.14.1

  -- Git SHA1           : 83f3a0752654771e2224e00eefc16fe9f96490ec

  -- System Memory size : 64188 MB

  -- Python Support     : YES

  -- Matlab Support     : NO

  -- CUDA Support       : YES ( -gencode arch=compute_35,code=sm_35 )

    * Number of CUDA capable devices: 1

      - Device 0: Tesla K40m

         + CUDA Driver Version / Runtime Version: 8.0/7.5

         + CUDA Capability Major/Minor version number: 3.5

         + Total amount of global GPU memory: 11439.9 MB

Both machines pass the test cases for Generic_Cartesian_Grappa.

Raw data can be found here: https://oxfile.ox.ac.uk/oxfile/work/extBox?id=411104DC0F10798AA6 (only the one large 3d dataset is pertinent here).

[Michael Hansen]

Which error?

In any case, you are probably running out of memory. You only have 8GB, which is probably not enough for a multi-coil 3D dataset. 

[Will Clarke]

Sorry, I should have been more specific.

2017-01-16 09:14:16 ERROR [hoNDArray_linalg.cpp:1330] info==0

2017-01-16 09:14:16 ERROR [hoNDArray_linalg.cpp:1334] Errors in posv(hoNDArray<T>& A, hoNDArray<T>& b) ...

2017-01-16 09:14:16 ERROR [hoNDArray_linalg.cpp:1927] posv failed in SolveLinearSystem_Tikhonov(... ) ... 

I'll try bumping the memory up.

[Will Clarke]

After establishing that this data worked on some others' installations (thanks to those on the hangout) and that it worked in the docker (1604 no_cuda) package. I was able to resolve this error by installing all the packages listed at the start of https://github.com/gadgetron/gadgetron/blob/master/docker/base/ubuntu_1604/Dockerfile and rebuilding gadgetron. I had followed this list on the server (connected to our scanner which had worked all along), but had previously set up my vm using the list here: https://github.com/gadgetron/gadgetron/wiki/Linux-Installation.

These two lists differ a little, e.g. the Docker list has the following packages which the wiki does not have: emacs, jq, libatlas-base-dev, libatlas-dev, libdcmtk-dev, libfreetype6-dev, liblapacke-dev libzmq-dev and pkg-config. I don't know which of these in particular solved the problem.

Will