BIDS + XNAT

[Kate Alpert]

Dear all,

I remember some talk of BIDS and XNAT… Is there anything going on in this arena?

I am thinking about creating a pipeline to run upon archive that would do a DICOM -> NIFTI conversion, using a customizable look-up table for BIDS naming based on scan type or series description. Before I went ahead and made this, I figured I should check to see if anyone else has done something similar. 

Thanks!

Kate 

[Lauren Wallace]

Hello Kate,

 

I’ve been working on BIDS support—in fact you asked at just the right time, I just checked in the 1.7 plugin component a few minutes ago. J I have two components planned for BIDS in XNAT—one is a pipeline that uses dcm2niix to convert the data to NIFTI with BIDS sidecar, naming them according to a lookup table, and generates rudimentary session-level BIDS files, and the other is a plugin that will allow you to package a download in the BIDS format. The downloader is probably still a couple of weeks away from availability at this point, and will only be a 1.7 plugin. I’m also working on a script to import existing BIDS data into an XNAT instance via the REST API.

 

You can find the pipeline here:

 

https://bitbucket.org/nrg_customizations/nrg_pipeline_dicomtobids (Pipeline, both 1.6 and 1.7)

https://bitbucket.org/nrg_customizations/nrg_webapp_dicomtobids (Webapp screens, 1.6 module version)

https://bitbucket.org/nrg_customizations/nrg_plugin_dicomtobids (Webapp screens, 1.7 plugin version)

 

Because this is still in development and hasn’t been integrated into XNAT, you will also need to manually add a launcher method to the BuildAction.java class:

 

    public void doDicomtobids(RunData data, Context context){

        try {

            DicomToBIDSLauncher dcm = new DicomToBIDSLauncher();

            boolean success = dcm.launch(data, context);

            if (success ) {

                data.setMessage("<p><b>The build process was successfully launched. Status email will be sent upon its completion.</b></p>");

            }else {

                data.setMessage("<p><b>The build process could not be launched. </b></p>");

            }

            data.setScreenTemplate("ClosePage.vm");

        }catch(Exception e) {

            logger.error(e);

        }

    }

 

For XNAT 1.7, you can (very soon! it’s not available in the Maven repo quite yet but should be by the end of day today) add the module to your build.gradle directly:

 

def vDicomToBids = '1.0.0-SNAPSHOT'

 

…

 

    compile ("org.nrg.xnat.plugin:dicom-to-bids:${vDicomToBids}") {

        transitive = false

    }

 

The pipeline requires Python, so after you install it make sure that you have setup scripts for dcm2niix and Python 2 (epd-python) pointing to the proper location in PIPELINE_HOME/scripts - a sample setup script for dcm2niix is included in the pipeline repository, I think the python one comes with XNAT. Make sure that your version of dcm2niix includes BIDS support (offers option -b).

 

To add the pipeline to your XNAT, use the following:

 

Enter Path to Pipeline descriptor xml: (Pipeline home here)/catalog/DicomToBIDS/DicomToBIDS.xml

Enter Name of the custom webpage to launch this pipeline: PipelineScreen_DicomToBIDS.vm

 

For configuration, the pipeline uses a simple sitewide JSON mapping file that is stored with the config service at this URI:

 

https://(your XNAT here)/REST/config/bids/bidsmap

 

The mapping file is simple JSON that takes a series description (will fall back on scan type if series description is not populated) and a “bids name”, which can include the task for functional MRI or the acquisition etc:

 

[

                { "series_description": "3DT2", "bidsname": "T2w" },

                { "series_description": "BOLD", "bidsname": "task-rest_bold" },

                { "series_description": "ep2d_bold_connect", "bidsname": "task-rest_bold" },

                { "series_description": "MPRAGE", "bidsname": "T1w" },

                { "series_description": "MPRAGE GRAPPA2", "bidsname": "T1w" },

                { "series_description": "t1_mpr_1mm_p2", "bidsname": "T1w" },

                { "series_description": "t2_spc_1mm_p2", "bidsname": "T2w" },

                { "series_description": "T2_SPC_1MM_P2", "bidsname": "T2w" },

                { "series_description": "t2_spc_sag_p2_isoWU", "bidsname": "T2w" },

 

]

 

You will want to include “task” or “acq” as appropriate for your BIDS names in the “bidsname” field in this file, but the converter will automatically detect and correctly name multiple runs/occurrences of a sequence and multiple echoes within in a series for you.

 

I believe this is everything! I will be out of the office next week through Wednesday, returning Thursday, but after I return I will be happy to answer any questions you have :)

Thanks,

 

Lauren Wallace

NRG Developer

[Kate Alpert]

Thanks, Lauren! This is great.

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussion+unsubscribe@googlegroups.com.
To post to this group, send email to xnat_discussion@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[landge]

Hi,

I was wondering if this plugin could be used by end-users already?

It sounds really great and would be of great value for our study.

Thanks,

landge

[Raminder Singh]

This is great!!   Lauren, Did you developed the download script and a script to upload existing BIDS data?

Thanks

Raminder

[Haofei FENG]

Hi Lauren,

Do you have any more updates about the BIDS and XNAT you mentioned ? Thanks.

Also, looks like the wki.xnat.org is down.

[Lauren Wallace]

Hello everyone,

I apologize for the delay on this, I was moved onto another project in the lab for a little while but have had some time the past couple of weeks to put together a beta version of the 1.7 Custom Downloader plugin, available for download as a jar here: https://wustl.box.com/s/r4kqgk1io4i86wahtwra2qxzm8cr61rk

The downloader uses a simple sitewide JSON mapping file that is stored with the config service at this URI:

 

https://(your XNAT here)/REST/config/downloader/rulesets/BIDS

 

The downloader ruleset is simple JSON that takes a list of series descriptions and for each one selects resources (for BIDS download, use NIFTI,BIDS to get both the images and the additional metadata) and puts them in an output directory (one of the BIDS subdirectories)- here is an excerpt from ours that you can extend and modify as needed:

{
    "filename": "{subjectLabel}-BIDS.zip",
    "key": "series_description",
    "root_directory": "sub-{subjectLabel}/",
    "compression": "zip",
    "series": [
        {
            "series_description": "3DT2",
            "resources": "NIFTI,BIDS",
            "destination": "anat/"
            },
            {
            "series_description": "3-pl T2* FGRE",
            "resources": "NIFTI,BIDS",
            "destination": "anat/"
            },
            {
            "series_description": "ASL Gp1 100% gap 10 slices",
            "resources": "NIFTI,BIDS",
            "destination": "func/"
            }]
}

I will be attending the BIDS Coding Sprint next week (working on a prototype of a utility for the other direction, getting BIDS data into XNAT), and when I return I will make sure this plugin is available through the usual distribution channels for everyone using XNAT 1.7, but the jar attached to this message should be good enough for testing it out. Please let me know if you have any questions.

Thank you,

Lauren

[lipinski...@gmail.com]

Dear Kate,

me and my colleague were working on bids-xnat support on 2nd CRN Coding Sprint, maybe you would like to check it out? https://github.com/kamillipi/2bids

Best,

Kamil

[Kate Alpert]

Hi Kamil,

Thanks - how does this compare to Lauren’s pipeline (https://bitbucket.org/nrg_customizations/nrg_pipeline_dicomtobids)?

Kate

--

[mmok...@research.baycrest.org]

Hi everybody, 

I am a biomedical engineer newly switched to the field of neuroimaging. I am planning to build a dicom to bids pipeline for XNAT. Was browsing this forum where found these codes that Kate has shared. Wanted to ask if anybody has tried them; and if so, how should I get them to work? Because I noticed there is no readme file or comment associated with the scripts. 

Many thanks, 

Maryam

On Friday, September 1, 2017 at 3:47:19 PM UTC-4, Kate wrote:

Hi Kamil,

Thanks - how does this compare to Lauren’s pipeline (https://bitbucket.org/nrg_customizations/nrg_pipeline_dicomtobids)?

Kate

On Fri, Sep 1, 2017 at 10:47 AM, <lipinski...@gmail.com> wrote:

Dear Kate,

me and my colleague were working on bids-xnat support on 2nd CRN Coding Sprint, maybe you would like to check it out? https://github.com/kamillipi/2bids

Best,

Kamil

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.

[Mohana Ramaratnam]

There is a BIDS Importer script (that is, BIDS formatted data is imported into XNAT) here:

Download the bids importer jar (  bids-dataimport-1.0.2-all.jar  ) from:

https://bitbucket.org/mohanar_radiologics/xnat_bids_importer/downloads 

Launch the app using:

java -jar PATH_TO_JAR/bids-dataimport-1.0.2-all.jar URL_XNAT_HOST XNAT_USERNAME XNAT_PASSWORD PATH_TO_ROOT_FOLDER_BIDS_DATASET PATH_TO_IMPORT_LOG

Mohana

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussion+unsubscribe@googlegroups.com.
To post to this group, send email to xnat_discussion@googlegroups.com.

[Maryam Mokhberi]

Many thanks. I will try this and let you know if I had more questions. 

Cheers,

Maryam

--

Maryam Mokhberi 

Research Assistant at Strother Lab 

Room 422, Rotman Research Institute, Baycrest Hospital

Tel: (416)785-2500 x6385

[David Meunier]

Dear Mohana,

Thanks so much for providing the bids inporter for Xnat, that's very helpful. I have tried to use it, and it appears I have an issue on the way the xnat architecture is defined in your code (as I can see from your bitbucket):
 the architecture you provide (hard coded) imposes a fix architecture

"/data/archive/projects/"

For example, on line 358 of xnat_bids_importer / src / main / java / com / radiologics / bids / dataimport / ImportProject.java:

String url_postfix =  "/data/archive/projects/" + _projectId + "/subjects/" + mr.getSubjectId() + "/experiments/" +mr.getLabel() + "/scans/" + scan.getId() + "/resources/" + resource + "/files?overwrite=true&extract=true&format="+format+"&content="+content+"&tags=BIDS";
	    	   					 response = _manager.importResource(connection, url_postfix , zipFile);
whereas my architecture is 

"/data/xnat/archive"
with projectId following directly (no directory named projects/ after archive).

Would it be possible to add to the jar you provided a variable path_to_xnat_projects   (= "/data/archive/projects/" by default, by that maybe user defined?)

Or directly in the xnat url ?

Let me know what you think, and again many thanks for your work!

Best

David

[David Meunier]

Hi Mohana,

Actually I am not sure this was the issue I have anymore. My data file is named "raw" and contains one subject with several sessions (dwi, func, etc.), as well as sourcedata and derivatives. I did not have a dataset_description.json , SO I copied one I found, I believe It has no importance, right?

I did a symbolic link between /data/xnat/archive -> /data/archive/projects and I believe it went further in the import; however I have another bug, corresponding to :

400 null
400 null
org.springframework.web.client.HttpClientErrorException: 400 null
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:63)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:531)
    at com.radiologics.bids.services.RemoteRESTServiceImpl.importImageSessionWithoutRetry(RemoteRESTServiceImpl.java:192)
    at com.radiologics.bids.services.RemoteRESTServiceImpl.importImageSession(RemoteRESTServiceImpl.java:142)
    at com.radiologics.bids.connection.RemoteConnectionManager.importImageSession(RemoteConnectionManager.java:236)
    at com.radiologics.bids.dataimport.ImportProject.importMrSession(ImportProject.java:330)
    at com.radiologics.bids.dataimport.ImportProject.save(ImportProject.java:305)
    at com.radiologics.bids.dataimport.BIDSRootFolder.importData(BIDSRootFolder.java:213)
    at com.radiologics.bids.importer.BidsDataImporter.importData(BidsDataImporter.java:67)
    at com.radiologics.bids.importer.BidsDataImporterApp.main(BidsDataImporterApp.java:19)
Log files generated in  /data/xnat/home/

In terms of generation in the archive, the program copied the sourcedata in "ressources", but that's all I see has been done.

I attech the files in BIDS_DATA_IMPORT directory, first the .log, and then the two .xml files in failures. I seems to me that the problem comes from the xnat:projectData, but I am not sure excactly what went wrong?

Let me know if you need more info for understanding what is being wrong here.

Thanks again for your help.

Best,

David

[Mohana Ramaratnam]

Hi David,

The code is using a REST URI which always starts with /data/archive/.... This has nothing to do with how the data is organized on the file-system. The REST URI would work even if you have an archive at /timbuktu - as long as XNAT knows where the archive is, a REST URI would look like /data/archive....

The BIDS Data Importer assumes that the data that you are trying to import is a valid BIDS data (that is, it would pass a BIDS data validator). Does your BIDS data folder conform to the BIDS standard?

These are the instructions to test the BIDS Importer:

a) Stop tomcat

b) Replace xnat-data-models-1.7.4.jar from 

https://bitbucket.org/mohanar_radiologics/xnat-data-models/downloads/xnat-data-models-1.7.4.jar

to

TOMCAT_HOME/webapps/XNAT_WEBAPP_NAME/WEB-INF/lib

c) Replace core jar from 

https://bitbucket.org/mohanar_radiologics/xdat/downloads/core-1.7.4.jar

to 

TOMCAT_HOME/webapps/XNAT_WEBAPP_NAME/WEB-INF/lib

d) Restart tomcat

e) Download the latest BIDS Importer (version 1.0.2) from

https://bitbucket.org/mohanar_radiologics/xnat_bids_importer/downloads/

f) Rerun the BIDS Import (make sure you rename the root folder name if you are uploading  an already uploaded data set, as the project in XNAT would exist and nothing would be imported.)

Mohana

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussion+unsubscribe@googlegroups.com.
To post to this group, send email to xnat_discussion@googlegroups.com.

[David Meunier]

Dear Mohana,

Thanks for your answer! Indeed with the new version of data models and core, I could get further in the import.

Concerning your quick remark, so far I am trying to incorporate a new dataset in Xnat, so no duplicated project in my case. However, it brings to the error I have now:

Error creating zip file: java.util.zip.ZipException: duplicate entry: sub-02_task-identification_run-01_bold.json
Unable to create/save zip file duplicate entry: sub-02_task-identification_run-01_bold.json

(And there is no several files with the same name)

Could it be related to the login/passwd I am using in the parameters of the .jar ? Because so far the project do not exists in Xnat, no rights /no user have been given to it, etc. and I am testing the import with admin login.

It makes me think to a "owner rights" issue because when I use another subject from this dataset to test, I find that the error is always about the first scan that appear in FAILURES/..._failed.xml for the corresponding  xnat:subjectData.

Could you give me more information about what is expected in terms of xnat_user and rights to the project (creating/existing, etc.).

Or could it be something completely unrelated?

[Mohana Ramaratnam]

Hi David,

The FAILURES folder should have xml's that failed to be stored by XNAT. It would help to look at the structure of XML to investigate. Could you send me one such failed one - does it reference the duplicate json?

Is it possible that  sub-02_task-identification_run-01_bold.json  is present in multiple locations?

[David Meunier]

Hi Mohana,

Please find attached the two files from the FAILURE directory (02_MR1_failed.xml and raw_failed.xml). I also attach the corresponding "_import.log" file.

As I was menitionning in my previous message, the  sub-02_task-identification_run-01_bold.json corresponds to the first bloc on info in the XML file. What I am not sure to understand is that is seems the crashes happens with the projectData from the import.log, but it is really the subjectData that seems to crash (see screen capture where the session are read correctly, but the scans are empty. Indeed in the archive I have directory "ressources" with what was in sourcedata im my original dataset, but no "scans" directory).

And also, there is no duplicate json in the data (see capture of the directory attached), I have done a search and that's the only place where the file leading to error is. However, is it possible that after multiple try it is still present in the database even if I remove the test projects I have done from the GUI? How can I check this?

Thanks again for your help

[David Meunier]

Actually after several tests, it seems the issue is that the json and .nii.gz have the same name (before the extension);  if I remove all .json, I skip the Zip error; In this case I get a directory called NIFTI that is created (but no BIDS, as it seems the other files should be stored  - from the comment in source code FileZipUtils.java). Do you have any ideas what may be wrong in my json? Did you have associated .json in your example datasets?

[Mohana Ramaratnam]

Hi David,

You caught a bug! I have fixed the code and uploaded the latest jar - please try to import your dataset again using the new importer jar from:

https://bitbucket.org/mohanar_radiologics/xnat_bids_importer/downloads/bids-dataimport-all-1.0.3.jar

On Fri, May 4, 2018 at 7:16 PM, David Meunier <david.mai...@gmail.com> wrote:

Actually after several tests, it seems the issue is that the json and .nii.gz have the same name (before the extension);  if I remove all .json, I skip the Zip error; In this case I get a directory called NIFTI that is created (but no BIDS, as it seems the other files should be stored  - from the comment in source code FileZipUtils.java). Do you have any ideas what may be wrong in my json? Did you have associated .json in your example datasets?

--

[David Meunier]

Hi Mohana,

Sorry I did not have time to try yout new script earlier. It seems something is missing in the new jar (version 0.1.3) as I now have this error:

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/springframework/context/support/AbstractApplicationContext
    at java.lang.Class.getDeclaredMethods0(Native Method)

I extracted the archive and compared to the previous version, and indeed the version 0.1.3. continas only file bidsImporterLog4j.properties and directories com and META-INF, whereas previous version contained 17 directories and 4 files in total, so I guess the problem comes from the wrapping.

Many thanks

[Mohana Ramaratnam]

HI David,

Please try now:

https://bitbucket.org/mohanar_radiologics/xnat_bids_importer/downloads/bids-dataimport-1.0.3-all.jar

Mohana

--

[David Meunier]

Hi Mohana,

Yes it is working perfectly now! Many thanks for your help!

David

[Ernandez Santos]

Dear I installed the plugin and try to convert a sequence into bids, however only processing and not delivering the result. Can someone help me.

[Ernandez Santos]

Dear Lauren 

 I installed Pipeline and the plugin according to the links that you indicated. 

 Sorry, how do I manually add the initializer method to the BuildAction.java class?

[mmok...@research.baycrest.org]

Hi Mohana, 

Hope you are doing well!

After somewhile, I am back again on this bids conversion project in XNAT. So I would like to ask a coulple of question about this code.

First, I was wondering where this BIDS imported folder sits in XNAT? At "Actions" section under "Manage Files" tab or somewhere else?

Second, I would like to know about the download options. Once the BIDS folder is imported to XNAT, do the other xnat user have the option to select and download arbitrary number of subjects and sessions within a project? or they have to download the whole project? 

Thanks,

Maryam