Saving subject data in XNAT using Xnatpy

[andres rojas]

Hello Guys,

I have been using Xnatpy to upload and download EEG data in Xnat. I have been sucessfull at doing it. But right now I am on a second iteration of this development in which I would like to save project and subject metadata, in order to perform searches later on using these saved data. 

I have seen Xnat already has some fields for project like Keywords and for subjecst such as: Age, Gender, Handedness, Education, Race, etc. 

I was expecting to be able to find these medata data fields accessible from the Xnatpy class, so I could just directly save it. But I haven't been able to save these data and then get to see the output in the Xnat interface.

Can you share a code snippet of how to save subject meta data using Xnatpy ? 

if possible this info should be visualized in the Xnat appilcation like in the attached screen shoot, where the data is saved using the Xnat interface. 

Thanks a lot in advance!

Andrés 

 

[Chidi Ugonna]

Hi Andres,

This is probably not the definite or ideal way using xnatpy but it seemed to work in my development space. This was done with xnatpy version 0.5.2 in python 3.8. The version of xnatpy you use may be different which might make a difference.

I had to do some jiggery-pockery with height and weight which looks weird to me - it actually looks like I was able to set other fields that way which is a little strange. So for example subject.demographics.weight.set("handedness","right") 

set the handedness even though I was calling a set method on weight.

with xnat.connect(server="http://192.168.0.28",user='admin',password='admin') as connection:

    subject = connection.projects["PROJECT_ID"].subjects["SUBJECT_LABEL"]

    subject.demographics.gender = "male"

    subject.demographics.handedness = "right"

    subject.demographics.education = 11

    subject.demographics.race = "Black"

    subject.demographics.ethnicity = "African"

    subject.demographics.height.set("height","70")

    subject.demographics.weight.set("weight","140")

    subject.src = "h"

    subject.group = "Control"

    subject.demographics.dob = "19751215"

[andres rojas]

Thank you for your reply. Indeed it was possible just by changing the Xnatpy version! I was using (0.5.1). I tried again with the 0.5.2 and it worked straigh away. You are right to poing out the jiggling with height and weight it does not work straigh forward like the other demographic values. 

I have another question: Have you tried to add new metadata fields ? do you know how this can be done by chance?

In the case of having subjects from clinical studies it would be very useful to save the drug, the dose, the behavioral scores, etc. This metada is very specific for each data set. So it should be saved as an addec custom field, but I don't know how to save it with Xnatpy. 

Thanks a lot in advance!

Andrés 

[Chidi Ugonna]

Hi Andres,

So I assume you want to take advantage of the new custom forms functionality provided I think since 1.8.8. This is described here Custom Forms and the API is described here Custom Forms API. As far as I can tell XNATpy does not provide helper functions for the custom forms API yet but using XNATpy it is fairly straightforward to access the API using low level REST like put, get  etc. If you want to use the old custom fields approach (it will be deprecated and so I would not advise this) then XNATpy has a fairly nice way to access those as detailed here XnatPy Custom Variables.

This is a brief example of how I have been able  to add custom variables using the new custom forms functionality  and populate these variables using the low-level put function in XNATpy.

(a) In the Custom forms manager I have designed 2 forms at the subject-level (Behavioral Scores and Intervention), each with some fields in each of the forms. These forms appear as tabs in a custom field set as show below.

(b) Each form has a UUID which is identifiable on the form design window as well as on the viewing window. I have highlighted it  above and below - we need these UUIDs to change field values in the form.

(c) Each field within the form has an API reference name which we will use to update the field. Click on API tab in the Form design window to see this name. For example "4 Minute Gait Speed Test" is reference as "MinuteGaitSpeedTest" by the API

(d) Essentially what we need to do is send  json definitions to XNAT to tell it what forms we want to update and what the new field values are going to be. To update both forms with new values then we need to send this Json construct. The key to each form is it's UUID.

{

'9ea210bb-4ca3-410b-bc61-0c01a50e1fff': {'faceName': 12, 'workingMemory': 24, 'MinuteGaitSpeedTest': 5},

'1a3adfc1-0143-46dd-9d8d-72eb19c12d8b': {'drugtype': 'TestDrug', 'drugDose': 3}

}

(e)  The XNATpy code we need to use to actually send this json expression to XNAT is as follows:

connection.put("/xapi/custom-fields/projects/PROJECT_ID/subjects/SUBJECT_LABEL/fields" , json=[json expression defined above])

(f) The code below recapitulates the above:

import xnat

subject_label="SUB001"

project_id="TEST

UUID_beh="9ea210bb-4ca3-410b-bc61-0c01a50e1fff"

beh_field1="faceName"

beh_field2="workingMemory"

beh_field3="MinuteGaitSpeedTest"

UUID_int="1a3adfc1-0143-46dd-9d8d-72eb19c12d8b"

int_field1 = "drugtype"

int_field2= "drugDose"

customjson={}

customjson[UUID_beh]={}

customjson[UUID_beh][beh_field1]=12

customjson[UUID_beh][beh_field2]=24

customjson[UUID_beh][beh_field3]=5

customjson[UUID_int]={}

customjson[UUID_int][int_field1]="TestDrug"

customjson[UUID_int][int_field2]=3

apistring='/xapi/custom-fields/projects/{}/subjects/{}/fields'.format(project_id,subject_label)

with xnat.connect(server="http://192.168.0.28",user='admin',password='admin') as connection:

    resp=connection.put(apistring,json=customjson)

The same approach works for Custom forms based on other XNAT Datatypes - you will need to change the `apistring` to access the XNAT Datatype on which you have defined your custom form which you can find at the Custom API reference provided further above.

[andres rojas]

Hi Chidi,

Thank you very much for such a clear explanation. It worked very well. I didn't know I should create the custom forms first from the Xnat interface. 

But your example of how to save the data with the raw api call was very useful. 

Andrés 

[andres rojas]

Hello Chidi,

After uploading the subject's metadata to custom fields I am now facing the problem of searching and filtering. I would like to access the subject custom fields in my searches to filter out subjects of interest. For example: subjects that satisfy a given condition such as: cognitive decline where the value is Yes or No.  Do you know how these searches could be carried out by any chance ?

I just opened a new thread for this here: https://groups.google.com/g/xnat_discussion/c/xMaFl3tlMuE

In case you have any suggestions I would highly appreciate it ! 

Thanks a lot in advance,

Andrés

--
You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/O_MY-QcUerg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/bb70df91-81c3-4074-ac32-1a167f77973cn%40googlegroups.com.

--

Andrés Rojas

Research Engineer

Starlab

Follow Starlab on Twitter!

__________________________________________________

Barcelona Offices:

Av. Tibidabo, 47 Bis
08035 Barcelona, Spain

Tel: +34 93 254 03 70

Fax: +34 93 212 64 45

[Chidi Ugonna]

Hi Andres,

Apologies for the delay in replying. So I am not sure if you made further progress with this but I did some digging around.

So I went head and used XNATpy 0.5.2 (I think 0.5.1 will also work without issues but I didn't test this)  and used Hakim's suggestion to get past the "*** TypeError: 'NoneType' object is not iterable" exception that was thrown when trying to query on an XNAT class. This is  a hack and I'm sure that with the next iteration of XNATpy this will be fixed. To make that change:

1. Quick hack for searching on 0.5.2

Find  module `search.py` on your python path for example `lib/python3.9/site-packages/xnat/search.py` (there is another search,py at /xnat/client/search.py that you dont want). In the definition for Class Query change fields=None to fields = ()

so before looks like this:

class Query(object):
    def __init__(self, queried_class, xnat_session, fields=None, constraints=None):
        self.queried_class = queried_class

after looks like this:

class Query(object):
    def __init__(self, queried_class, xnat_session, fields=(), constraints=None):
        self.queried_class = queried_class

2. What can I search on?

XNATpy's documentation on searching is really comprehensive and explains much  https://xnat.readthedocs.io/en/latest/static/tutorial.html#searching. To inspect what fields are available in a connection class e.g SubjectData - we can use the vars() method as shown below - I have highlighted the Custom Fields that I created using the Custom Form

import xnat

from pprint import pprint

connection = xnat.connect(server="http://127.0.0.1",user='admin',password='admin')
SubjectData = connection.classes.SubjectData

pprint(vars(SubjectData))

This is a trunctated output showing custom fields and standard fields:

mappingproxy({'ADD_IDS': <SearchField xnat:subjectData/ADD_IDS>,
              'AGE': <SearchField xnat:subjectData/AGE>,
              'AGE_BRACKET': <SearchField xnat:subjectData/AGE_BRACKET>,
              'DOB': <SearchField xnat:subjectData/DOB>,
              'EDUC': <SearchField xnat:subjectData/EDUC>,
              'ETHNICITY': <SearchField xnat:subjectData/ETHNICITY>,
              'GENDER_TEXT': <SearchField xnat:subjectData/GENDER_TEXT>,
              'HANDEDNESS_TEXT': <SearchField xnat:subjectData/HANDEDNESS_TEXT>,
              'INSERT_DATE': <SearchField xnat:subjectData/INSERT_DATE>,
              'INSERT_USER': <SearchField xnat:subjectData/INSERT_USER>,
              'INVEST_CSV': <SearchField xnat:subjectData/INVEST_CSV>,
              'PROJECT': <SearchField xnat:subjectData/PROJECT>,
              'PROJECTS': <SearchField xnat:subjectData/PROJECTS>,
              'RACE': <SearchField xnat:subjectData/RACE>,
              'SES': <SearchField xnat:subjectData/SES>,
              'SUB_GROUP': <SearchField xnat:subjectData/SUB_GROUP>,
              'XNAT:SUBJECTDATA_CUSTOM-FORM_0E51F111-E2AA-4671-A0C8-B5C50981F332_DRUGDOSE': <SearchField xnat:subjectData/XNAT:SUBJECTDATA_CUSTOM-FORM_0E51F111-E2AA-4671-A0C8-B5C50981F332_DRUGDOSE>,
              'XNAT:SUBJECTDATA_CUSTOM-FORM_0E51F111-E2AA-4671-A0C8-B5C50981F332_DRUGTYPE': <SearchField xnat:subjectData/XNAT:SUBJECTDATA_CUSTOM-FORM_0E51F111-E2AA-4671-A0C8-B5C50981F332_DRUGTYPE>,

3. Constructing a search using custom fields (from custom form) and standard fields?

because of the format of the custom fields we will access them using the getattr() method instead. So for a combined search on gender and drug dose for example we can do this:

sublist = SubjectData.query().filter((SubjectData.GENDER_TEXT == "M") & (getattr(SubjectData,"XNAT:SUBJECTDATA_CUSTOM-FORM_0E51F111-E2AA-4671-A0C8-B5C50981F332_DRUGDOSE") > 20))
sublist.all()

[<SubjectData SUB001 (XNAT_S00001)>, <SubjectData SUB004 (XNAT_S00004)>]