python interfaces to XNAT

[Open Minds Lab]

Hi Everyone!

I am just getting started with using XNAT programmatically, and trying to learn python interfaces. My main goal is to implement some routine pipelines to do some QA and QC in the end, but I'd start with checking some basic parameter values and ensure they are what they are supposed to be etc (let me know if there are any existing tools for this).  There seems to be two with similar names pyxnat and xnatpy, and from what I gather one offers a more RESTful API and the other a more pythonic interface. Is that it, or are there any differences I need to be aware of?

also, are there limits to python interfaces in terms of what we can achieve with XNAT?

Thanks,

Pradeep

[Martin Boswell]

Pradeep,  

I was hoping that someone with more experience than I would have responded to you.  I would have liked to have heard the perspective of veteran XNAT pipeline coders.  But since nobody else has chimed in yet...  I've only been using XNAT for about 18 months (administering an XNAT server, uploading data, building and running pipelines, etc.).  Most of my XNAT work has been under considerable time pressure to just-get-it-done-and-move-on, so I haven't been able to be as thorough as I'd like when making decisions such as choosing which technology to use for programatic control of my XNAT server.  

I began by trying to figure out the differences between pyXNAT and XNATpy and understand what tools like DAX offered.  I had a lot of data to store and process and I didn't want to start down the wrong path, but the pros and cons of the various choices were not clear, and with a beginner's perspective, it was all quite overwhelming.  

To make a long story short... I decided to use XNATpy, but right away I had trouble figuring out how to perform some tasks with it.  Then I had trouble installing it in the environment of the compute cluster that I was going to use for my pipelines (the trouble was really with a dependency, the six.moves.collections_abc module), and on top of that, from something that I'd read I was concerned that it wasn't going to be able to manipulate all of the data that I needed (I can't figure out from my notes specifically what I was concerned with, maybe a custom plugin?).  Anyway, after struggling with these issues for a while, I threw in the towel and decided to just use straight python (and the requests library) to access XNAT through the RESTful API.  I combined this with using cURL commands from the shell (esp. for uploading data or bulk operations on long lists of experiments or subjects).  This way I thought that I wouldn't have to learn yet another tool's interface, and I wouldn't have to worry about limitations or bugs of any specific tool.  And that's what I did, writing a lot of CRUD functions for different XNAT objects.

Since then, I've started to use XNATpy again, and have found it to be very useful.  With more XNAT experience under my belt it was easier to understand how to use it.  A lot of my early confusion was with XNAT concepts (e.g. the data model) rather than XNATpy itself.  I realize now how much XNATpy would have simplified some of my pipelines and saved me a lot of time sifting through JSON output and catching http exceptions.

So, in retrospect after all of this time, I wish that I'd stuck with XNATpy, and moving forward, I'll probably start using XNATpy for more of my new development.  

- Martin

[Open Minds Lab]

Thanks Martin for sharing your experience. You're right about the learning curve, and its better to stick to one or the other than starting from scratch, even if they thrown bugs at us in our environment. I am inclined towards the more pythonic pyxnat (than the RESTful xnatpy if I understand them correctly), but I thought I will check with other users/experts before I put too much time into it.

Hoping others chime in as well

[Alexander Bartnik]

I'm in a pretty similar situation to Martin, started about a year and a half ago and have been using XNATpy.

From what I've seen, XNATpy feels more "object oriented, " and I'm more comfortable thinking about things that way. I will say that handling individual scans and experiments seems easier in pyxnat, since with XNATpy I find myself iterating over all of a subjects scans/experiments to find the ones I want. XNATpy does tend to expose sessions/scans as dictionaries, which is pretty to work with overall, though.

[Hakim Achterberg]

I haven't really said anything before as I am the author of xnatpy, so I am definitely biased. So I want to mention two features of xnatpy that might help you. xnatpy is the one that is more python-object style, pyxnat follows the rest api more.

First off, if you want to do anything that is not supported by the higher level interface, there are simpler functions that you can use on the session, which still does the authentication management and and some error checking etc:

https://xnat.readthedocs.io/en/latest/xnat.html#xnat.session.XNATSession.get

https://xnat.readthedocs.io/en/latest/xnat.html#xnat.session.XNATSession.put

Actually the get, head, put, post, delete are all available, also some functions to upload data. You can use just the relative path after the connection has been made:

e.g. you can do

response = connection.get_json('/data/projects/projectid')

To get a requests response object, or

data = connection.get_json('/data/projects/projectid')

To get the json data for the project, already unpacked.

The other trick is that you can create xnatpy object from a url directly without having to go through all data. I really should update the documentation to reflect this. You can do:

experiment_object = connection.create_object('/data/projects/$project_id/experiments/$experiment_id')

and it should get the data and automatically figure out it is an experiment object and wrap it properly. This allows you to start at random places in the data tree. You could still query the subject or scans of this experiment object just like you would if you would have gotten it from a listing. I realise now that this might be a nice feature but it currently completely undocumented, the code is here: https://gitlab.com/radiology/infrastructure/xnatpy/-/blob/master/xnat/session.py#L768

I should really update the documentation, still hope this helps a bit.