Re: REST query of SOP Instance UID

[Adam Harding]

> ...ways to query through the REST API to see if a particular image exists inside XNAT based on its SOP Instance UID?
No*. The key point is that XNAT is not a DICOM PACS and doesn't behave like one beyond being able to receive (and scrape/parse) DICOM transfers of image data, thereafter caring nothing that the data it received happens to be DICOM. XNAT has its own data model that isn't DICOM. XNAT doesn't even (by default, anyway) store all the _interesting_ metadata present in DICOM, let alone DICOM's lexically meaningless book-keeping arcana. Such metadata doesn't get scraped from the files when XNAT stores them to disk, so XNAT doesn't "know" about it.

> ...would like to achieve the /same/ /thing/ with XNAT somehow if possible.
OK, so your goal is to answer the question "What difference is there between this list of data and the list of data in my repository?", and you're accustomed to answering it by comparing lists of DICOM SOP instance UIDs by uttering DICOM. *You could customize your XNAT to store UIDs that you'd then populate alonglside your data and query RESTfully, but it would probably be better to enumerate the subjects/sessions/scans in XNAT using REST and compare these with the list of those from the elements in your source DICOM. Some XNAT users do just this kind of thing.

XNAT-DICOM Gateway is an option for mapping between DICOM requests and XNAT requests, but I think even then you'd still end up doing something like the above to get what you want. Maybe someone else has experience with this approach and would speak to the contrary.

Adam

On Thursday, July 19, 2012 10:04:04 AM UTC-5, Justin Kirby wrote:

Hi all,

I was looking at http://xnat15.wikispaces.com/XNAT+REST+API.  Is there any ways to query through the REST API to see if a particular image exists inside XNAT based on its SOP Instance UID?  It doesn't look like there is in v1.5.  What about in v1.6?

I ask because CTP has something called a "Database Verifier" and it allows you to confirm everything that was de-identified and sent out via CTP has been received by your archive software.  It does this by cross checking the list of SOP Instance UIDs in both locations to make sure nothing got lost along the way.  We've found this to be incredibly valuable when sending large amounts of data to NBIA, and would like to achieve the same thing with XNAT somehow if possible.

Thanks,
Justin

[Simon Doran]

Adam,

  I think that your answer gave an impression that was somewhat more negative than I would wish to do. What Justin is asking for is actually a very reasonable request and, whilst not possible directly from the REST-API in a single call, is certainly do-able via a combination of REST and Java.

  I don't agree with the phrase "let alone DICOM's lexically meaningless book-keeping arcana". Yes, the DICOM UID system can appear rather clumsy at times, but can also be rather useful. In fact, all the required information is already stored in XNAT. Both the StudyInstanceUID and the SeriesInstanceUID are at the places one would expect in the XNAT schema: at xnat:imageSessionData/UID and xnat:imageScanData/UID respectively. To get the SOPInstanceUID's, you have to dig down to the catalog level, but they are there.

  I do something very similar to what Justin wants in my data-uploader application. The use-case is as follows. I have the result of some piece of data-processing - doesn't matter what it is - that I wish to upload to the repository and at the same time, I want to parse the significant metadata from the object and store it as a custom qcAssessmentData "object" in the SQL database, However, I disallow the upload if the original image file dependencies that were used to perform the processing are not already present in XNAT.

  Justin, below is a code snippet that might help give a rough idea of how I do things. You can't directly use it, because there is a whole infrastructure sitting below it to parse the various XML files that the REST API generates, but it might help a little by showing how I went about doing the REST queries. If you need more of the code, please get in touch. Most of it will get uploaded onto XNAT Marketplace soon.

  Hope some of this is useful.

  Best wishes,

Simon

/**

    * Check that all the relevant studies, series and SOPInstances have been

    * loaded into the XNAT database.

    * @return true if all studies are present, false if not or on error. 

    */

   protected void dependenciesInDatabase(XNATProfile xnprf)

                  throws XNATException, Exception   

   {

      // Normally, the reason we are creating an RTStruct is to upload a given

      // file to a particular project and so the given profile will be supplied

      // with only one element in the project list. However, this might change ...

      XNATProjectID = xnprf.getProjectList().get(0);

      

      String                RESTCommand;

      XNATRESTToolkit       xnrt = new XNATRESTToolkit(xnprf);

      Vector2D<String>      result;

      String[][]            parseResult;

      Document              resultDoc;

      XNATNamespaceContext XNATns = new XNATNamespaceContext();

      

      try

      {         

         // Find information on all the studies in the database.

         RESTCommand = "/data/archive/projects/" + XNATProjectID + "/experiments"

                       + "?xsiType=xnat:imageSessionData"

                       + "&columns=xnat:imageSessionData/UID,xnat:imageSessionData/subject_ID"

                       + "&format=xml";

         result      = xnrt.RESTGetResultSet(RESTCommand);

      }

      catch (XNATException exXNAT)

      {

         throw exXNAT;

      }

      

      // Are all the DICOM studies from the uploaded ROI file present

      // in the database? Note that studyUIDs has previously been populated from the

      // file we want to upload.

      for (int i=0; i<studyUIDs.size(); i++)

      {

         if (!result.columnContains(1, studyUIDs.get(i)))

            throw new Exception("The DICOM study with UID " + studyUIDs.get(i) + "\n"

                            + " is referenced by the file you are loading,"

                            + " but it is not yet in the database.\n"

                            + "Please ensure that all files on which this"

                            + " structure set is dependent are already loaded into XNAT.");

            

            

         // Now check that all the series are present.

         int row = result.indexOfForCol(1, studyUIDs.get(i));

         

          

         String XNATExp  = result.atom(0, row);

         String XNATSubj = result.atom(2, row);

         

         // Note that we have an incompatibility between the possibility in

         // DICOM that the structure set can reference more than one DICOM

         // study and the XNAT requirement to archive the structure set file

         // with a particular XNAT Experiment (imageSession). Resolve this

         // by arbitrarily choosing the first.

         if (i == 0)

         {   

            XNATExperimentID = XNATExp;

            XNATSubjectID    = XNATSubj;

            XNATScanID       = new ArrayList<String>();

         }

         try

         {

            RESTCommand = "/data/archive/projects/" + XNATProjectID

                          + "/subjects/"    + XNATSubj

                          + "/experiments/" + XNATExp

                          + "?format=xml";

            resultDoc   = xnrt.RESTGetDoc(RESTCommand);

            parseResult = XMLUtilities.getAttributes(resultDoc, XNATns, "xnat:scan",

                                                new String[] {"ID", "UID"});

         }

         catch (XNATException exXNAT)

         {

            throw exXNAT;

         }

         

         boolean present = false;

         for (int j=0; j<parseResult.length; j++)

         {

            // Not all of the experiments returned are scans. Some might be

            // assessors, with no UID. These need to be screened out.

            if (parseResult[j][1] != null)

            {

               if (parseResult[j][1].equals(seriesUIDs.get(i)))

               {

                  present = true;

                  XNATScanID.add(parseResult[j][0]);

               }

            }

         }

      

         if (!present)

            throw new Exception("The DICOM series with UID " + studyUIDs.get(i) + "\n"

                            + " is referenced by the file you are loading,"

                            + " but it is not yet in the database.\n"

                            + "Please ensure that all files on which this"

                            + "structure set is dependent are already loaded into XNAT.\n'n");

      }

      

      // Finally, we need a list of the actual data files in the repository

      // that are referenced, to go in the "in" section of qcAssessmentData.

      // See the Class DICOMFileListWorker for an example of how to do this

      // both if the files are local or remote. Here, for simplicity, I don't

      // assume anything and use the REST method whether the files are local

      // or remote. 

      fileSOPMap = new HashMap<String, String>();

      

      for (int i=0; i<XNATScanID.size(); i++)

      {

         try

         {

            RESTCommand = "/data/archive/projects/"    + XNATProjectID

                             + "/subjects/"            + XNATSubjectID

                             + "/experiments/"         + XNATExperimentID

                             + "/scans/"               + XNATScanID.get(i)

                             + "/resources/DICOM?format=xml";

            resultDoc   = xnrt.RESTGetDoc(RESTCommand);

            parseResult = XMLUtilities.getAttributes(resultDoc, XNATns, "cat:entry",

                                                     new String[] {"URI", "UID"});

         }

         catch(XNATException exXNAT)

         {

            throw exXNAT;

         }

         

         // Cater for the obscure case where parseResult comes back null. This

         // happened to me after I had (manually) screwed up the data repository.

         if (parseResult == null)

            throw new Exception("There are no relevant DICOM image files. This might be an"

                            + " inconsistent condition in the repository and be"

                            + " worth investigating further.");

         for (int j=0; j<parseResult.length; j++)

         {

            if (SOPInstanceUIDs.contains(parseResult[j][1]))

               fileSOPMap.put(parseResult[j][1], parseResult[j][0]);

         }

      }

      

      // Retrieve some further demographic information so that it is available

      // for output where necessary.

      try

      {

         RESTCommand      = "/data/archive/projects/" + XNATProjectID

                                + "/subjects/"        + XNATSubjectID

                                + "?format=xml";

         resultDoc        = xnrt.RESTGetDoc(RESTCommand);

         XNATSubjectLabel = XMLUtilities.getAttribute(resultDoc, XNATns,

                                                   "xnat:Subject", "label")[0];

         XNATDateOfBirth  = XMLUtilities.getElementText(resultDoc, XNATns, "xnat:dob")[0];

         XNATGender  = XMLUtilities.getElementText(resultDoc, XNATns, "xnat:gender")[0];      

      }

      catch(XNATException exXNAT)

      {

         throw exXNAT;

      }

 

   } 

[Adam Harding]

Hi Simon & Justin,

Thanks for the concrete use-case, Simon. I didn't intend to give a negative impression, but to highlight that the different "worlds" are being bridged and that the task may be faster / more maintainable / easier by re-thinking the problem. The payoff may be smaller if all your imaging workflows revolve around DICOM, but circumstances change over time. We even create some (reasonably compliant!) DICOM from non-DICOM imaging data before storing in XNAT to allow DICOM tool use. I certainly agree that the approach at hand is reasonable, but with some caveats.

DICOM's book-keeping metadata is indeed useful, but so by virtue of semantics upon it (imposed by tools; emphasis Simon), while being "lexically meaningless" (hence "arcana", as the "labels" aren't symbolic without that context, which XNAT doesn't directly provide; my emphasis). I actually don't mean that pejoratively-- UIDs are the right choice for the design goal. It's just that there are real situations where this doesn't help: as soon as the "same" data mistakenly gets stamped with a different UID after being sent through a scrubber multiple times, the distinction between humans' and DICOM semantics' notions of sameness comes to issue. My recollection is that we had some variant of this happen on a moderately large scale; certain metadata gave an illusory picture of the data.

> To get to the SOPInstance UID's, you have to dig down to the catalog level...
They are, but they're only sort of "in XNAT", which depends on context. Hence my comment regarding XNAT's "knowing" about them. [Something I perceive that most users consider] A major part of XNAT is its search functionality, and perhaps I misapprehend DICOM users' expectations when first interacting with XNAT. I suppose I should have said something like "You can list them, but avoid the impression that XNAT 'uses' them, and I think you could make your process more robust by comparing elements [symbolically rather than lexically]..." or so.

Thanks for being a continued source of good bits, Simon. I'm sure many of us will benefit from your contributions to the marketplace.

Adam

[Simon Doran]

Adam,

  Sorry about the late reply. On annual leave ... and just been to the Olympics :-) ... and been rained on :-(  ... bit of a change from 105 degrees in St Louis, but a lot of fun anyway.

  I agree with pretty much all the points you make. Particularly, when you say:

> It's just that there are real situations where this doesn't help: as soon as the "same" data mistakenly gets stamped with a different UID after being sent through a scrubber multiple times, the distinction between humans' and DICOM semantics' notions of sameness comes to issue.

  The mutability of DICOM files does concern me. The same mechanism that allows one to replace header fields to perform anonymisation also allows the UID of a file from one dataset to be pasted into another. This has the potential to cause a lot of problems when one is using the UID to decide whether an ROI can validly be superimposed on an image. I have equally heard of people who, in a shortcut to create a conforming DICOM header, will use an acquired image to create a template and then simply replace the pixel data together with a few key fields.

  I guess this is where having a rigorous control of one's procedures is really important.

Simon