Purpose of <xnat:field>

[Simon Doran]

Hi All,

  We have a question about the use of the schema element <xnat:field>.

  The XML for an XNAT Experiment has a section that looks like this:

<xnat:fields>

    <xnat:field name="Field name 1">Content 1</xnat:field>

    <xnat:field name="Field name 2">Content 2</xnat:field>

    .

    .

    .

</xnat:fields>

  It's clear that, where custom variables are used, the fields coincide with those custom variables. However, they are by no means limited to custom variables. As far as I can see, there isn't a REST API per se for creating and altering either custom variables or fields. However, XNATpy, for example, provides a mechanism that makes it very simple to list the fields that exist for an experiment and to change them. It is trivially possible to use XNATpy to create fields that don't correspond to what is seen in the UI, just by a command like this:

experiment.fields['non-existent custom variable'] = 'some value'

  So, I've got several questions:

1. Custom variables can be enumerated. Does XNAT apply any checks on field contents when they are uploaded via the technique that XNATpy uses?
   
   
2. Was the xnat:field schema element originally designed for any purposes other than custom variables?
   
   
3. If so, how was/is it envisaged that fields be used?
   
   
4. If not, is the mismatch that we see simply an artefact of the fact that the custom variable API is under-developed and that the fields concept can be "abused". 
   
   
5. Suppose a user has created a new field in an experiment as above, which doesn't correspond to a custom variable originally created by the project owner (and thus is not displayed in the UI). Can the information retrospectively be displayed by the project owner creating a new custom variable with that name?

  Thanks for taking a look.

  Best wishes,

Simon

[Moore, Charlie]

Hi Simon,

I can't say I was around when they were added so I can't speak to the intentions, but I think I've used them enough that I could answer some of these:

It's important to keep in mind the distinction between adding (or defining, whatever term you want) a custom variable versus populating a custom variable. When you click on "Manage Custom Variables" in the UI and add a new custom variable, all this is really doing is driving the UI, telling it:

• Display variables x, y, z on report and edit pages
• Variable x is of type String, and required, etc.

What this means is that "custom variables" along with the metadata you define for them is really a UI concept. As you've noted, you're free to set whatever fields you want on experiments/subjects, regardless of custom variable status. There is no API to add a new custom variable or variable set​ (it's actually my number 1 personal feature request for XNAT), but there is an API to populate a custom variable​. I suspect XNATpy is indeed using the API. If you would find a curl example useful, I've included it along with caveats at the very end.

 The consequences are therefore (to answer your questions):

1. No. Setting a field is not necessarily tied to a custom variable. The enumeration (or type) is not checked in the REST API (or XML).
2. I can't say for certain. but I suspect it was mainly for custom variables or other "ad hoc" type of data capture.
3. Same as 2.
4. Hard to say. I would certainly love a full custom variable API.
5. Absolutely. I will add there are some oddities I'll explore in the curl example that may be problematic.

curl -X PUT "http://10.1.100.17/data/projects/TEST_PROJECT/subjects/Sample_Patient/experiments/Sample_ID?xnat:experimentData/fields/field%5Bname%3Dnumrois%5D/field=1000" -u admin:admin

Some notes about this:

1. Using xnat:experimentData makes it work for any subject assessor regardless of the particular data type of the subject assessor.
2. This call is very difficult to get right as there's a lot of ugly URL-encoding going on. If you have trouble parsing it, this example is setting a field called "numrois" to 1000.
3. The field name ("numrois") should always be lowercase​ when entering it. I don't know why it behaves like that, but it does. If you create a custom variable called "NUMROIS", it will still work and show up in the UI as "NUMROIS", but you have to set it as an xnat:field with name "numrois".
4. And yes, it will be invisible until you actually create a custom variable to match it 🙂.

Hopefully that helps a bit.

Thanks,

Charlie

---

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of Simon Doran <simon...@icr.ac.uk>
Sent: Wednesday, November 10, 2021 9:22 AM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: [XNAT Discussion] Purpose of <xnat:field>

 

* External Email - Caution *

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/a37a1e4d-c769-435b-a60c-4aa2502d642cn%40googlegroups.com.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Simon Doran]

Hi Charlie,

  Thanks. That's all really useful information. Particularly note 4 at the end.

  Best wishes,

Simon

[Hakim Achterberg]

Hi Charlie,

I looked into this when I was working on that part of xatpy, but I always have had a bit of an itch about not being able to add custom variable to UI from xnatpy.

Also I haven't found a way of getting the display settings for custom variables, because there is a type associated with them in the UI, but in the database it's all a string. It would be nice to expose the custom variables with a type conversion/checking. Was my assessment that that is impossible with the API correct?

I actually use hidden fields/custom variables as a feature sometimes, to store data that I think isn't needed to be user facing ;-)

Cheers,

Hakim

PS Simon, if you ever want to know which requests xnatpy uses, if you change the loglevel of the last release to INFO, it will print every request it makes.

[Moore, Charlie]

Hi Hakim,

Custom variable definitions are actually stored in the project representation in either JSON or XML (/data/projects/ID?format=xml|json). The problem is they're kind of a nightmare to parse out. I've included an example of a project I just made with 2 simple custom variables. Likewise, it might be possible to define custom variables by injecting them into the project's XML and reuploading, but that sounds extremely difficult. Having a real API would be much better.

Thanks,

Charlie

P.S. our email filter really doesn't like XML attachments so that's why this is a txt.

---

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of Hakim Achterberg <hakim.ac...@gmail.com>
Sent: Thursday, November 11, 2021 6:45 AM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: Re: [XNAT Discussion] Purpose of <xnat:field>

 

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/84e92496-7c3d-4b99-8236-903f7aae363fn%40googlegroups.com.

[Hakim Achterberg]

Hi Charlie,

Thank you for pointing that out, even though it is tricky to parse out, I might be able to make a custom_variables property for subjects and experiments that gets the definition on the project. However, the type is custom for both, does that mean it is just a string? I noticed in my own project some strings can have possible values, so I guess that is how the enums are stored.

That makes me wonder if I could manipulate that data via posting to the project endpoint, but it would probably be really tricky to do correctly.

Anyways, food for thought.

Cheers,

Hakim

You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/92lxYp3aYwI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/CH2PR02MB6710B5219845C7252733C432FE949%40CH2PR02MB6710.namprd02.prod.outlook.com.

[Moore, Charlie]

Hi Hakim,

Take a look at the "datatype" property at that level. That seems to have string for the string variable and integer for the integer variable.

Thanks,

Charlie

---

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of Hakim Achterberg <hakim.ac...@gmail.com>

Sent: Thursday, November 11, 2021 10:14 AM
To: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com>

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/CAOc%3DDcUFnG2juf4iQ%2BxKOsu_6Hr57kC3XFF5z50mYh1aTGQzuw%40mail.gmail.com.