Re: [XNAT Discussion] Contributing to XNAT

[John Flavin]

Thanks to all of you who have expressed your interest in contributing! It is heartening to see that people do want to help.

We have talked about this a lot internally—though we don't have anything to announce right now. I do want to say that while we 100% want people in the community to be able to contribute to XNAT, and clearly there are people in the community who want to be able to contribute to XNAT, it's a harder problem than you might think. It's a large idiosyncratic project that has been developed over a long time, so there is a lot of cultural knowledge—like how to build or develop things, or how the application works and why it works this particular way—that gets passed around among people without necessarily being formally communicated or even written down. Bringing new people in to the project so that they can learn enough to effectively contribute is a tough process even when they're full-time employees with nothing to do but work on XNAT. Providing an on-ramp for community members is another order of magnitude harder.

Which is not to say it is impossible, or that we can't or don't want to, just that it is going to take a lot of work to get there. One step that has been accomplished recently was this pull request from Paul Smith: https://bitbucket.org/xnatdev/xnat-web/pull-requests/399, who took some information from the discussion group and turned it into a contributing.md file in the XNAT repo. That has been merged to the develop branch, but not yet released. (...though what does it really mean to "release" a markdown file in the repo?) Now we can hopefully build on that success by using that as a starting point for other people to get their own local XNAT dev environments running and making their own changes and contributions.

We still have a long way to go to get to fully functional community support, but this is a move in the right direction. I hope we can do more soon to reduce the barrier to contributions.

John Flavin

On May 6, 2024 at 9:59:03 AM, "Battiston, Christopher" <chris.b...@wchospital.ca> wrote:

Just wanted to jump back on this and see if there's been any further discussion.  Would really love to get into this with other XNAT institutions. 

Thanks, 

Chris 

Women’s College Hospital is located on the traditional lands of the Anishinaabe, Mississauga, Haudenosaunee, Ojibway/Chippewa and Huron-Wendat nations. 

---

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of Mark Janse <marku...@gmail.com>
Sent: Monday, April 22, 2024 05:41
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: Re: [External]:Re: [XNAT Discussion] Contributing to XNAT

 

You don't often get email from marku...@gmail.com. Learn why this is important

CAUTION: This email originated from outside of Women's College Hospital. Do not click links, open attachments or respond to requests for information unless you recognize the sender and know the content is safe.

Hi all,

If a developer workshop is held, I'd love to attend as well.

My organization's main interest is the long-term maintainability of XNAT, updating the dependencies (Java 8, Tomcat 9) is something we'd like to contribute to.

Kind regards,

Mark Janse

Health-RI

Op vrijdag 19 april 2024 om 19:18:45 UTC+2 schreef Ford McDonald:

Thanks for the helpful information!

I'd also like to express interest in contributing to future development and attending a developer's workshop. 

Best,

Ford McDonald

On Monday, April 15, 2024 at 11:04:29 AM UTC-4 Battiston, Christopher wrote:

Just coming on to say I'd be possibly able to help with testing and validation of new features.  I'm not a programmer but I am very experienced in testing web apps (specifically REDCap) and would be happy to help with developing / running scripts and other documentation.  

Thanks, 

Chris 

---

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of Paul Smith <paul.14...@gmail.com>
Sent: Monday, April 15, 2024 10:29
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: [External]:Re: [XNAT Discussion] Contributing to XNAT

 

You don't often get email from paul.14...@gmail.com. Learn why this is important

CAUTION: This email originated from outside of Women's College Hospital. Do not click links, open attachments or respond to requests for information unless you recognize the sender and know the content is safe.

Hi,

Sorry for the very slow reply to this, but thanks so much John for the detailed info!

I managed to build `xnat-web` following the steps you outlined. I also opened a PR to add this info to the docs.

Martin and Guus, nice to know others are interested in contributing too! I agree with Guus that whilst we can view open issues, it would also be nice if we could submit new issues too.

Guus, what were the issues you ran into when creating the plugin? We have a bit of experience in the group of making plugins. It might be fun to work together on one and perhaps improve the docs around creating plugins.

On the topic of docs, is there anywhere we can make or suggest improvements to the wiki? 

Best wishes,

Paul

On Monday 16 October 2023 at 08:29:37 UTC+1 Guus Kolpa wrote:

I'd also like to express my interest in contributing. I ran into the exact same scenarion as outlined above:

"I got down the Rabbit hole of designing my own [plugin], and got lost in java, groovy, gradle, maven and then stopped"

One of the reasons I did not pursue any further involvement is that development seems to be closed-source on purpose. While the XNAT code-base and surface-level issues ( https://radiologics.atlassian.net/jira/software/c/projects/XNAT/issues/?filter=allissues ) are open to the public, other pages such as https://radiologics.atlassian.net/projects/XNAT/versions/14885/tab/release-report-all-issues require a radiologics account, giving me much more of a distant observer / read-only perspective than that of an open invitation to join in on contributing to open-source software. 

On Tuesday, October 10, 2023 at 8:29:51 PM UTC+2 Martin Boswell wrote:

John,

I'd also like to express my interest in resources for XNAT development.  Thanks for sharing this info; it's very handy summary. 

Is there any detailed documentation of the postgres schema?  Or even just brief descriptions of the purpose of each of the tables?  I've tried to pick some information out of there and quickly gotten confused.  

  - Martin

Martin S. Boswell
Cellular and Tissue Imaging Department
Molecular Biophysics and Integrated Bioimaging Division
Lawrence Berkeley National Laboratory
One Cyclotron Road, MS55R0121
Berkeley, CA   94720

On Thursday, October 5, 2023 at 9:37:49 AM UTC-7 John Flavin wrote:

Hi Paul,

TL;DR

• No plans right now for Developer Workshop or similar, but anyone else interested should let us know.
• Documentation of the code itself and the application architecture is sparse.
• I give some developer setup instructions below.

Full Answer

First let me say thank you for this question. It is exciting to see members of the community interested enough and engaged enough to try to start digging into how to develop XNAT. A community-supported XNAT is absolutely the future we want to create, and I'm grateful that you seem to want to get started on that future right now. I do too!

But you've quickly encountered one of the first hurdles to developing XNAT: it is not easy to figure out how to start. You’re definitely not the only person who encounters this problem. In fact there was another discussion board post just recently that ran up against a similar problem; when the post author looked into building a plugin to do some thing that XNAT doesn't do yet, their (entirely reasonable) response was

I got down the Rabbit hole of designing my own [plugin], and got lost in java, groovy, gradle, maven and then stopped.

That's just the first step in the process, and it was already too difficult (or too poorly documented) for a presumably smart, capable person to be able to make that commitment of time and energy. That experience will need to get better if we expect more people to be involved even deeper in XNAT's development.

Back to your specific questions. We don't currently have any plans for a Developer Workshop or any webinars.  That's not to say that we won't do those things or haven't discussed them, just that we don't have any plans as of today. We have definitely noted your interest, though, and if others also let us know they are interested that could influence our prioritization.

I'll go through your other questions at a high level. I do think a mini-workshop would be a better way to deliver this information, but I can give you a bit to go on now.

- the structure of the codebase

We don't have this documented as well as we should. As you found already, we have the XNAT Developer Documentation, but it is light on these kinds of fundamental details like the structure of the code base and the architecture of the application.

We have Understanding the Components of XNAT, but that's written for an audience of XNAT administrators, not developers. Might be helpful, might not.

The XNAT Under the Hood session from the 2016 Workshop seems likely to help with this. But I haven't seen it since I was there live and my memory of its contents is a bit hazy.

- how to set up a development environment

This is something we do have some internal documentation about that isn't available anywhere public. It doesn't have any kind of sensitive content or anything, so I can summarize it here.

You'll need git installed. I think it's likely that you already have it so I'll leave the details out.

You should get a Java IDE. Most of the core XNAT team use IntelliJ. The free version will almost certainly be enough to get you started.

Another thing you will need is the Java 8 JDK. That's the version that we build everything with. How to download it is actually far more complex than I would expect because there are versions from multiple vendors with varying coverage for different platforms. It could be as simple as using your package manager of choice, i.e. brew install openjdk@8 on a mac (but only if it has an intel chip).

If you have IntelliJ, you can install a JDK through it. Open a project (could be anything, even an empty one). From there open the project settings (⌘+; on a mac). From there open Platform Settings > SDKs on the left pane. Click the + button in the middle pane and select Download JDK…. Select version 1.8. Pick a vendor. I’m not sure if the particular selection matters or not, but you need to select one that matches your architecture. For instance, I'm on an M1 mac, so I need an aarch64 version. I go with the Amazon Corretto version, myself, but I really am not sure it matters.

Once you have a JDK you should install the build tools. First, maven. It's available via package managers. It's called "maven" but on the terminal the command is mvn which is odd but it is what it is. We also use gradle as a build tool but you won't need to install it manually because we use "wrapper scripts" in each repo that download gradle for you. You'll run, say, ./gradlew war to build the XNAT WAR file and it will work.

Also note that you will need to set the JAVA_HOME environment variable to point to the JDK you installed. There is a magical command that will figure it out for you. Put this command into your shell's configuration so it runs every time you open a terminal:

export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

(Assuming bash or zsh or similar. Adapt for different shells as needed.)

You know you have everything set up correctly when you can run mvn -version or ./gradlew --version (from within a repo directory) and it prints out the correct JDK version you installed.

- how to run tests

We have a mix of tests happening at different levels. Most of the core repositories include unit tests that run at build time, which are definitely easier and faster to use during development but give you a limited picture of the impact of a change on the whole application. You could likely run these using whatever build tool is used in the repo, either mvn test for maven or ./gradlew test for gradle, though the details may vary a bit by repo.

We also have the XNAT REST Tests, which are a suite of integration tests that run against a live XNAT. It's harder to set all this up and they are too slow to run manually as a regular part of development, but they are a very valuable part of automated testing infrastructure.

- what issues are good for first time contributors

Another good question. I don't have anything at hand. But we might want to start tagging issues that would be good for this purpose.

I think that's all for now. Again, thank you for asking this question. I think that connecting with people who want to get started developing on XNAT itself is a wonderful idea, and I would love it if we could organize something like that in the future. The information I've given here is admittedly not that much but hopefully it helps a little. 

John Flavin

Backend Team Lead

johnf...@flywheel.io

He/Him

flywheel.io

On Sep 28, 2023 at 9:11:11 AM, Paul Smith <paul.14...@gmail.com> wrote:

Hi,

I'm Paul, a software developer supporting the XNAT service we run at UCL.

At the XNAT Workshop in London last year, it was mentioned that there is a goal to have XNAT be fully maintained and developed by the community once current grant funding runs out.

I would be happy to contribute to the development of XNAT, but I'm not sure where to start. I've seen there is some developer documentation (https://wiki.xnat.org/documentation/xnat-developer-documentation), but I'm still not too sure where to begin.

I'm wondering whether there are any plans to run a Developer Workshop where these topics are introduced? Or perhaps a series of webinars? I'd love to learn about the internals of XNAT and how I can start to contribute. I think covering the following topics would be super useful:

- the structure of the codebase

- how to set up a development environment

- how to run tests

- what issues are good for first time contributors

I think a good goal would be for the attendees of the workshop / webinars to help update the existing developer docs to make it easier for first-time contributors to get started.

Apologies if all this information is already available - I may well be missing something!

Thanks

Paul

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/57b1e234-4bdf-40a9-aa02-b6b066c5f52en%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/6dc372ca-a6fc-4c97-9a0e-5a52a54d41d2n%40googlegroups.com.

CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

AVIS DE CONFIDENTIALITÉ : Ce message électronique, ainsi que tout fichier qui y est joint, est réservé à l’usage exclusif du destinataire visé et peut contenir des renseignements confidentiels et privilégiés. Toute lecture, utilisation, divulgation ou distribution non autorisée est interdite. Si vous n’êtes pas le destinataire visé, veuillez en aviser l’expéditeur par retour de courriel et détruire toutes les copies du message original.

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/29bf60f4-e1b9-49b6-a906-0d2fa80712ecn%40googlegroups.com.

CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

AVIS DE CONFIDENTIALITÉ : Ce message électronique, ainsi que tout fichier qui y est joint, est réservé à l’usage exclusif du destinataire visé et peut contenir des renseignements confidentiels et privilégiés. Toute lecture, utilisation, divulgation ou distribution non autorisée est interdite. Si vous n’êtes pas le destinataire visé, veuillez en aviser l’expéditeur par retour de courriel et détruire toutes les copies du message original.

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/YT2PR01MB8759C0B1CFF603E3252AB209F71C2%40YT2PR01MB8759.CANPRD01.PROD.OUTLOOK.COM.

[Battiston, Christopher]

Thanks John – I’ll keep an eye out for anything else, and appreciate your time!

 

Thanks, 

Chris 

 

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/CADwXCSo-aW_4OuhFY-KU%2B21Wf4VdrqTxXvM1v9UdP0EHWugzdw%40mail.gmail.com.

[James]

Hey,

One thing I would like to suggest that I think could go a long way is a public issue tracker for each repository. Obviously you guys use JIRA internally, and that's fine an make sense for the core team. But us other users can't really report bugs there without first requesting access. I think this discourages users helping each other out.

I understand bugs are mostly reported on the mailing list, but this has a number of problems. For one, bugs relating to all the many software components are on the same list, rather than on a repo by repo basis a la github/gitlab. Secondly it doesn't have good integration with the source code repository. referencing specific files, blocks of code, or other issues is cumbersome and amounts to posting URLS that other users have to open in a separate browser window. Moreover it is easier to keep track of the status of an issue in a real issue tracker. Finally, mail doesn't support Markdown, which I know is a minor gripe but I have got used formatting my bug reports over the years.

I think a great example of a good issue tracker is the one in use at https://github.com/NrgXnat/xnat-docker-compose/issues. I have had a great experience reporting bugs there, it has active users who are willing to share their patches, knowledge and solutions with each other, and its easy to search for issue specifically related to the xnat-docker-compose project there.

I know xnat related code is mostly hosted on bitbucket, and and i'm sure not if bitbucket offer an issue tracker similar to github/gitlab. But you could consider hosting a mirror on github/gitlab? This is fairly easy to automate and shouldn't cause too many head-aches if you are just using it for issues rather than PRs.

Anyway this was just a thought. And I'd also like to add I too would be interested in contributing bug fixes back to XNAT. I have a number of bug fixes monkey-patched into my plugins which I would love to see merged upstream one day.

All the best,
James

[Battiston, Christopher]

I realise that this is probably farfetched but I love this idea.  I'm hoping to start using XNAT for regulated clinical research, and the more documentation and such is done / available, the better and easier it will be to get the application approved.  

Thanks, 

Chris 

Pronouns: He / Him  

REDCap Administrator / Research Data Analyst, Research Operations 

The Center for Health Research Information Systems 

Please know that I honour and respect boundaries around personal time, well-being, caretaking, and rest. Should you receive correspondence from me during a time that you're engaging in any of the above, please protect your time and wait to respond until you're next working or in front of a computer. Prioritize joy, not email, when and where you can. 

  

Women’s College Hospital is located on the traditional lands of the Anishinaabe, Mississauga, Haudenosaunee, Ojibway/Chippewa and Huron-Wendat nations. 

---

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of James <james....@med.uni-goettingen.de>
Sent: Thursday, May 9, 2024 12:26

To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: Re: [External]:Re: [XNAT Discussion] Contributing to XNAT

 

You don't often get email from james....@med.uni-goettingen.de. Learn why this is important

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/f74bfc4d-0fe2-4c89-9532-74c3fd1cb60an%40googlegroups.com.