integration with Open OnDemand, for compute, via Loop

[Philip Durbin]

Hello Dataverse enthusiasts!

You may have heard me mention an integration we've been working on between Dataverse and Open OnDemand. I gave a talk[1] about it at the 2025 Dataverse Community Meeting at UNC, for example.

Two weeks ago Aday Bujeda, one of the lead developers of this new integration (the other being David Verdú), demo'ed how it all works to the Open OnDemand community during their monthly Tips and Trips call, which was recorded. Here's a direct link to the recording: https://drive.google.com/file/d/1ItFEz7TiwjFuWBLGjXfOBWpehGN6-8Fu/view?usp=drive_link

You'll probably want to skip ahead to 24:20. Aday introduces the new application called OnDemand Loop (just Loop, for short), so called because we are creating a loop between a compute environment and a data repository and back again. Download a dataset from Dataverse, remix it, and publish it back as a new dataset to Dataverse. Something like that. There are plenty of use cases to explore.

Naturally, I also put the video on DataverseTV: https://dataverse.org/dataversetv

I'll also attach his slides.

According to https://openondemand.org Open OnDemand is being used by over 1600 organizations. Harvard is one of them. Maybe your org is already using it!

OnDemand Loop is open source, of course. The code is at https://github.com/IQSS/ondemand-loop and the guides can be found at https://iqss.github.io/ondemand-loop/

Enjoy! Please let us know if you have any questions, comments, or use cases to share!

Thanks,

Phil

1. My slides from the 2025 Dataverse Community Meeting at UNC: https://docs.google.com/presentation/d/1gzINek2-x8HLdVkpV7lQg2OIK6VoQmXzAE3bU8k7CeU/edit?usp=sharing

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Philip Durbin]

Hello Dataverse Enthusiasts!

tl;dr: Datasets on https://demo.dataverse.org now have a "Explore in Open OnDemand" button (with the computation only available to Harvard affiliates).

As I mentioned in the previous message in this thread, Open OnDemand is a web frontend to High Performance Computing (HPC) resources that is newly integrated with Dataverse through an application we've built called OnDemand Loop ("Loop" for short). Loop is a way to bring data alive by computing on it, and you can deposit data back into Dataverse from the computational environment. (That's why we call it "Loop".)

As you may know, https://demo.dataverse.org is the place where we encourage people to try Dataverse. We've just added Loop as a dataset-level external tool, which means you'll see "Explore in Open OnDemand" on every dataset.

I wish I could tell you that everyone will be able to play around with this new toy on the demo server, but because it requires computational resources, it's limited to Harvard affiliates.

Everyone is welcome to find a dataset with files at https://demo.dataverse.org and click "Access Dataset" followed by "Explore in Open OnDemand". You'll be taken to this landing page: https://hmdc.github.io/fasrc-ondemand-loop/dv_external_tools/demo_landing.html

For all of you, you can use the landing page to learn more about Loop and especially the documentation on how to install it at your own site, since it's open source, of course! Here's a direct link to those docs: https://iqss.github.io/ondemand-loop/

For Harvard affiliates only, when you click "Explore in Open OnDemand" you're welcome to sign into the VPN and click "Continue" to actually try out the integration.

Either way, we'd love your feedback on this new integration!

Thanks!

Phil