Re: [Dataverse-Users] Is anyone using SWORD to connect to Globus for large files?

51 views
Skip to first unread message

Philip Durbin

unread,
Nov 5, 2015, 5:00:55 PM11/5/15
to dataverse...@googlegroups.com
Not that I know of. Globus is interesting though... it came up recently in a kickoff meeting for this project: https://hms.harvard.edu/news/dropbox-structural-biologists

Currently large files are not handled any differently than small files in Dataverse. Part of the project mentioned in the story above is about handling large biomedical datasets. It's such a new project that I don't have any details to share but I'm *very* interested in ideas people in the Dataverse community about how best to support large files. I'll try to remember to reply to this thread when we have a GitHub issue or Functional Requirements Document to read! In the meantime (everybody), please tell us what you're thinking! :)

Phil

On Thu, Nov 5, 2015 at 1:43 PM, susan borda <mutan...@gmail.com> wrote:
Hi-
We are currently using DSpace for our IR, and are considering using Dataverse for our data repository. How are large files handled in Dataverse? Apparently Uni of Exeter is using SWORD to connect Globus: https://www.globusworld.org/files/2013/02-Taylor-Plugging_the_BIG_DATA_Gap_in_DSpace_Using_SWORD_and_Globus.pdf

Is anyone doing this with Dataverse?

Thanks,
susan

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/11f77ea5-97e6-4d35-9fee-e2c994bef652%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

susan borda

unread,
Nov 5, 2015, 5:50:41 PM11/5/15
to Dataverse Users Community, philip...@harvard.edu
Hi Phil-
Thanks for the info. I'm just learning about Dataverse, I didn't realize that harvesting was part of it if you're running your own instance. We are setting up an server for archived data (large files) and the plan is to use a Globus link to connect DSpace to the file. I was hoping this sort of feature was available with Dataverse. The majority of our datasets will be a reasonable size and can exist with a system such as Dataverse but some will not and I'm just trying to figure out what the best approach would be in this scenario.

Thanks,
susan


On Thursday, November 5, 2015 at 3:00:55 PM UTC-7, Philip Durbin wrote:
Not that I know of. Globus is interesting though... it came up recently in a kickoff meeting for this project: https://hms.harvard.edu/news/dropbox-structural-biologists

Currently large files are not handled any differently than small files in Dataverse. Part of the project mentioned in the story above is about handling large biomedical datasets. It's such a new project that I don't have any details to share but I'm *very* interested in ideas people in the Dataverse community about how best to support large files. I'll try to remember to reply to this thread when we have a GitHub issue or Functional Requirements Document to read! In the meantime (everybody), please tell us what you're thinking! :)

Phil
On Thu, Nov 5, 2015 at 1:43 PM, susan borda <mutan...@gmail.com> wrote:
Hi-
We are currently using DSpace for our IR, and are considering using Dataverse for our data repository. How are large files handled in Dataverse? Apparently Uni of Exeter is using SWORD to connect Globus: https://www.globusworld.org/files/2013/02-Taylor-Plugging_the_BIG_DATA_Gap_in_DSpace_Using_SWORD_and_Globus.pdf

Is anyone doing this with Dataverse?

Thanks,
susan

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

Philip Durbin

unread,
Nov 5, 2015, 5:58:37 PM11/5/15
to dataverse...@googlegroups.com
Ok, this is starting to make more sense. Please feel free to create an issue at https://github.com/IQSS/dataverse/issues for your use case. As much detail as possible would be appreciated!

On Thu, Nov 5, 2015 at 5:50 PM, susan borda <mutan...@gmail.com> wrote:
Hi Phil-
Thanks for the info. I'm just learning about Dataverse, I didn't realize that harvesting was part of it if you're running your own instance. We are setting up an server for archived data (large files) and the plan is to use a Globus link to connect DSpace to the file. I was hoping this sort of feature was available with Dataverse. The majority of our datasets will be a reasonable size and can exist with a system such as Dataverse but some will not and I'm just trying to figure out what the best approach would be in this scenario.

Thanks,
susan

On Thursday, November 5, 2015 at 3:00:55 PM UTC-7, Philip Durbin wrote:
Not that I know of. Globus is interesting though... it came up recently in a kickoff meeting for this project: https://hms.harvard.edu/news/dropbox-structural-biologists

Currently large files are not handled any differently than small files in Dataverse. Part of the project mentioned in the story above is about handling large biomedical datasets. It's such a new project that I don't have any details to share but I'm *very* interested in ideas people in the Dataverse community about how best to support large files. I'll try to remember to reply to this thread when we have a GitHub issue or Functional Requirements Document to read! In the meantime (everybody), please tell us what you're thinking! :)

Phil
On Thu, Nov 5, 2015 at 1:43 PM, susan borda <mutan...@gmail.com> wrote:
Hi-
We are currently using DSpace for our IR, and are considering using Dataverse for our data repository. How are large files handled in Dataverse? Apparently Uni of Exeter is using SWORD to connect Globus: https://www.globusworld.org/files/2013/02-Taylor-Plugging_the_BIG_DATA_Gap_in_DSpace_Using_SWORD_and_Globus.pdf

Is anyone doing this with Dataverse?

Thanks,
susan

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/6113de65-62bd-4b93-831a-9b8ee46da32a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Eugene Barsky

unread,
Nov 6, 2015, 10:04:15 AM11/6/15
to Dataverse Users Community, philip...@harvard.edu, Leanne Trimble, Chuck Humphrey
Folks:

Here in Canada, Compute Canada has adopted Globus to move large files through our HPC infrastructure. We will be connecting our Dataverse to Globus for large files in the near future. As I learn more about Globus in the next few months, I will be able to update the community - https://www.computecanada.ca/research-portal/national-services/globus-portal/

Interesting times...

Eugene (UBC)



On Thursday, 5 November 2015 14:58:37 UTC-8, Philip Durbin wrote:
Ok, this is starting to make more sense. Please feel free to create an issue at https://github.com/IQSS/dataverse/issues for your use case. As much detail as possible would be appreciated!
On Thu, Nov 5, 2015 at 5:50 PM, susan borda <mutan...@gmail.com> wrote:
Hi Phil-
Thanks for the info. I'm just learning about Dataverse, I didn't realize that harvesting was part of it if you're running your own instance. We are setting up an server for archived data (large files) and the plan is to use a Globus link to connect DSpace to the file. I was hoping this sort of feature was available with Dataverse. The majority of our datasets will be a reasonable size and can exist with a system such as Dataverse but some will not and I'm just trying to figure out what the best approach would be in this scenario.

Thanks,
susan

On Thursday, November 5, 2015 at 3:00:55 PM UTC-7, Philip Durbin wrote:
Not that I know of. Globus is interesting though... it came up recently in a kickoff meeting for this project: https://hms.harvard.edu/news/dropbox-structural-biologists

Currently large files are not handled any differently than small files in Dataverse. Part of the project mentioned in the story above is about handling large biomedical datasets. It's such a new project that I don't have any details to share but I'm *very* interested in ideas people in the Dataverse community about how best to support large files. I'll try to remember to reply to this thread when we have a GitHub issue or Functional Requirements Document to read! In the meantime (everybody), please tell us what you're thinking! :)

Phil
On Thu, Nov 5, 2015 at 1:43 PM, susan borda <mutan...@gmail.com> wrote:
Hi-
We are currently using DSpace for our IR, and are considering using Dataverse for our data repository. How are large files handled in Dataverse? Apparently Uni of Exeter is using SWORD to connect Globus: https://www.globusworld.org/files/2013/02-Taylor-Plugging_the_BIG_DATA_Gap_in_DSpace_Using_SWORD_and_Globus.pdf

Is anyone doing this with Dataverse?

Thanks,
susan

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

Esther Dzale

unread,
Mar 9, 2016, 10:18:18 AM3/9/16
to Dataverse Users Community, philip...@harvard.edu, leanne....@utoronto.ca, chuck.h...@ualberta.ca
Hi Eugene,
just to know if you managed to connect your Dataverse to Globus? I would be glad to have your feedback. Thanks.
Esther

Eugene Barsky

unread,
Mar 9, 2016, 2:14:55 PM3/9/16
to dataverse...@googlegroups.com, Hlady, Jason, Jeffrey, Keith, Philip Durbin, Leanne Trimble, Chuck Humphrey
Hello Esther:

I am away from my office till late March and have an intermittent access to the Web. Sorry for this short message...

Our dear colleagues in Compute Canada / Globus project are planning to explore forking Globus Publications to Dataverses API during this year. I am referring you to the project leader - Jason Hlady or project manager - Keith Jeffrey to ask about the details.

With thanks,

Eugene




On Wed, Mar 9, 2016 at 5:18 PM, Esther Dzale <estd...@gmail.com> wrote:
Hi Eugene,
just to know if you managed to connect your Dataverse to Globus? I would be glad to have your feedback. Thanks.
Esther



--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Dataverse Users Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dataverse-community/Pe6znGA000Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

Mercè Crosas

unread,
Mar 10, 2016, 4:14:21 PM3/10/16
to dataverse...@googlegroups.com, Philip Durbin, Leanne Trimble, Chuck Humphrey, Pete Meyer
Eugene, Esther,

Are any of you planning to go to the GlobusWorld conference this April:


Two people from our Dataverse Harvard group are planning to go. It could be a chance to have a discussion about integrating Dataverse with Globus.

Let us know!
Merce


Mercè Crosas, Ph.D.
Chief Data Science and Technology Officer, IQSS
Harvard University

Esther
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages