Large file (> 2GB) upload experiences with DVN

206 views
Skip to first unread message

Joerg Messer

unread,
Sep 23, 2014, 4:13:59 PM9/23/14
to dataverse...@googlegroups.com
Greetings,

I was wondering if anyone has managed to upload files larger than 2GB to DVN?  I did some tests using the latest Chrome, Firefox, Opera and Explorer running on a 4GB 32 bit Windows 7 client and our Red Hat 5.10 48GB DVN v3.6 server.  We have a fairly fast network.  Uploads in the 1GB range were no problem and loaded on the first attempt.  Getting up to 1.4GB was a tad more problematic and sometimes required a second attempt.  I was only able to upload a 2GB file with Explorer.  Has anyone else managed to consistently load files greater than 2GB in size through the web?  At the moment we recommend that users split files but there are times when this is inconvenient.

Also, are there any plans for including a status widget to indicate upload progress in the coming versions of DVN? 

//Joerg

Philip Durbin

unread,
Sep 23, 2014, 8:16:45 PM9/23/14
to dataverse...@googlegroups.com
It's something of a known issue that DVN 3.x doesn't support files greater that 2 GB being uploaded via the GUI:

"There is a maximum file size of 2 gigabytes for each file that you upload" https://github.com/IQSS/dvn/blob/develop/doc/sphinx/source/dataverse-user-main.rst#create-new-study

That said, I've had decent luck uploading larger files via the Data Deposit API. I think the largest file I tried was 4 GB.

With regard to Dataverse 4.0, I don't believe we're aware of any file size upload limit in either the GUI or any API. The file upload widget does show a progress bar as files are being uploaded.

Phil



--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/05f4e126-4e01-4f52-bc8b-ecab70bf21c5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Joerg Messer

unread,
Sep 23, 2014, 8:23:16 PM9/23/14
to dataverse...@googlegroups.com, philip...@harvard.edu
Hi Phil,

I forgot about the SWORD API.  That sounds encouraging.  It will be interesting to see what v4.0 brings to the table.  Looking forward to it.  Thanks.

//Joerg


On Tuesday, 23 September 2014 17:16:45 UTC-7, Philip Durbin wrote:
It's something of a known issue that DVN 3.x doesn't support files greater that 2 GB being uploaded via the GUI:

"There is a maximum file size of 2 gigabytes for each file that you upload" https://github.com/IQSS/dvn/blob/develop/doc/sphinx/source/dataverse-user-main.rst#create-new-study

That said, I've had decent luck uploading larger files via the Data Deposit API. I think the largest file I tried was 4 GB.

With regard to Dataverse 4.0, I don't believe we're aware of any file size upload limit in either the GUI or any API. The file upload widget does show a progress bar as files are being uploaded.

Phil


On Tue, Sep 23, 2014 at 4:13 PM, Joerg Messer <joerg....@gmail.com> wrote:
Greetings,

I was wondering if anyone has managed to upload files larger than 2GB to DVN?  I did some tests using the latest Chrome, Firefox, Opera and Explorer running on a 4GB 32 bit Windows 7 client and our Red Hat 5.10 48GB DVN v3.6 server.  We have a fairly fast network.  Uploads in the 1GB range were no problem and loaded on the first attempt.  Getting up to 1.4GB was a tad more problematic and sometimes required a second attempt.  I was only able to upload a 2GB file with Explorer.  Has anyone else managed to consistently load files greater than 2GB in size through the web?  At the moment we recommend that users split files but there are times when this is inconvenient.

Also, are there any plans for including a status widget to indicate upload progress in the coming versions of DVN? 

//Joerg

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

Philip Durbin

unread,
Sep 23, 2014, 8:27:21 PM9/23/14
to dataverse...@googlegroups.com
Oh sure. You can give feedback about the direction Dataverse 4.0 is taking by trying out http://dataverse-demo.iq.harvard.edu

Obviously, the earlier we get feedback the easier it will be to make any changes!

See also the wonderful blog posts that we're making regularly: http://datascience.iq.harvard.edu/blog

Phil

p.s. Thanks to all for various feedback so far! :)

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages