Dataverse maximum uploadable file size

977 views
Skip to first unread message

Michel Bamouni

unread,
Aug 10, 2017, 10:07:03 AM8/10/17
to Dataverse Users Community
Hi,

When creating dataset in dataverse, on the tab "Files", what the maximum size of the file I can upload?


Best regards

Michel

Brian Silverstein

unread,
Aug 10, 2017, 10:49:39 AM8/10/17
to Dataverse Users Community
Hi Michel,

The maximum file upload size is actually configurable for your installation through a database option :MaxFileUploadSizeInBytes -- this can be set with a curl command shown below:


Note that the 2147483648 is the number of bytes you are setting as the maximum, which in this case would be 2GB. Harvard Dataverse has their maximum set to about 2.5GB as I recall.

If you have any other questions regarding database settings, check out:

and feel free to ask here in the group as well!

Best,

Brian Silverstein
IQSS

Leonhard Maylein

unread,
Aug 10, 2017, 11:46:47 AM8/10/17
to dataverse...@googlegroups.com
Hi all,

we have changed MaxFileUploadSizeInBytes to 35 GByte:

curl -X PUT -d 37580963840
http://localhost:8080/api/admin/settings/:MaxFileUploadSizeInBytes

Unfortunately this helps only for files between 2 GByte and
10 GByte (roughly spoken).

For larger files we run into another problem. During the
upload, the progress bar looks good (100 % after a few minutes)
but then we are not able to save the upload.

server.log:

[2017-07-27T15:16:33.386+0200] [glassfish 4.1] [SEVERE] []
[org.glassfish.jersey.server.ServerRuntime$Responder] [tid: _ThreadID=51
_ThreadName=jk-connector(4)]
[timeMillis: 1501161393386] [levelValue: 1000] [[
An I/O error has occurred while writing a response message entity to
the container output stream.
org.glassfish.jersey.server.internal.process.MappableException:
java.io.IOException: Connection reset by peer
[...]

Are there any solutions for very big files?


Leonhard Maylein

Heidelberg University Library



Am 10.08.2017 um 16:49 schrieb Brian Silverstein:
> Hi Michel,
>
> The maximum file upload size is actually configurable for your
> installation through a database option*:MaxFileUploadSizeInBytes* --
> this can be set with a curl command shown below:
>
> |curl -X PUT -d 2147483648
> http://localhost:8080/api/admin/settings/:MaxFileUploadSizeInBytes|
>
>
> Note that the 2147483648 is the number of bytes you are setting as the
> maximum, which in this case would be 2GB. Harvard Dataverse has their
> maximum set to about 2.5GB as I recall.
>
> If you have any other questions regarding database settings, check out:
> http://guides.dataverse.org/en/latest/installation/config.html#database-settings
>
> and feel free to ask here in the group as well!
>
> Best,
>
> Brian Silverstein
> IQSS
>
>
>
> On Thursday, August 10, 2017 at 10:07:03 AM UTC-4, Michel Bamouni wrote:
>
> Hi,
>
> When creating dataset in dataverse, on the tab "Files", what the
> maximum size of the file I can upload?
>
>
> Best regards
>
> Michel
>
> --
> You received this message because you are subscribed to the Google
> Groups "Dataverse Users Community" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dataverse-commu...@googlegroups.com
> <mailto:dataverse-commu...@googlegroups.com>.
> To post to this group, send email to
> dataverse...@googlegroups.com
> <mailto:dataverse...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dataverse-community/722f6380-edaa-48a3-ad0d-848efd1b68c6%40googlegroups.com
> <https://groups.google.com/d/msgid/dataverse-community/722f6380-edaa-48a3-ad0d-848efd1b68c6%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


Philip Durbin

unread,
Aug 10, 2017, 12:27:48 PM8/10/17
to dataverse...@googlegroups.com
I'd say a 35 GB file big enough that in the future, you might be interested in evaluating the rsync solution currently under development under the name "Data Capture Module". It's highly experimental at this point but you can read a little bit about it at http://guides.dataverse.org/en/4.7.1/installation/data-capture-module.html

Today I would suggested trying to upload big files like this via API. 35 GB is much bigger than what I've ever tested with though.

I also described a replace-the-file hack here, if that's of interest: https://groups.google.com/d/msg/dataverse-community/DDIGmmRe8oE/xMLMjOuxBgAJ

I hope this helps!

Phil

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com <mailto:dataverse-community+unsubs...@googlegroups.com>.
To post to this group, send email to dataverse-community@googlegroups.com <mailto:dataverse-community@googlegroups.com>.
--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsubscribe...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/c362175b-67b5-620d-cf60-8b5cd64ca14a%40ub.uni-heidelberg.de.

For more options, visit https://groups.google.com/d/optout.

Michel Bamouni

unread,
Aug 11, 2017, 4:56:17 AM8/11/17
to Dataverse Users Community, philip...@harvard.edu
Hi,

Thanks all for answers.

Regards,

Michel
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com <mailto:dataverse-community+unsub...@googlegroups.com>.
To post to this group, send email to dataverse...@googlegroups.com <mailto:dataverse...@googlegroups.com>.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.

Amber Leahey

unread,
Aug 15, 2017, 1:59:55 PM8/15/17
to Dataverse Users Community, May...@ub.uni-heidelberg.de
We've noticed the same thing after increasing limit to 10GB. 

Can someone confirm whether Harvard is at 2gb or 10gb - I saw on a list earlier this year that Harvard was now at 10gb. 

We don't see any noticeable difference after increasing size limit. 

@Leonhard can you describe your system setup to work with up to 10GB files? e.g. local storage / hardware, etc. 

Many thanks, 
Amber

Philip Durbin

unread,
Aug 15, 2017, 2:30:23 PM8/15/17
to dataverse...@googlegroups.com
Harvard Dataverse says, "File upload limit is 2,684,354,560 bytes per file."

> To post to this group, send email to

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/4bf10eab-6ee4-4707-ae94-9fc55bd4c3d7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Leonhard Maylein

unread,
Aug 16, 2017, 2:42:00 AM8/16/17
to dataverse...@googlegroups.com
Our system setup:

Virtual machine, 4 (virtual) CPUs, 8 GByte RAM.
File storage via NFS.

I don't know the exact limit. The 10 GByte are
an estimation.

Leonhard Maylein
> <http://localhost:8080/api/admin/settings/:MaxFileUploadSizeInBytes%7C>
> > an email to dataverse-commu...@googlegroups.com
> <javascript:>
> > <mailto:dataverse-commu...@googlegroups.com
> <javascript:>>.
> > To post to this group, send email to
> > dataverse...@googlegroups.com <javascript:>
> > <mailto:dataverse...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/dataverse-community/722f6380-edaa-48a3-ad0d-848efd1b68c6%40googlegroups.com
> <https://groups.google.com/d/msgid/dataverse-community/722f6380-edaa-48a3-ad0d-848efd1b68c6%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/dataverse-community/722f6380-edaa-48a3-ad0d-848efd1b68c6%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/dataverse-community/722f6380-edaa-48a3-ad0d-848efd1b68c6%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>


Reply all
Reply to author
Forward
0 new messages