large files

[August Muench]

Hi,

Have other users run into the 1 Gb file size restriction stated on this guide page:  http://guides.thedata.org/book/create-new-study ?

We are about to crash into it big time in theastrodata with an ingest that contains four tables between 1.4 and 1.7G and 10 image files with sizes of 2.5-2.7G. 

I predict we will hit it regularly in the future.

 - Gus

[Condon, Kevin]

Gus,

The current production file size upload limit is 2GB. The documentation needs to be updated to reflect that. That said, this upper limit is a limitation of the upload control we're using. Until recently, there hasn't been much call for larger files. We have been investigating ways to handle very large files so we expect to improve this is an upcoming release.

There have been a few users who want to upload larger files, one recent user wanted to upload a single 100GB file. In most of those cases, the users were able to subdivide the file to sizes that did fit.

Regards,

Kevin

[August Muench]

Kevin,

Thanks.  

I thought about subdivision but it really doesn't work, esp with the images.  They are conceptually complete "things" -- image mosaics -- that do not lend themselves to division; which is to "undo" their creation! :)

Are upload, download and internal storage separate issues?  I mean that the backend servers might have no problem with 100 Gb files but the upload process might be bottlenecked by the server or just too inefficient (same with download).  Put another way, if we took "upload" off the table by instantiating the dataset another, physical, way, then would the other potential problems -- server storage and download -- remain?

- gus

[Condon, Kevin]

Gus,

I would have to check with the group to provide an accurate answer but my impression was this was a limitation of file upload only.

Kevin

[Gustavo Durand]

Storage should definitely not be an issue, provided you have the space.

Download should also be OK, but of course, the larger the file, the
more potential for the user to have to wait a long time , have the
download interrupted, etc.