Web browser upload limits?

[Patrick Gavin]

We are having problems uploading larger files to our guest collection using the 'upload' button. Files under a gig work fine but larger files cause some kind of error and do not transfer. I'm not seeing anything in our GCS logs. Transferring larger files works if using GCP.

Is there some kind of limit as to what can be transferred with the 'upload' button?

Thanks,

-P

[Richard Wagner]

Hi Patrick,

The upload button is going to do an HTTP PUT to your server through Apache, at least for collections tied to a POSIX  storage gateway. (I’m not sure if Globus uses other HTTP servers for different Linux distributions.)

The upload will show up in the Apache logs, which on an Ubuntu instance is at
  /var/log/apache2/other_vhosts_access.log

If something went wrong, you’ll probably find a hint in the error.log file in the same directory.

I was able to upload a 5.5GB file from my Mac through Safari to a GCS DTN running on Ubuntu 22.04LTS. I have old memories of 2GB or similar limits on HTTP PUTs and GETs, which may not be accurate.

Hopefully this gets you looking in the right place for troubleshooting.

—Rick

[Patrick Gavin]

Hi Rick-

That was exactly it.

Thanks for your help!

Patrick Gavin (He/Him)
Systems Administrator
Central IT Systems & Services
Cal Poly Humboldt
Email: Patric...@humboldt.edu

This e-mail was written without the assistance of ChatGPT

[Anthony Weaver]

I understand this thread is almost a year old and we are having this issue with web uploads and we'd like some guidance.  

Our GCS is running Debian 12 with GCS 5.4.811 cli 1.0.56.

Similar to Patrick we are unable to upload files larger than 1GB.  Transfer is file and uploading a folder whose contents are greater than a1GB 

also work (with the caveat that no one individual file was larger than 1GB).  We tried with different browser with no success.

We did look at the other_vhosts_access.log but it did not provide any error or useful information

Our DTN has been running setup and going for a few years now, so we haven't mucked around with settings in GCS any time recently.

If the answer is that larger (than 1GB files) need to be transferred, not uploaded, then fine.  It might be helpful to put that

in the documentation

[Lev Gorenstein]

If the answer is that larger (than 1GB files) need to be transferred, not uploaded, then fine.

It's not, and you totally should be able to upload larger files (with all the caveats of plain HTTPS upload). The crux is that when you do that, your GCS is not involved - this becomes a pure Apache action. So the limit you see is not a GCS limit, but rather an Apache one: https://httpd.apache.org/docs/2.4/mod/core.html#limitrequestbody

We have an FAQ on that (but I agree that it's not very easy to find without the magic keyword).

Lev

--

Lev Gorenstein
Solutions Architect
Globus // University of Chicago

e: l...@globus.org

[Anthony Weaver]

Lev,

Thank you.  That is very helpful

[Anthony Weaver]

We modified our apache.conf adding the following line

LimitRequestBody 0

and restarted apache.  We were then unable to upload large files.  The Apache documentation seems to at least suggest the it is 0 by default but

in our case that was not true as we did not have any limit in our apache.conf

[Lev Gorenstein]

I suspect distributions may choose to set their own default values - plus there's been a change in Apache's internal default (docs say it was 0 before 2.4.53, then changed to 1 GB). So depending on the distro and Apache version, sites can see different behavior - which of course adds to the confusion.

Lev

[Anthony Weaver]

There was a slight typo in my message.  Once we modified our apache.conf, we were ABLE to do large file uploads.  My original message mistakenly said unable