Failing Uploading Files More Than 1MB
Tumie Mosweu
I am Itumeleng from Botswana Harvard Partnership, we have a dataverse running at https://dataverse.bhp.org.bw. We are having trouble uploading files on the instance that are larger than 1MB, the files just load the status bar to full and never give the option to specify metadata. I have attached a screenshot to give a visual idea of the problem. Kindly assist, thanks.
James Myers
Itumeleng,
In v4.9.4, any failure of the upload http call can cause the symptom you’re seeing. The cause is usually a timeout that needs to be increased, especially when the problem only appears as files get larger.
Since this is a common issue, I’m going to try and summarize lots of the standard guidance (developed by multiple people over time) here:
There are a few timeouts that could be a problem:
If you are using Apache (httpd or apache2) there are timeouts for http connections and for the proxy connection from apache to Dataverse/glassfish. These are
Timeout 6000
ProxyTimeout 6000
Values are in seconds – these settings are for 6000 seconds (100 minutes). They would be added to a .conf file usually /etc/httpd/conf.d/ssl.conf.
There is also a timeout in glassfish that can be viewed and set with the following (the solution was found in Issue #6927, @Venki18 provided these commands there):
Change GlassFish's timeout setting as follows:
GET CURRENT VALUE
asadmin get server-config.network-config.protocols.protocol.http-listener-1.http.request-timeout-seconds
SET NEW VALUE. e.g. 3600s
asadmin set server-config.network-config.protocols.protocol.http-listener-1.http.request-timeout-seconds=3600
You can just try setting these values to times longer than the time your uploads are taking. If you want to verify that it is a timeout and not some other network error, the best way to debug the issue on older Dataverse versions is to open the dev console in your browser. For Chrome, you can right click on the page and select ‘Inspect’. If you then click on the ‘Network’ tab, you’ll see a list of the individual http calls being made to the server as you upload. If you try your upload again, you should see new http calls happening. You’ll also see one or more fail when you have the problem.
If I disconnect my network, I see an entry like
Where the status is (failed).
If there’s a timeout, you’ll see a numeric code: 200 means success and timeouts usually appear as 504. In that case, if you click on the specific http call that is failing, the panel that appears will tell you more about the error. In the example below (which only shows a 200 (success) response because I haven’t tried to create a timeout to get a picture), you can see that ‘server’ sending that response is listed (where I have the blue error). For instances with complex network setups, a 504 response could come from a load balancer (e.g. on Amazon AWS) or other proxy (nginx?) and seeing which server responded would guide you towards the right element in your system to look at.
If the error you see is 500, that indicates a problem in the Dataverse server itself and the next step would be to look in the /usr/local/glassfish4/glassfish/domains/domain1/logs/server.log file.
In later versions of Dataverse, any of these types of errors are caught and displayed in the UI, as in the image below:
This change just retrieves the same info that is available in the dev console.
Hope this helps,
-- Jim
--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
dataverse-commu...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/dataverse-community/e78c0ca8-8982-4495-a0ed-ff14c0472594o%40googlegroups.com.
Tumie Mosweu
client_max_body_size in the nginx.conf file and now I am able to upload files larger than 1MB! Thanks once again.To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
James Myers
Great! I’d like to turn the guidance below into a community google doc (or part of the guides?) so I’m glad you reported another potential failure mode we can add!
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/dataverse-community/e78c0ca8-8982-4495-a0ed-ff14c0472594o%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
dataverse-commu...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/dataverse-community/610dd4d3-4d73-43d3-b088-5acead0ead29o%40googlegroups.com.