Artifact size limitation in GO 20.6.0?

[Chris Payne]

We upgraded to 20.6.0 from 19.12 over the weekend and we're now getting problems uploading large-ish artifacts (35MB+). This didn't happen before... seeing these errors in our logs:

[go] Artifact upload for file ####### (Size: 38003857) was denied by the server. This usually happens when server runs out of disk space.

[go] Failed to upload #####

Is there some new limit imposed? Is there any way to configure this? My searching on this turns up the problem reported with the Linux tmp directory, but I'm not sure how that applies here at all. I don't see any other settings in the config files or in the Tanuki wrapper that could tweak this. The OS didn't change, and we have plenty of storage space on the server (over 160GB). 

Thanks!

[Jason Smyth]

Hi Chris,

The last time someone brought up a space issue like this, it turned out that the server had plenty of space but it was allocated elsewhere on the file system.

Have you checked the server and confirmed that there are no volumes that are running out of space? I suggest looking specifically at the mount points that hold /tmp and GoCD's server directory.

Regards,

Jason

[Chris Payne]

Well we're on a Windows server. We have over 160GB free space on a 1TB drive. Everything is stored on the single C drive. As far as I can tell, everything looks fine on the server itself. But maybe there's something hidden I'm missing.

[Jason Smyth]

Hi Chris,

Are there any quotas in place on the file system that the GoCD might be hitting? Are there any error messages in the server logs when the artifact upload fails? Are there any errors in Event Log when the artifact upload fails?

Unless there is a quota in place, it sounds like you're right about there being plenty of disk space so maybe the logs will have more information that can point you in the right direction.

Regards,

Jason

[Chris Payne]

Unfortunately, the logs aren't giving me much more info. 

The agent log simply reports the same message in my original message, with a stacktrace:

java.lang.RuntimeException: Artifact upload for file #### (Size: 38003857) was denied by the server. This usually happens when server runs out of disk space..  HTTP return code is 413
at com.thoughtworks.go.util.ExceptionUtils.bomb(ExceptionUtils.java:29)
at com.thoughtworks.go.publishers.GoArtifactsManipulator.publish(GoArtifactsManipulator.java:102)
at com.thoughtworks.go.work.DefaultGoPublisher.upload(DefaultGoPublisher.java:68)
at com.thoughtworks.go.domain.ArtifactPlan.uploadArtifactFile(ArtifactPlan.java:169)
at com.thoughtworks.go.domain.ArtifactPlan.publishBuildArtifact(ArtifactPlan.java:143)
at com.thoughtworks.go.domain.ArtifactPlan.publishBuiltInArtifacts(ArtifactPlan.java:131)
at com.thoughtworks.go.remote.work.artifact.ArtifactsPublisher.publishArtifacts(ArtifactsPublisher.java:75)
at com.thoughtworks.go.remote.work.BuildWork.completeJob(BuildWork.java:197)
at com.thoughtworks.go.remote.work.BuildWork.build(BuildWork.java:129)
at com.thoughtworks.go.remote.work.BuildWork.doWork(BuildWork.java:79)
at com.thoughtworks.go.agent.JobRunner.run(JobRunner.java:53)
at com.thoughtworks.go.agent.AgentHTTPClientController.retrieveWork(AgentHTTPClientController.java:149)
at com.thoughtworks.go.agent.AgentHTTPClientController.work(AgentHTTPClientController.java:121)
at com.thoughtworks.go.agent.AgentController.loop(AgentController.java:86)
at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

The server log doesn't report anything of significance:

2020-08-10 15:46:20,983 INFO  [qtp536519735-422] BuildRepositoryRemoteImpl:99 - [Agent [XXX, XXX, 63e8d22e-b360-4b84-bcc5-847ec9fb55f4, 4b178349-8d56-43cb-8d25-dc048a4d84fa]] is reporting result [Passed] for [Build [XXX]]
2020-08-10 15:46:39,061 INFO  [qtp536519735-422] BuildRepositoryRemoteImpl:99 - [Agent [XXX, XXX, 63e8d22e-b360-4b84-bcc5-847ec9fb55f4, 4b178349-8d56-43cb-8d25-dc048a4d84fa]] is reporting status and result [Completed, Failed] for [Build [XXX]]
2020-08-10 15:46:39,061 INFO  [qtp536519735-422] Stage:234 - Stage is being completed by transition id: 209109

There's no quotas on the file system, and I'm not even sure of a way to do that in Windows. 

Windows Event logs on the GO Server and GO Agent computers show nothing at that time either. 

I assumed there was some undocumented setting that changed in a recent GO version that now limits artifact upload size. 

[Ketan Padegaonkar]

If you're fronting GoCD with a reverse proxy (apache/NGINX/some LB), usually they come with some values of an upload size limit.

To check if that's actually the case, you could try using the upload artifact API to upload a large file directly to the gocd server, bypassing the reverse proxy to eliminate if the root cause is the reverse proxy.

You could also enable GoCD's HTTP request logging to see if it is the gocd server rejecting these requests.

- Ketan

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/75f9852f-4f25-4b88-8140-59ff0fde7085o%40googlegroups.com.

[Chris Payne]

Oh man, I'm an idiot. You are 100% right. We added a reverse proxy this time to add TLS support that GO dropped. The limit was in there, adjusted it, and now we're all good. Sorry for wasting everyone's time.

On Tuesday, August 11, 2020 at 11:24:09 PM UTC-4, Ketan Padegaonkar wrote:

If you're fronting GoCD with a reverse proxy (apache/NGINX/some LB), usually they come with some values of an upload size limit.

To check if that's actually the case, you could try using the upload artifact API to upload a large file directly to the gocd server, bypassing the reverse proxy to eliminate if the root cause is the reverse proxy.

You could also enable GoCD's HTTP request logging to see if it is the gocd server rejecting these requests.

- Ketan

On Tue, Aug 11, 2020 at 8:35 PM Chris Payne <clp...@outlook.com> wrote:

We upgraded to 20.6.0 from 19.12 over the weekend and we're now getting problems uploading large-ish artifacts (35MB+). This didn't happen before... seeing these errors in our logs:

[go] Artifact upload for file ####### (Size: 38003857) was denied by the server. This usually happens when server runs out of disk space.

[go] Failed to upload #####

Is there some new limit imposed? Is there any way to configure this? My searching on this turns up the problem reported with the Linux tmp directory, but I'm not sure how that applies here at all. I don't see any other settings in the config files or in the Tanuki wrapper that could tweak this. The OS didn't change, and we have plenty of storage space on the server (over 160GB). 

Thanks!

--
You received this message because you are subscribed to the Google Groups "go-cd" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go...@googlegroups.com.