PR validation failing due to apparent Artifactory repo problem

[Jason Zaugg]

For the past 24 hours, we've been seeing Scala PR validation failing with:

[artifact:deploy] Uploading: org/scala-lang/scala-library/2.12.0-362aa5c-SNAPSHOT/scala-library-2.12.0-362aa5c-20141110.055550-1.jar to repository pr-scala at http://private-repo.typesafe.com/typesafe/scala-pr-validation-snapshots/
[artifact:deploy] Transferring 5437K from pr-scala
[artifact:deploy] Unexpected end of file from server
[artifact:deploy] An error has occurred while processing the Maven artifact tasks.
[artifact:deploy]  Diagnosis:
[artifact:deploy] 
[artifact:deploy] Error deploying artifact 'org.scala-lang:scala-library:jar': Error deploying artifact: Error transferring file
[artifact:deploy] Unexpected end of file from server

I lodged a support ticket about 12 hours ago. It is the hands of an engineer at JFrog, but we are still awaiting a diagnosis / fix.

Thanks for you patience,

-jason

​

[Grzegorz Kossakowski]

Hi guys!

Here's the response we've got:

Following our investigation and after analyzing logs of the relevant components and testing the issue in our end, here is what we found:

This issue appeared after we have upgraded the Artifactory online Tomcat to a newer version that includes a few important security bug fixes.

It seems that the Maven Ant Task is not using preemptive authentication so it sends an unauthenticated PUT request and start streaming the content of the resource. Artifactory respond to the PUT request with 401 challenge that requires the client to provide credentials, however, the newer version of Tomcat has a feature that cut the connection of the upload when responding with 401 challenge to the client. It seems that the client intercept the response incorrectly and don't know how do deal with it, therefore due to the connection that been closed, the client report on 'Error deploying artifact: Error transferring file, Unexpected end of file from server' and not answer the challenge with a new request that includes the authentication.

From our tests, it seems that other clients other clients such as Maven and Ivy knows how to deal with it and have no similar issues.

If it is possible, we suggest to check the option of upgrade the Maven Ant Task to a newer version and see if it knows how to deal with this situation.

Another important direction is check if preemptive authentication can be enabled in Maven Ant Task can support  so the client will send the authentication with the first request instead sending unauthenticated request and waiting to a 401 challenge. If this is possible, it will probably be the best solution.

Looking at our Ant macros I think ultimately it's maven-gpg-plugin that is responsible for uploading artifacts:

 <artifact:mvn failonerror="true">

  <arg value="org.apache.maven.plugins:maven-gpg-plugin:1.3:sign-and-deploy-file" />

  <arg value="-Durl=${repo}" />

  <arg value="-DrepositoryId=${repository.credentials.id}" />

  <arg value="-DpomFile=${path}-pom-filtered.xml" />

  <arg value=   "-Dfile=${path}.jar" />

  <arg value="-Dsources=${path}-src.jar" />

  <arg value="-Djavadoc=${path}-docs.jar" />

  <arg value="-Pgpg" />

  <arg value="-Dgpg.useagent=true" />

</artifact:mvn>

I'm checking if upgrade to latest (1.5) version helps here: https://github.com/scala/scala/pull/4126

Any ideas/help is greatly appreciated.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Grzegorz Kossakowski

Scalac hacker at Typesafe

twitter: @gkossakowski

github: @gkossakowski

[Grzegorz Kossakowski]

Hi!

I wanted to give you a quick update on PR validation failures. PR validation stability has been restored. However, we are running a slimmer version of validation that runs only partest tests. Other parts of PR validation that depend on publishing artifacts to JFrog's repository has been temporarily disabled until we find a way to publish artifacts again. In particular, IDE validation has been disabled. We understand this is not an ideal situation and we are working on restoring full PR validation.

[Grzegorz Kossakowski]

Hi guys,

We are running regular PR validation again. The problem with uploading to Artifactory has been resolved by excellent JFrog support that deployed a workaround for us on their side. It took us a while to come to that resolution and I apologize everyone for the inconvenience. This incident shows the best it's a liability for us to keep using Ant. The world has moved on and so we should. Our answer to Ant-related pain is an sbt build.

Once JFrog deployed their workaround we found out that IDE validation broke. I immediately unplugged that component from the validation job to unblock people hacking on Scala. A few days we fixed IDE validation and it's plugged in again. This is going to be standard practice from now on. If some component of our infrastructure breaks, we'll be doing whatever is necessary to restore project's operations ASAP. This way we can investigate the root cause of the problem without pressure and without wasting other people's time.