Error "Could not write response part ChunkedMessageEnd(,List())" when serving static content

[Alexander Semenov]

Hello all.

I get this error while serving one of my static files:

"Could not write response part ChunkedMessageEnd(,List()), aborting connection" (for each request for that file)

From the browser side it looks like 'CONNECTION RESET' from the server.

So, some machines (~5%) can't download that file (URL like http://10.99.22.88:8080/js/libs/moment/moment-with-langs-2.7.0.min.js) but most of then can. I don't use server caching.
No idea what do do with that. I use latest Spray on Akka 2.3 and Scala 2.11.1.

Any help is very appreciated.

[Alexander Semenov]

As a temporary workaround I set

spray.routing {

  file-chunking-threshold-size = 256k

}

moment-with-langs-2.7.0.min.js is 136kb which is more than default Spray's 128kb. So, with disabled chunking the problem resolves (for a while).

[Johannes Rudolph]

Hi Alexander,

we'd need at least some clue how to reproduce this issue, better would
be a project we could clone to look into it.

Thanks,
Johannes

> --
> You received this message because you are subscribed to the Google Groups
> "spray.io User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spray-user+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/spray-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/spray-user/32da16c6-34f9-416c-a9dd-4f71e3c74877%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[Alexander Semenov]

Hi, Johannes.

I can share my project: https://bitbucket.org/artezio/art-team branch 'spray-problem'.

Just run ArtTeamApp in app module (sbt "project app" run) having mongo db running on localhost. 5-10% of clients can't open http://localhost:8080/js/libs/moment/moment-with-langs-2.7.0.min.js due to this chunking problem.

Not sure if it will be helpful.

Regards,
Alexander.

[Alexander Semenov]

I see that this problem is reproduced more frequently on clients with slower connection.

[Alexander Semenov]

Hi, Johannes.

Will it be helpful if I create a small subset of my application (just Spray serving static resource)?

[Alexander Semenov]

Interesting that this problem can only be reproduced when server runs on a Windows machine. Perhaps, I will spent some time debugging manually, though I'm not an akka expert...

[Johannes Rudolph]

Hi Alexander,

sorry, for not yet looking into it (though, I wouldn't have had a
Windows machine at hand anyways...).

You could try setting

akka.loglevel = DEBUG

and

akka.io.tcp.trace-logging = on

This will produce lots of debugging output so make sure to minimize
the application running. Maybe we can then see what's going on.

Johannes

[Erdem Agaoglu]

Hi,

I don't know if it's OK to revive threads, but i can reproduce this.

A simple routing application like:

path("index.html") {

  get {

    getFromFile("index.html")

  }

}

Or (in my situation) a bare spray-can app:

case _: HttpRequest =>

  sender ! HttpResponse(200, HttpEntity(HttpData(new java.io.File("index.html"))))

fails with

Could not write response part HttpResponse(200 OK,HttpEntity(application/octet-stream,...

for about 1 in 10000 requests. I can create the situation using wrk like:

wrk -c 1 -d 1s -t 1 http://127.0.0.1:8080/index.html

which opens 1 connection and bombards it with requests for 1 second. When the problem happens, report shows socket read errors and the application logs the given line.

I enabled trace-logging and akka-io said "Dropping write because queue is full". I tried changing pipelining-limit, file-io-dispatcher and jvm version. While the latter two didn't make any difference, incresing pipelining-limit made things worse, causing exceptions like:

java.lang.IllegalStateException

at spray.can.server.OpenRequestComponent$EmptyOpenRequest$.nextIfNoAcksPending(OpenRequest.scala:240)

at spray.can.server.OpenRequestComponent$EmptyOpenRequest$.nextIfNoAcksPending(OpenRequest.scala:232)

at spray.can.server.ServerFrontend$$anon$2$$anon$1$$anonfun$1.apply(ServerFrontend.scala:77)

at spray.can.server.ServerFrontend$$anon$2$$anon$1$$anonfun$1.apply(ServerFrontend.scala:60)

at spray.can.server.ServerFrontend$$anon$2$$anon$1$$anonfun$1.apply(ServerFrontend.scala:62)

at spray.can.server.ServerFrontend$$anon$2$$anon$1$$anonfun$1.apply(ServerFrontend.scala:60)

With further debugging, my best guess is the problem lies in TcpConnection.PendingWriteFile [1] where after transferring file to socket, it acks first and updates pendingWrite status later. I'll try some changes around there but it's better to get an expert opinion before diving in.

BTW i'm on spray-can 1.3.1, akka 2.3.6, archlinux kernel 3.16.3 using jvm 7u65 and 6u51

[1]: https://github.com/akka/akka/blob/master/akka-actor/src/main/scala/akka/io/TcpConnection.scala#L430

[Johannes Rudolph]

Hi Erdem,

do you also get the ChunkedMessageEnd error from the OP or just what you show?

Johannes

> https://groups.google.com/d/msgid/spray-user/878cde0b-1e74-4178-8a3f-793489c4dae2%40googlegroups.com.

[Johannes Rudolph]

Hi Erdem,

On Fri, Sep 26, 2014 at 11:05 AM, Erdem Agaoglu <erdem....@gmail.com> wrote:

> With further debugging, my best guess is the problem lies in
> TcpConnection.PendingWriteFile [1] where after transferring file to socket,
> it acks first and updates pendingWrite status later. I'll try some changes
> around there but it's better to get an expert opinion before diving in.

Yes, thanks, that's indeed a likely problem. I filed a ticket:

https://github.com/akka/akka/issues/15991

[Erdem Agaoglu]

Hi Johannes,

I haven't tried chunked responses till now, but using a ~600kB file for getFromFile(), things seem OK. I didn't try to craft a chunked response using spray-can directly. The problem i mentioned occurs on small files like ~2kB. I guess it's not the same situation Alexander says.

BTW, i know it's probably not a viable one but just to see where it goes, i tried cut/pasting the ack line after pendingWrite-update and release in TcpConnection and it solves the issue.

Thanks for your interest,

Cheers,

[Johannes Rudolph]

Hi Erdem,

On Mon, Sep 29, 2014 at 11:33 AM, Erdem Agaoglu <erdem....@gmail.com> wrote:
> BTW, i know it's probably not a viable one but just to see where it goes, i
> tried cut/pasting the ack line after pendingWrite-update and release in
> TcpConnection and it solves the issue.
> Thanks for your interest,

Yes, while it makes sense that it solves the issue in some cases, Akka
doesn't make the kind of message ordering guarantee needed to prevent
this situation in every context.

I tried a safer approach here:

https://github.com/akka/akka/pull/15992

Maybe you could test this one as well.

In any case, thanks for the report!

[Erdem Agaoglu]

Hi Johannes,

I tried the patch and it works as expected.

Thanks again,

Cheers,

[Matt Hughes]

What version of Akka was this fixed in?  Best I can tell from the Akka ticket, it was 2.3.7.  I'm still getting it with Akka 2.3.7 and Spray 1.3.1.  Increasing the chunked threshold (aka turning it off) does avoid the problem.

[Johannes Rudolph]

On Thu, Feb 12, 2015 at 2:50 AM, Matt Hughes <hughe...@gmail.com> wrote:
> What version of Akka was this fixed in? Best I can tell from the Akka
> ticket, it was 2.3.7. I'm still getting it with Akka 2.3.7 and Spray 1.3.1.
> Increasing the chunked threshold (aka turning it off) does avoid the
> problem.

The fix was about the WriteFile command. Do you have this problem with
sending files?

[Matt Hughes]

Yes, sending a 242KB file. I have spray routing configured to not chunk this sized file:

spray.routing {
  file-chunking-threshold-size = 512k
}

yet getFromFile still attempts chunking and I get the error mentioned in this thread.

Strangely, getFromResource correctly observes this setting and does not even attempt to chunk it. Looking at the two methods, I don’t see any glaring differences:

  def getFromFile(file: File, contentType: ContentType)
                 (implicit settings: RoutingSettings, refFactory: ActorRefFactory): Route =
    get {
      detach() {
        if (file.isFile && file.canRead) {
          autoChunked.apply {
            conditionalFor(file.length, file.lastModified).apply {
              withRangeSupport() {
                complete(HttpEntity(contentType, HttpData(file)))
              }
            }
          }
        } else reject
      }
    }
   
  def getFromResource(resourceName: String, contentType: ContentType)
                     (implicit refFactory: ActorRefFactory): Route =
    if (!resourceName.endsWith("/"))
      get {
        detach() {
          val theClassLoader = actorSystem(refFactory).dynamicAccess.classLoader
          theClassLoader.getResource(resourceName) match {
            case null ⇒ reject
            case url ⇒
              val (length, lastModified) = {
                val conn = url.openConnection()
                try {
                  conn.setUseCaches(false) // otherwise the JDK will keep the JAR file open when we close!
                  val len = conn.getContentLength
                  val lm = conn.getLastModified
                  len -> lm
                } finally { conn.getInputStream.close() }
              }
              implicit val bufferMarshaller = BasicMarshallers.byteArrayMarshaller(contentType)
              autoChunked.apply { // TODO: add implicit RoutingSettings to method and use here
                conditionalFor(length, lastModified).apply {
                  withRangeSupport() {
                    complete {
                      // readAllBytes closes the InputStream when done, which ends up closing the JAR file
                      // if caching has been disabled on the connection
                      val is = url.openStream()
                      try { FileUtils.readAllBytes(is) }
                      finally { is.close() }
                    }
                  }
                }
              }
          }
        }
      }
    else reject // don't serve the content of resource "directories"

--
You received this message because you are subscribed to a topic in the Google Groups "spray.io User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spray-user/ktEUtjs9Nks/unsubscribe.
To unsubscribe from this group and all its topics, send an email to spray-user+...@googlegroups.com.

Visit this group at http://groups.google.com/group/spray-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/spray-user/CAKF7HnfuXa8Bv%2B4nbR20JfFHLHgDK3_p1R0rt_J%3DMZ7%3DYrwXQQ%40mail.gmail.com.

[Matt Hughes]

So the odd thing is, this only seems broken over SSL.  Works fine when serving the same resource up over non-ssl.  Any ideas as to what might cause that?

To unsubscribe from this group and all its topics, send an email to spray-user+unsubscribe@googlegroups.com.

[Leon]

Hi Matt

We have meet the same situation, non-ssl is fine, but SSL is broken

Do you have any solution?

在 2015年3月23日星期一 UTC+8下午10:57:57，Matt Hughes写道：

To unsubscribe from this group and all its topics, send an email to spray-user+...@googlegroups.com.