Error "Could not write response part ChunkedMessageEnd(,List())" when serving static content

528 views
Skip to first unread message

Alexander Semenov

unread,
Jul 11, 2014, 5:11:40 AM7/11/14
to spray...@googlegroups.com
Hello all.

I get this error while serving one of my static files:

"Could not write response part ChunkedMessageEnd(,List()), aborting connection" (for each request for that file)

From the browser side it looks like 'CONNECTION RESET' from the server.

So, some machines (~5%) can't download that file (URL like http://10.99.22.88:8080/js/libs/moment/moment-with-langs-2.7.0.min.js) but most of then can. I don't use server caching.
No idea what do do with that. I use latest Spray on Akka 2.3 and Scala 2.11.1.

Any help is very appreciated.

Alexander Semenov

unread,
Jul 11, 2014, 6:03:24 AM7/11/14
to spray...@googlegroups.com
As a temporary workaround I set

spray.routing {
  file-chunking-threshold-size = 256k
}

moment-with-langs-2.7.0.min.js is 136kb which is more than default Spray's 128kb. So, with disabled chunking the problem resolves (for a while).

Johannes Rudolph

unread,
Jul 11, 2014, 6:06:02 AM7/11/14
to spray...@googlegroups.com
Hi Alexander,

we'd need at least some clue how to reproduce this issue, better would
be a project we could clone to look into it.

Thanks,
Johannes
> --
> You received this message because you are subscribed to the Google Groups
> "spray.io User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spray-user+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/spray-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/spray-user/32da16c6-34f9-416c-a9dd-4f71e3c74877%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

Alexander Semenov

unread,
Jul 11, 2014, 6:34:31 AM7/11/14
to spray...@googlegroups.com, johannes...@googlemail.com
Hi, Johannes.

I can share my project: https://bitbucket.org/artezio/art-team branch 'spray-problem'.

Just run ArtTeamApp in app module (sbt "project app" run) having mongo db running on localhost. 5-10% of clients can't open http://localhost:8080/js/libs/moment/moment-with-langs-2.7.0.min.js due to this chunking problem.

Not sure if it will be helpful.

Regards,
Alexander.

Alexander Semenov

unread,
Jul 11, 2014, 6:46:36 AM7/11/14
to spray...@googlegroups.com, johannes...@googlemail.com
I see that this problem is reproduced more frequently on clients with slower connection.

Alexander Semenov

unread,
Jul 12, 2014, 6:19:44 AM7/12/14
to spray...@googlegroups.com, johannes...@googlemail.com
Hi, Johannes.

Will it be helpful if I create a small subset of my application (just Spray serving static resource)?

Alexander Semenov

unread,
Jul 14, 2014, 5:10:49 PM7/14/14
to spray...@googlegroups.com, johannes...@googlemail.com
Interesting that this problem can only be reproduced when server runs on a Windows machine. Perhaps, I will spent some time debugging manually, though I'm not an akka expert...

Johannes Rudolph

unread,
Jul 15, 2014, 4:39:37 AM7/15/14
to Alexander Semenov, spray...@googlegroups.com
Hi Alexander,

sorry, for not yet looking into it (though, I wouldn't have had a
Windows machine at hand anyways...).

You could try setting

akka.loglevel = DEBUG

and

akka.io.tcp.trace-logging = on

This will produce lots of debugging output so make sure to minimize
the application running. Maybe we can then see what's going on.

Johannes

Erdem Agaoglu

unread,
Sep 26, 2014, 5:05:56 AM9/26/14
to spray...@googlegroups.com
Hi,

I don't know if it's OK to revive threads, but i can reproduce this.
A simple routing application like:

path("index.html") {
  get {
    getFromFile("index.html")
  }
}

Or (in my situation) a bare spray-can app:

case _: HttpRequest =>
  sender ! HttpResponse(200, HttpEntity(HttpData(new java.io.File("index.html"))))

fails with
Could not write response part HttpResponse(200 OK,HttpEntity(application/octet-stream,...

for about 1 in 10000 requests. I can create the situation using wrk like:


which opens 1 connection and bombards it with requests for 1 second. When the problem happens, report shows socket read errors and the application logs the given line.
I enabled trace-logging and akka-io said "Dropping write because queue is full". I tried changing pipelining-limit, file-io-dispatcher and jvm version. While the latter two didn't make any difference, incresing pipelining-limit made things worse, causing exceptions like:
java.lang.IllegalStateException
at spray.can.server.OpenRequestComponent$EmptyOpenRequest$.nextIfNoAcksPending(OpenRequest.scala:240)
at spray.can.server.OpenRequestComponent$EmptyOpenRequest$.nextIfNoAcksPending(OpenRequest.scala:232)
at spray.can.server.ServerFrontend$$anon$2$$anon$1$$anonfun$1.apply(ServerFrontend.scala:77)
at spray.can.server.ServerFrontend$$anon$2$$anon$1$$anonfun$1.apply(ServerFrontend.scala:60)
at spray.can.server.ServerFrontend$$anon$2$$anon$1$$anonfun$1.apply(ServerFrontend.scala:62)
at spray.can.server.ServerFrontend$$anon$2$$anon$1$$anonfun$1.apply(ServerFrontend.scala:60)

With further debugging, my best guess is the problem lies in TcpConnection.PendingWriteFile [1] where after transferring file to socket, it acks first and updates pendingWrite status later. I'll try some changes around there but it's better to get an expert opinion before diving in.

BTW i'm on spray-can 1.3.1, akka 2.3.6, archlinux kernel 3.16.3 using jvm 7u65 and 6u51

Johannes Rudolph

unread,
Sep 29, 2014, 4:52:59 AM9/29/14
to spray...@googlegroups.com
Hi Erdem,

do you also get the ChunkedMessageEnd error from the OP or just what you show?

Johannes
> https://groups.google.com/d/msgid/spray-user/878cde0b-1e74-4178-8a3f-793489c4dae2%40googlegroups.com.

Johannes Rudolph

unread,
Sep 29, 2014, 4:57:58 AM9/29/14
to spray...@googlegroups.com
Hi Erdem,

On Fri, Sep 26, 2014 at 11:05 AM, Erdem Agaoglu <erdem....@gmail.com> wrote:
> With further debugging, my best guess is the problem lies in
> TcpConnection.PendingWriteFile [1] where after transferring file to socket,
> it acks first and updates pendingWrite status later. I'll try some changes
> around there but it's better to get an expert opinion before diving in.

Yes, thanks, that's indeed a likely problem. I filed a ticket:

https://github.com/akka/akka/issues/15991

Erdem Agaoglu

unread,
Sep 29, 2014, 5:33:05 AM9/29/14
to spray...@googlegroups.com
Hi Johannes,

I haven't tried chunked responses till now, but using a ~600kB file for getFromFile(), things seem OK. I didn't try to craft a chunked response using spray-can directly. The problem i mentioned occurs on small files like ~2kB. I guess it's not the same situation Alexander says.
BTW, i know it's probably not a viable one but just to see where it goes, i tried cut/pasting the ack line after pendingWrite-update and release in TcpConnection and it solves the issue.
Thanks for your interest,

Cheers,

Johannes Rudolph

unread,
Sep 29, 2014, 5:38:02 AM9/29/14
to spray...@googlegroups.com
Hi Erdem,

On Mon, Sep 29, 2014 at 11:33 AM, Erdem Agaoglu <erdem....@gmail.com> wrote:
> BTW, i know it's probably not a viable one but just to see where it goes, i
> tried cut/pasting the ack line after pendingWrite-update and release in
> TcpConnection and it solves the issue.
> Thanks for your interest,

Yes, while it makes sense that it solves the issue in some cases, Akka
doesn't make the kind of message ordering guarantee needed to prevent
this situation in every context.

I tried a safer approach here:

https://github.com/akka/akka/pull/15992

Maybe you could test this one as well.

In any case, thanks for the report!

Erdem Agaoglu

unread,
Sep 29, 2014, 5:58:42 AM9/29/14
to spray...@googlegroups.com
Hi Johannes,

I tried the patch and it works as expected.
Thanks again,

Cheers,

Matt Hughes

unread,
Feb 11, 2015, 8:50:05 PM2/11/15
to spray...@googlegroups.com
What version of Akka was this fixed in?  Best I can tell from the Akka ticket, it was 2.3.7.  I'm still getting it with Akka 2.3.7 and Spray 1.3.1.  Increasing the chunked threshold (aka turning it off) does avoid the problem.

Johannes Rudolph

unread,
Feb 12, 2015, 3:52:25 AM2/12/15
to spray...@googlegroups.com
On Thu, Feb 12, 2015 at 2:50 AM, Matt Hughes <hughe...@gmail.com> wrote:
> What version of Akka was this fixed in? Best I can tell from the Akka
> ticket, it was 2.3.7. I'm still getting it with Akka 2.3.7 and Spray 1.3.1.
> Increasing the chunked threshold (aka turning it off) does avoid the
> problem.

The fix was about the WriteFile command. Do you have this problem with
sending files?

Matt Hughes

unread,
Feb 12, 2015, 9:17:04 AM2/12/15
to Johannes Rudolph, spray...@googlegroups.com

Yes, sending a 242KB file. I have spray routing configured to not chunk this sized file:

spray.routing {
  file-chunking-threshold-size = 512k
}

yet getFromFile still attempts chunking and I get the error mentioned in this thread.

Strangely, getFromResource correctly observes this setting and does not even attempt to chunk it. Looking at the two methods, I don’t see any glaring differences:

  def getFromFile(file: File, contentType: ContentType)
                 (implicit settings: RoutingSettings, refFactory: ActorRefFactory): Route =
    get {
      detach() {
        if (file.isFile && file.canRead) {
          autoChunked.apply {
            conditionalFor(file.length, file.lastModified).apply {
              withRangeSupport() {
                complete(HttpEntity(contentType, HttpData(file)))
              }
            }
          }
        } else reject
      }
    }
   
  def getFromResource(resourceName: String, contentType: ContentType)
                     (implicit refFactory: ActorRefFactory): Route =
    if (!resourceName.endsWith("/"))
      get {
        detach() {
          val theClassLoader = actorSystem(refFactory).dynamicAccess.classLoader
          theClassLoader.getResource(resourceName) match {
            case null ⇒ reject
            case url ⇒
              val (length, lastModified) = {
                val conn = url.openConnection()
                try {
                  conn.setUseCaches(false) // otherwise the JDK will keep the JAR file open when we close!
                  val len = conn.getContentLength
                  val lm = conn.getLastModified
                  len -> lm
                } finally { conn.getInputStream.close() }
              }
              implicit val bufferMarshaller = BasicMarshallers.byteArrayMarshaller(contentType)
              autoChunked.apply { // TODO: add implicit RoutingSettings to method and use here
                conditionalFor(length, lastModified).apply {
                  withRangeSupport() {
                    complete {
                      // readAllBytes closes the InputStream when done, which ends up closing the JAR file
                      // if caching has been disabled on the connection
                      val is = url.openStream()
                      try { FileUtils.readAllBytes(is) }
                      finally { is.close() }
                    }
                  }
                }
              }
          }
        }
      }
    else reject // don't serve the content of resource "directories"

--
You received this message because you are subscribed to a topic in the Google Groups "spray.io User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spray-user/ktEUtjs9Nks/unsubscribe.
To unsubscribe from this group and all its topics, send an email to spray-user+...@googlegroups.com.

Matt Hughes

unread,
Mar 23, 2015, 10:57:57 AM3/23/15
to spray...@googlegroups.com, johannes...@googlemail.com
So the odd thing is, this only seems broken over SSL.  Works fine when serving the same resource up over non-ssl.  Any ideas as to what might cause that?
To unsubscribe from this group and all its topics, send an email to spray-user+unsubscribe@googlegroups.com.

Leon

unread,
Dec 7, 2016, 3:24:36 AM12/7/16
to spray.io User List, johannes...@googlemail.com
Hi Matt

We have meet the same situation, non-ssl is fine, but SSL is broken
Do you have any solution?

在 2015年3月23日星期一 UTC+8下午10:57:57,Matt Hughes写道:
To unsubscribe from this group and all its topics, send an email to spray-user+...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages