Spray-can: Help understanding curl request time out with status code 504 GATEWAY_TIMEOUT

[Lakshmi]

Hi,

I would like some help with request timeouts. Please excuse the length of the post as I have included the relevant code.

I am using spray-can with spray-routing [v 1.3.3] to handle a mix of APIs that include normal routes + chunked requests for file uploads up to 100MB (using the DemoService and FileUploadHandler as a base idea). This works fine on local. I am basically keeping the 100 MB in memory until fully received before writing it to S3. [And this is fine for now because we will not be getting a lot of these type of large uploads].

When I deploy the service to a remote host and call curl using the same command, I am getting a 504 GATEWAY TIMEOUT error and the curl client closes prematurely. I also see the actor processing the file upload (FileUploadActor) has gone into limbo and the service itself becomes completely unresponsive. I am not seeing any logs or processing after that point. Up until that point, the FileUploadActor is obviously processing the chunks much slower than local. So I cannot reproduce this scenario on local.

I have tried setting the request-timeout and idle-timeout to 'infinite' for just this type of request in FileUploadActor.scala. This is the same as in the spray FileUploadHandler example, but I don't know if it is taking effect. 

- Is this problem external to spray, and something to do with my curl command?

- Do I also need to handle additional Timeout messages in the HttpRequestHandler and FileUploadActor. If so, what messages and what needs to be done when they are received? 

Details below:

curl output:

curl -vvv -X PUT -H "Content-Type: image/png" --upload-file '/Users/abc/Downloads/test.png' 'http://urlx/upload/resource/627514df'

* Hostname was NOT found in DNS cache

*   Trying 1.2.3.4...

* Connected to urlx (1.2.3.4) port 80 (#0)

> PUT /upload/resource/627514df HTTP/1.1

> User-Agent: curl/7.37.1

> Host: urlx

> Accept: */*

> Content-Type: image/png

> Content-Length: 99614720

> Expect: 100-continue

> 

< HTTP/1.1 100 Continue

< HTTP/1.1 504 GATEWAY_TIMEOUT

< Content-Length: 0

< Connection: keep-alive

* HTTP error before end of send, stop sending

< 

* Closing connection 0

$

Corresponding spray service logs:

2015-08-26 14:17:30.249 14:17:30.231UTC [A4Resource] INFO  c.l.a4resource.actor.FileUploadActor A4Resource-akka.actor.default-dispatcher-2 akka://A4Resource/user/http_request_handler/$a - Set request-timeout to Duration.Inf

2015-08-26 14:17:30.251 14:17:30.248UTC [A4Resource] WARN  s.can.server.HttpServerConnection A4Resource-akka.actor.default-dispatcher-2 akka://A4Resource/user/IO-HTTP/listener-0/13 - command pipeline: dropped CommandWrapper(SetIdleTimeout(Duration.Inf))

Relevant Code:

application.conf:

spray.can.server {
 request-timeout = 20 s
 idle-timeout = 60s
}

HttpRequestCustomHandler.scala:

class HttpRequestCustomHandler(routes: Route)
 extends HttpServiceActor
 with FileUploadService
 with ActorLogging
 with GlobalConfig {

 val normal = routes
 val chunked = chunkedRoute()

 val customReceive: Receive = {
 case _: Http.Connected =>
 sender ! Http.Register(self)

 case r: HttpRequest =>
 normal(RequestContext(r, sender(), r.uri.path).withDefaultSender(sender()))

 case s@ChunkedRequestStart(HttpRequest(PUT, path, _, _, _)) =>
 chunked(RequestContext(s.request, sender(), s.request.uri.path).withDefaultSender(sender()))

 case Timedout(HttpRequest(method, uri, _, _, _)) =>
 sender() ! HttpResponse(status = 500, entity = "The " + method + " request to '" + uri + "' has timed out...")
 
}

 override def receive: Receive = customReceive orElse super.runRoute(routes)
}

FileUploadHandler.scala:

class FileUploadActor(client: ActorRef, request: HttpRequest, ctx: RequestContext)
 extends Actor with ActorLogging with GlobalConfig {

 import request._

 var bytes: Array[Byte] = "".getBytes

 log.info(s"Set request-timeout to ${Duration.Inf}")

 client ! CommandWrapper(SetRequestTimeout(Duration.Inf))
 client ! CommandWrapper(SetIdleTimeout(Duration.Inf))

 def receive = {
   case c: MessageChunk =>
     bytes ++= c.data.toByteArray

   case e: ChunkedMessageEnd =>
     log.info(s"Got end of chunked request $method $uri.")

     Try(saveBinaryFile(bytes)) match {
       case Success(_) =>
         client ! HttpResponse(StatusCodes.Created, entity = "success")
       case Failure(f) =>
         f.printStackTrace(); ctx.complete(HttpResponse(StatusCodes.InternalServerError, entity = "failure"))
     }

     client ! CommandWrapper(SetRequestTimeout(SprayCanServerRequestTimeout)) // reset timeout to original value
     client ! CommandWrapper(SetRequestTimeout(SprayCanServerIdleTimeout)) // reset timeout to original value
     context.stop(self)

   case x => log.error(s"Unknown message: $x" )
 }
}

Thanks in advance for any help!

Lakshmi