HTTP Chunk Size Exceeds the Configured Limit

[bord...@gmail.com]

Hello,

I have API response coming as a JSON string where Server won't support chunked response. I'm using below sample code to parse the JSON using Jackson parser.

type Result = Map[String, List[(String, JValue)]]

private def apiResult(resp : HttpResponse): Future[Result] = {
  Unmarshal(resp.entity.withoutSizeLimit).to[String] map {
    jsonResp =>
     getResultMap(jsonResp) // JSON parsing using Jackson
  }
}

I'm hitting API for for 10k different results. Some of the JSON results are very large, I'm finding below error ultimately (same error encounters even after removing withoutSizeLimit in response entity) :

[ERROR] [11/09/2017 11:33:58.337] [myactor-threadpool-dispatcher-7] [TaskInvocation] HTTP chunk size exceeds the configured limit of 1048576 bytes

akka.http.scaladsl.model.EntityStreamException: HTTP chunk size exceeds the configured limit of 1048576 bytes

at akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$2.applyOrElse(HttpMessageParser.scala:329)

at akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$2.applyOrElse(HttpMessageParser.scala:327)

at akka.stream.impl.fusing.Collect$$anon$2.$anonfun$wrappedPf$1(Ops.scala:212)

at akka.stream.impl.fusing.SupervisedGraphStageLogic.withSupervision(Ops.scala:173)

at akka.stream.impl.fusing.Collect$$anon$2.onPush(Ops.scala:214)

at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:499)

at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:462)

at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:368)

at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:571)

at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:457)

at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:546)

at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:728)

at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:743)

at akka.actor.Actor.aroundReceive(Actor.scala:517)

at akka.actor.Actor.aroundReceive$(Actor.scala:515)

at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:653)

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)

at akka.actor.ActorCell.invoke(ActorCell.scala:496)

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)

at akka.dispatch.Mailbox.run(Mailbox.scala:224)

at akka.dispatch.Mailbox.exec(Mailbox.scala:234)

at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

How can i solve my problem? 

Do you think, JSON Streaming can help here for me (Entity size might again be an issue as size is restricted to certain limit) ?

Regards,

Aditya Borde

[bord...@gmail.com]

I saw, sometimes, I get into below issue -

I use : where emptyResult of type : Map[String, List[(String, JValue)]]

Note that, Maps acc and curr are getting bigger and bigger. It might have some GB of data while combining at last step.

Sink.fold(Future{emptyResult}){
  case (acc, curr) => {
    acc zip curr map Function.tupled(_ ++ _)}

}

At ultimate step, it fails. These list has  

[Kyrylo Stokoz]

Chunk size and entity size are 2 different configuration settings.

You can easily set bigger chunk size for all connections in your application config see akka.http.parsing.max-chunk-size.

If you want to do it programmatically i think right now it is only possible via connection:

private final val requestResponseSink: Sink[HttpRequest, Future[HttpResponse]] = {
  val connectionSettings = ClientConnectionSettings(system)
  val parserSettings     = connectionSettings.parserSettings.withMaxChunkSize(100000000)

  Http().outgoingConnectionHttps(
    host = host,
    settings = connectionSettings.withParserSettings(parserSettings)
  ).toMat(Sink.head)(Keep.right)