Akka HTTP usage of HttpEntity.toStrict

[Yannick Lazzari]

Hi,

There is an opened ticket in akka-http regarding exceptions being thrown after using HttpEntity.toStrict: https://github.com/akka/akka-http/issues/1177.

My question is regarding Johannes Rudolph's comment about it being a workaround:

Using toStrictEntity is basically only a workaround and not a solution because it will mean that all entity data will be buffered in memory which means that your server will need more memory per concurrent request than necessary.

Let's say you have the need to log the complete request payload of all requests coming in to a service. If you want to allow the logging logic to fully consume to request payload and still allow the entity to be unmarshalled to the desired object later on, what would be another way to ensure this never fails other than using toStrict, forcing the entire payload into memory and allowing it to be consumed more than once?

For example, consider the following route:

val route = extractExecutionContext & extractMaterializer & extractLog { (ec, mat, log) =>

  logRequest(LoggingMagnet(_ => logRequestDetails(_)(ec, mat, log))) {

    path("someapi") {

      post {

        entity(as[SomeInputClass]) { input =>

          // do something with input

        }

      }

    } 

  }

}

def logRequestDetails(req: HttpRequest)(implicit ec: ExecutionContext, materializer: Materializer, log: LoggingAdapter): Unit =

  Unmarshal(req.entity).to[String].map(b => s"The request body is $b").onComplete {

    case Success(s) => log.info(s)

    case Failure(e) => log.error(e, "Error")

  }

This is obviously a contrived example, but this logs the request body. It works, but we've seen such a pattern inconsistently yield java.lang.IllegalStateException: Substream Source cannot be materialized more than once errors. Isn't this a case where converting to a strict entity the correct way to address such problems, i.e. allow the request entity to be consumed more than once? If not, what would be the correct way?

Thanks for any insight you might have!

Yannick

[johannes...@lightbend.com]

Hi Yannick,

if you want to log the complete request contents, then there is no other way than to collect anything into memory (actually, that's a consequence of logging, not of the API).

In that case, you can use toStrict method or the toStrictEntity directive at the root of your routing tree to make sure you load the whole contents into memory and use it from different places.

Johannes

[Yannick Lazzari]

Thanks for confirming. It's just that with the way you said it on the ticket, calling the usage of toStrict a "workaround", you made it sound like even if toStrict was in the API, all its uses were kind of wrong and workarounds for a "proper" and "streaming" way to consume the entity body more than once. Then I started wondering if you hadn't come up with a better alternative. Couldn't figure out how without loading all of the entity content into memory, but who knows what tricks you could have come up with!

Thanks for the reply!

[Roland Kuhn]

Hi Yannik,

if your logging facility were capable of streaming then you could also avoid the double materialization issue using the .alsoTo combinator: you send the bytes to the logger while processing them in your business logic. This is also beneficial if the logger compacts the data, reducing peak memory usage.

Regards,

Roland 

Sent from my iPhone

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.