Using akka streams with a REST end point

[Bhashit Parikh]

Hey folks,

I am trying to send a response for a REST end-point in lift, and the data that I receive is an akka stream `Source[Byte, _]`.

I am not sure how I can send a streaming response back without blocking somewhere. For ex. if I use `OutputStreamResponse`, I will have to get the contents of the entire stream in memory and then write them to the OutputStream. If I use `StreamingResponse` class, the `read` calls would have to be blocking. Either way, I can't seem to be able to find a way to do it in a non-blocking manner. 

For ex. one of the (not completely fleshed out) idea was that if I could get access to the OutputStream for the response, I could just keep writing to it whenever I get some data from the the akka stream Source. This would still be blocking. But hopefully, I won't have to load everything in memory at once. Although, this is very similar to what we'd have to do with OutputStreamResponse. But I am hoping that I could avoid hogging the thread somehow while waiting for the data to arrive. I am not completely sure if that's even possible.

I also thought about using ServletOutputStream.html#setWriteListener, but I can't figure out an easy way to get access to the ServletOutputStream. And even if I get access to it, I don't know how useful it would be for async responses with akka streams.

Has anybody else encountered something like this before? If so, how did you handle this? Would it be possible to somehow leverage NIO to send the data in a non-blocking, async manner?

[Antonio Salazar Cardozo]

I actually don't think you can do this straight-up nonblocking in the sense of not tying up the response thread

while Akka is processing next steps. However, it is not true that you need to load all data in memory. You can

use the streaming response variants to pull data from Akka as the output stream can take it, and presumably

Akka will release this data's memory once you've written it.

It seems like the comet mechanism in its current state could probably be adapted to dealing with wake-up-then-

fall-back-asleep-to-wait-for-more-data-type interactions like this one, but it would require some work in Lift

proper.

Thanks,

Antonio

[Bhashit Parikh]

Thanks Antonio.

About not loading everything in memory: yeah, I mean, that would work with the OutputStreamResponse. I was just hoping not to block on it.

I can maybe use the PipedInputStream with the StreamingResponse to make sure not everything gets loaded in memory at once. The default converters for InputStreams load everything at once IIUC.

It seems like the comet mechanism in its current state could probably be adapted to dealing with wake-up-then-

fall-back-asleep-to-wait-for-more-data-type interactions like this one, but it would require some work in Lift

proper.

Interesting idea. I'll try to spend some brain cycles on it.

Thanks again.