is endless streaming possible?

[Dennis Haupt]

a quick google search didn't find anything

what i want to do it this: (pseudocode)

val stream = ConnectToServer("http://somewhere:8080/get/some/huge/json/array")

stream.toValidJsonChunks.map(parseJson).process(doThisWithEachElement)

i want the data to be streamed to the client. all examples i found involve "bytestring" which (as far as i understood) always contains the entire http response, but this is exactly what i want to avoid. the response can be very big (several gb) and all the elements can be processed individually, so i want to load a few mb, process a batch, load the next batch, process it and so on.

is this possible?

[Richard Bradley]

Bytestring is just a byte array, not necessarily the whole response.

Akka on its default settings should do just what you want here, i.e. process the response in a streaming fashion.

Check out https://doc.akka.io/docs/alpakka/current/data-transformations/json.html#streaming-of-nested-structures for a streaming JSON parser