[Proposal] Make `StringIO` accept stream as an input

[Łukasz Niemier]

The reason for this is simple. There are some functionalities in Erlang that do not take Elixir's Streams into consideration (no surprise there), but it still would be useful to utilise them. In my particular case I am fetching file from S3 using ExAWS, which streams file content. That particular file is a Gzipped TAR, which I want to pass to `erl_tar` module. However the only way to do it right now is either create the file on disk or to fetch whole file into memory. But `erl_tar` accepts IO server as a source for the TAR file, which mean that if there would be a way to transform `Stream` into `StringIO` I would have a stream that could pass there and achieve streaming TAR decompression and unpacking.

[José Valim]

That sounds reasonable and I would be curious to see how a proof of concept of this would look like in practice. Given we can suspend streams, we should be able to consume/emit items one by one. This is similar how to GenStage converts streams into stages.

José Valim

https://dashbit.co/

On Fri, Feb 7, 2025 at 11:58 Łukasz Niemier <luk...@niemier.pl> wrote:

The reason for this is simple. There are some functionalities in Erlang that do not take Elixir's Streams into consideration (no surprise there), but it still would be useful to utilise them. In my particular case I am fetching file from S3 using ExAWS, which streams file content. That particular file is a Gzipped TAR, which I want to pass to `erl_tar` module. However the only way to do it right now is either create the file on disk or to fetch whole file into memory. But `erl_tar` accepts IO server as a source for the TAR file, which mean that if there would be a way to transform `Stream` into `StringIO` I would have a stream that could pass there and achieve streaming TAR decompression and unpacking.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/elixir-lang-core/4b1d51a2-336d-48d3-ba8d-f21555b2fb0dn%40googlegroups.com.

[Łukasz Niemier]

> That sounds reasonable and I would be curious to see how a proof of concept of this would look like in practice. Given we can suspend streams, we should be able to consume/emit items one by one. This is similar how to GenStage converts streams into stages.

Can we? I haven't seen such functionality in `Stream` module.

--

Łukasz Niemier
luk...@niemier.pl

[José Valim]

Look at how zip is implemented. We don't expose it directly because any suspended stream/enumerable needs to be eventually terminated, so it is easy to mess things up, but if we encapsulate it in a process, we can guarantee we do so. You can also look at GenStage stream integration.

José Valim

https://dashbit.co/

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/elixir-lang-core/6D3F3D69-61DF-40BF-8ABB-2D4C2B928D63%40niemier.pl.