Vectored I/O?

[Carl Hörberg]

We have a use case where we write multiple parts of a file from different strings/buffers. We think vectored I/O may be beneficial to us here and could increase the our IO speed (which is the bottleneck), but it seems like Crystal doesn't expose a way to perform writev or readv (https://linux.die.net/man/2/writev) through its standard lib. Would it be possible to add or is it something in the architecture that makes it difficult/unfeasible? I guess it would be pretty simple to add to LibC and then expose it in IO::FileDescriptor? I could try make a PR if you think it's a good idea? 

[Chris Hobbs]

I’ve thought about implementing this but it’s a bit far down my to-do list.

PRs are welcome, the interface would be to add a IO#write(*slices : Bytes) stub which would emulate writev using IO#write, then override it in IO::FileDescriptor to take advantage of writev. I thought about readv too but don’t think that’s worth doing for now since the usecase is less clear to me.

On 07/05/18 13:44, Carl Hörberg wrote:

We have a use case where we write multiple parts of a file from different strings/buffers. We think vectored I/O may be beneficial to us here and could increase the our IO speed (which is the bottleneck), but it seems like Crystal doesn't expose a way to perform writev or readv (https://linux.die.net/man/2/writev) through its standard lib. Would it be possible to add or is it something in the architecture that makes it difficult/unfeasible? I guess it would be pretty simple to add to LibC and then expose it in IO::FileDescriptor? I could try make a PR if you think it's a good idea? 

--
You received this message because you are subscribed to the Google Groups "Crystal" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crystal-lang...@googlegroups.com.
To post to this group, send email to crysta...@googlegroups.com.
Visit this group at https://groups.google.com/group/crystal-lang.
To view this discussion on the web visit https://groups.google.com/d/msgid/crystal-lang/b6f930c7-d30b-45c1-afe1-ef19a653d294%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

​

[Carl Hörberg]

ok! I'll give it a shot.

[Carl Hörberg]

Not IO#write(slices : Array(Bytes))? It has to be converted to a heap allocated array at some point anyway as the writev syscall takes a pointer to an array of "IoVec"s. (Can of course implement IO#write(*slices : Bytes) too but which in turn calls IO#write(slices : Array(Bytes)). It will also make "flush_on_newline" in IO#Buffered easier, if it's an array already.

[Chris Hobbs]

Yeah, you’re right, you should implement both.

*slices : Bytes might perform better if you can trick the compiler into generating the array of iovecs on the stack (hint: Tuple#map). For that matter, you could accept an Enumerable(Bytes) restriction, and call map on both: in the tuple case it should end up on the stack and in the array case it’ll end up on the heap. I might be wrong that you can magic it onto the stack though.

To view this discussion on the web visit https://groups.google.com/d/msgid/crystal-lang/e8459c31-bd40-495d-8632-11251d27fd8c%40googlegroups.com.

​