Mojo DataPipe use cases and reasoning

[Xiaohan Wang (王消寒)]

Hello,

Currently we use mojo DataPipe to transfer compressed media (audio/video) data. This design dates back to the early days of mojo in Chromium and the reason is that DataPipe is more performant for transporting large data (e.g. 1090p video at 60fps). I checked the code base and we use DataPipe for media, network and storage, which all make sense.

But recently I got a question on exactly how performant DataPipe is compared to just sending data over MessagePipe, and I couldn't find any answer in existing documents in the code base. Basically I am looking for:

• Recommendations on when to use DataPipe vs MessagePipe
• Advantages of using DataPipe, e.g. performance numbers.

Is there any doc available on these topics? Thanks!

Best,

Xiaohan

[John Abd-El-Malek]

I don't have data unfortunately, but I know long ago we had timed this with the old IPC system. Both are based on pipes, so the performance would be similar. Sending > 100KB or so over pipes was slow, which is why we've always recommended using shared memory for large data.

--
You received this message because you are subscribed to the Google Groups "chromium-mojo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-moj...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/CAF1j9YPwyUbjh7U%3DC8zXAiJUKi_rj0oKZhmx%3DRZMkKXY6xCH%2BQ%40mail.gmail.com.

[Xiaohan Wang (王消寒)]

Thanks! Then what's the use case for DataPipe now, assuming we should just use MessagePipe for small data for simplicity, and use shared memory for large data for performance?

[John Abd-El-Malek]

Data pipes use shared memory; compared to using shared memory directly they provide convenience about messaging when data is available etc. Without them, each callsite of shared memory has to roll their own parallel message pipe.

[Ken Rockot]

Data pipes are for streaming data that either isn't completely available when consumption starts (large disk or network transfers), or when data is large enough that holding it so in memory at once might needlessly waste resources.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/CAF1j9YNmzKUpeq93hj%3DLwWbw1r5oqX6vbjtDZTx%2BzSMjpn_tig%40mail.gmail.com.

[Ken Rockot]

And to clarify, as John says, data pipes do also use shared memory. Rather than being one big buffer to hold all the data, it's a configurable fixed-size ring buffer shared by producer and consumer.

Performance will vary widely depending on usage and environment, but in general the performance advantage comes in the form of fewer memory allocations and fewer copies throughout the transfer process.

[Xiaohan Wang (王消寒)]

Ah, good to know that data pipes also use shared memory. That's the point I was missing. Thank you all for the clarification!