Native messaging host message queue

[Asesh Shrestha]

Hello, I was going through the Chromium source code of how messages are read by Chromium when a native messaging host sends messages to extensions. I was really surprised when I found out that Chromium reads those messages in a 4 KB chunk, so that means if I send 500 KB data then there will be 500/4 read calls to read the whole data. So I wanted to know if I send multiple messages parallelly to Chromium from the same native messaging, then will those data be read parallely too or in a queue? We have a requirement that we send multiple types of data which could be in the range of 0-1 MB from the same native messaging host. So if those data are read in a queue then I see a bottleneck.

Thanks

[Sergey Ulanov]

Large messages are read sequentially in 4kB chunks. The API is not designed for high throughput (note that all messages have to be serialized in JSON). Still, I don't expect any issues if you just need to send ~1MB of data, unless you need to do that more than a few times per second. These reads take data from a local pipe, so they should be relatively inexpensive (e.g. compared to reading that data from the disk).

On Mon, Jul 27, 2020 at 9:00 PM Asesh Shrestha <asesh.s...@cloudfactory.com> wrote:

Hello, I was going through the Chromium source code of how messages are read by Chromium when a native messaging host sends messages to extensions. I was really surprised when I found out that Chromium reads those messages in a 4 KB chunk, so that means if I send 500 KB data then there will be 500/4 read calls to read the whole data. So I wanted to know if I send multiple messages parallelly to Chromium from the same native messaging, then will those data be read parallely too or in a queue? We have a requirement that we send multiple types of data which could be in the range of 0-1 MB from the same native messaging host. So if those data are read in a queue then I see a bottleneck.

Thanks

”This email and any files transmitted with it may be confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender.”

[Asesh Shrestha]

Thanks for replying but your answer doesn't answer the other question. For instance there's a method which will send a message to a specific native messaging host:

void send_message(const std::string& message) {...}

And there are  a couple of threads which will use that method to send data which could be in the range 0-1 MB and they could send those data at any time. Let's assume there are 5 threads which will parallely send message to a specific native messaging host at the same time:

Thread 1 calls send_message with 100 KB of data
Thread 2 calls send_message with 300 KB of data
Thread 3 calls send_message with 400 KB of data
Thread 4 calls send_message with 600 KB of data
Thread 5 calls send_message with 800 KB of data

Now will Chromium read read all of those data sequentially or parallely? I tried to find it out by debugging but I still can't reach a conclusion.

I have noticed there's a bug that's very hard to replicate when sending huge chunk of data parallely to native messaging host. Even though we have limited this JSON data (which includes binary data encoded in base64 so that we could send those data in JSON format) to 1 MB, Chromium will rarely report it was trying to send x GB of data, even though it's always less than 1 MB because of the default native message host limitation. I did try to debug that issue but have not been able to fix it as it rarely happens on some of our users' machine. This is what happens at that specific time: The contents of memory where 32-bit length of data is initially stored in Chromium will different at that time, than what was sent before sending the actually JSON serialized data. It gets overwritten, not sure why. So I am guessing it's most probably because of the above mentioned scenario?

On Tuesday, July 28, 2020 at 1:09:59 PM UTC+5:45, Sergey Ulanov wrote:

Large messages are read sequentially in 4kB chunks. The API is not designed for high throughput (note that all messages have to be serialized in JSON). Still, I don't expect any issues if you just need to send ~1MB of data, unless you need to do that more than a few times per second. These reads take data from a local pipe, so they should be relatively inexpensive (e.g. compared to reading that data from the disk).

[Manjunath E]

Chromium uses a single anonymous named pipe for reading(windows). So whatever data has been written by native messaging host will be read in the same sequence it has been written by the native messaging host. 
Your individual write call will write the data into the pipe completely in a single call. Only the read will be done in 4kb chunks and that shouldn't affect the order of read even if the write id one parallely from multiple threads.

[Asesh Shrestha]

Got it, thanks

[Sergey Ulanov]

It sounds like you have multiple threads writing messages to the same pipe. This is obviously a race condition. It's more likely to cause problems when sending messages larger than the size of the buffer used for the pipe because in that case each message needs to be split between multiple write() calls. If parts of different messages are interleaved then the receiver code on the Chrome's side won't be able to deserialize these messages correctly. It will try to interpret a wrong part of the stream as a message header. That yields invalid message size, which is consistent with the behaviour you see. To avoid this problem each message needs to be written all the way to the end before the first byte of the following message. The best way to achieve that is to write all messages from the same thread.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/5a3ab5e9-47ba-47bf-b90e-584392b7f637o%40chromium.org.

[Asesh Shrestha]

Just what I was guessing because we only came across this issue after we started sending bigger chunks of data from native messaging host and they are sent from multiple threads without any locks because those data should not be placed in a queue. Thanks a lot, that clears up everything :)