Native messaging host termination signals by Chromium

[Asesh Shrestha]

Hi, few years back I had posted this question: https://groups.google.com/a/chromium.org/g/chromium-dev/c/e1p_FvRpYtY/m/PXvo_hDgBAAJ 

Even though we are sending messages to extensions from our native messaging host sequentially, our native messaging host is still being terminated randomly on a few machines. We are using our own Chromium fork, while we can't replicate this issue on our machines but few users are affected by this bug randomly almost every day which is giving us a big problem right now. 

To mitigate this issue, I have modified our fork to ignore messages greater than 1024*1024 from our specific extension only as shown below:

ProcessIncomingData method from src\chrome\browser\extensions\api\messaging\native_message_process_host.cc:

   if (message_size > kMaximumNativeMessageSize) {
     // Don't send exit signal to WM, as this bug could have bee
     // by a bug in Chromium: https://cloudfactory.atlassian.net
     if (source_extension_id_ .compare("ddmbapcgpfbnlhildjojbgcg
       // Clear the buffer to prevent infinite loop and discard
       incoming_data_.clear();
       LOG(WARNING) <<
         "Exit signal bug triggered in the WSB. This message wil
       return;
     }

     LOG(ERROR) << "Native Messaging host tried sending a messag
                << message_size << " bytes long.";
     Close(kHostInputOutputError);
     return;
   }

   if (incoming_data_.size() < message_size + kMessageHeaderSize
     return;

   client_->PostMessageFromNativeHost(
       incoming_data_.substr(kMessageHeaderSize, message_size));

in our test which we managed to do so by explicitly sending messages greater than 1 MB, it won't be terminated. 

But our native messaging host won't send messages greater than 1 MB at all as we have designed it not to do so. Despite that it's being terminated stating that it was trying to send x length data where x is a random integer which specifies it's length, which we managed to find out by checking the log of one of our users.

So is there any other places where this termination happens besides shown above? I have been trying to find haven't been able to find till now.

Thanks

[Asesh Shrestha]

looks like our code wasn't posted properly, so here's the code:

    size_t message_size =
        *reinterpret_cast<const uint32_t*>(incoming_data_.data());

    if (message_size > kMaximumNativeMessageSize) {
      // Don't send exit signal to WM, as this bug could have been triggered
      // by a bug in Chromium: https://cloudfactory.atlassian.net/browse/PRODDEV-2002
      if (source_extension_id_ .compare("ddmbapcgpfbnlhildjojbgcgcefnindd") == 0) {
        // Clear the buffer to prevent infinite loop and discard this data
        incoming_data_.clear();
        LOG(WARNING) <<
          "Exit signal bug triggered in the WSB. This message will be discarded";
        return;
      }

      LOG(ERROR) << "Native Messaging host tried sending a message that is "

                 << message_size << " bytes long.";
      Close(kHostInputOutputError);
      return;
    }

    if (incoming_data_.size() < message_size + kMessageHeaderSize)

      return;

    client_->PostMessageFromNativeHost(
        incoming_data_.substr(kMessageHeaderSize, message_size));

[Sergey Ulanov]

Hi Asesh,

AFAICT from the that old thread in 2020 the issue was that the host was sending messages to chrome from multiple threads at the same time. This won't work, particularly for messages that are larger than the pipe buffer allocated by the OS. When multiple threads try to write to stdout chunks from different messages may be in interleaved, resulting in the behavior you are observing. Is the native messaging host writing all messages from the same thread?

”This email and any files transmitted with it may be confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender.”

[Asesh Shrestha]

Hi, yes it sends all the messages in a queue from the same thread. We made that change after your reply around 2020. So that basically means, all the messages from our native messaging host will be queued and sent once at a time, so no parallel messages are sent to our browser. We have also ensured that messages that exceed 1 MB will be discarded. Like I mentioned above,  out of thousands of users, it happens on around 1-15 users randomly everyday, native messaging host was exiting because it was sent an exit command (-1) from our browser. So we made the above change in Chromium too, despite that it still happens. There was one user where it happened frequently, I had enabled logging (--enable-logging) on her machine, that's when we found out that our browser had sent an exit command. Also we have enabled logging in our native messaging host, so Chromium is sending an exit signal to our app when we have designed it to close on browser close only.

Like I mentioned above, we cannot replicate this issue on our machines at all. So is that exit command being sent from somewhere else too? We tried to find it but haven't been able to do so. I have been studying the flow of how extensions and native messaging hosts are executed and terminated but still can't find it. Please let us know if there are other places where Chromium sends -1 to native messaging host.

Thanks

[Asesh Shrestha]

I should have posted this too. This is our method to send data from native messaging host to Chromium:
    
// Send the specified message to extension
void CStdioMessageQueue::send_message_to_extension(const std::string& message) {

   std::size_t message_length = message.length();
   if (message_length > max_message_size || message_length <= 0) {
       LOG_ERROR(g_invalid_message_length.data());
       return;
   }

   // We need to send 4 bytes of length information and then send JSON
   // formatted message as ANSI string/char array
   std::cout
       << char(message_length & 0xFF)
       << char((message_length >> 8) & 0xFF)
       << char((message_length >> 16) & 0xFF)
       << char((message_length >> 24) & 0xFF)
       << message.c_str() << std::flush;

#ifdef _DEBUG

   std::cerr << std::endl << "Sending data to extension:" << std::endl
       << message.c_str() << std::endl
       << "Length: " << message_length << std::endl;

#endif // _DEBUG
}

All the calls to this method will be queued to ensure no parallel messages are sent to Chromium. Let me know if you need more information. 

Thanks

[PhistucK]

Have you considered memory pressure issues? Do you know whether the users experiencing this have a relatively weak machine, or have a low memory machine?

☆PhistucK

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/1dce5333-d324-4d2a-9000-051d884f5353n%40chromium.org.

[Asesh Shrestha]

Hi, actually I don't exactly remember memory pressure when checking this issue on their machines but they had a low-mid spec machines. We have tried to replicate the same issue on virtual machines too with two cores and 4 GB memory assigned--couldn't replicate it at all. We have tried flooding the channel from our messaging host for hours to Chromium--still won't replicate. If we explicitly send messages more than 1 MB in our fork, those messages will be discarded as we have modified our fork but none of the messages smaller than 1 MB will send exit signal to our app. Now the only solution we have is to look into Chromium source code deeply.

Thanks

[PhistucK]

Have you generated high memory pressure in those virtual machines? Have lots of heavy tabs open, Facebook playing videos, YouTube playing videos, GMail, some web games...

☆PhistucK

[Asesh Shrestha]

Hi, not really but this is something that we will try in this sprint.

Thanks

[Asesh Shrestha]

We just tried it on a VM with dual core CPU and 2 GB RAM with Windows 10 64-bit installed. We ran so many apps and played YouTube videos in so many tabs and windows to ensure CPU usage was always 100% and memory usage exceeded 90% for so many hours (more than 4 hours) continuously. Looks like we won't be able to replicate this issue on a constrained virtual machine either. I have attached a screenshot below, the highlighted app in the Task Manager is our native messaging host, it continuously sent messages to our Chromium fork all the time:

We just need a pointer to where the Chromium is sending exit signal from besides that place that we have mentioned above. That should be more than enough for us to fix this issue.

Thanks

[tom mak]

Hi, did you fix the problem?

[Asesh Shrestha]

This bug probably exists right now in Chromium too, not sure though. So, rather than fixing this bug in Chromium, which we tried to, but as we couldn't replicate it on our machines; we added a polling mechanism in our extensions which detects if all the extensions are working properly or not. That was the fix that we implemented for this bug.