How to handle malicious or buggy clients?

[Bruno JA]

Hi,

Let me start by saying that gRPC is awesome :) Now to my issue: 

I have implemented a gRPC server that has a single streaming RPC, in C++ (just like the getting started tutorials). 

Then, I started multiple channels and connected to the server to see how many connections it can handle, before even dealing with the streams, along the lines of (one std::thread per connection):

std::shared_ptr<grpc::Channel>
    channel = grpc::CreateCustomChannel("localhost:8080", credentials, channel_arguments);
while (true) {
  channel->GetState(true);  // Tries to connect.
  absl::SleepFor(absl::Seconds(1));
}

I started 1000 concurrent connections, and I can verify they are connected with:

$ netstat -anp |grep 57751 |wc -l
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
3964

The problem is, as I increase the number of concurrent connections, the server crashes with:

terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable

I imagine that is due the socket limits of my OS (Ubuntu 18.04), which is set around 1024 (ulimit -n, unsure why it allowed 3964 connections but the magnitude makes sense). Even if I raise that limit, it is too easy to just increase the number of concurrent connections and make it crash again (tried with 100k). My use case is that I want to bring up a service that will have very few servers running, but, it seems too easy to bring it down if I do the described above.

My questions are:

• Is there a way to handle the incoming connections at the gRPC library level?
• Is there a way to make gRPC not crash and instead just not accept new connections if the limit is reached?
• Would using iptables and limiting connections by IP be a temporary solution?
• Related to the above, could I set some parameter that would disconnect connection from channels at the server side that aren't activated (e.g.: by calling a stream)? That way I could also use iptables and limit the max number of connections from an IP to X per second (better than limiting the number of connections by IP, I think).
• Are there other ways and gRPC best practices to prevent abuse or a buggy client that could start too many connections? (The docs in grpc.io don't cover that).
• Are there any open issues in GitHub about this problem?

Thanks in advance,

Bruno

[c.mau...@gmail.com]

Hi Bruno,

our experiences are very similar it seems...

I located three separate issues in my setup:

1) when the servers threadlimit (ulimit -u) is reached, the server gets unresponsive

see https://github.com/grpc/grpc/issues/20366

=> maybe the added PR is useful to you

2) when the servers open file limit (ulimit -n) is reached, it gets unresponsive

see https://github.com/grpc/grpc/issues/20418

3) when the clients open file limit (ulimit -n) is reached, it aborts with SIGABRT in pollset_set_del_pollset

see https://github.com/grpc/grpc/issues/19767

i guess the solution for 2+3 would be to change the method-signatures of the polling engines, but how this should be done eludes me at the moment... here i hope for the reaction of a code owner... :)

Kind regards,

Christian

[Mark D. Roth]

Hey Bruno!  Sorry for the late reply; this message somehow got lost in our queue.

gRPC currently does not expose any server-side API for handling individual connections.  However, there are settings that will allow you to close idle connections and to forcibly close connections after some fixed time limit.  For details, see:

https://github.com/grpc/proposal/blob/master/A9-server-side-conn-mgt.md

The fact that the server crashes under load instead of just closing connections does sound like a bug.  Please file an issue for that.  Thanks!

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/0076d023-331a-4413-9d97-e5677f458f72%40googlegroups.com.

--

Mark D. Roth <ro...@google.com>
Software Engineer
Google, Inc.