E0505 17:00:34.036852668 47929 chttp2_transport.cc:2880] ipv6:[::1]:50051: Keepalive watchdog fired. Closing transport

charles...@gmail.com

unread,

May 7, 2020, 1:15:34 PM5/7/20

to grpc.io

### What version of gRPC and what language are you using?

```

Go (server): google.golang.org/grpc v1.27.1

Python (client): grpcio==1.27.2

```

### What operating system (Linux, Windows,...) and version?

Linux Ubuntu 18.04

```

# uname -a

4.15.0-99-generic #100-Ubuntu SMP Wed Apr 22 20:32:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

```

### What did you do?

Go (server)

```

// MaxMessageSizeBytes 2GB to support large data size

const MaxMessageSizeBytes = 2097152000

// KeepalivePolicyMinTime The minimum time between keepalive pings. Client pings can not be more frequent than this.

const KeepalivePolicyMinTime = 5 * time.Second

// KeepalivePolicyPermitWithoutStream Permit keepalive pings even with no inflight RPCs.

const KeepalivePolicyPermitWithoutStream = true

// KeepaliveParamTime If the server doesn't see any activity after this time, it

// pings the client to see if the transport is still alive.

const KeepaliveParamTime = 5 * time.Second

// KeepaliveParamTimeOut After having pinged for keepalive check, the server waits for this duration and

// the connection is closed if no activity is seen after that.

const KeepaliveParamTimeOut = 5 * time.Second

...

grpcListener, err := net.Listen(network, address)

if err != nil {

return nil, err

}

keepaliveParameters := keepalive.ServerParameters{

Time: KeepaliveParamTime,

Timeout: KeepaliveParamTimeOut,

}

keepalivePolicy := keepalive.EnforcementPolicy{

MinTime: KeepalivePolicyMinTime,

PermitWithoutStream: KeepalivePolicyPermitWithoutStream,

}

var (

recoveryFunc grpc_recovery.RecoveryHandlerFunc

)

opts := []grpc_recovery.Option{

grpc_recovery.WithRecoveryHandler(recoveryFunc),

}

grpcServer := grpc.NewServer(

grpc_middleware.WithUnaryServerChain(

grpc_recovery.UnaryServerInterceptor(opts...),

),

grpc_middleware.WithStreamServerChain(

grpc_recovery.StreamServerInterceptor(opts...),

),

grpc.MaxRecvMsgSize(MaxMessageSizeBytes),

grpc.KeepaliveEnforcementPolicy(keepalivePolicy),

grpc.KeepaliveParams(keepaliveParameters),

)

// followed by grpcServer.Serve

```

Python (client)

```

import grpc

import time

from my_pb

from my_grpc

GRPC_CHANNEL_OPTIONS = [

('grpc.keepalive_time_ms', 5000),

('grpc.keepalive_timeout_ms', 5000),

('grpc.keepalive_permit_without_calls', True),

('grpc.http2.max_pings_without_data', 0),

('grpc.http2.min_time_between_pings_ms', 5000),

('grpc.http2.min_ping_interval_without_data_ms', 5000),

('grpc.max_send_message_length', 2000 * 1024 * 1024),

('grpc.max_receive_message_length', 2000 * 1024 * 1024)]

if __name__ == "__main__":

url = "localhost:50051"

my_channels = [grpc.insecure_channel(url, options=GRPC_CHANNEL_OPTIONS) for i in range(10)]

my_stubs = [my_grpc.MyStub(my_channels[i]) for i in range(10)]

print(f"Setup grpc channel + stubs... {len(my_channels)}")

# make a request on each connection

for my_stub in my_stubs:

try:

request = my_pb.GetRequest()

request.id = 'abc'

response = my_stub.Get(request)

print(response)

except Exception as e:

# print(e)

pass

# test: grpc should keepalive with the connection

# simulate the scenario when there are no activity on the channel

while True:

print("Sleeping ...")

time.sleep(10)

# ... code to use the channel

```

### What did you expect to see?

Keepalive maintains the connection when no request/response flows through the channel.

And I also do not expect to see network/connection/watchdog error received on the client.

### What did you see instead?

Errors seen in the client. (No error on server.)

```

Setup grpc channel + stubs... 10

Sleeping ...

E0505 16:54:08.762435018 47441 chttp2_transport.cc:2893] keepalive_ping_end state error: 0 (expect: 1)

Sleeping ...

E0505 16:54:18.762541095 47452 chttp2_transport.cc:2893] keepalive_ping_end state error: 0 (expect: 1)

Sleeping ...

E0505 16:54:28.766357886 47452 chttp2_transport.cc:2893] keepalive_ping_end state error: 0 (expect: 1)

Sleeping ...

E0505 16:54:38.765747109 47441 chttp2_transport.cc:2893] keepalive_ping_end state error: 0 (expect: 1)

Sleeping ...

E0505 16:54:48.766456333 47452 chttp2_transport.cc:2880] ipv6:[::1]:50051: Keepalive watchdog fired. Closing transport.

Sleeping ...

```

Setup grpc channel + stubs... 10

Sleeping ...

E0505 17:00:34.036852668 47929 chttp2_transport.cc:2880] ipv6:[::1]:50051: Keepalive watchdog fired. Closing transport.

Sleeping ...

```

----

I've read https://github.com/grpc/grpc-go/issues/2202 and tried other configuration. However, I am still seeing the same error.

In production environment, I have also seen the same error message when using the above configuration -- thought reporting a different error code `E0330` and using a older version of grpcio client library (grpcio==1.23.0):

```

E0330 19:01:16.746628681 10533 chttp2_transport.cc:2825] ipv4:10.50.40.102:50051: Keepalive watchdog fired. Closing transport.

```

------

I have also tried the following configuration but also encountered the watchdog error.

Go (server)

```

// KeepalivePolicyMinTime The minimum time between keepalive pings. Client pings can not be more frequent than this.

// const KeepalivePolicyMinTime = 5 * time.Second

const KeepalivePolicyMinTime = 10 * time.Second

// KeepalivePolicyPermitWithoutStream Permit keepalive pings even with no inflight RPCs.

const KeepalivePolicyPermitWithoutStream = true

// KeepaliveParamTime If the server doesn't see any activity after this time, it

// pings the client to see if the transport is still alive.

// const KeepaliveParamTime = 5 * time.Second

const KeepaliveParamTime = 10 * time.Second

// KeepaliveParamTimeOut After having pinged for keepalive check, the server waits for this duration and

// the connection is closed if no activity is seen after that.

const KeepaliveParamTimeOut = 5 * time.Second

```

Python (client)

```

GRPC_CHANNEL_OPTIONS = [

('grpc.keepalive_time_ms', 10000),

('grpc.keepalive_timeout_ms', 5000),

('grpc.keepalive_permit_without_calls', True),

('grpc.http2.max_pings_without_data', 0),

('grpc.http2.min_time_between_pings_ms', 10000),

# ('grpc.http2.min_ping_interval_without_data_ms', 10000), # N/A on client side

('grpc.max_send_message_length', 2000 * 1024 * 1024),

('grpc.max_receive_message_length', 2000 * 1024 * 1024)]

```

1. Could you please help me understand what is the correct way to configure these options so that I would not encounter the `Keepalive watchdog fired. Closing transport` error? Thanks.

2. Also, could you please clarify the difference between the following [options](https://github.com/grpc/grpc/blob/master/doc/keepalive.md):

> GRPC_ARG_KEEPALIVE_TIME_MS

> This channel argument controls the period (in milliseconds) after which a keepalive ping is sent on the transport.

and

> GRPC_ARG_HTTP2_MIN_SENT_PING_INTERVAL_WITHOUT_DATA_MS

> If there are no data frames being received on the transport, this channel argument controls the minimum time (in milliseconds) gRPC Core will wait between successive pings.

Does GRPC_ARG_KEEPALIVE_TIME_MS send a keepalive ping regardless if there is data provided GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA = true?

Does that mean GRPC_ARG_HTTP2_MIN_SENT_PING_INTERVAL_WITHOUT_DATA_MS is only applicable when GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA = false?

Thanks

Sidhartha Thota

unread,

May 7, 2020, 2:55:40 PM5/7/20

to charles...@gmail.com, grpc.io

Hi,

I asked similar question for GRPC C++ and here is the answer from one of the contributor: #22698. Hoping this will be helpful.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/c5455e6c-f5e2-484c-9d04-1d219963a0f4%40googlegroups.com.

--

Thanks,
Sidhartha Thota.

charles...@gmail.com

unread,

May 7, 2020, 3:08:51 PM5/7/20

to grpc.io

Thanks. I had a brief read but I don't think it is the same scenario.

I did stumbled across a "working" configuration: https://github.com/grpc/grpc/issues/22877#issuecomment-625383964 and saw an open bug in the C++ core that might be related https://github.com/grpc/grpc/issues/16210

I am still looking for a clearer explanation of the settings so I can understand why certain configurations don't work and how to adjust them properly.

Charles

Sidhartha Thota

unread,

May 7, 2020, 5:02:39 PM5/7/20

to charles...@gmail.com, grpc.io

Hi,

Just curious, may I know if you are using asynchronous/synchronous client and server?

--

You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/87229792-a035-4126-9e40-b0820442f527%40googlegroups.com.

--

Thanks,
Sidhartha Thota.

charles...@gmail.com

unread,

May 7, 2020, 5:08:46 PM5/7/20

to grpc.io

I believe I am using synchronous comm. (Go server/Python client)

On Thursday, May 7, 2020 at 2:02:39 PM UTC-7, Sidhartha Thota wrote:

Hi,

Just curious, may I know if you are using asynchronous/synchronous client and server?

On Thu, May 7, 2020 at 12:08 PM <charle...@gmail.com> wrote:

Thanks. I had a brief read but I don't think it is the same scenario.

I did stumbled across a "working" configuration: https://github.com/grpc/grpc/issues/22877#issuecomment-625383964 and saw an open bug in the C++ core that might be related https://github.com/grpc/grpc/issues/16210

I am still looking for a clearer explanation of the settings so I can understand why certain configurations don't work and how to adjust them properly.

Charles

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/87229792-a035-4126-9e40-b0820442f527%40googlegroups.com.

--
Thanks,
Sidhartha Thota.

Reply all

Reply to author

Forward