HTTP/2 RESOURCE_EXHAUSTED and goaway responses

899 views
Skip to first unread message

Pavani Manthana

unread,
May 2, 2018, 4:34:11 PM5/2/18
to Glowroot
Hi Trask - We have connection to agents dropping out of central collectors with the following error...

018-05-02 14:23:20.611 WARN  o.g.a.s.i.g.netty.NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}

2018-05-02 14:23:20.614 WARN  o.g.a.s.i.g.internal.AtomicBackoff - Increased keepalive time nanos to 60,000,000,000

2018-05-02 14:23:20.623 WARN  o.g.agent.central.CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): RESOURCE_EXHAUSTED: Bandwidth exhausted

HTTP/2 error code: ENHANCE_YOUR_CALM

Received Goaway

too_many_pings


Based on my research looks like you have used KeepAliveTime of 30 secs and this was in issue with grpc before 1.6 but glowroot seems to use 1.10. Any idea on what might be causing this and if we can configure any timeouts to prevent it.


Thanks,

Pavani

Trask Stalnaker

unread,
May 2, 2018, 4:42:45 PM5/2/18
to Glowroot
Hi Pavani,

Thanks, for reporting this.

What version is the agent?

Also, it looks like this error is on the initial connection?  Is that true?  Can you include more of the agent log (from it's startup) so I can see?

Trask

Pavani Manthana

unread,
May 2, 2018, 5:46:53 PM5/2/18
to Glowroot, Trask Stalnaker
Askd the client to send the logs but meanehile noticed these errors in the agent logs that we configured for central colletors as well...

2018-05-01 15:14:25.209 WARN  o.g.agent.central.CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 77 warnings were suppressed since it was last logged)
2018-05-01 15:15:24.217 WARN  o.g.agent.central.CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 3183 warnings were suppressed since it was last logged)
2018-05-01 15:15:27.294 WARN  o.g.agent.central.CentralConnection - unable to send data to the central collector: UNKNOWN (this warning will be logged at most once a minute, 54 warnings were suppressed since it was last logged)
2018-05-01 15:16:25.421 WARN  o.g.agent.central.CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 889 warnings were suppressed since it was last logged)
2018-05-01 15:16:38.916 WARN  o.g.agent.central.CentralConnection - unable to send data to the central collector: UNKNOWN (this warning will be logged at most once a minute, 10 warnings were suppressed since it was last logged)
2018-05-01 15:17:26.194 WARN  o.g.agent.central.CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 17 warnings were suppressed since it was last logged)
2018-05-01 15:18:26.209 WARN  o.g.agent.central.CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 19 warnings were suppressed since it was last logged)
2018-05-01 15:18:59.628 WARN  o.g.agent.central.CentralConnection - unable to send data to the central collector: UNKNOWN (this warning will be logged at most once a minute, 1 warnings were suppressed since it was last logged)
2018-05-01 15:19:26.222 WARN  o.g.agent.central.CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 14 warnings were suppressed since it was last logged)
2018-05-01 15:20:26.237 WARN  o.g.agent.central.CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 19 warnings were suppressed since it was last logged)
2018-05-01 15:21:03.280 WARN  o.g.a.s.i.g.netty.NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-01 15:21:03.281 WARN  o.g.a.s.i.g.internal.AtomicBackoff - Increased keepalive time nanos to 60,000,000,000
2018-05-01 15:21:03.285 WARN  o.g.agent.central.CentralConnection - unable to send data to the central collector: RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings (this warning will be logged at most once a minute)
2018-05-01 15:22:41.490 WARN  o.g.a.s.i.g.netty.NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-01 15:22:41.493 WARN  o.g.agent.central.CentralConnection - unable to send data to the central collector: RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings (this warning will be logged at most once a minute, 99 warnings were suppressed since it was last logged)
2018-05-01 15:25:53.605 WARN  o.g.a.s.i.g.netty.NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-01 15:25:53.606 WARN  o.g.a.s.i.g.internal.AtomicBackoff - Increased keepalive time nanos to 120,000,000,000
2018-05-01 15:25:53.608 WARN  o.g.agent.central.CentralConnection - unable to send data to the central collector: RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings (this warning will be logged at most once a minute, 8 warnings were suppressed since it was last logged)
2018-05-01 15:29:04.434 WARN  o.g.a.s.i.g.netty.NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-01 15:29:04.437 WARN  o.g.agent.central.CentralConnection - unable to send data to the central collector: RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings (this warning will be logged at most once a minute, 53 warnings were suppressed since it was last logged)
2018-05-01 15:30:56.640 WARN  o.g.agent.central.CentralConnection - unable to send data to the central collector: UNAVAILABLE: HTTP/2 error code: NO_ERROR
Received Goaway (this warning will be logged at most once a minute, 62 warnings were suppressed since it was last logged)


Thanks,
Pavani

On Wednesday, May 2, 2018, 1:42:47 PM PDT, Trask Stalnaker <trask.s...@gmail.com> wrote:


Hi Pavani,

Thanks, for reporting this.

What version is the agent?

Also, it looks like this error is on the initial connection?  Is that true?  Can you include more of the agent log (from it's startup) so I can see?

Trask


On Wednesday, May 2, 2018 at 1:34:11 PM UTC-7, Pavani Manthana wrote:
Hi Trask - We have connection to agents dropping out of central collectors with the following error...

018-05-02 14:23:20.611 WARN  o.g.a.s.i.g.netty. NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}

2018-05-02 14:23:20.614 WARN  o.g.a.s.i.g.internal. AtomicBackoff - Increased keepalive time nanos to 60,000,000,000

2018-05-02 14:23:20.623 WARN  o.g.agent.central. CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): RESOURCE_EXHAUSTED: Bandwidth exhausted

HTTP/2 error code: ENHANCE_YOUR_CALM

Received Goaway

too_many_pings


Based on my research looks like you have used KeepAliveTime of 30 secs and this was in issue with grpc before 1.6 but glowroot seems to use 1.10. Any idea on what might be causing this and if we can configure any timeouts to prevent it.


Thanks,

Pavani

--
You received this message because you are subscribed to a topic in the Google Groups "Glowroot" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/glowroot/HPU_q_fs7Zw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to glowroot+u...@googlegroups.com.
To post to this group, send email to glow...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/glowroot/4ceeab3d-a806-4882-8f7a-72fb50e10544%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Pavani Manthana

unread,
May 2, 2018, 9:12:28 PM5/2/18
to Glowroot, Trask Stalnaker
Client logs are similar as well...Here it is....

2018-05-02 14:05:16.237 INFO  org.glowroot - Glowroot version: 0.10.4, built 2018-04-06 21:32:49 +0000
2018-05-02 14:05:16.240 INFO  org.glowroot - Java version: 1.8.0_144
2018-05-02 14:21:46.867 INFO  org.glowroot - Glowroot version: 0.10.4, built 2018-04-06 21:32:49 +0000
2018-05-02 14:21:46.870 INFO  org.glowroot - Java version: 1.8.0_144
2018-05-02 14:21:48.793 INFO  org.glowroot - plugins loaded: Cassandra Plugin, Elasticsearch Plugin, Executor Plugin, Grails Plugin, Hibernate Plugin, HTTP Client Plugin, Java HTTP Server Plugin, JAX-RS Plugin, Jdbc Plugin, JMS Plugin, JSF Plugin, JSP Plugin, Logger Plugin, Play Plugin, Netty Plugin, Quartz Plugin, Redis Plugin, Servlet Plugin, Spring Plugin, Struts Plugin
2018-05-02 14:21:49.672 INFO  org.glowroot - agent id: CSORPT-24168-tomcat::zld01929-tomcat
2018-05-02 14:23:20.611 WARN  o.g.a.s.i.g.netty.NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-02 14:23:20.614 WARN  o.g.a.s.i.g.internal.AtomicBackoff - Increased keepalive time nanos to 60,000,000,000
2018-05-02 14:23:20.623 WARN  o.g.agent.central.CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings
2018-05-02 14:24:51.765 WARN  o.g.a.s.i.g.netty.NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-02 14:24:51.767 WARN  o.g.agent.central.CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings
2018-05-02 14:28:08.687 WARN  o.g.a.s.i.g.netty.NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-02 14:28:08.688 WARN  o.g.a.s.i.g.internal.AtomicBackoff - Increased keepalive time nanos to 120,000,000,000
2018-05-02 14:28:08.690 WARN  o.g.agent.central.CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings
2018-05-02 14:31:22.344 WARN  o.g.a.s.i.g.netty.NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-02 14:31:22.348 WARN  o.g.agent.central.CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings
2018-05-02 14:35:39.937 WARN  o.g.agent.central.CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): UNAVAILABLE: Keepalive failed. The connection is likely gone
2018-05-02 14:42:10.156 WARN  o.g.agent.central.CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute)
2018-05-02 14:43:10.174 WARN  o.g.agent.central.CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 12 warnings were suppressed since it was last logged)
2018-05-02 14:43:16.303 WARN  o.g.agent.central.CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): UNAVAILABLE: Network closed for unknown reason
2018-05-02 14:43:26.357 INFO  org.glowroot - connected to the central collector http://xx.xx.xx.xx:8181, version 0.10.4, built 2018-05-01 18:01:09 +0000
2018-05-02 17:01:01.527 INFO  org.glowroot - Glowroot version: 0.10.4, built 2018-04-06 21:32:49 +0000
2018-05-02 17:01:01.530 INFO  org.glowroot - Java version: 1.8.0_144
2018-05-02 17:01:19.365 INFO  org.glowroot - Glowroot version: 0.10.4, built 2018-04-06 21:32:49 +0000
2018-05-02 17:01:19.367 INFO  org.glowroot - Java version: 1.8.0_144
2018-05-02 17:01:21.224 INFO  org.glowroot - plugins loaded: Cassandra Plugin, Elasticsearch Plugin, Executor Plugin, Grails Plugin, Hibernate Plugin, HTTP Client Plugin, Java HTTP Server Plugin, JAX-RS Plugin, Jdbc Plugin, JMS Plugin, JSF Plugin, JSP Plugin, Logger Plugin, Play Plugin, Netty Plugin, Quartz Plugin, Redis Plugin, Servlet Plugin, Spring Plugin, Struts Plugin
2018-05-02 17:01:22.033 INFO  org.glowroot - agent id: CSORPT-24168-tomcat::zld01929-tomcat
2018-05-02 17:01:22.869 INFO  org.glowroot - connected to the central collector http://135.69.200.233:8181, version 0.10.4, built 2018-05-01 18:01:09 +0000




Trask Stalnaker

unread,
May 8, 2018, 12:29:57 AM5/8/18
to Glowroot
Hi Pavani,

I think this should be resolved now by 62961e98e9c8538f3a3277e599d6f5f7b0542718.

Thanks,
Trask
To unsubscribe from this group and all its topics, send an email to glowroot+unsubscribe@googlegroups.com.

To post to this group, send email to glow...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/glowroot/4ceeab3d-a806-4882-8f7a-72fb50e10544%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Glowroot" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/glowroot/HPU_q_fs7Zw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to glowroot+unsubscribe@googlegroups.com.

Pavani Manthana

unread,
May 8, 2018, 12:56:45 PM5/8/18
to Glowroot, Trask Stalnaker
Thanks Trask. Will make the change and test.

On Monday, May 7, 2018, 9:29:59 PM PDT, Trask Stalnaker <trask.s...@gmail.com> wrote:


Hi Pavani,

I think this should be resolved now by 62961e98e9c8538f3a3277e599d6f5f7b0542718.

Thanks,
Trask


On Wednesday, May 2, 2018 at 6:12:28 PM UTC-7, Pavani Manthana wrote:
Client logs are similar as well...Here it is....

2018-05-02 14:05:16.237 INFO  org.glowroot - Glowroot version: 0.10.4, built 2018-04-06 21:32:49 +0000
2018-05-02 14:05:16.240 INFO  org.glowroot - Java version: 1.8.0_144
2018-05-02 14:21:46.867 INFO  org.glowroot - Glowroot version: 0.10.4, built 2018-04-06 21:32:49 +0000
2018-05-02 14:21:46.870 INFO  org.glowroot - Java version: 1.8.0_144
2018-05-02 14:21:48.793 INFO  org.glowroot - plugins loaded: Cassandra Plugin, Elasticsearch Plugin, Executor Plugin, Grails Plugin, Hibernate Plugin, HTTP Client Plugin, Java HTTP Server Plugin, JAX-RS Plugin, Jdbc Plugin, JMS Plugin, JSF Plugin, JSP Plugin, Logger Plugin, Play Plugin, Netty Plugin, Quartz Plugin, Redis Plugin, Servlet Plugin, Spring Plugin, Struts Plugin
2018-05-02 14:21:49.672 INFO  org.glowroot - agent id: CSORPT-24168-tomcat::zld01929- tomcat
2018-05-02 14:23:20.611 WARN  o.g.a.s.i.g.netty. NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-02 14:23:20.614 WARN  o.g.a.s.i.g.internal. AtomicBackoff - Increased keepalive time nanos to 60,000,000,000
2018-05-02 14:23:20.623 WARN  o.g.agent.central. CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings
2018-05-02 14:24:51.765 WARN  o.g.a.s.i.g.netty. NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-02 14:24:51.767 WARN  o.g.agent.central. CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings
2018-05-02 14:28:08.687 WARN  o.g.a.s.i.g.netty. NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-02 14:28:08.688 WARN  o.g.a.s.i.g.internal. AtomicBackoff - Increased keepalive time nanos to 120,000,000,000
2018-05-02 14:28:08.690 WARN  o.g.agent.central. CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings
2018-05-02 14:31:22.344 WARN  o.g.a.s.i.g.netty. NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-02 14:31:22.348 WARN  o.g.agent.central. CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings
2018-05-02 14:35:39.937 WARN  o.g.agent.central. CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): UNAVAILABLE: Keepalive failed. The connection is likely gone
2018-05-02 14:42:10.156 WARN  o.g.agent.central. CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute)
2018-05-02 14:43:10.174 WARN  o.g.agent.central. CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 12 warnings were suppressed since it was last logged)
2018-05-02 14:43:16.303 WARN  o.g.agent.central. CentralConnection - unable to establish connection with the central collector http://135.69.200.233:8181 (will keep trying...): UNAVAILABLE: Network closed for unknown reason
2018-05-02 14:43:26.357 INFO  org.glowroot - connected to the central collector http://xx.xx.xx.xx:8181, version 0.10.4, built 2018-05-01 18:01:09 +0000
2018-05-02 17:01:01.527 INFO  org.glowroot - Glowroot version: 0.10.4, built 2018-04-06 21:32:49 +0000
2018-05-02 17:01:01.530 INFO  org.glowroot - Java version: 1.8.0_144
2018-05-02 17:01:19.365 INFO  org.glowroot - Glowroot version: 0.10.4, built 2018-04-06 21:32:49 +0000
2018-05-02 17:01:19.367 INFO  org.glowroot - Java version: 1.8.0_144
2018-05-02 17:01:21.224 INFO  org.glowroot - plugins loaded: Cassandra Plugin, Elasticsearch Plugin, Executor Plugin, Grails Plugin, Hibernate Plugin, HTTP Client Plugin, Java HTTP Server Plugin, JAX-RS Plugin, Jdbc Plugin, JMS Plugin, JSF Plugin, JSP Plugin, Logger Plugin, Play Plugin, Netty Plugin, Quartz Plugin, Redis Plugin, Servlet Plugin, Spring Plugin, Struts Plugin
2018-05-02 17:01:22.033 INFO  org.glowroot - agent id: CSORPT-24168-tomcat::zld01929- tomcat
2018-05-02 17:01:22.869 INFO  org.glowroot - connected to the central collector http://135.69.200.233:8181, version 0.10.4, built 2018-05-01 18:01:09 +0000


On Wednesday, May 2, 2018, 2:46:54 PM PDT, 'Pavani Manthana' via Glowroot <glow...@googlegroups.com> wrote:


Askd the client to send the logs but meanehile noticed these errors in the agent logs that we configured for central colletors as well...

2018-05-01 15:14:25.209 WARN  o.g.agent.central. CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 77 warnings were suppressed since it was last logged)
2018-05-01 15:15:24.217 WARN  o.g.agent.central. CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 3183 warnings were suppressed since it was last logged)
2018-05-01 15:15:27.294 WARN  o.g.agent.central. CentralConnection - unable to send data to the central collector: UNKNOWN (this warning will be logged at most once a minute, 54 warnings were suppressed since it was last logged)
2018-05-01 15:16:25.421 WARN  o.g.agent.central. CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 889 warnings were suppressed since it was last logged)
2018-05-01 15:16:38.916 WARN  o.g.agent.central. CentralConnection - unable to send data to the central collector: UNKNOWN (this warning will be logged at most once a minute, 10 warnings were suppressed since it was last logged)
2018-05-01 15:17:26.194 WARN  o.g.agent.central. CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 17 warnings were suppressed since it was last logged)
2018-05-01 15:18:26.209 WARN  o.g.agent.central. CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 19 warnings were suppressed since it was last logged)
2018-05-01 15:18:59.628 WARN  o.g.agent.central. CentralConnection - unable to send data to the central collector: UNKNOWN (this warning will be logged at most once a minute, 1 warnings were suppressed since it was last logged)
2018-05-01 15:19:26.222 WARN  o.g.agent.central. CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 14 warnings were suppressed since it was last logged)
2018-05-01 15:20:26.237 WARN  o.g.agent.central. CentralConnection - not sending data to the central collector because pending request limit (100) exceeded (this warning will be logged at most once a minute, 19 warnings were suppressed since it was last logged)
2018-05-01 15:21:03.280 WARN  o.g.a.s.i.g.netty. NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-01 15:21:03.281 WARN  o.g.a.s.i.g.internal. AtomicBackoff - Increased keepalive time nanos to 60,000,000,000
2018-05-01 15:21:03.285 WARN  o.g.agent.central. CentralConnection - unable to send data to the central collector: RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings (this warning will be logged at most once a minute)
2018-05-01 15:22:41.490 WARN  o.g.a.s.i.g.netty. NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-01 15:22:41.493 WARN  o.g.agent.central. CentralConnection - unable to send data to the central collector: RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings (this warning will be logged at most once a minute, 99 warnings were suppressed since it was last logged)
2018-05-01 15:25:53.605 WARN  o.g.a.s.i.g.netty. NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-01 15:25:53.606 WARN  o.g.a.s.i.g.internal. AtomicBackoff - Increased keepalive time nanos to 120,000,000,000
2018-05-01 15:25:53.608 WARN  o.g.agent.central. CentralConnection - unable to send data to the central collector: RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings (this warning will be logged at most once a minute, 8 warnings were suppressed since it was last logged)
2018-05-01 15:29:04.434 WARN  o.g.a.s.i.g.netty. NettyClientHandler - Received GOAWAY with ENHANCE_YOUR_CALM. Debug data: {1}
2018-05-01 15:29:04.437 WARN  o.g.agent.central. CentralConnection - unable to send data to the central collector: RESOURCE_EXHAUSTED: Bandwidth exhausted
HTTP/2 error code: ENHANCE_YOUR_CALM
Received Goaway
too_many_pings (this warning will be logged at most once a minute, 53 warnings were suppressed since it was last logged)
2018-05-01 15:30:56.640 WARN  o.g.agent.central. CentralConnection - unable to send data to the central collector: UNAVAILABLE: HTTP/2 error code: NO_ERROR

For more options, visit https://groups.google.com/d/ optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Glowroot" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/glowroot/HPU_q_fs7Zw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to glowroot+u...@googlegroups.com.

To post to this group, send email to glow...@googlegroups.com.

Visvanathan

unread,
Oct 9, 2023, 3:29:59 AM10/9/23
to Glowroot
Hi I am Using glowroot-central-0.14.0. But I am getting the below error when we get more logs.
WARN org.glowroot.agent.central.CentralConnection - unable to send data to the central collector: org.glowroot.agent.shaded.io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: collection rejected due to backlog (this warning will be logged at most once a minute)

Reply all
Reply to author
Forward
0 new messages