My app hangs up at some point in net/http/h2_bundle.go. What is the better way to debug?

185 views
Skip to first unread message

Denis Telyukh

unread,
Nov 9, 2020, 11:49:57 AM11/9/20
to golang-nuts
Hello!

I use third-party software with my own code to build reverse-proxy, which supports http/2. Sometimes requests hang up with infinite waiting. With Delve I found, that it happens in https://golang.org/src/net/http/h2_bundle.go in writeHeaders or in writeDataFromHandler. But I have no any idea why it happens. Which tools should I use to find the reason of such behaviour?

Denis

seank...@gmail.com

unread,
Nov 9, 2020, 1:20:42 PM11/9/20
to golang-nuts
maybe setting GODEBUG=http2debug=2 to spit out some logs?

Denis Telyukh

unread,
Nov 9, 2020, 10:25:58 PM11/9/20
to seank...@gmail.com, golang-nuts
Thanks! Maybe it will help.

вт, 10 нояб. 2020 г. в 01:21, seank...@gmail.com <seank...@gmail.com>:
--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/CJwsb1H9mGg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/94e4b5cd-6005-4a52-916f-9e1a88137399n%40googlegroups.com.

Denis Telyukh

unread,
Nov 10, 2020, 5:45:27 AM11/10/20
to golang-nuts
More complex problem description with images: https://caddy.community/t/sometimes-http-2-request-hangs-up-need-help-with-debugging/10439/2 

вторник, 10 ноября 2020 г. в 10:25:58 UTC+7, Denis Telyukh:

Robert Engels

unread,
Nov 10, 2020, 7:10:44 AM11/10/20
to Denis Telyukh, golang-nuts
You’ve most likely exhausted the network capacity. You need to use tools like netperf and wireshark. Start with netstat and review the outgoing queue sizes. 

On Nov 10, 2020, at 4:46 AM, Denis Telyukh <telyuk...@gmail.com> wrote:

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a9ac428d-1389-4278-be9a-273ae95751b0n%40googlegroups.com.

Denis Telyukh

unread,
Nov 10, 2020, 10:36:37 AM11/10/20
to Robert Engels, golang-nuts
Thanks for your advice. But I have this problem on localhost also, I doubt that problem in network capacity :-(

вт, 10 нояб. 2020 г. в 19:09, Robert Engels <ren...@ix.netcom.com>:

Robert Engels

unread,
Nov 10, 2020, 11:12:21 AM11/10/20
to Denis Telyukh, golang-nuts
You can easily have the same problem on localhost if the producer is faster than the consumer. 

You can use netstat in the localhost as well. 

On Nov 10, 2020, at 9:36 AM, Denis Telyukh <telyuk...@gmail.com> wrote:



Denis Telyukh

unread,
Nov 11, 2020, 12:38:53 AM11/11/20
to golang-nuts
1. Got some stuck requests
2. netstat -atp
tcp       25      0 dionysius-pc:52340      server-52-85-47-1:https CLOSE_WAIT  1364/brave --type=u 
tcp       25      0 dionysius-pc:40296      server-13-33-240-:https CLOSE_WAIT  1364/brave --type=u 
3. A few second later netstat doesn't show any non-empty queues
4. But requests continue to hang (it will be endless)


вторник, 10 ноября 2020 г. в 23:12:21 UTC+7, ren...@ix.netcom.com:

Kurtis Rader

unread,
Nov 11, 2020, 1:00:08 AM11/11/20
to Denis Telyukh, golang-nuts
On Tue, Nov 10, 2020 at 9:39 PM Denis Telyukh <telyuk...@gmail.com> wrote:
1. Got some stuck requests
2. netstat -atp
tcp       25      0 dionysius-pc:52340      server-52-85-47-1:https CLOSE_WAIT  1364/brave --type=u 
tcp       25      0 dionysius-pc:40296      server-13-33-240-:https CLOSE_WAIT  1364/brave --type=u 
3. A few second later netstat doesn't show any non-empty queues
4. But requests continue to hang (it will be endless)

The `netstat` command does not show "non-empty queues". It shows connections that exist between processes on your system and remote systems. The fact "a few second later netstat doesn't show any non-empty queues" means that your system has no open connections. Also, what platform (OS) are you using? The `netstat -atp` command makes no sense on macOS or Linux.

Denis Telyukh

unread,
Nov 11, 2020, 1:21:56 AM11/11/20
to Kurtis Rader, golang-nuts
OS: Manjaro 20.2 Nibia
Kernel: x86_64 Linux 5.4.74-1-MANJARO

ср, 11 нояб. 2020 г. в 12:59, Kurtis Rader <kra...@skepticism.us>:

Denis Telyukh

unread,
Nov 12, 2020, 4:21:53 AM11/12/20
to golang-nuts
Better problem description after deeper debugging: https://github.com/golang/go/issues/42534

среда, 11 ноября 2020 г. в 13:21:56 UTC+7, Denis Telyukh:

Robert Engels

unread,
Nov 12, 2020, 9:14:05 AM11/12/20
to Denis Telyukh, golang-nuts
Your issue report describes the same issue. 

If you switch to http rather than http2 do you have the same issue?

I am guessing that your consumer is blocked which eventually blocks your producer when the buffers fill. 

I would continue to use the networking tools to diagnose. 

On Nov 12, 2020, at 3:22 AM, Denis Telyukh <telyuk...@gmail.com> wrote:

Better problem description after deeper debugging: https://github.com/golang/go/issues/42534
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

Robert Engels

unread,
Nov 12, 2020, 9:34:35 AM11/12/20
to Denis Telyukh, golang-nuts
One thing to note, gRPC uses http2. If there were a bug like this lurking in the library it would be far more common, so the most likely cause is specific to your code. 

On Nov 12, 2020, at 8:13 AM, Robert Engels <ren...@ix.netcom.com> wrote:



Denis Telyukh

unread,
Nov 12, 2020, 9:40:20 AM11/12/20
to golang-nuts
Could you, please, explain me what is producer in this case and what is consumer? Is it web browser and web server? And what kind of anomalies do you think I can find with netstat? I'm not sure what to look at, the traffic seems fine to me.

Maybe this issue is related the same problem: https://github.com/golang/go/issues/23559

четверг, 12 ноября 2020 г. в 21:34:35 UTC+7, ren...@ix.netcom.com:

Robert Engels

unread,
Nov 12, 2020, 9:52:55 AM11/12/20
to Denis Telyukh, golang-nuts
After reading that issue and the related ones there does appear to be a live lock under heavy stress. I didn’t read too deeply because I am on a phone. 

I am guessing a small reproducible test case is required. Since you encounter it on localhost I assume you can extract what is needed based on the other issues cited. 

On Nov 12, 2020, at 8:42 AM, Denis Telyukh <telyuk...@gmail.com> wrote:



Brian Candler

unread,
Nov 12, 2020, 10:06:26 AM11/12/20
to golang-nuts
On Thursday, 12 November 2020 14:34:35 UTC, Robert Engels wrote:
One thing to note, gRPC uses http2. If there were a bug like this lurking in the library it would be far more common, so the most likely cause is specific to your code.

To the OP: have you tried running your code under the race detector?

Denis Telyukh

unread,
Nov 12, 2020, 10:15:21 AM11/12/20
to Brian Candler, golang-nuts
> If you switch to http rather than http2 do you have the same issue?
No, I haven't.


>  have you tried running your code under the race detector?
Yes, I did.


> Since you encounter it on localhost I assume you can extract what is needed based on the other issues cited.
It's not easy, but I could try.

чт, 12 нояб. 2020 г. в 22:07, Brian Candler <b.ca...@pobox.com>:
On Thursday, 12 November 2020 14:34:35 UTC, Robert Engels wrote:
One thing to note, gRPC uses http2. If there were a bug like this lurking in the library it would be far more common, so the most likely cause is specific to your code.

To the OP: have you tried running your code under the race detector?

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/CJwsb1H9mGg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8e2463b4-da04-4f1b-b09f-948d45954088o%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages