Tcp proxy load balancer random disconnect
232 views
Skip to first unread message
ting chiang
Feb 7, 2021, 12:41:20 AM2/7/21
to gce-discussion
I'm using tcp proxy load balancer , and need connection long-alive.
But always random time disconnect.
The time was very randomly , sometimes after 30 minutes. sometimes after 30 hours.
When it happen , not all connection disconnect , only 1~2 connection disconnect.
For example:
2021/02/03 19:00:00 create 16 connection
2021/02/04 02:02:29 disconnect
2021/02/04 20:50:42 disconnect
2021/02/04 22:26:37 disconnect
2021/02/04 22:26:38 disconnect
Those are my tried , but useless.
Pedro Moreno Bonelo
Feb 11, 2021, 3:56:35 AM2/11/21
to gce-discussion
Which server are you using ? There is a similar issue with Nginx server related to the keep_alive timeout .
ting chiang
Feb 11, 2021, 5:56:37 AM2/11/21
to gce-discussion
I'm using a mutiplayer game server. it written in C/C++.
There is no timeout setting.
Random disconnect only happened in tcp proxy load balancer.
If my architecture is Client -> Server or Client -> Haproxy -> Server , random disconnect would not happen.
moreno...@google.com 在 2021年2月11日 星期四下午4:56:35 [UTC+8] 的信中寫道：
Pedro Moreno
Feb 12, 2021, 3:55:35 AM2/12/21
to gce-discussion
I am a little bit confused , Please could you provide a little bit more information about the infrastructure that you are using.
How many servers are running behind the load balancer? If you have multiple servers you might be interested in using MIG to set up multiple servers.
Could you get any other log error network side related or from the game server, that could help to troubleshoot the issue?
Here there is an example of how to configure a TCP Cloud Load Balancing. Not sure if it will help you, they are talking about the timeout_sec default value equal to 5 seconds.
Another doubt that I have is that I usually understand that games traffic is usually UDP. If you are unsure about this you could check the following video of choosing the right load balancer.
ting chiang
Feb 13, 2021, 6:16:27 AM2/13/21
to gce-discussion
Thanks for your reply.
Original I'm using MIG like this
Client => tcp proxy load balancer => 2~8 linux+haproxy forwarding => game server
After found random disconnect issue, I started looking for which layer was the problem.
So I modify architecture to cross test.
1. Client => game server , no problem
2. Client => 1 linux+haproxy forwarding => game server , no problem
3. Client => tcp proxy load balancer => 1 linux+haproxy forwarding => game server , random disconnect
4. Client => tcp proxy load balancer => game server , random disconnect
If I used tcp proxy load balancer , it cause me got problem.
My client every 12 seconds will send a keep alive packet.
I also changed to 500ms~1500ms. But still random disconnect.
I capture the packet by windump.
(02/13/2021 17:45:00) start create 10 session
(02/13/2021 18:19:22) session3 disconnect
(02/13/2021 18:19:23) session4 disconnect
(02/13/2021 18:19:23) session1 disconnect
server-dump: FIN(Connection Finish) & RST(Connection reset)
client-dump: FIN(Connection Finish)
server-dump:
18:19:22.732214 IP 20-11-191-35.1e100.net.61331 > backend.c.testserver-182413.internal.9300: F 942:942(0) ack 996 win 324 <nop,nop,timestamp 143487325 1515307>
0x0000: 4560 0034 0000 4000 7f06 c1d0 23bf 0b14 E`.4..@.....#...
0x0010: 0a8c 0035 ef93 2454 9f10 1e3f cc5e 03c6 ...5..$T...?.^..
0x0020: 8011 0144 015c 0000 0101 080a 088d 715d ...D.\........q]
0x0030: 0017 1f2b ...+
18:19:22.732257 IP backend.c.testserver-182413.internal.9300 > 20-11-191-35.1e100.net.61331: . ack 943 win 512 <nop,nop,timestamp 1515426 143487325>
0x0000: 4500 0034 46af 4000 8006 0000 0a8c 0035 E..4F.@........5
0x0010: 23bf 0b14 2454 ef93 cc5e 03c6 9f10 1e40 #...$T...^.....@
0x0020: 8010 0200 39ba 0000 0101 080a 0017 1fa2 ....9...........
0x0030: 088d 715d ..q]
18:19:22.733577 IP backend.c.testserver-182413.internal.9300 > 20-11-191-35.1e100.net.61331: P 996:1008(12) ack 943 win 512 <nop,nop,timestamp 1515426 143487325>
0x0000: 4500 0040 46b0 4000 8006 0000 0a8c 0035 E..@F.@........5
0x0010: 23bf 0b14 2454 ef93 cc5e 03c6 9f10 1e40 #...$T...^.....@
0x0020: 8018 0200 39c6 0000 0101 080a 0017 1fa2 ....9...........
0x0030: 088d 715d b600 70ab 8e06 b600 7da7 8e06 ..q]..p.....}...
18:19:22.733595 IP backend.c.testserver-182413.internal.9300 > 20-11-191-35.1e100.net.61331: F 1008:1008(0) ack 943 win 512 <nop,nop,timestamp 1515426 143487325>
0x0000: 4500 0034 46b1 4000 8006 0000 0a8c 0035 E..4F.@........5
0x0010: 23bf 0b14 2454 ef93 cc5e 03d2 9f10 1e40 #...$T...^.....@
0x0020: 8011 0200 39ba 0000 0101 080a 0017 1fa2 ....9...........
0x0030: 088d 715d ..q]
18:19:22.733862 IP 20-11-191-35.1e100.net.61331 > backend.c.testserver-182413.internal.9300: R 2668633664:2668633664(0) win 0
0x0000: 4500 0028 0000 4000 7f06 c23c 23bf 0b14 E..(..@....<#...
0x0010: 0a8c 0035 ef93 2454 9f10 1e40 0000 0000 ...5..$T...@....
0x0020: 5004 0000 a514 0000 0000 0000 0000 P.............
18:19:22.733863 IP 20-11-191-35.1e100.net.61331 > backend.c.testserver-182413.internal.9300: R 2668633664:2668633664(0) win 0
0x0000: 4500 0028 0000 4000 7f06 c23c 23bf 0b14 E..(..@....<#...
0x0010: 0a8c 0035 ef93 2454 9f10 1e40 0000 0000 ...5..$T...@....
0x0020: 5004 0000 a514 0000 0000 0000 0000 P.............
18:19:23.909199 IP 29-11-191-35.1e100.net.53383 > backend.c.testserver-182413.internal.9300: F 942:942(0) ack 996 win 324 <nop,nop,timestamp 2386125925 1515007>
0x0000: 4560 0034 0000 4000 7f06 c1c7 23bf 0b1d E`.4..@.....#...
0x0010: 0a8c 0035 d087 2454 e6f8 f859 c3b1 d370 ...5..$T...Y...p
0x0020: 8011 0144 bfd5 0000 0101 080a 8e39 6465 ...D.........9de
0x0030: 0017 1dff ....
18:19:23.909235 IP backend.c.testserver-182413.internal.9300 > 29-11-191-35.1e100.net.53383: . ack 943 win 513 <nop,nop,timestamp 1515544 2386125925>
0x0000: 4500 0034 3150 4000 8006 0000 0a8c 0035 E..41P@........5
0x0010: 23bf 0b1d 2454 d087 c3b1 d370 e6f8 f85a #...$T.....p...Z
0x0020: 8010 0201 39c3 0000 0101 080a 0017 2018 ....9...........
0x0030: 8e39 6465 .9de
18:19:23.910495 IP backend.c.testserver-182413.internal.9300 > 29-11-191-35.1e100.net.53383: P 996:1008(12) ack 943 win 513 <nop,nop,timestamp 1515544 2386125925>
0x0000: 4500 0040 3151 4000 8006 0000 0a8c 0035 E..@1Q@........5
0x0010: 23bf 0b1d 2454 d087 c3b1 d370 e6f8 f85a #...$T.....p...Z
0x0020: 8018 0201 39cf 0000 0101 080a 0017 2018 ....9...........
0x0030: 8e39 6465 b600 70ab 8e06 b600 7da7 8e06 .9de..p.....}...
18:19:23.910513 IP backend.c.testserver-182413.internal.9300 > 29-11-191-35.1e100.net.53383: F 1008:1008(0) ack 943 win 513 <nop,nop,timestamp 1515544 2386125925>
0x0000: 4500 0034 3152 4000 8006 0000 0a8c 0035 E..41R@........5
0x0010: 23bf 0b1d 2454 d087 c3b1 d37c e6f8 f85a #...$T.....|...Z
0x0020: 8011 0201 39c3 0000 0101 080a 0017 2018 ....9...........
0x0030: 8e39 6465 .9de
18:19:23.910795 IP 29-11-191-35.1e100.net.53383 > backend.c.testserver-182413.internal.9300: R 3875076186:3875076186(0) win 0
0x0000: 4500 0028 0000 4000 7f06 c233 23bf 0b1d E..(..@....3#...
0x0010: 0a8c 0035 d087 2454 e6f8 f85a 0000 0000 ...5..$T...Z....
0x0020: 5004 0000 a214 0000 0000 0000 0000 P.............
18:19:23.910796 IP 29-11-191-35.1e100.net.53383 > backend.c.testserver-182413.internal.9300: R 3875076186:3875076186(0) win 0
0x0000: 4500 0028 0000 4000 7f06 c233 23bf 0b1d E..(..@....3#...
0x0010: 0a8c 0035 d087 2454 e6f8 f85a 0000 0000 ...5..$T...Z....
0x0020: 5004 0000 a214 0000 0000 0000 0000 P.............
18:19:23.927233 IP 133-0-191-35.1e100.net.59622 > backend.c.testserver-182413.internal.9300: F 942:942(0) ack 996 win 325 <nop,nop,timestamp 3116433034 1515188>
0x0000: 4560 0034 0000 4000 7f06 cc5f 23bf 0085 E`.4..@...._#...
0x0010: 0a8c 0035 e8e6 2454 c9e9 80bf 84b0 89d5 ...5..$T........
0x0020: 8011 0145 08f2 0000 0101 080a b9c0 fe8a ...E............
0x0030: 0017 1eb4 ....
18:19:23.927262 IP backend.c.testserver-182413.internal.9300 > 133-0-191-35.1e100.net.59622: . ack 943 win 512 <nop,nop,timestamp 1515546 3116433034>
0x0000: 4500 0034 7356 4000 8006 0000 0a8c 0035 E..4sV@........5
0x0010: 23bf 0085 2454 e8e6 84b0 89d5 c9e9 80c0 #...$T..........
0x0020: 8010 0200 2f2b 0000 0101 080a 0017 201a ..../+..........
0x0030: b9c0 fe8a ....
18:19:23.928320 IP backend.c.testserver-182413.internal.9300 > 133-0-191-35.1e100.net.59622: P 996:1008(12) ack 943 win 512 <nop,nop,timestamp 1515546 3116433034>
0x0000: 4500 0040 7357 4000 8006 0000 0a8c 0035 E..@sW@........5
0x0010: 23bf 0085 2454 e8e6 84b0 89d5 c9e9 80c0 #...$T..........
0x0020: 8018 0200 2f37 0000 0101 080a 0017 201a ..../7..........
0x0030: b9c0 fe8a b600 70ab 8e06 b600 7da7 8e06 ......p.....}...
18:19:23.928336 IP backend.c.testserver-182413.internal.9300 > 133-0-191-35.1e100.net.59622: F 1008:1008(0) ack 943 win 512 <nop,nop,timestamp 1515546 3116433034>
0x0000: 4500 0034 7358 4000 8006 0000 0a8c 0035 E..4sX@........5
0x0010: 23bf 0085 2454 e8e6 84b0 89e1 c9e9 80c0 #...$T..........
0x0020: 8011 0200 2f2b 0000 0101 080a 0017 201a ..../+..........
0x0030: b9c0 fe8a ....
18:19:23.928625 IP 133-0-191-35.1e100.net.59622 > backend.c.testserver-182413.internal.9300: R 3387523264:3387523264(0) win 0
0x0000: 4500 0028 0000 4000 7f06 cccb 23bf 0085 E..(..@.....#...
0x0010: 0a8c 0035 e8e6 2454 c9e9 80c0 0000 0000 ...5..$T........
0x0020: 5004 0000 28f7 0000 0000 0000 0000 P...(.........
18:19:23.928626 IP 133-0-191-35.1e100.net.59622 > backend.c.testserver-182413.internal.9300: R 3387523264:3387523264(0) win 0
0x0000: 4500 0028 0000 4000 7f06 cccb 23bf 0085 E..(..@.....#...
0x0010: 0a8c 0035 e8e6 2454 c9e9 80c0 0000 0000 ...5..$T........
0x0020: 5004 0000 28f7 0000 0000 0000 0000 P...(.........
Client side:
18:19:21.016451 IP DESKTOP-R8MXEAJ.23330 > 141.195.102.34.bc.googleusercontent.com.9300: P 912:918(6) ack 966 win 1026
0x0000: 4500 002e 166b 4000 4006 0000 c0a8 0168 E....k@.@......h
0x0010: 2266 c38d 5b22 2454 af73 0b94 c785 f729 "f..["$T.s.....)
0x0020: 5018 0402 a824 0000 d35c 9086 6701 P....$...\..g.
18:19:21.023553 IP 141.195.102.34.bc.googleusercontent.com.9300 > DESKTOP-R8MXEAJ.23330: . ack 918 win 256
0x0000: 4500 0028 cc7a 0000 3a06 0c52 2266 c38d E..(.z..:..R"f..
0x0010: c0a8 0168 2454 5b22 c785 f729 af73 0b9a ...h$T["...).s..
0x0020: 5010 0100 0d9d 0000 0000 c79a 12f0 P.............
18:19:21.023624 IP 141.195.102.34.bc.googleusercontent.com.9300 > DESKTOP-R8MXEAJ.23330: P 966:972(6) ack 918 win 256
0x0000: 4500 002e cc7b 0000 3a06 0c4b 2266 c38d E....{..:..K"f..
0x0010: c0a8 0168 2454 5b22 c785 f729 af73 0b9a ...h$T["...).s..
0x0020: 5018 0100 cc56 0000 7f00 db36 e700 P....V.....6..
18:19:21.070486 IP DESKTOP-R8MXEAJ.23330 > 141.195.102.34.bc.googleusercontent.com.9300: . ack 972 win 1026
0x0000: 4500 0028 166c 4000 4006 0000 c0a8 0168 E..(.l@.@......h
0x0010: 2266 c38d 5b22 2454 af73 0b9a c785 f72f "f..["$T.s...../
0x0020: 5010 0402 a81e 0000 P.......
18:19:22.083835 IP 141.195.102.34.bc.googleusercontent.com.9300 > DESKTOP-R8MXEAJ.23334: F 972:972(0) ack 918 win 256
0x0000: 4500 0028 cea1 0000 3a06 0a2b 2266 c38d E..(....:..+"f..
0x0010: c0a8 0168 2454 5b26 180d f88b 2a95 e9bd ...h$T[&....*...
0x0020: 5011 0100 6269 0000 0000 0922 5c3e P...bi....."\>
18:19:22.083882 IP DESKTOP-R8MXEAJ.23334 > 141.195.102.34.bc.googleusercontent.com.9300: . ack 973 win 1026
0x0000: 4500 0028 166d 4000 4006 0000 c0a8 0168 E..(.m@.@......h
0x0010: 2266 c38d 5b26 2454 2a95 e9bd 180d f88c "f..[&$T*.......
0x0020: 5010 0402 a81e 0000 P.......
18:19:22.099552 IP DESKTOP-R8MXEAJ.23334 > 141.195.102.34.bc.googleusercontent.com.9300: F 918:918(0) ack 973 win 1026
0x0000: 4500 0028 166e 4000 4006 0000 c0a8 0168 E..(.n@.@......h
0x0010: 2266 c38d 5b26 2454 2a95 e9bd 180d f88c "f..[&$T*.......
0x0020: 5011 0402 a81e 0000 P.......
18:19:22.101941 IP 141.195.102.34.bc.googleusercontent.com.9300 > DESKTOP-R8MXEAJ.23334: . ack 919 win 256
0x0000: 4500 0028 0000 4000 3a06 98cc 2266 c38d E..(..@.:..."f..
0x0010: c0a8 0168 2454 5b26 180d f88c 2a95 e9be ...h$T[&....*...
0x0020: 5010 0100 6268 0000 0000 38da a60d P...bh....8...
18:19:22.905191 IP 141.195.102.34.bc.googleusercontent.com.9300 > DESKTOP-R8MXEAJ.23317: F 972:972(0) ack 918 win 256
0x0000: 4500 0028 d072 0000 3a06 085a 2266 c38d E..(.r..:..Z"f..
0x0010: c0a8 0168 2454 5b15 5ea8 c176 d7b9 45d3 ...h$T[.^..v..E.
0x0020: 5011 0100 49ba 0000 0000 8af2 5e4a P...I.......^J
18:19:22.905230 IP DESKTOP-R8MXEAJ.23317 > 141.195.102.34.bc.googleusercontent.com.9300: . ack 973 win 1026
0x0000: 4500 0028 166f 4000 4006 0000 c0a8 0168 E..(.o@.@......h
0x0010: 2266 c38d 5b15 2454 d7b9 45d3 5ea8 c177 "f..[.$T..E.^..w
0x0020: 5010 0402 a81e 0000 P.......
18:19:22.916524 IP DESKTOP-R8MXEAJ.23317 > 141.195.102.34.bc.googleusercontent.com.9300: F 918:918(0) ack 973 win 1026
0x0000: 4500 0028 1670 4000 4006 0000 c0a8 0168 E..(.p@.@......h
0x0010: 2266 c38d 5b15 2454 d7b9 45d3 5ea8 c177 "f..[.$T..E.^..w
0x0020: 5011 0402 a81e 0000 P.......
18:19:22.919801 IP 141.195.102.34.bc.googleusercontent.com.9300 > DESKTOP-R8MXEAJ.23317: . ack 919 win 256
0x0000: 4500 0028 0000 4000 3a06 98cc 2266 c38d E..(..@.:..."f..
0x0010: c0a8 0168 2454 5b15 5ea8 c177 d7b9 45d4 ...h$T[.^..w..E.
0x0020: 5010 0100 49b9 0000 0000 eb45 5114 P...I......EQ.
18:19:22.975858 IP 141.195.102.34.bc.googleusercontent.com.9300 > DESKTOP-R8MXEAJ.23330: F 972:972(0) ack 918 win 256
0x0000: 4500 0028 d0a4 0000 3a06 0828 2266 c38d E..(....:..("f..
0x0010: c0a8 0168 2454 5b22 c785 f72f af73 0b9a ...h$T[".../.s..
0x0020: 5011 0100 0d96 0000 0000 b8a9 68e8 P...........h.
18:19:22.975895 IP DESKTOP-R8MXEAJ.23330 > 141.195.102.34.bc.googleusercontent.com.9300: . ack 973 win 1026
0x0000: 4500 0028 1671 4000 4006 0000 c0a8 0168 E..(.q@.@......h
0x0010: 2266 c38d 5b22 2454 af73 0b9a c785 f730 "f..["$T.s.....0
0x0020: 5010 0402 a81e 0000 P.......
18:19:22.983389 IP DESKTOP-R8MXEAJ.23330 > 141.195.102.34.bc.googleusercontent.com.9300: F 918:918(0) ack 973 win 1026
0x0000: 4500 0028 1672 4000 4006 0000 c0a8 0168 E..(.r@.@......h
0x0010: 2266 c38d 5b22 2454 af73 0b9a c785 f730 "f..["$T.s.....0
0x0020: 5011 0402 a81e 0000 P.......
18:19:22.987719 IP 141.195.102.34.bc.googleusercontent.com.9300 > DESKTOP-R8MXEAJ.23330: . ack 919 win 256
0x0000: 4500 0028 0000 4000 3a06 98cc 2266 c38d E..(..@.:..."f..
0x0010: c0a8 0168 2454 5b22 c785 f730 af73 0b9b ...h$T["...0.s..
0x0020: 5010 0100 0d95 0000 0000 1394 982e P.............
moreno...@google.com 在 2021年2月12日 星期五下午4:55:35 [UTC+8] 的信中寫道：
mfl...@google.com
Feb 15, 2021, 1:51:50 PM2/15/21
to gce-discussion
As this is a technical very specific use-case question, and as this forum is meant for general questions about the platform, please post your question and all information you consider may help to better understand the behavior you are noticing at serverfault.com where you have access to a large community of enthusiasts and experts to share ideas with and get support from.
Google group is a discussion forum where you would likely find information like service status updates and release notes, and ranging from book recommendations to creative shortcuts[1]. Considering this, it might not be the right platform to report this type of situation.
[1] Google Groups:
https://cloud.google.com/support/docs/community#discussions_google_groups
Reply all
Reply to author
Forward
0 new messages