[long STW] I have a problem with STW, which PauseNs max > 4s

[kcn]

go version = 1.9

system: centos6

cpu: Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz

ram: 256G

When I just start the program is normal, but after a week, I have got a serious problem.

use this command:

curl -s http://localhost:6060/debug/pprof/heap?debug=1 | grep -A 100 runtime.MemStats

# runtime.MemStats

# Alloc = 9152929216

# TotalAlloc = 24497596006112

# Sys = 14643025808

# Lookups = 24640021

# Mallocs = 314983702799

# Frees = 314890606713

# HeapAlloc = 9152929216

# HeapSys = 13834256384

# HeapIdle = 3843006464

# HeapInuse = 9991249920

# HeapReleased = 784695296

# HeapObjects = 93096086

# Stack = 14942208 / 14942208

# MSpan = 143016040 / 172310528

# MCache = 27776 / 131072

# BuckHashSys = 9887243

# GCSys = 577380352

# OtherSys = 34118021

# NextGC = 12637023712

# LastGC = 1576051983906690682

# PauseNs = [203828819 833122 44497010 159325516 319338315 375776346 208292466 133034718 141559087 247169403 224303382 512235442 165825001 89714485 129990939 148599231 490780744 133338423 154782262 812745 60532623 436341938 238022496 319964271 454474982 518773328 332954413 368718104 858266 457445676 464195037 834154 505901643 936110 3611034 495067537 289896022 524796682 140914877 848957 168438529 318007899 404385567 431675930 288005687 177508673 978077 91788246 340347753 540147043 421783343 264586748 90422906 6624034 417814921 53855855 453283135 40211925 118634926 96312914 368066690 235327760 100536648 1045337 257594391 1074007 66337537 142560983 480373240 392492774 418955865 32178967 26348115 293009917 394083984 219093549 803052 182271365 10284976 534806307 346830504 500074245 1021712 322887198 162602787 457591186 958799 283020383 28184437 345535081 828362 2446129720 889084 441424548 50855043 205107794 733906 415573895 1017907 455905395 879362 380071322 66247272 476012181 877939 521751696 1032588 30851995 485583989 2458300127 115004157 437713696 230794838 810304 246137432 939081 312121432 903018 951949 93242582 464245951 798639 288326403 235891135 121719955 64269077 815271 328020282 520650549 223382584 1004947 199729943 889266 278668733 432818802 98651203 311842240 499257411 260718907 269543337 819032 419187197 73016408 844057 396913394 117294816 96768203 277352403 235586928 349492503 880960 244180721 138064221 257537378 336688385 158606093 2762971 931411 2478892018 417227872 220396927 2446837344 314316950 54964367 67772899 153282247 1047737 208754885 919618 82454641 69999744 341117367 486141465 430776285 895392 135104892 954775 303320998 293714345 174196878 340615474 261347792 345574367 501635317 181690032 85475951 496206341 493427235 451805458 445636806 258290995 83218325 343698814 2502440845 81088232 459480913 368038349 42508786 508837597 326986278 461034688 72136756 957324 411684055 494013055 927757 95024833 208899784 326070156 315869167 129176389 1243124 483039370 453715852 815319 444773746 222427607 273437516 9229028 978643 356583344 359076061 499989934 438760605 791907 29874462 821385 260563387 318759644 302369941 213534383 43918041 101796913 104227525 324251770 38787048 159455648 409513370 3015769897 87574452 281620809 995836 142908763 382185848 9009796 334681354 142779759 100080809 312912939 190165085 319839487 127021161 1058189 469217422 929459 93705930]

My max PauseNs is large than 4s. I know golang's STW is very efficient，but I don't know what happens about this, when I restart the program, everything returns normal. 

Before and after a restart, the inuse_objects and  inuse_space is same as the alloc_objects and alloc_space.

[Wojciech S. Czarnecki]

On Wed, 11 Dec 2019 00:20:16 -0800 (PST)
kcn <xvsf...@gmail.com> wrote:

> go version = 1.9
This Go version is no longer supported. That's why you did not get other answers yet.

> I know golang's STW is very efficient，but I don't know what happens about this,
> when I restart the program, everything returns normal.

Go runtime garbage collector has improved much since 1.9. and way too many changes
went to the runtime since 1.9 to have someone investigate further.

> when I restart the program, everything returns normal.

A good news, likely. Get cron do that restart for you :)

The second and more sane solution is to recompile with current Go version and
raise issue if your core problem would persist.

Hope this helps,

--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE

[kcn test]

Thank you for your help! 

Wojciech S. Czarnecki <oh...@fairbe.org> 于2019年12月12日周四 下午4:30写道：