Upgrade grpcio (+grpcio-tools) from 1.48.1 to 1.54.2 became the reason massive memory leak

[Jerry]

Some instances of my app, green and blue full rollback to prev version, for yellow - downgrade  grpcio (+grpcio-tools) to 1.48.1 only. Trying other versions older than 1.48.1 - the same result - massive memory leak.

If allowed, I can provide a link to my project on GitHub

Regard's, Jerry.

[Richard Belleville]

> If allowed, I can provide a link to my project on GitHub

Absolutely. Please share. There's not much to investigate given just the information so far.

> Trying other versions older than 1.48.1 - the same result - massive memory leak.

Is this a typo? Do you mean versions newer than 1.48.1 show a memory leak? Or do you really mean "older."

[Jerry]

Richard hi!

Thanks for reaction.

> Trying other versions older than 1.48.1 - the same result - massive memory leak.

Is this a typo? Do you mean versions newer than 1.48.1 show a memory leak? Or do you really mean "older."

Sorry, it's my typo, really I try versions newer  than 1.48.1

App for this trouble. It's stable release, I'm use it in product without any issues with memory leak.

At 1.3.0b11 I'm update ```requirements.txt``` for last version used packages:

and began to have problems with memory leakage.

After some investigations and downgrading ver. for:

```

grpcio==1.48.1
grpcio-tools==1.48.1

```

everything became OK.

I have no problem with version 1.48.1

But gRPC is a cool project, I want to help figure out what the reason is.  Get in touch, I'll do my best.

Regards, Jerry.

[Jerry]

Hi! I'm upgrade gprcio to 1.56.0 and have not any problem with memory.

Perhaps initially the problem was caused by what I try used with version 1.54.2 protobuff compiled grpc-tool v1.48.1

Regard's, Jerry.

среда, 7 июня 2023 г. в 21:39:00 UTC+3, Jerry:

[Jerry]

Hi!

Problem with memory leak on grpcio after 1.48.1 persistent.

OS - Ubuntu 22.04, The blue and green line it is martin-binance 1.3.4 prod instance, where grpcio==1.48.1. It is stable. For yellow instance I try upgrade grpcio to 1.57.0 and get problem with memory.

And it is not in Python code:

```

def heap():
    h = guppy.hpy()
    logger.info(str(h.heap()))

```

```

[2023-08-31 13:13:17,863: INFO] Partition of a set of 374326 objects. Total size = 45706311 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  48350  13 14826776  32  14826776  32 dict (no owner)
     1 172365  46 12183673  27  27010449  59 str
     2  41081  11  2909376   6  29919825  65 tuple
     3  10939   3  2108276   5  32028101  70 types.CodeType
     4   1840   0  1709432   4  33737533  74 type
     5  19872   5  1676867   4  35414400  77 bytes
     6  10684   3  1538496   3  36952896  81 function
     7  14518   4  1477488   3  38430384  84 list
     8   1840   0   926008   2  39356392  86 dict of type
     9    453   0   708800   2  40065192  88 dict of module
<641 more rows. Type e.g. '_.more' to view.>

```

and

```

def pstats():
    process = psutil.Process(os.getpid())
    logger.info(
        {"rss": f"{process.memory_info().rss / 1024 ** 2:.2f} MiB",
         "vms": f"{process.memory_info().vms / 1024 ** 2:.2f} MiB",
         "shared": f"{process.memory_info().shared / 1024 ** 2:.2f} MiB",
         "open file descriptors": process.num_fds(),
         "threads": process.num_threads()}
    )

```

return

```

[2023-08-31 13:13:17,865: INFO] {'rss': '1208.87 MiB', 'vms': '2553.13 MiB', 'shared': '23.82 MiB', 'open file descriptors': 135, 'threads': 12}

```

After I try use `valgring` and get:

```

==1318226== 7,374,928 bytes in 11 blocks are possibly lost in loss record 1,264 of 1,264
==1318226==    at 0x48487A9: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1318226==    by 0x83FF781: _upb_Arena_SlowMalloc (in /home/ubuntu/.local/lib/python3.10/site-packages/google/_upb/_message.abi3.so)
==1318226==    by 0x83D9F33: PyUpb_PyToUpb (in /home/ubuntu/.local/lib/python3.10/site-packages/google/_upb/_message.abi3.so)
==1318226==    by 0x83E4CC4: PyUpb_RepeatedScalarContainer_Append (in /home/ubuntu/.local/lib/python3.10/site-packages/google/_upb/_message.abi3.so)
==1318226==    by 0x273F89: ??? (in /usr/bin/python3.10)
==1318226==    by 0x250F51: _PyEval_EvalFrameDefault (in /usr/bin/python3.10)
==1318226==    by 0x28587F: ??? (in /usr/bin/python3.10)
==1318226==    by 0x36213E: ??? (in /usr/bin/python3.10)
==1318226==    by 0x4036FA: ??? (in /usr/bin/python3.10)
==1318226==    by 0x7684DD8: ??? (in /home/ubuntu/.local/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so)
==1318226==    by 0x7684DA7: ??? (in /home/ubuntu/.local/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so)
==1318226==    by 0x7684DA7: ??? (in /home/ubuntu/.local/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-x86_64-linux-gnu.so)
==1318226==
==1318226== LEAK SUMMARY:
==1318226==    definitely lost: 2,000 bytes in 23 blocks
==1318226==    indirectly lost: 5,440 bytes in 17 blocks
==1318226==      possibly lost: 19,236,245 bytes in 428 blocks
==1318226==    still reachable: 6,232,524 bytes in 11,413 blocks
==1318226==                       of which reachable via heuristic:
==1318226==                         stdstring          : 3,949 bytes in 72 blocks
==1318226==                         newarray           : 1,416 bytes in 1 blocks
==1318226==         suppressed: 0 bytes in 0 blocks
==1318226== Reachable blocks (those to which a pointer was found) are not shown.
==1318226== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==1318226==
==1318226== For lists of detected and suppressed errors, rerun with: -s
==1318226== ERROR SUMMARY: 155 errors from 155 contexts (suppressed: 0 from 0)

```

In order to get a more specific result, I need to use the grpcio assembly with the debug switch -g
 I am trying to figure out how to do this.  This is not my area, maybe you can tell me how to move on.

Regard's, Jerry.

среда, 19 июля 2023 г. в 12:14:30 UTC+3, Jerry:

[Jerry]

Ok, I'm build from source following the directions in this link: https://github.com/grpc/grpc/tree/master/src/python/grpcio

When compile use command `GRPC_PYTHON_BUILD_WITH_CYTHON=1 GRPC_PYTHON_ENABLE_CYTHON_TRACING=1 pip install .`

I hope this is correct.

Then I run my app under valgrind: `$valgrind --leak-check=full --show-leak-kinds=all  --log-file=valgrind-out.txt exchanges-wrapper-srv`

and get valgrind log, it is begin and end:

==468238== Memcheck, a memory error detector
==468238== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==468238== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==468238== Command: /home/ubuntu/.local/bin/exchanges-wrapper-srv
==468238== Parent PID: 461924
==468238==
==468238== Warning: set address range perms: large range [0x2cd64040, 0x3ed64030) (undefined)
==468238== Warning: set address range perms: large range [0x59c87040, 0x7dc87030) (undefined)
==468238== Warning: set address range perms: large range [0x59c87028, 0x7dc87048) (noaccess)
==468238== Warning: set address range perms: large range [0x2cd64028, 0x3ed64048) (noaccess)
==468238==
==468238== HEAP SUMMARY:
==468238==     in use at exit: 3,188,356 bytes in 3,123 blocks
==468238==   total heap usage: 40,205,330 allocs, 40,202,207 frees, 461,970,792,989 bytes allocated

...

==468238== LEAK SUMMARY:
==468238==    definitely lost: 4,560 bytes in 43 blocks
==468238==    indirectly lost: 0 bytes in 0 blocks
==468238==      possibly lost: 274,971 bytes in 73 blocks
==468238==    still reachable: 2,908,825 bytes in 3,007 blocks
==468238==                       of which reachable via heuristic:
==468238==                         newarray           : 1,416 bytes in 1 blocks
==468238==         suppressed: 0 bytes in 0 blocks

The full log there: https://drive.google.com/file/d/1LX553LqJ_91VhyLvNIyI5DVrnKVeJ3ME/view?usp=sharing

What else can I do to get this problem resolved?

Regard's, Jerry.

пятница, 1 сентября 2023 г. в 14:23:24 UTC+3, Jerry:

[Jerry]

Hi!

Tried combination `Python3.11` + `grpcio 1.58.0rc1 ` and get the same result:

Regards, Jerry.

понедельник, 4 сентября 2023 г. в 15:21:48 UTC+3, Jerry: