Auto join() or asyncjoin()

[Sergiy Lozovsky]

Hi,

is it possible to avoid using join()? In my project greenlets are started on demand, do they job and has nothing meaningful to report back to the join().

Greenlets are good for async tasks, but join() requires code to be synchronous - stay and wait for greenlets to join, which doesn't make any sense if the return value is not used.

If join() is not used the code will leak memory as a structure associated with agreenlet will not be freed without explicit join().

Is there a way to avoid this synchronous join()? Or make it event driven - join() would be called when the greenlet ends to free the memory?

Thanks,

Sergiy. 

[Jason Madden]

Can you provide a reference for that? I don’t think that’s true. Greenlets may be prematurely collected under cpython, but as long as they don’t participate in reference cycles I don’t  think they leak if you don’t join them. 

--
You received this message because you are subscribed to the Google Groups "gevent: coroutine-based Python network library" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gevent+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Grady Player]

I am unaware of a memory issue with non joined greenlets...

you could make a wrapper function that adds the greenlet to a global done pool after the action is complete that could be periodically joined ... it shouldn't take any time to join a completed greenlet...

[Sergiy Lozovsky]

This the test:

import gevent

import os

_proc_status = '/proc/%d/status' % os.getpid()

_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,

          'KB': 1024.0, 'MB': 1024.0*1024.0}

def _VmB(VmKey):

    '''Private.

    '''

    global _proc_status, _scale

     # get pseudo file  /proc/<pid>/status

    try:

        t = open(_proc_status)

        v = t.read()

        t.close()

    except:

        return 0.0  # non-Linux?

     # get VmKey line e.g. 'VmRSS:  9999  kB\n ...'

    i = v.index(VmKey)

    v = v[i:].split(None, 3)  # whitespace

    if len(v) < 3:

        return 0.0  # invalid format?

     # convert Vm value to bytes

    return float(v[1]) * _scale[v[2]]

def memory(since=0.0):

    '''Return memory usage in bytes.

    '''

    return _VmB('VmSize:') - since

def resident(since=0.0):

    '''Return resident memory usage in bytes.

    '''

    return _VmB('VmRSS:') - since

def stacksize(since=0.0):

    '''Return stack size in bytes.

    '''

    return _VmB('VmStk:') - since

def func(a):

    #print "Func is done"

    return a + 1

sn = memory()

print sn

for x in range(0, 3000):

    #ge = gevent.spawn_later(10, func, 5)

    ge = gevent.spawn(func, 5)

    #gevent.sleep(5)

    #ge.kill()

    #ge.join()

gevent.sleep(5)

print "memdiff:", memory(sn)

With looping 3 times:

$ python test10.py 

129015808.0

memdiff: 4546560.0

Looping 3000 times:

$ python test10.py 

129011712.0

memdiff: 11939840.0

5 seconds is more than enough for all greenlets to end, but memdiff grows with the number of greenlets. 

There is a question of calculating the memory used. Code in the test is probably not perfect (Linux process doesn't return memory to the system). Are there better ways to find if all greenlet memory is freed after it is ended and there are no references to it?

[Sergiy Lozovsky]

It's an additional headache. Service is constantly running spawning greenlets from time to time. All I need actually is to execute some function later on - spawn_later(). From what I understand - underlying greenlets don't have join() at all. join() is introduced by gevent. It can be useful in some cases, but gets in a way in many others. As I understand Python threading has an option "daemon" which clears memory used by a thread without join().

[Jason Madden]

> On Jan 24, 2018, at 17:42, Sergiy Lozovsky <sergiy....@gmail.com> wrote:
>
>
> With looping 3 times:
>
> $ python test10.py
> 129015808.0
> memdiff: 4546560.0
>
> Looping 3000 times:
>
> $ python test10.py
> 129011712.0
> memdiff: 11939840.0
>
> 5 seconds is more than enough for all greenlets to end, but memdiff grows with the number of greenlets.

I believe what you're seeing is simply the internals of CPython getting warmed up. Things like the tuple and list and frame object freelists are getting populated, the integer object cache being populated, things like that. Not to mention the C standard library data, shared objects being mapped in (and copied to physical memory if written to), things like that. If that's true, then eventually the process will reach a steady state. One iteration of the outer loop won't account for that.

> There is a question of calculating the memory used. Code in the test is probably not perfect (Linux process doesn't return memory to the system). Are there better ways to find if all greenlet memory is freed after it is ended and there are no references to it?

I recommend the portable `psutil` and `objgraph` libraries, the former for operating system level stats, the later for Python stats.

I wrote a script using them to examine this behaviour in more depth. It's down below, but I want to go over the output first. The script collects and prints memory and object snapshots before doing *anything*, after spawning 3000 Greenlets, and then after sleeping to let those greenlets run. It does this repeatedly. Every 10 runs, it shows the difference since the script was started.

Here's the first iteration on macOS 10.13.2 with CPython 3.6.4:

>> ============================
>> Beginning iteration 0
>> ===============
>> Before spawn
>> Num greenlets 1
>> Num Greenlets 0
>> Delta RSS 0.00390625
>> Delta VMS 0.0
>> ===============
>> Before sleep
>> Num greenlets 1
>> Num Greenlets 3000
>> Delta RSS 1.7578125
>> Delta VMS 9.5
>> ===============
>> After sleep
>> Num greenlets 1
>> Num Greenlets 0
>> Delta RSS 0.015625
>> Delta VMS 7.75
>> ============================
>>
>> ===============
>> Since beginning at 0
>> Num greenlets 1
>> Num Greenlets 0
>> Delta RSS 2.0546875
>> Delta VMS 17.25

Before we sleep you can see that we've added 3000 Greenlet objects, exactly as expected. The RSS (non-swapped physical memory) has gone up by 1.75MB, and the total virtual memory (VMS) has gone up by 9.5MB. That's more or less expected too. After we sleep and the greenlets have exited, we can see that the are no longer found as Python objects, and yet our VMS has increased more. That's likely the result of mapping in the code needed to execute the greenlets. In total, we've added 2.05MB of physical memory and 17MB of virtual address space since the process started.

The second, third and fourth iterations show small increases in RSS and VMS, but by the fifth iteration, we've reached steady state:

>> ============================
>> Beginning iteration 4
>> ===============
>> Before spawn
>> Num greenlets 1
>> Num Greenlets 0
>> Delta RSS 0.0
>> Delta VMS 0.0
>> ===============
>> Before sleep
>> Num greenlets 1
>> Num Greenlets 3000
>> Delta RSS 0.0
>> Delta VMS 0.0
>> ===============
>> After sleep
>> Num greenlets 1
>> Num Greenlets 0
>> Delta RSS 0.0
>> Delta VMS 0.0
>> ============================

The final report shows us very similar to where we were after our first iteration (after accounting for the small increases in the next few iterations):

>> ===============
>> Final report
>> Num greenlets 1
>> Num Greenlets 0
>> Delta RSS 3.38671875
>> Delta VMS 17.5

So if greenlets leak, it's an immeasurably small one :)

YMMV, of course, and your numbers won't match mine and they may vary substantially between runs on one machine (for example, a separate run of the test reported only 1.8 and 1.25 MB increases in RSS and VMS, respectively, after the first iteration), but I think the overall trend to steady state should be the same.

(On Ubuntu 16.04 with CPython 3.5.2 steady state looks different; the deltas added are exactly offset by the deltas removed:

>> ============================
>> Beginning iteration 40
>> ===============
>> Before spawn
>> Num greenlets 1
>> Num Greenlets 0
>> Delta RSS 0.0
>> Delta VMS 0.0
>> ===============
>> Before sleep
>> Num greenlets 1
>> Num Greenlets 3000
>> Delta RSS 0.51171875
>> Delta VMS 0.65234375
>> ===============
>> After sleep
>> Num greenlets 1
>> Num Greenlets 0
>> Delta RSS -0.51171875
>> Delta VMS -0.65234375
>> ============================
)

There are occasionally things I cannot explain. For example, in a separate run of the test, I saw a large increase in VMS at iteration 15:

>> ============================
>> Beginning iteration 15
>> ===============
>> Before spawn
>> Num greenlets 1
>> Num Greenlets 0
>> Delta RSS 0.0
>> Delta VMS 0.0
>> ===============
>> Before sleep
>> Num greenlets 1
>> Num Greenlets 3000
>> Delta RSS 0.26953125
>> Delta VMS 9.0
>> ===============
>> After sleep
>> Num greenlets 1
>> Num Greenlets 0
>> Delta RSS 0.0
>> Delta VMS 0.0
>> ============================


I've no idea what happened there. Maybe a Python process in the background exited and we'd been sharing mappings with it, and now that they were no longer shared they were accounted for here? No clue.

Jason

PS: The test script:

>> import objgraph
>> import psutil
>> import gevent
>>
>> # Warm up the hub. We
>> # should now have one greenlet
>> gevent.sleep(1)
>>
>> proc = psutil.Process()
>>
>> def f(x):
>> return x + 1
>>
>> def report(title, mem_before):
>> mem_now = proc.memory_info()
>>
>> print("===============")
>> print(title)
>> print("Num greenlets", objgraph.count('greenlet'))
>> print("Num Greenlets", objgraph.count('Greenlet'))
>> print("Delta RSS ", (mem_now.rss - mem_before.rss) / 1024 / 1024)
>> print("Delta VMS ", (mem_now.vms - mem_before.vms) / 1024 / 1024)
>> return mem_now
>>
>> def spawn(iter_number, mem_before):
>> print("============================")
>> print("Beginning iteration %s" % (iter_number))
>> mem_now = report("Before spawn", mem_before)
>>
>> for i in range(3000):
>> gevent.spawn(f, i)
>>
>> mem_after_spawn = report("Before sleep", mem_now)
>> gevent.sleep(3)
>> mem_after_sleep = report("After sleep", mem_after_spawn)
>> print("============================")
>> return mem_after_sleep
>>
>> def test():
>> ultimate_mem_before = mem_before = proc.memory_info()
>>
>> for iter_number in range(50):
>> mem_before = spawn(iter_number, mem_before)
>> if iter_number % 10 == 0:
>> print()
>> report("Since beginning at %i" % (iter_number), ultimate_mem_before)
>> print()
>>
>> print()
>> report("Final report", ultimate_mem_before)
>>
>>
>> if __name__ == '__main__':
>> test()

[Sergiy Lozovsky]

Hi Jason,

thanks for the detailed analysis. It looks like resources are freed. There would be a steady memory growth otherwise.

At the same time, does anyone know how gevent internals work? That would give the definite answer (in case if there are no bugs causing leaking :-).

I looked at at geven sources, but can't claim that understand it well enough. It would be nice to update documentation on what join() does exactly.

Thanks,

Sergiy.

[Jason Madden]

> On Jan 25, 2018, at 13:03, Sergiy Lozovsky <sergiy....@gmail.com> wrote:
>
> At the same time, does anyone know how gevent internals work? That would give the definite answer (in case if there are no bugs causing leaking :-).

As the most active maintainer of gevent, I certainly hope I know how gevent internals work, at least more-or-less. If not, maybe I should rethink my plans to release 1.3a1 soon :)

> I looked at at geven sources, but can't claim that understand it well enough. It would be nice to update documentation on what join() does exactly.
>

The documentation for Greenlet.join says "Wait until the greenlet finishes or timeout expires. Return None regardless." As far as I can tell, that's pretty much exactly what it does. If you can think of an important clarification, I would review a PR.

Jason

[Sergiy Lozovsky]

Hi Jason,

I didn't know that you are gevent maintainer. Sorry.

In that case I can take your word without any tests (which were used to figure out greenlet memory management in empirical way without understanding of the internals). Is it true that join() is not required to clean up the memory after greenlet was ended and the reference to it was cleared?

As for documentation - if I didn't miss anything ALL examples use join(). It makes it look as join() is required for proper greenlet lifecycle.

It can be useful to add such clarification, that use of join() is optional.

Thanks,

Sergiy.

[Jason Madden]

> On Jan 25, 2018, at 14:31, Sergiy Lozovsky <sergiy....@gmail.com> wrote:
>
> I didn't know that you are gevent maintainer. Sorry.

No worries!

>
> In that case I can take your word without any tests (which were used to figure out greenlet memory management in empirical way without understanding of the internals). Is it true that join() is not required to clean up the memory after greenlet was ended and the reference to it was cleared?

Yes, that is true. Greenlet.join() has nothing to do with freeing memory.

> As for documentation - if I didn't miss anything ALL examples use join(). It makes it look as join() is required for proper greenlet lifecycle.

Much of the examples are also tests, so it's important that things are in a known state, hence the use of join().

join is also useful because it ensures that the greenlet object is still alive until it's done. The greenlet docs[1] spell out that if all references to a greenlet go away while it is running, it will be killed (that's what I meant by "Greenlets may be prematurely collected"). That's almost never what you want, so it's a common practice to make sure that you keep a reference to the greenlet so it doesn't happen---calling join() gives that reference something to do (in addition to actually joining the greenlet, of course, which some algorithms require).

Now, gevent is set up in such a way that you can usually ignore that if you're using gevent APIs. Spawning a greenlet makes gevent keep a reference to it until it starts running, blocking for IO or a lock or what have you also keeps references. But if you ever drop down to the low-level greenlet APIs, that's something you have to remember.

[1] http://greenlet.readthedocs.io/en/latest/#garbage-collecting-live-greenlets

> It can be useful to add such clarification, that use of join() is optional.

I think that's pretty much the default for *every* API: Unless otherwise documented as being necessary, it's optional, called if needed. Otherwise every program would have to call every API, which doesn't make much sense. :)

[Sergiy Lozovsky]

Thanks for the clarification. You refer to greenlet documentation. My understanding is that gevent is an additional abstraction layer on top of greenlets and could add additional allocated structures.