About iterators and memory deallocation

[Nathann Cohen]

Hello everybody,

I come to you with a problem, without knowing what a Cython answer might be.

iterators that leak memory:

I often have to create iterators (i.e. use the yield keyword) in a function
that allocates heap memory. At the end of the function I can free it, but of
course that part of the code is never read if the iterator is not read
"until the end". Thus, iterators that are not read to their end never free
their memory.

def my_iterator():
<allocate_memory>
while True:
<do_something>
yield <something>
<deallocate_memory>

Usual workaround:

This problem can be solved by creating a custom class, with a `__cinit__`
(to allocate the memory) function, a `__iter__` function (which contains the
yield) and a `__dealloc__` (to free the memory) to replace the iterator. But
that is seriously more code, more doctests, more documentation to write when
all we need is a 'free'.

It is easier to deal with exceptions:

When a code that allocates memory is interrupted by an exception, it is not
a problem to free the memory. For instance:

def my_function():
try:
<allocate memory>
<do whatever you like>
finally:
<free memory>

Would it be possible to have a similar syntax for iterators? Would it be
possible that the iterator function itself would trigger a
"IteratorIsBeingDeallocatedException", that we could use to deallocate memory?

I really have no idea how this could be made available. I just thought that the
exception mechanism made things so simple that it could be used to solve this
problem.

Thaaaaaaaaaaanks,

Nathann

[Stefan Behnel]

The canonical Python answer to this problem is to call the close() method
when you're done with it, which will then raise a GeneratorExit exception
in the generator's current yield position.

Stefan

[Nathann Cohen]

Hellooooooooooooo,

The canonical Python answer to this problem is to call the close() method
when you're done with it, which will then raise a GeneratorExit exception
in the generator's current yield position.

And it seems that this .close() function is also called by Python when it gets rid of the generator object. Sigh. Sorry for the loss of time, and thank you very very very much for your answer!

Nathann