Okay, I got a problem now.

Basically, I wanted to simplify the usage of the yielded generator in the generator itself. The value of the first yield in the generator (=the generator itself) had basically the only usages to specify one if its method as parameter and the need to use get_next_method was annoying.

Since you can not set attributes of a generator I decided to send a wrapper of the generator instead. I decided to use a class because of two reasons:

I could define call with the util function that automatically calls next(generator) or generator.send(value) it it was called with a paratemer. I could optionally catch StopIteration exceptions so that the wrapper doesn’t have to end with yield . I also suspected that this could cause an issue with the generator never being gc’ed, but read on.

However, now I have a problem with garbage collecting and the weak reference of the generator. My theory is that generator.send (and generator.next or next) reference the original generator so that any function that currently holds it as callback will keep a reference on it until it finishes. The other part of this theory is that for as long as the generator is running, Python keeps a reference to it internally so that it doesn’t just stop and get gc’ed while running. This needs to happen because otherwise your initial proposal would fail at the second yield, which is after _send_self terminates and the refcount in code is reduced to 0.

So, because I pass the (weak reference) to the wrapper I now disturb this system by introducing a wrapper to the send and next methods. The problem is that the generator seems to be gc’ed the moment when I pass the wrapper with the weak reference as a callback to a function and wait in a yield . That results in a ReferenceError once the function attempts to call the callback.

Now, I could pass the non-proxied generator to the wrapper but that would mean unless the generator itself terminates (or removes the reference to the wrapper) it won’t be gc’ed. That ultimately leads to memory a leak when a function terminates and does not call the callback.

Solution idea: I need to somehow keep a weak reference of the generator in the generator itself but create a strong reference when I pass one of its attributes as callback. Any ideas?

Here is my current code:

# Based on https://gist.github.com/Varriount/aba020b9d43c13d2794b from weakref import proxy from functools import wraps class GeneratorWrapper(): """TODOC """ # An instance of this will be sent to the generator function. # "cont" holds a wrapper function that calls either next(gen) or gen.send(val). # Save some overhead here. Not exactly needed but you shouldn't mess with # instances of this class, only use or call. __slots__ = ('generator', 'catch_stopiteration') def __init__(self, generator, catch_stopiteration=True): self.generator = generator self.catch_stopiteration = catch_stopiteration def cont(self, value=None): try: if value is not None: return self.generator.send(value) else: return next(self.generator) except StopIteration: if self.catch_stopiteration: return None else: raise def throw(self, *args, **kwargs): if self.catch_stopiteration: try: return self.generator.throw(*args, **kwargs) except StopIteration: return None else: self.generator.throw(*args, **kwargs) def close(self): if self.catch_stopiteration: try: return self.generator.close() except StopIteration: return None else: self.generator.close() __call__ = cont def send_self(use_proxy=True, catch_stopiteration=True): """Decorator that sends a generator a wrapper of itself. The returned function instantiates and sends a generator a wrapper of itself via the first 'yield' used. The wrapper is an instance of GeneratorWrapper. Useful for creating generators that can leverage callback-based functions in a linear style, by passing the wrapper as callback in a yield statement. The generator wrapper wraps a weakly referenced proxy by default. To override this set use_proxy to `False`. The wrapper catches StopIteration exceptions by default. If you wish to have them propagated, set catch_stopiteration to `False`. """ # "use_proxy" needs to be the name of the first parameter. For clarity, we # mirror that to _first_param and override use_proxy later. _first_param = use_proxy # We either directly call this, or return it to be called by python's # decorator mechanism. def _send_self(func): @wraps(func) def send_self_wrapper(*args, **kwargs): nonlocal use_proxy, catch_stopiteration # optional but for clarity generator = func(*args, **kwargs) next(generator) # Start the generator. # Send generator wrapper to the generator. generator_ref = proxy(generator) if use_proxy else generator gen_wrapper = GeneratorWrapper(generator_ref, catch_stopiteration) generator.send(gen_wrapper) # = gen_wrapper(gen_wrapper) # gen_wrapper(proxy(gen_wrapper) if use_proxy else gen_wrapper) return send_self_wrapper # If the argument is a callable, we've been used without being directly # passed an argument by the user, and thus should call _send_self directly. if callable(_first_param): # No arguments, this is the decorator. use_proxy = True return _send_self(_first_param) else: # Someone has called @send_self(...) with parameters and thus we need to # return _send_self to be called indirectly. use_proxy = _first_param return _send_self

Edit: Found a work around and getting close.