Enclosing an iterable in a generator expression changes the order that contexts are cleaned up

Question

I have a Python application which has two nested with blocks, one of which occurs inside an object's __iter__ method. I have observed that wrapping the iterable object inside a generator expression changes the order these with blocks are finalised when an exception is raised, and I am very surprised by this.

I am using python 3.11.5 but have observed the same behaviour in 3.8.17.

Here is a demonstrator which reproduces the observed behaviour as well as the expected behaviour:

from contextlib import closing, contextmanager


class Reader:
    def close(self):
        print(f"CLOSE: {self.__class__.__name__}")

    @contextmanager
    def get_resource(self):
        try:
            yield 42
        finally:
            print(f"CLOSE: {self.__class__.__name__} releases resource")


class Container:
    def __init__(self, reader: Reader):
        self._reader = reader

    def __iter__(self):
        with self._reader.get_resource() as resource:
            for x in range(10):
                yield x


def my_code():
    with closing(Reader()) as reader:
        container = Container(reader)
        container_gen = (x for x in container)
        for thing in container_gen:
            assert False


def expected():
    with closing(Reader()) as reader:
        container = Container(reader)
        for thing in container:
            assert False


print(">>> What my code does:")
try:
    my_code()
except AssertionError:
    print("handle exception")

print()

print(">>> The context handling I expected:")
try:
    expected()
except AssertionError:
    print("handle exception")

This is the output I get from this code:

>>> What my code does:
CLOSE: Reader
handle exception
CLOSE: Reader releases resource

>>> The context handling I expected:
CLOSE: Reader releases resource
CLOSE: Reader
handle exception

It seems that wrapping the Container in a generator expression causes the with block in its __iter__ method to be cleaned up after the exception handling is complete, despite the fact that it's nested within another with block that gets cleaned up while the exception is being raised.

Why does wrapping the iterable in a generator expression change how the with block inside its __iter__ method is finalised during exception handling?

EDIT:

Thanks to the answer from @user2357112 I see that omitting the local variable bound to the generator expression gives the context handling I expected:

def my_code_without_local():
    with closing(Reader()) as reader:
        container = Container(reader)
        for thing in (x for x in container):
            assert False

This gives this output:

CLOSE: Reader releases resource
CLOSE: Reader
handle exception

This behaviour is really surprising to me, as I expected the active with-block in the named generator expression container_gen to be cleaned up in the same way as the active with-block in my_code, not that it would hang around and only get cleaned up once the local name container_gen goes out of scope.

Answer 1

You're relying on a with in a generator, and those are awkward, because cleanup can easily get delayed like this. Specifically, I'm talking about your __iter__ method, which you wrote as a generator. Not the genexp - I'll get to that later.

If you don't fully loop over your __iter__ generator, the with block cleanup only runs when the generator is closed. Like most people, you never explicitly closed your generator. You probably never even thought about it. The only time it gets closed is in the generator's __del__ method, which gets called when Python determines the generator is unreachable.

With expected, the only reference to your __iter__ generator is the internal reference the for loop keeps to its iterator. That reference dies before the with closing(Reader()) as reader: block is exited, and since CPython has reference counting, the interpreter immediately detects that the generator is unreachable, so it calls __del__, triggering the cleanup you expected. (On a non-refcounting implementation, like PyPy, this cleanup wouldn't be so prompt.)

With my_code, your __iter__ generator is reachable from the generator created by your genexp, which is reachable through the container_gen local variable. That local variable is reachable through the exception traceback, so it doesn't get cleared until the except block finishes and the exception object dies. (This is true even though your except block doesn't have an as clause - the exception and its traceback are still available through sys.exc_info() during the except block.)

Your generator ends up living much longer than you expected it to. Since the cleanup you were expecting only gets triggered by the generator's __del__ method, the cleanup gets delayed just as long.