numba jit mode memory leak issue?

Question

I am currently facing a weird memory leak problem when having jit decorator on a function which uses heapify takings results from another jitted function which return results in yield style

here is the idea of the design to indicate the issue

@njit
def main_ ():
    ss = []
    heapq.heapify (ss)
    yd = generator_()
    result = 0
    while 1:
        result = next (yd)
        if result == []:
             break
        heapq.heappush (result)
    return list (ss)

@njit
def generator_():
    for i in range (100000000):
        yd2 = generator_2 (i)
        while 1:
            result = next (yd2)
            if result == []:
                break
            yield [i, result]
    yield []

@njit
def generator_2(ii):
    for i in range (100000000):
        yield ([ii, i*2, i*i])
    yield []

If the function main_ having the njit or jit removed, the memory issue would not happened. I have also called gc collection but it seems like aint doing much. I have also found some discussion shows it would be relate to llvm as it store everything it compiled.

Would anyone would be able to give me some opinion what exact the problem is.

Thanks you

Answer 1

I've just faced the same issue

import gc
import numpy as np
import numba


@numba.jit(cache=False)
def main(x):
    x**2


gc.enable(  )
gc.set_debug(gc.DEBUG_SAVEALL)
x = np.random.random((10,))
main(x)
gc.collect()
assert not len(gc.garbage), f"found {len(gc.garbage)} objects."

raises AssertionError