Memory leak in CIMultiDict.pop() method?

[F1int0m]

I noticed a big memory leak in my application.

With tracemalloc I found that memory is leaking little by little when calling the .pop() method.
I used multidict version: 6.6.3

From #1134 I took an example for the script and got this:

import os

import psutil
from multidict import MultiDict, __version__ as multidict_version

print(f"multidict version: {multidict_version}")


def get_memory_usage():
    process = psutil.Process(os.getpid())
    memory_info = process.memory_info()
    return memory_info.rss / (1024 * 1024)


keys = [f"X-Any-{i}" for i in range(1000)]
headers = {key: key * 2 for key in keys}


def main():
    for _ in range(1000):
        for _ in range(1000):
            result = MultiDict(headers)

            for key in keys:
                result.pop(key, None) # leak
                # result.popall(key) # leak
                # del result[key] # leak

                # result.pop('123321', None) # very slow leak

        del result
        usage = get_memory_usage()
        print(f"Memory usage: {usage:.2f} MB")
        if usage > 200:
            raise MemoryError("Memory usage exceeded 200 MB")


if __name__ == "__main__":
    main()

From the notable:

1. If you increase if usage > 200 to a super large value, the script runs without errors (but should this script really consume so much memory? The maximum I saw was about 6000 MB, and at the end of the script the memory stabilizes and remains at some still large value)
2. Just in case, I tried different functions, most of them behave the same. But if you do .pop from a non-existent key, then the memory leaks very slowly, about 0.5MB in 1-2 minute.

[Vizonex]

@F1int0m Interesting find let me test this script you've made on my computer to see if I notice anything.

[Vizonex]

I have about 20GB of ram so I should be able to test this safely.

[Vizonex]

@F1int0m You were right, I'm going to censor my path but yes this is true.

multidict version: 6.6.3
Memory usage: 61.18 MB
Memory usage: 106.79 MB
Memory usage: 152.39 MB
Memory usage: 198.01 MB
Memory usage: 243.62 MB
Traceback (most recent call last):
  File "-", line 39, in <module>
    main()
  File "-", line 35, in main
    raise MemoryError("Memory usage exceeded 200 MB")
MemoryError: Memory usage exceeded 200 MB

Let me see if I or someone else wants to take a shot at fixing this bug.

[Vizonex]

@F1int0m I think the issue might be a releasing bug in the keys. when a dictionary deletes itself I tried putting the result into the while loop and confirmed deleting result in there is also not releasing/freeing the entries @asvetlov Seems someone found a bug in popping and freeing entries. I haven't figured out where it is exactly but I would assume it's in the dealloc code.

[Vizonex]

@F1int0m I wonder if the new 6.7 update is affected by this bug since I contributed a pretty large portion of it which introduces the ability to optionally interact with the code in C, If I had to guess 80% was me and asvetlov helped me come up with the baseplate and cherry-pick my ideas with it. I'll test 6.7 tomorrow it's very late for me.

[F1int0m]

I'll wait for the results and the release)

I couldn't find a version that doesn't have this problem (at the same time, version 6.3.0 didn't behave like this until some time ago. I didn't understand what change in the code led to the leaks. But I have an old and stable image without leaks, I can try to compare it with the current one to compare the change in the environment)

In the meantime, I patched up the production a little like this (just so as not to delete something that definitely doesn't exist and can be left alone), hopefully not for long 😄 :

if header in headers:
    headers.popall(header, None)

[Vizonex]

@F1int0m I want to see if the problem might be GC Related. I was thinking about that when I had to patch up some problems with allocating memory to multidict from C's end.

[Vizonex]

@F1int0m Seems 6.7 is also affected I'll attempt to patch this for now on my end and maybe see about writing a pytest later today for it.

[Vizonex]

@F1int0m Before I make others aware of this I want to try using tracemalloc on this code snippet you gave me because otherwise it's GC Related.

[F1int0m]

I wrote this script based on the previous one.

It's very strange, but tracemalloc shows a leak when creating a MultiDict, and not when deleting elements from it

import os
import tracemalloc

import psutil
from multidict import MultiDict, __version__ as multidict_version

print(f"multidict version: {multidict_version}")
tracemalloc.start()


def get_memory_usage():
    process = psutil.Process(os.getpid())
    memory_info = process.memory_info()
    return memory_info.rss / (1024 * 1024)


keys = [f"X-Any-{i}" for i in range(1000)]
headers = {key: key * 2 for key in keys}


def main():
    for i in range(10000):
        for _ in range(1000):
            result = MultiDict(headers)

            for key in keys:
                result.pop(key, None)  # leak
                # result.popall(key) # leak
                # del result[key] # leak

                # result.pop('123321', None) # very slow leak

        del result
        if i % 100 == 0:
            usage = get_memory_usage()
            print(f"Memory usage: {usage:.2f} MB")

            snapshot = tracemalloc.take_snapshot()
            top_stats = snapshot.statistics('lineno')

            print("[ Top 10 ]")
            for stat in top_stats[:10]:
                print(stat)
            print('-' * 30)


if __name__ == "__main__":
    main()

[F1int0m]

But in this example, memory (if you believe psutil) is slowly leaking, although tracemalloc shows nothing.

import os
import tracemalloc

import psutil
from multidict import MultiDict, __version__ as multidict_version

print(f"multidict version: {multidict_version}")
tracemalloc.start()


def get_memory_usage():
    process = psutil.Process(os.getpid())
    memory_info = process.memory_info()
    return memory_info.rss / (1024 * 1024)

print('create dict')
headers = MultiDict({f"X-Any-{i}": i for i in range(1000000)})
keys = list(headers.keys())


def main():
    for i, key in enumerate(keys):

        headers.pop(key, None)  # leak

        if i % 10 == 0:
            usage = get_memory_usage()
            print(f"Memory usage: {usage:.2f} MB")

            snapshot = tracemalloc.take_snapshot()
            top_stats = snapshot.statistics('lineno')

            print("[ Top 10 ]")
            for stat in top_stats[:10]:
                print(stat)
            print('-' * 30)


if __name__ == "__main__":
    main()

[Vizonex]

But in this example, memory (if you believe psutil) is slowly leaking, although tracemalloc shows nothing.

import os
import tracemalloc

import psutil
from multidict import MultiDict, version as multidict_version

print(f"multidict version: {multidict_version}")
tracemalloc.start()

def get_memory_usage():
process = psutil.Process(os.getpid())
memory_info = process.memory_info()
return memory_info.rss / (1024 * 1024)

print('create dict')
headers = MultiDict({f"X-Any-{i}": i for i in range(1000000)})
keys = list(headers.keys())

def main():
for i, key in enumerate(keys):

    headers.pop(key, None)  # leak

    if i % 10 == 0:
        usage = get_memory_usage()
        print(f"Memory usage: {usage:.2f} MB")

        snapshot = tracemalloc.take_snapshot()
        top_stats = snapshot.statistics('lineno')

        print("[ Top 10 ]")
        for stat in top_stats[:10]:
            print(stat)
        print('-' * 30)

if name == "main":
main()

Whelp that confirms that it could be gc related. I'll if the other maintainers would like deviate away from garbage collection or not. Otherwise I would try disabling the garbage collector.