RecursionError caused by AtomicCache.get_many

[sutao]

What happened?

Got RecursionError during any request that requires DB access.
The service is currently configured to have the following related components enabled:

• Django Debug Toolbar
• Django Silk
• Django Cachalot
• Django Redis

Looking at the stack trace it seems get_many was called 890 times, causing Python to exceed the 1000 recursion limit.
Cronjob tasks triggered by Django Celery Beats do not have this issue. Currently, it appears only external HTTP requests have this issue. One suspicion is some kind of conflict with Django Debug Toolbar's cache panel.

Steps to reproduce

Django 5.0.5
Django Debug Toolbar 4.2.0
Django Silk 5.1.0
Django Cachalot 2.6.2
Django Redis 5.4.0

Stack trace:
Notice in the middle it said : [Previous line repeated 896 more times]

2024-08-26 10:38:12,840 <9> [ERROR] Internal Server Error: /index/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/middleware.py", line 175, in __call__
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/silk/middleware.py", line 70, in __call__
    self.process_request(request)
  File "/usr/local/lib/python3.11/site-packages/silk/profiling/profiler.py", line 57, in wrapped_target
    result = target(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/silk/middleware.py", line 104, in process_request
    if not _should_intercept(request):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/silk/middleware.py", line 41, in _should_intercept
    if not config.SILKY_INTERCEPT_FUNC(request):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/central/central/core/utils/functional.py", line 24, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/central/central/core/monitor/silk.py", line 11, in silk_should_intercept
    decision = DebugRule.decide(method, full_path, "silk")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/central/central/apps/base/models/debug.py", line 98, in decide
    rules = list(DebugRule.objects.all())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 400, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/silk/sql.py", line 100, in execute_sql
    return self._execute_sql(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cachalot/monkey_patch.py", line 37, in inner
    return original(compiler, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cachalot/monkey_patch.py", line 96, in inner
    return _get_result_or_execute_query(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cachalot/monkey_patch.py", line 46, in _get_result_or_execute_query
    data = cache.get_many(table_cache_keys + [cache_key])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cachalot/transaction.py", line 18, in get_many
    data.update(self.parent_cache.get_many(missing_keys))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cachalot/transaction.py", line 18, in get_many
    data.update(self.parent_cache.get_many(missing_keys))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cachalot/transaction.py", line 18, in get_many
    data.update(self.parent_cache.get_many(missing_keys))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 896 more times]   ### !!! <---- Notice this line indicating where infinite recursion happens
  File "/usr/local/lib/python3.11/site-packages/debug_toolbar/panels/cache.py", line 40, in wrapper
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django_redis/cache.py", line 29, in _decorator
    return method(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django_redis/cache.py", line 122, in get_many
    return self.client.get_many(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django_redis/client/default.py", line 485, in get_many
    results = client.mget(*map_keys)
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/redis/commands/core.py", line 2015, in mget
    return self.execute_command("MGET", *args, **options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/redis/__init__.py", line 231, in sentry_patched_execute_command
    with hub.start_span(op=OP.DB_REDIS, description=description) as span:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/hub.py", line 448, in start_span
    return scope.start_span(span=span, instrumenter=instrumenter, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/scope.py", line 789, in start_span
    new_child_span = active_span.start_child(**kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/tracing.py", line 255, in start_child
    child = Span(
            ^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/tracing.py", line 146, in __init__
    self.span_id = span_id or uuid.uuid4().hex[16:]
                              ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/uuid.py", line 723, in uuid4
    return UUID(bytes=os.urandom(16), version=4)
                      ^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while calling a Python object