IndexError: pop from an empty deque from _kernel_wake

[mwerezak]

Got this exception from inside curio/sched.py

Not sure yet as to the cause.

Traceback (most recent call last):
  File "...", line 72, in <module>
    curio.run(main)
  File "...\venv\Lib\site-packages\curio\kernel.py", line 824, in run
    return kernel.run(corofunc, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\venv\Lib\site-packages\curio\kernel.py", line 146, in run
    task = self._runner(coro)
           ^^^^^^^^^^^^^^^^^^
  File "...\venv\Lib\site-packages\curio\kernel.py", line 768, in kernel_run
    traps[trap[0]](*trap[1:])
  File "...\venv\Lib\site-packages\curio\kernel.py", line 519, in trap_sched_wake
    tasks = sched._kernel_wake(n)
            ^^^^^^^^^^^^^^^^^^^^^
  File "...\venv\Lib\site-packages\curio\sched.py", line 88, in _kernel_wake
    task, = self._queue.popleft()
            ^^^^^^^^^^^^^^^^^^^^^
IndexError: pop from an empty deque

ntasks is 1, but self._queue is empty.

__exception__ = {tuple: 3} (<class 'IndexError'>, pop from an empty deque, <traceback object at 0x00000261BE8AA500>)
ntasks = {int} 1
self = {SchedFIFO: 0} SchedFIFO<0 tasks waiting>
  _abc_impl = {_abc_data} <_abc._abc_data object at 0x000002619CFB9E80>
  _actual_len = {int} 0
  _queue = {deque: 0} deque([])
task = {NoneType} None
tasks = {list: 0} []
 __len__ = {int} 0

[mwerezak]

I have no idea where to even start looking for the cause of this. Any insight would be appreciated.

[mwerezak]

It seems related to creating a lot of timeouts using curio.timeout_after()

This appears to be the code in my application that triggers it:

        async with self._condition:
            while not self._bus_available():
                try:
                    async with curio.timeout_after(self.wait_interval):
                        await self._condition.wait()
                except curio.TaskTimeout:
                    _log.debug(f"{client}: wait.")
                    await client.send_message(WAIT_MSG)

I'm trying to wait for a condition but I also want to send regular messages back to the client so they know the connection isn't timed out

[dabeaz]

What version of curio? What version of Python? What operating system? Is the bug predictably reproducible or is it random?

[mwerezak]

Hi, sorry for forgetting to include that information.

curio version 1.6
Python version 3.11.5

From memory the bug was pretty predictable but depended on workload... that snippet above was being hit ~10-100/second, with the timeout value being about 5 seconds IIRC... I guess that means several hundred timeouts active at once. Under lesser loads the bug did not manifest, but I don't recall the exact number.

We implemented a workaround by frequently polling (and checking the timeout manually using time()) instead of blocking on a condition variable. In retrospect maybe a MPSC setup using a Queue might have been a better solution than either of these.

[x42005e1f]

The cause is actually quite simple. SchedFIFO, which is used by Condition, does not check the waiting queue length at all, and as a result cond.notify() tries to perform self._queue.popleft() for the empty queue, thus raising IndexError. Below is a fairly simple code that reproduces the issue.

import curio


async def main():
    async with curio.Condition() as cond:
        await cond.notify()  # IndexError + RuntimeError


if __name__ == "__main__":
    curio.run(main)

The quickest solution is to add a check that the queue is not empty to the loop condition. Or add a check for SchedFIFO length to the Condition methods themselves in the spirit of the other primitives, but this seems like an ugly solution to me, especially because of memory leaks on cancellations (1-item lists are not removed).

[mwerezak]

I guess the question is, what should happen when you try notify a SchedFIFO that is empty? Does it make sense for the alert to become a no-op in that case?

[x42005e1f]

Does it make sense for the alert to become a no-op in that case?

Almost correct. That is not an alert, but a bug that causes Curio to die. In the example shown, the output will look like this:

Traceback (most recent call last):
  File "/home/user/workspace/test.py", line 10, in <module>
    curio.run(main)
  File "/home/user/.local/lib/python3.12/site-packages/curio/kernel.py", line 824, in run
    return kernel.run(corofunc, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/curio/kernel.py", line 146, in run
    task = self._runner(coro)
           ^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/curio/kernel.py", line 768, in kernel_run
    traps[trap[0]](*trap[1:])
  File "/home/user/.local/lib/python3.12/site-packages/curio/kernel.py", line 519, in trap_sched_wake
    tasks = sched._kernel_wake(n)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.12/site-packages/curio/sched.py", line 88, in _kernel_wake
    task, = self._queue.popleft()
            ^^^^^^^^^^^^^^^^^^^^^
IndexError: pop from an empty deque
Exception ignored in: <function Task.__del__ at 0x7ff187ebed40>
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.12/site-packages/curio/task.py", line 157, in __del__
  File "/home/user/.local/lib/python3.12/site-packages/curio/task.py", line 222, in exception
RuntimeError: Task not terminated
sys:1: ResourceWarning: unclosed <socket.socket fd=5, family=1, type=1, proto=0>
sys:1: ResourceWarning: unclosed <socket.socket fd=4, family=1, type=1, proto=0>

[mwerezak]

I think you misunderstood me. What you are talking about is an exception that is raised, which is not what I meant by "alert". By "alert" I was referring to the mechanism that causes a sleeping task to be woken up (I only chose that word because I saw it being used with a similar meaning in the comments for UniversalQueueBase). The way SchedFIFO is written, it appears to expect every sleeping task (suspend) to be paired with an alert (wake). Imagine now that due to concurrency a suspend/wake pair gets reordered such that the wake arrives before the suspend. Should the the wake be converted into a no-op because the queue is empty? Or should it wake the next suspended task to arrive? I don't know because I don't have enough context. I don't know the intention of SchedFIFO. I don't know all the situations in which it is used. This is why I'm hoping for someone with more familiarity to join in the conversation.

…

On Thu., Aug. 21, 2025, 7:43 a.m. Ilya Egorov, ***@***.***> wrote: *x42005e1f* left a comment (dabeaz/curio#365) <#365 (comment)> Does it make sense for the alert to become a no-op in that case? Almost correct, but that is not an alert, but a bug that causes Curio to die <https://github.com/dabeaz/curio/blob/751b0d313c25ac03d57f475c8aafa1fca6005785/curio/kernel.py#L762-L772>. In the example shown, the output will look like this: Traceback (most recent call last): File "/home/user/workspace/test.py", line 10, in <module> curio.run(main) File "/home/user/.local/lib/python3.12/site-packages/curio/kernel.py", line 824, in run return kernel.run(corofunc, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/.local/lib/python3.12/site-packages/curio/kernel.py", line 146, in run task = self._runner(coro) ^^^^^^^^^^^^^^^^^^ File "/home/user/.local/lib/python3.12/site-packages/curio/kernel.py", line 768, in kernel_run traps[trap[0]](*trap[1:]) File "/home/user/.local/lib/python3.12/site-packages/curio/kernel.py", line 519, in trap_sched_wake tasks = sched._kernel_wake(n) ^^^^^^^^^^^^^^^^^^^^^ File "/home/user/.local/lib/python3.12/site-packages/curio/sched.py", line 88, in _kernel_wake task, = self._queue.popleft() ^^^^^^^^^^^^^^^^^^^^^IndexError: pop from an empty dequeException ignored in: <function Task.__del__ at 0x7ff187ebed40>Traceback (most recent call last): File "/home/user/.local/lib/python3.12/site-packages/curio/task.py", line 157, in __del__ File "/home/user/.local/lib/python3.12/site-packages/curio/task.py", line 222, in exceptionRuntimeError: Task not terminated — Reply to this email directly, view it on GitHub <#365 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB3F7FCVWWAYHXN6TSUECL3OWWFNAVCNFSM6AAAAACAKYEQ5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJQGE4TIMRUGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[x42005e1f]

I think you misunderstood me.

Yes, I misunderstood you, because you used the word "notify" in the previous sentence.

Imagine now that due to concurrency a suspend/wake pair gets reordered such that the wake arrives before the suspend.

This is not true for Kernel code. We are dealing with coroutines (cooperative multitasking) rather than threads (preemptive multitasking). Therefore, we do not need to worry about waking up the next arrivals. Moreover, neither asyncio.Condition nor even threading.Condition do this.

If user code relies on such behavior, then it is simply using Condition incorrectly - it is not a primitive that counts the number of wakes like a semaphore or its friends. Calling notify() when there are no waiters is always expected to be a no-op.

[mwerezak]

Ok, good to know.

…

On Thu., Aug. 21, 2025, 8:47 a.m. Ilya Egorov, ***@***.***> wrote: *x42005e1f* left a comment (dabeaz/curio#365) <#365 (comment)> I think you misunderstood me. Yes, I misunderstood you, because you used the word "notify" in the previous sentence. Imagine now that due to concurrency a suspend/wake pair gets reordered such that the wake arrives before the suspend. This is not true for Kernel code. We are dealing with coroutines (cooperative multitasking) rather than threads (preemptive multitasking). Therefore, we do not need to worry about waking up the next arrivals. Moreover, neither asyncio.Condition nor even threading.Condition do this. If user code relies on such behavior, then it is simply using Condition incorrectly - it is not a primitive that counts the number of wakes like a semaphore or its friends. Calling notify() when there are no waiters is always expected to be a no-op. — Reply to this email directly, view it on GitHub <#365 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB3F7H44SCNI3RV7WJNE5T3OW5VRAVCNFSM6AAAAACAKYEQ5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJQGQ3DANZZGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>