Possible event loss when using raise_event on suspended orchestration #42
Description
The CI has been a bit flakey lately.
Here was the output of a recent tests/test_orchestration_e2e.py::test_suspend_and_resume test.
2025-02-13 23:18:15.166 durabletask-worker INFO: Starting gRPC worker that connects to localhost:4001
2025-02-13 23:18:15.166 durabletask-client INFO: Starting new 'orchestrator' instance with ID = '6a0f557f863f41bba1d78f6fffaaa15f'.
2025-02-13 23:18:15.167 durabletask-worker INFO: Successfully connected to localhost:4001. Waiting for work items...
2025-02-13 23:18:15.167 durabletask-client INFO: Waiting up to 30s for instance '6a0f557f863f41bba1d78f6fffaaa15f' to start.
2025-02-13 23:18:15.181 durabletask-worker INFO: 6a0f557f863f41bba1d78f6fffaaa15f: Orchestrator yielded with 0 task(s) and 1 event(s) outstanding.
2025-02-13 23:18:15.183 durabletask-client INFO: Suspending instance '6a0f557f863f41bba1d78f6fffaaa15f'.
2025-02-13 23:18:15.230 durabletask-worker INFO: 6a0f557f863f41bba1d78f6fffaaa15f: Execution suspended.
2025-02-13 23:18:15.230 durabletask-worker INFO: 6a0f557f863f41bba1d78f6fffaaa15f: Orchestrator yielded with 0 task(s) and 1 event(s) outstanding.
2025-02-13 23:18:15.284 durabletask-client INFO: Raising event 'my_event' for instance '6a0f557f863f41bba1d78f6fffaaa15f'.
2025-02-13 23:18:15.285 durabletask-client INFO: Waiting 3s for instance '6a0f557f863f41bba1d78f6fffaaa15f' to complete.
2025-02-13 23:18:15.337 durabletask-worker INFO: 6a0f557f863f41bba1d78f6fffaaa15f: Orchestrator yielded with 0 task(s) and 1 event(s) outstanding.
2025-02-13 23:18:18.286 durabletask-worker INFO: Stopping gRPC worker...
2025-02-13 23:18:18.287 durabletask-worker INFO: Disconnected from localhost:4001
2025-02-13 23:18:18.287 durabletask-worker INFO: No longer listening for work items
2025-02-13 23:18:18.287 durabletask-worker INFO: Worker shutdown completed
It's possible that this could be related to the backend, but it's unclear. Either way this needs to be investigated.