Skip to content

Conversation

@ota42y
Copy link

@ota42y ota42y commented Feb 18, 2022

Original PR #44

We want to reduce redis load by delete-queue command when the queue has a large number of tasks.
*a large number is about 10^6

I implemented following improvements:

  • add --without-transaction option to large queue delete

    • if the user doesn't set this option, keep old feature.
    • if the queue have many tasks and without this option, raise error.
  • use SScan instead of SMembers command to fetch task UIDs

  • use UnLink instead of Del command and bulk deleting keys by chunk size

  • WIP points

    • Two pieces of code that are doing almost the same thing.
      • with / without transaction
    • Testing the corner case
      • i.e. abort and resume deleting
    • Performance test
      • I checked this PR work well for over 1234500 tasks
      • But it's take 20 minutes so it's not good for unite test.

fixes #28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The same problem as #25 occurs with delete-queue for large queues

2 participants