Rework get_task and steal_task to better interact with out_of_work checks

[akukanov]

This PR "revives" the improvement originated in #417, but takes a different approach to the problem.

To remind, when a thread looks for a task to take from a task pool, it might skip some tasks due to affinity or isolation restrictions, The task pool still contains pointers to the tasks, but the observable limits of the pool (the head, modified by thieves, and the tail, modified by the owning thread) might temporary exclude the skipped tasks. Due to that, another thread that inspects the arena for work availability might find the task pool "empty" and potentially mark the whole arena empty, causing premature leaving of worker threads. The current implementation mitigates that by issuing a "work advertisement" signal when the skipped tasks are "returned" to the observable pool.

The PR #417 tried to improve the implementation by adding "shadow" head and tail indexes for the slot inspection, which are not changed until an operation on the task pool is complete, and so they should never exclude "skipped" tasks. In my opinion, however, it puts the burden on the wrong side and complicates the arbitration protocol between the pool owner and thieves. As implemented, it also does not achieve the goal, as in the case of pool "exhaustion" the shadow limits would be temporarily reset, similar to the real limits.

This PR takes a different approach and puts more burden on the inspecting thread, which anyway has no tasks to execute. If that thread suspects the task pool to be empty after comparing its head and tail, it locks the pool and re-reads its state. By locking, any temporary modifications by stealing threads are prevented. To coordinate the inspection with changes made by the owning thread, a new flag is added into arena_slot. The flag is set by the owning thread in get_task if it skips one or more tasks, and is reset once the pool limits are restored. The flag is read and tested by the inspecting thread, and the slot is only considered empty when both the pool limits show no tasks and the skipping flag is not set.

Tests

• [ X ] not needed, existing tests should be sufficient

Documentation

• [ X ] not needed

Breaks backward compatibility?

• [ X ] No - the changes are not exposed in API or ABI

[akukanov]

@kboyarinov @isaevil @dnmokhov Please take a look.

[kboyarinov]

Just to clarify my understanding of the problem statement and the proposed fix (may duplicate the description in a bit:) ).

Let's assume that the current slot contain a single task, isolation of which prevents the owning thread to execute it.
The owning thread skips this task while looking for a work to do and temporarily decrements the tail (head equal to tail in our case).
Between the decrement and the restore of the tail, the stealing thread can inspect the current slot in arena::out_of_work and see head equal to tail meaning the current slot is out of tasks.

If all other slots in arena are out of tasks, the stealing thread signals to workers to leave the arena.

When the restoration of head and tail is done by the owning thread, the advertise_new_work will request the workers to re-join the arena.

Am I correct that the idea of the patch is to prevent workers from leaving the arena in this case and double-check by reading the flag has_skipped_tasks that should be set by the owning thread while the tasks are skipped?

[akukanov]

That's correct overall, with a couple of comments to be made.

Between the decrement and the restore of the tail, the stealing thread can inspect the current slot in arena::out_of_work...

Technically, it's not a stealing thread, as it does not at that point attempt to steal anything. I would call it an inspecting thread, to disambiguate with other threads that can really try stealing at the same time.

The second comment is that stealing threads might also skip tasks and temporarily make the slot appear empty. In the current implementation, they also call advertise_new_work afterwards. In the patch, the inspecting thread has to lock the pool for final inspection, therefore no stealing can happen at the same time.

[kboyarinov]

Do I understand correctly that we do not need to restore head here since it is a stealing thread responsibility and is done in the steal_task?

[akukanov]

That is correct.

Generally, H0 represents the state of the pool head as it was seen by the owner; it might get outdated at any time. The core principle therefore is that the owner only works with the tail and does not change the head.

Indeed, if there was no conflict for the last task, the owner has no idea what the proper value for the head should be. And in case of a conflict the pool lock is taken and the head is re-read, and we can be sure that there is no skipped tasks beyond the head, so there is no need to change anything.

Prior to the patch, there is a store of H0 to the head - but it is done at the point where the pool is temporarily quiescent, and therefore it is safe. It "optimizes" the case when the task at the head is taken while others were skipped. In the patch, the pool is not reset if tasks were skipped, as that would also mislead observers. So this optimization cannot be safely performed anymore.

[akukanov]

The commit 796db43 is for code refactoring and is not strictly necessary. It significantly changes get_task, and though the core logic there remains the same, some code blocks and checks had to be reordered. It's likely better to review it separately, on top of the previous commits. I can as well revert it in case you prefer to keep refactoring separate from the substantial changes.