[question] High CPU usage when calling ABT_cond_timedwait from a non-ULT thread

[tsukada-ascade]

Summary

I am using ABT_cond_timedwait() in a single-threaded program where the main() thread calls it directly to wait for a condition with a timeout (no other thread signals the condition, so it always times out). However, I notice that the program consumes 100% CPU during the wait, which is unexpected—I would have assumed it to be blocking with minimal CPU usage.

Reproducer Overview

#include <assert.h>
#include <stdio.h>
#include <time.h>
#include <abt.h>

ABT_mutex_memory mutex_mem = ABT_MUTEX_INITIALIZER;
ABT_cond_memory cond_mem = ABT_COND_INITIALIZER;

int main(int argc, char *argv[])
{
    ABT_mutex mutex = ABT_MUTEX_MEMORY_GET_HANDLE(&mutex_mem);
    ABT_cond cond = ABT_COND_MEMORY_GET_HANDLE(&cond_mem);
    struct timespec ts;
    int ret;

    ABT_init(argc, argv);

    clock_gettime(CLOCK_REALTIME, &ts);
    ts.tv_sec += 30;

    ABT_mutex_lock(mutex);
    ret = ABT_cond_timedwait(cond, mutex, &ts); // Always times out
    ABT_mutex_unlock(mutex);

    ABT_finalize();
    return 0;
}

Questions

1. Is ABT_cond_timedwait() expected to work correctly when called from a non-ULT thread, such as the main POSIX thread (i.e., outside of a user-level thread)?
2. If not, is this limitation documented anywhere? I couldn’t find any explicit mention of this constraint in the official documentation.
3. Is the high CPU usage in this case an expected behavior or a bug?

Any clarification would be greatly appreciated.

[carns]

Hi @tsukada-ascade , to answer your questions:

1. yes, effectively any Argobots function is legal to call from main(); when you initialize Argobots it is implicitly configured like a normal Argobots execution stream.
2. (see above, though maybe we could better document that this is not a constraint)
3. This is expected default behavior, but you can alter that behavior. More info below.

The default scheduler in Argobots (ABT_SCHED_BASIC) will busy spin looking for work to do if there are not work units available to execute. The main() routine is all that the scheduler has to run in the example you gave above. When you call ABT_Cond_timedwait() it is suspended, and the scheduler begins spinning to look for other work to do.

You can alter this behavior by switching to the ABT_SCHED_BASIC_WAIT scheduler. There is an example in https://github.com/pmodels/argobots/blob/main/test/basic/sched_basic_wait.c#L54 that shows how to alter the scheduler associated with the main() thread.

One other note, is that if you have multiple execution streams that may signal each other, then you also need to use the ABT_POOL_FIFO_WAIT pool type, which is designed to integrate with the BASIC_WAIT scheduler and wake it up when external work arrives. I think there are some examples of how to configure this in the tests/ subdirectory as well, but we can help answer questions if you have any.

[tsukada-ascade]

Hi @carns , thank you for the helpful explanation earlier.

I tried modifying the original minimal reproducer I posted earlier based on your suggestion using ABT_SCHED_BASIC_WAIT. However, the CPU usage remains high (close to 100%) during the ABT_cond_timedwait() call.

Here’s the relevant diff:

--- re-1st.c    2025-06-27 00:37:43.273318849 +0000
+++ re-2nd.c    2025-06-27 00:39:16.759185413 +0000
@@ -12,9 +12,24 @@ int main(int argc, char *argv[])
     ABT_cond cond = ABT_COND_MEMORY_GET_HANDLE(&cond_mem);
     struct timespec ts;
     int ret;
+    ABT_sched sched, get;
+    ABT_xstream xstream;

     ABT_init(argc, argv);

+    ABT_sched_create_basic(ABT_SCHED_BASIC_WAIT, 0, NULL, ABT_SCHED_CONFIG_NULL, &sched);
+    ABT_xstream_self(&xstream);
+
+    /* before change */
+    ABT_xstream_get_main_sched(xstream, &get);
+    ABT_info_print_sched(stdout, get);
+
+    ABT_xstream_set_main_sched(xstream, sched);
+
+    /* after */
+    ABT_xstream_get_main_sched(xstream, &get);
+    ABT_info_print_sched(stdout, get);
+
     clock_gettime(CLOCK_REALTIME, &ts);
     ts.tv_sec += 30;

From ABT_info_print_sched() it looks like the scheduler switch was successful:

== SCHED (0x5654eac54680) ==
kind     : 7f5bb052f300 (BASIC)
used     : MAIN
automatic: TRUE
...
== SCHED (0x5654eac54e00) ==
kind     : 7f5bb052f340 (BASIC_WAIT)
used     : MAIN
automatic: TRUE
...

I also suspected that the pool associated with the scheduler might need to be wait-aware, so I further modified the code to use ABT_POOL_FIFO_WAIT as shown here:

--- re-2nd.c    2025-06-27 00:39:16.759185413 +0000
+++ re-3rd.c    2025-06-27 00:49:00.931698168 +0000
@@ -12,12 +12,14 @@ int main(int argc, char *argv[])
     ABT_cond cond = ABT_COND_MEMORY_GET_HANDLE(&cond_mem);
     struct timespec ts;
     int ret;
+    ABT_pool pool;
     ABT_sched sched, get;
     ABT_xstream xstream;

     ABT_init(argc, argv);

-    ABT_sched_create_basic(ABT_SCHED_BASIC_WAIT, 0, NULL, ABT_SCHED_CONFIG_NULL, &sched);
+    ABT_pool_create_basic(ABT_POOL_FIFO_WAIT, ABT_POOL_ACCESS_MPMC, ABT_TRUE, &pool);
+    ABT_sched_create_basic(ABT_SCHED_BASIC_WAIT, 1, &pool, ABT_SCHED_CONFIG_NULL, &sched);
     ABT_xstream_self(&xstream);

     /* before change */

Despite this, the CPU usage during the timed wait is still high.

Do you see anything I might be missing in the way I’m using the scheduler and pool?
Should it be possible to use ABT_cond_timedwait() from a non-ULT context (like main()) with ABT_SCHED_BASIC_WAIT and expect low CPU usage?

Thank you again for your time and help!

[carns]

Thanks for trying that out, and especially for providing the code to reproduce it @tsukada-ascade . Your code looks correct to me, so you may have found a bug after all.

Unfortunately I had some travel problems this weekend and now I am leaving for vacation until July 8. I would like to try to run your example and look at it but I won't be able to until I return.

[tsukada-ascade]

Hi @carns , thank you very much for your reply, and I appreciate your willingness to look into it despite your busy schedule.

Please enjoy your vacation — no rush at all. I'll look forward to hearing back after you return. Safe travels!

[carns]

Ok, I did try your reproducer (all three versions with the default scheduler, the wait scheduler, and the wait scheduler plus wait pool), and confirmed that all three of them take 100% cpu for me as well.

Have you already tried this in a scenario where the cond_timedwait is called from a different ult or execution stream? From the title of the issue, it sounds like you may have already narrowed it down to occurring specifically when called from the main thread.

[tsukada-ascade]

I tested again, and this time ABT_cond_timedwait() is called from a ULT (not from main()), but the CPU is still getting quite the workout! That was definitely a surprise. I also want to apologize for the misleading title of the issue — I initially assumed the behavior was specific to main() due to incomplete testing on my part.

Could this be a bug in the current implementation of ABT_cond_timedwait, or is there something subtle I might be missing?

I applied the following patch to the official test program test/basic/cond_timedwait.c from the Argobots repository:

--- cond_timedwait.c    2025-07-12 05:59:31.207154744 +0000
+++ cond_timedwait_oversleep.c  2025-07-12 06:03:17.868232132 +0000
@@ -9,6 +9,7 @@
 #include <sys/time.h>
 #include "abt.h"
 #include "abttest.h"
+#include <unistd.h>

 #define DEFAULT_NUM_XSTREAMS 4
 #define DEFAULT_NUM_THREADS 4
@@ -35,7 +36,7 @@

     ts.tv_sec = tv.tv_sec;
     ts.tv_nsec = tv.tv_usec * 1000;
-    ts.tv_sec += 1;
+    ts.tv_sec += 100;

     ret = ABT_mutex_lock(mutex);
     ATS_ERROR(ret, "ABT_mutex_lock");
@@ -122,6 +123,7 @@
     }

     ABT_thread_yield();
+    sleep(30);

     ret = ABT_mutex_lock(mutex);
     ATS_ERROR(ret, "ABT_mutex_lock");

[luobuda]

I also encountered this problem. I use the following configure can reduce CPU when pool is empty：

./configure --enable-sched-sleep=yes --prefix=/usr/local --enable-debug=most --enable-wait-policy=passive

I wonder if there are any other questions.

[carns]

For another data point, @tsukada-ascade , I took your example that used the wait scheduler and wait pool, and modified it to do a normal ABT_cond_wait() (no timeout) as follows:

#include <assert.h>
#include <stdio.h>
#include <time.h>
#include <abt.h>

ABT_mutex_memory mutex_mem = ABT_MUTEX_INITIALIZER;
ABT_cond_memory cond_mem = ABT_COND_INITIALIZER;

int main(int argc, char *argv[])
{
    ABT_mutex mutex = ABT_MUTEX_MEMORY_GET_HANDLE(&mutex_mem);
    ABT_cond cond = ABT_COND_MEMORY_GET_HANDLE(&cond_mem);
    struct timespec ts;
    int ret;
    ABT_sched sched, get;
    ABT_xstream xstream;
    ABT_pool pool;

    ABT_init(argc, argv);

    ABT_pool_create_basic(ABT_POOL_FIFO_WAIT, ABT_POOL_ACCESS_MPMC, ABT_TRUE, &pool);
    ABT_sched_create_basic(ABT_SCHED_BASIC_WAIT, 1, &pool, ABT_SCHED_CONFIG_NULL, &sched);
    ABT_xstream_self(&xstream);

    /* before change */
    ABT_xstream_get_main_sched(xstream, &get);
    ABT_info_print_sched(stdout, get);

    ABT_xstream_set_main_sched(xstream, sched);

    /* after */
    ABT_xstream_get_main_sched(xstream, &get);
    ABT_info_print_sched(stdout, get);

    clock_gettime(CLOCK_REALTIME, &ts);
    ts.tv_sec += 30;

    ABT_mutex_lock(mutex);
    ret = ABT_cond_wait(cond, mutex);
    ABT_mutex_unlock(mutex);

    ABT_finalize();
    return 0;
}

This example does not consume CPU. (It also deadlocks because of course nothing will signal the condition, so you have to ctrl-c the program, but still interesting as a data point).

This seems to indicate that Argobots is capable of doing the right thing here at a fundamental level, but the timed version of the function must not be implemented the same way. The underlying pool has a pop_wait function with a timeout that I would have expected to be used in this case but it isn't (or maybe it can't be, for a reason that I don't understand yet; I'm not familiar with this code path).

The solution pointed out by @luobuda likely also works, but I think there might be a tradeoff here, because I think in that mode the scheduler does brief (non-interruptable) sleeps when idling, which will indeed reduce CPU usage but may have some latency penalty when it is time for the scheduler to resume work. It depends on if this scenario is on a critical path or not.

[tsukada-ascade]

To understand why ABT_cond_timedwait() causes high CPU usage even when using ABT_SCHED_BASIC_WAIT, I investigated the scheduler behavior using gdb.

When using ABT_cond_wait() (which does not consume CPU), I obtained the following backtrace:

(gdb) bt
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fd3241fdf20, expected=0, futex_word=0x1a1cfd0)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7fd3241fdf20, mutex=0x1a1cf80, cond=0x1a1cfa8)
    at pthread_cond_wait.c:462
#2  __pthread_cond_timedwait (cond=0x1a1cfa8, mutex=0x1a1cf80, abstime=0x7fd3241fdf20)
    at pthread_cond_wait.c:508
#3  0x00007fd326b62bc8 in pool_pop_wait (pool=<optimized out>, time_secs=0.10000000000000001, context=<optimized out>)
    at pool/fifo_wait.c:182
#4  0x00007fd326b66576 in ABTI_pool_pop_wait (context=0, time_secs=0.10000000000000001, p_pool=0x1a1ce00)
    at ../src/include/abti_pool.h:83
#5  sched_run (sched=0x1a1d140) at sched/basic_wait.c:133  // <- calling pool_pop_wait from here!
#6  0x00007fd326b544f8 in thread_main_sched_func (arg=<optimized out>) at ../src/include/abti_sched.h:35
#7  0x00007fd326b603f8 in ABTD_ythread_func_wrapper (p_arg=0x7fd3241fe0a0) at arch/abtd_ythread.c:12
#8  0x00007fd326b53f29 in ABTD_ythread_context_func_wrapper (p_fctx=<optimized out>)
    at ../src/include/abtd_fcontext.h:74
#9  0x0000000000000000 in ?? ()

This shows that the scheduler correctly reaches the !run_cnt_nowait path in basic_wait.c:

        /* Block briefly on pop_wait() if we didn't find work to do in main loop
         * above. */
        if (!run_cnt_nowait) {
            ABTI_pool *p_pool = ABTI_pool_get_ptr(pools[0]);
            ABT_thread thread;
            if (p_pool->optional_def.p_pop_wait) {
                thread = ABTI_pool_pop_wait(p_pool, 0.1,    // <-
                                            ABT_POOL_CONTEXT_OP_POOL_OTHER);
                [...]
            }

However, when using ABT_cond_timedwait(), the scheduler never enters this !run_cnt_nowait branch.
This appears to be because ABTI_pool_pop() keeps returning some work unit every time:

        /* Execute one work unit from the scheduler's pool */
        for (i = 0; i < num_pools; i++) {
            ABT_pool pool = pools[i];
            ABTI_pool *p_pool = ABTI_pool_get_ptr(pool);
            /* Pop one work unit */
            ABT_thread thread =
                ABTI_pool_pop(p_pool, ABT_POOL_CONTEXT_OP_POOL_OTHER);
            if (thread != ABT_THREAD_NULL) {
                ABTI_thread *p_thread = ABTI_thread_get_ptr(thread);
                ABTI_ythread_schedule(p_global, &p_local_xstream, p_thread);
                run_cnt_nowait++; // <- probably incremented here when using ABT_cond_timedwait()
                break;
            }
        }

To explore this further, I added a sleep(1) in fifo_wait.c:pool_push():

diff --git a/src/pool/fifo_wait.c b/src/pool/fifo_wait.c
index 3f6c541..e216574 100644
--- a/src/pool/fifo_wait.c
+++ b/src/pool/fifo_wait.c
@@ -128,6 +128,8 @@ static void pool_push(ABT_pool pool, ABT_unit unit, ABT_pool_context context)
     data_t *p_data = pool_get_data_ptr(p_pool->data);
     ABTI_thread *p_thread = ABTI_unit_get_thread_from_builtin_unit(unit);

+    sleep(1);
+
     pthread_mutex_lock(&p_data->mutex);
     thread_queue_push_tail(&p_data->queue, p_thread);
     pthread_cond_signal(&p_data->cond);

This confirmed that the scheduler is continuously processing work units pushed to the pool. Here's the backtrace from gdb while paused in sleep(1):

(gdb) bt
#0  0x00007fd854df9148 in __GI___nanosleep (requested_time=0x7fd8527fdeb0, remaining=0x7fd8527fdeb0)
    at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x00007fd854df907e in __sleep (seconds=0, seconds@entry=1) at ../sysdeps/posix/sleep.c:55
#2  0x00007fd8550ea699 in pool_push (pool=<optimized out>, unit=0x7fd85347ff81, context=<optimized out>) // <- ???
    at ../src/include/abti_unit.h:40
#3  0x00007fd8550e864f in switch_with_call_fcontext () at pool/fifo_wait.c:170
#4  0x0000000000000004 in ?? ()
#5  0x00000000023ed140 in ?? ()
#6  0x00000000023ed3c0 in ?? ()
#7  0x00007fd8527fe040 in ?? ()
#8  0x00000000023ed3c8 in ?? ()
#9  0x00000000023eca80 in ?? ()
#10 0x00007fd8550ee60f in ABTD_ythread_context_switch (p_new=<optimized out>, p_old=0x7fd85347ffe0)
    at ../src/include/abtd_fcontext.h:146
#11 ABTI_ythread_context_switch (p_new=<optimized out>, p_old=0x7fd85347ff80, p_local_xstream=0x37f00001f80)
    at ../src/include/abti_ythread.h:93
#12 ABTI_ythread_switch_to_child_internal (p_new=<optimized out>, p_old=0x7fd85347ff80,
    pp_local_xstream=<synthetic pointer>) at ../src/include/abti_ythread.h:157
#13 ABTI_ythread_run_child (p_child=<optimized out>, p_self=0x7fd85347ff80, pp_local_xstream=<synthetic pointer>)
    at ../src/include/abti_ythread.h:234
#14 ABTI_ythread_schedule (p_thread=<optimized out>, pp_local_xstream=<synthetic pointer>, p_global=<optimized out>)
    at ../src/include/abti_ythread.h:665
#15 sched_run (sched=0x7fd8527fe040) at sched/basic_wait.c:121
#16 0x00007fd8550dc4f8 in thread_main_sched_func (arg=<optimized out>) at ../src/include/abti_sched.h:35
#17 0x00007fd8550e83f8 in ABTD_ythread_func_wrapper (p_arg=0x7fd8527fe0a0) at arch/abtd_ythread.c:12
#18 0x00007fd8550dbf29 in ABTD_ythread_context_func_wrapper (p_fctx=<optimized out>)
    at ../src/include/abtd_fcontext.h:74
#19 0x0000000000000000 in ?? ()

From this, it appears that switch_with_call_fcontext() transitions into pool_push(), suggesting that ABTI_pool_pop() never returns NULL, and the scheduler never calls ABTI_pool_pop_wait().

This may explain why ABT_cond_timedwait() causes high CPU usage even when using BASIC_WAIT.

Does this behavior match your( @carns ) expectations? I’m happy to run more tests if helpful.

[carns]

Thank you for the detailed analysis @tsukada-ascade . @yfguo do you have any insight into this?

[tsukada-ascade]

Hi, just checking in on this. I just wondering if there's been any further discussion or insight.

Thanks again for looking into it!

[carns]

Sorry for the delay. I'm not sure how to solve it, but I looked into the code and I think I understand why the behavior is different between the two calls.

In the cond_wait case, the code eventually ends up in ABTI_waitlist_wait_and_unlock() (

argobots/src/include/abti_waitlist.h

Line 21 in bb6faac

ABTI_waitlist_wait_and_unlock(ABTI_local **pp_local, ABTI_waitlist *p_waitlist,

) where it has an opportunity to fully suspend the caller at

argobots/src/include/abti_waitlist.h

Line 83 in bb6faac

ABTI_ythread_suspend_unlock(&p_local_xstream, p_ythread, p_lock,

. This makes sense; the semantics of cond_wait are such that it can block indefinitely until someone signals the condition and causes the ULT to resume executing.

In contrast, the cond_timedwait case ends up in ABT_bool ABTI_waitlist_wait_timedout_and_unlock (

argobots/src/include/abti_waitlist.h

Line 91 in bb6faac

static inline ABT_bool ABTI_waitlist_wait_timedout_and_unlock(

). Instead of suspending, it loops to continuously yield and recheck the condition. If there are no other threads for the scheduler to execute, then the yield doesn't really do anything. The calling thread remains at the top of the queue and keeps getting executed in a tight loop (thus the CPU utilization).

It would be better for CPU utilization to suspend the thread so that the scheduler has an opportunity to idle, but the cond_timedwait code path can't do this, because there isn't any other entity that will be able to resume the thread when the timeout condition is hit. It has to rely on itself to be able to execute and detect the timeout.

In theory maybe something like a timer_create() callback could be used as an event source so that Argobots could internally implement a "timed suspend" to automatically resume a thread after some duration (as an optional feature). I think that would be a complex change, and to use it in cond_timedwait you would also need to handle the race between the two completion paths (timeout or condition signal).

[tsukada-ascade]

Thanks a lot, @carns — really appreciate you digging into the code and explaining the two code paths.

Based on your explanation (and my own tracing), my understanding is:

1. With ABT_cond_timedwait(), the scheduler keeps yielding and re-checking the condition/deadline, so when there are no other runnable ULTs it gets immediately rescheduled and spins, driving CPU usage up.
2. That behavior stems from an Argobots design constraint: unlike pthread_cond_timedwait(), which can hand off “sleep until deadline” to the kernel (futex + timer) and be woken up externally, Argobots must manage this purely in user space. Without an external “waker,” a fully suspended “timed sleep” isn’t available in the current implementation, hence the cooperative polling approach.

From a practical standpoint, it seems wise to avoid ABT_cond_timedwait() in single-ULT / single-ES scenarios. In those cases, unless there’s a special reason not to, using kernel-backed primitives is safer and simpler—e.g., pthread_cond_timedwait(), or event loops based onpoll/epoll (with timerfd/eventfd), or sigtimedwait(). Does that interpretation sound right?

If so, could you consider adding a note to the ABT_cond_timedwait() documentation about the potential for 100% CPU spinning in single-ULT setups, along with suggested alternatives/workarounds?

Thank you for considering this.

[carns]

I think your summary is accurate @tsukada-ascade . I proposed an additional note for the documentation of that function in #406 .