failed-tests.txt before v0.18 release in x86_64

[honggyukim]

The current runtest.py failures are as follows.

$ cat failed-tests.txt
Compiler                  gcc                                           clang
Runtime test case         pg             finstrument-fu fpatchable-fun  pg             finstrument-fu fpatchable-fun
------------------------: O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os  O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os
124 exception1          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
182 thread_exit         : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK OK OK OK OK OK SG OK OK OK OK
185 exception2          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
186 exception3          : OK OK OK OK OK OK OK OK OK OK NG NG NG NG NG  OK OK OK OK OK NG NG NG NG NG NG NG NG NG NG
191 posix_spawn         : TM TM TM TM TM TM TM TM TM TM TM TM TM TM TM  TM TM OK OK OK OK OK OK OK OK OK OK OK OK OK
196 chrome_taskname     : OK OK OK NG OK OK OK OK OK OK OK OK OK OK OK  NG NG NG OK OK NG NG OK OK OK NG OK OK OK NG
204 arg_dwarf4          : OK OK NG NG OK SK SK SK SK SK OK OK NG NG OK  OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK
212 noplt_libcall       : OK OK OK OK OK NZ NZ NZ NZ NZ OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
220 trace_script        : NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG  NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG
251 exception4          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NZ NZ NZ NZ NZ OK OK OK OK OK
255 arg_dwarf6          : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK  NG OK OK OK OK SK SK SK SK SK NG OK OK OK OK
281 agent_trace_toggle  : NG NG NG OK OK NG NG NG NG NG OK NG NG NG NG  NG OK OK OK NG OK OK OK OK OK OK OK OK OK OK
282 agent_depth         : NG NG NG NG OK NG NG NG NG NG NG OK NG NG OK  NG NG NG OK OK OK OK OK OK OK OK OK OK OK OK
283 agent_time          : OK NG NG NG NG NG NG NG NG NG NG NG NG NG NG  OK NG NG NG OK OK OK OK OK OK OK OK OK OK OK
284 agent_filter        : NG NG NG OK NG NG NG NG NG OK NG OK OK OK OK  OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK
285 agent_caller_filter : NG NG NG NG OK OK OK NG NG NG NG NG NG OK OK  OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK
286 agent_trigger       : NG NG NG NG NG NG OK NG NG NG NG OK OK OK OK  OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK
289 exception5          : OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK

The version info is here.

$ uftrace --version
uftrace v0.17-43-gb6598 ( x86_64 dwarf python3 luajit tui perf sched dynamic kernel )

[honggyukim]

This is another result that I ran in a different desktop machine.

Compiler                  gcc                                           clang
Runtime test case         pg             finstrument-fu fpatchable-fun  pg             finstrument-fu fpatchable-fun
------------------------: O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os  O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os
003 thread              : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK NG OK OK OK OK OK OK OK OK OK NG OK
049 column_view         : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK NG OK OK OK OK OK OK OK OK OK NG OK
084 arg_mixed           : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK  OK NG NG NG OK SK SK SK SK SK OK NG NG NG OK
085 arg_reg             : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK  OK NG NG NG OK SK SK SK SK SK OK NG NG NG OK
124 exception1          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
176 arg_fptr            : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK NG OK OK OK OK OK OK OK OK OK NG OK
182 thread_exit         : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK NG OK OK OK OK OK OK OK NG OK OK OK OK OK
185 exception2          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
186 exception3          : OK OK OK OK OK OK OK OK OK OK NG NG NG NG NG  OK OK OK OK OK NG NG NG NG NG NG NG NG NG NG
204 arg_dwarf4          : OK OK NG NG OK SK SK SK SK SK OK OK NG NG OK  OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK
212 noplt_libcall       : OK OK OK OK OK NZ NZ NZ NZ NZ OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
220 trace_script        : NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG  NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG
238 report_field2       : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK NG OK OK OK OK OK OK OK OK OK OK OK OK OK
244 report_task_field   : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK NG OK OK OK OK OK OK OK OK OK NG OK
251 exception4          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NZ NZ NZ NZ NZ OK OK OK OK OK
255 arg_dwarf6          : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK  NG OK OK OK OK SK SK SK SK SK NG OK OK OK OK
273 agent_basic         : NZ OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK
282 agent_depth         : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK NZ OK OK OK OK OK OK OK OK OK OK OK OK OK
283 agent_time          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK OK OK NG OK OK OK OK OK OK OK
289 exception5          : OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK

The only difference is that this includes the fix 827a36a.

$ uftrace --version
uftrace v0.17-44-g827a ( x86_64 dwarf python3 luajit tui perf sched dynamic kernel )

[honggyukim]

After applying the following fix, then many failures are gone.

diff --git a/tests/s-thread.c b/tests/s-thread.c
index df834e6b..5a03d405 100644
--- a/tests/s-thread.c
+++ b/tests/s-thread.c
@@ -21,6 +21,9 @@ static int b(void *arg)
        return c(arg) + 1;
 }

+#if __clang__
+__attribute__((optnone))
+#endif
 static int c(void *arg)
 {
        return *(int *)arg;

This is the result with the fix.

$ cat failed-tests.txt
Compiler                  gcc                                           clang
Runtime test case         pg             finstrument-fu fpatchable-fun  pg             finstrument-fu fpatchable-fun
------------------------: O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os  O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os
084 arg_mixed           : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK  OK NG NG NG OK SK SK SK SK SK OK NG NG NG OK
085 arg_reg             : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK  OK NG NG NG OK SK SK SK SK SK OK NG NG NG OK
124 exception1          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
185 exception2          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
186 exception3          : OK OK OK OK OK OK OK OK OK OK NG NG NG NG NG  OK OK OK OK OK NG NG NG NG NG NG NG NG NG NG
204 arg_dwarf4          : OK OK NG NG OK SK SK SK SK SK OK OK NG NG OK  OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK
212 noplt_libcall       : OK OK OK OK OK NZ NZ NZ NZ NZ OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
220 trace_script        : NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG  NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG
251 exception4          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NZ NZ NZ NZ NZ OK OK OK OK OK
255 arg_dwarf6          : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK  NG OK OK OK OK SK SK SK SK SK NG OK OK OK OK
283 agent_time          : OK OK OK OK OK OK NG OK OK OK OK OK NG OK NG  OK OK NG OK OK OK OK OK OK OK OK OK OK OK NG
289 exception5          : OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK

[honggyukim]

Regarding 220 failures below.

$ ./runtest.py 220
Compiler                  gcc                                           clang
Runtime test case         pg             finstrument-fu fpatchable-fun  pg             finstrument-fu fpatchable-fun
------------------------: O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os  O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os
220 trace_script        : NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG  NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG

The following change makes 220 test pass.

diff --git a/tests/t220_trace_script.py b/tests/t220_trace_script.py
index 9930e74e..ab0be74d 100644
--- a/tests/t220_trace_script.py
+++ b/tests/t220_trace_script.py
@@ -14,9 +14,7 @@ class TestCase(TestBase):
             [28141] | main() {
             [28141] |   a() {
             [28141] |     b() {
-            [28141] |       c() {
-   0.753 us [28141] |         getpid();
-   1.430 us [28141] |       } /* c */
+   1.430 us [28141] |       c();
    1.915 us [28141] |     } /* b */
    2.405 us [28141] |   } /* a */
    3.005 us [28141] | } /* main */
@@ -38,7 +36,7 @@ class TestCase(TestBase):
         return TestBase.build(self, name, cflags, ldflags)

     def setup(self):
-        self.option = "--force -F main"
+        self.option = "--force -F main --no-libcall"
         if self.patchable:
             self.option += " -P .@t-abc"

$ ./runtest.py 220
Compiler                  gcc                                           clang
Runtime test case         pg             finstrument-fu fpatchable-fun  pg             finstrument-fu fpatchable-fun
------------------------: O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os  O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os
220 trace_script        : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK

But the real issue is that the -F main filter didn't work for previously running sh shell process here so not sure if this change is the real fix.

[honggyukim]

The 084 and 085 didn't show up in the previous result in another server, but it fails in some clang tests as follows.

$ ./runtest.py -c clang '084|085'
Start 2 tests with 2 worker

Compiler                  clang
Runtime test case         pg             finstrument-fu fpatchable-fun
------------------------: O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os
084 arg_mixed           : OK NG NG NG OK SK SK SK SK SK OK NG NG NG OK
085 arg_reg             : OK NG NG NG OK SK SK SK SK SK OK NG NG NG OK

$ ./runtest.py -c clang -pvd -O1 '084|085'
        ...
t084_arg_mixed: diff result of clang -pg -O1
--- expect      2025-06-29 09:20:37.516720370 +0900
+++ result      2025-06-29 09:20:37.516720370 +0900
@@ -2,3 +2,3 @@
    mixed_add(-1, 0.200000) = -0.800000;
-   mixed_sub(0x400000, 2048) = 0x3ff800;
+   mixed_sub.specialized.1();
    mixed_mul(-3.000000, 80000000000) = -240000000000;

084 arg_mixed           : NG
t085_arg_reg: diff result of clang -pg -O1
--- expect      2025-06-29 09:20:37.516720370 +0900
+++ result      2025-06-29 09:20:37.516720370 +0900
@@ -2,3 +2,3 @@
    mixed_add(-1, 0.200000);
-   mixed_sub(0x400000, 2048);
+   mixed_sub.specialized.1();
    mixed_mul(-3.000000, 80000000000);

085 arg_reg             : NG

The problem is that mixed_sub is renamed mixed_sub.specialized.1 here because of compiler optimization.

The following makes the tests pass.

diff --git a/tests/s-exp-mixed.c b/tests/s-exp-mixed.c
index 1e51173b..9f678937 100644
--- a/tests/s-exp-mixed.c
+++ b/tests/s-exp-mixed.c
@@ -7,6 +7,9 @@ double mixed_add(int a, float b)
        return a + b;
 }

+#if __clang__
+__attribute__((optnone))
+#endif
 long mixed_sub(void *a, unsigned long b)
 {
        return (long)a - b;

$ ./runtest.py -c clang '084|085'
Start 2 tests with 2 worker

Compiler                  clang
Runtime test case         pg             finstrument-fu fpatchable-fun
------------------------: O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os
084 arg_mixed           : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK
085 arg_reg             : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK

I wanted to fix it with fixup, but doesn't work.

[honggyukim]

I have fixed some failures with #2005.

[namhyung]

On my Fedora machine (with #2005 fixed).

runtime test stats
====================
total  8880  Tests executed (success: 85.68%)
  OK:  7462  Test succeeded
  OK:   146  Test succeeded (with some fixup)
  NG:   167  Different test result
  NZ:    10  Non-zero return value
  SG:    60  Abnormal exit by signal
  TM:     0  Test ran too long
  BI:    90  Build failed
  LA:     0  Unsupported Language
  SK:   945  Skipped

Compiler                  gcc                                           clang                                       
Runtime test case         pg             finstrument-fu fpatchable-fun  pg             finstrument-fu fpatchable-fun
------------------------: O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os  O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os
010 forkexec            : NG OK NG OK NG OK OK OK OK OK NG OK OK OK OK  OK NG OK NG OK OK OK OK NG OK OK NG OK OK OK
051 return              : OK OK NG NG NG OK OK OK OK OK OK OK NG NG NG  OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK
092 report_tid          : NG OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK OK NG OK OK OK OK OK OK OK OK
093 report_filter       : NG NG OK OK OK OK OK NG NG OK OK OK OK OK OK  OK OK OK OK NG OK OK OK OK OK NG OK OK OK OK
094 report_depth        : OK OK OK OK OK OK OK NG OK OK OK OK OK OK OK  OK OK OK OK OK OK OK OK NG OK OK NG OK OK OK
118 thread_tsd          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK OK OK OK OK OK OK OK OK OK NG
120 malloc_tsd          : OK OK OK OK OK NG OK NG OK OK NG OK OK NG OK  OK OK OK OK OK OK OK OK NG OK NG OK OK NG OK
124 exception1          : NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG  NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG
130 thread_exec         : NG OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK
147 event_sdt           : BI BI BI BI BI BI BI BI BI BI BI BI BI BI BI  BI BI BI BI BI BI BI BI BI BI BI BI BI BI BI
150 recv_event          : BI BI BI BI BI BI BI BI BI BI BI BI BI BI BI  BI BI BI BI BI BI BI BI BI BI BI BI BI BI BI
162 pltbind_now_pie     : BI BI BI BI BI BI BI BI BI BI BI BI BI BI BI  BI BI BI BI BI BI BI BI BI BI BI BI BI BI BI
165 graph_sched         : OK OK OK OK OK OK NG OK OK OK OK OK OK OK OK  OK OK OK OK OK NG OK OK OK OK OK OK OK OK OK
168 lib_nested          : SG SG SG SG SG SG SG SG SG SG SG SG SG SG SG  SG SG SG SG SG SG SG SG SG SG SG SG SG SG SG
181 graph_full          : NG OK OK OK OK OK OK OK OK OK OK OK OK OK NG  OK OK OK OK OK OK NG OK OK OK OK NG OK OK OK
182 thread_exit         : OK NG OK OK NG OK OK OK OK NG OK OK OK OK NG  OK OK OK OK OK OK OK OK OK OK NG OK OK OK OK
185 exception2          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
186 exception3          : OK OK OK OK OK OK OK OK OK OK NG NG NG NG NG  OK OK OK OK OK NG NG NG NG NG NG NG NG NG NG
192 lib_name            : SG SG SG SG SG SG SG SG SG SG SG SG SG SG SG  SG SG SG SG SG SG SG SG SG SG SG SG SG SG SG
196 chrome_taskname     : NG OK OK NG OK OK OK OK NG NG NG OK OK OK NG  OK OK OK OK NG NG NG OK NG NG NG OK OK OK NG
204 arg_dwarf4          : OK OK NG NG OK SK SK SK SK SK OK OK NG NG OK  OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK
212 noplt_libcall       : NG NG NG NG NG NZ NZ NZ NZ NZ OK OK OK OK OK  NG NG NG NG NG NG NG NG NG NG OK OK OK OK OK
251 exception4          : NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG  NG NG NG NG NG NZ NZ NZ NZ NZ NG NG NG NG NG
255 arg_dwarf6          : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK  NG OK OK OK OK SK SK SK SK SK NG OK OK OK OK
283 agent_time          : OK OK OK OK OK OK NG OK OK OK OK OK NG OK NG  OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK
289 exception5          : OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK

[honggyukim]

Another test result after having #2005 is as follows.

runtime test stats
====================
total  8880  Tests executed (success: 83.85%)
  OK:  7300  Test succeeded
  OK:   146  Test succeeded (with some fixup)
  NG:    87  Different test result
  NZ:    12  Non-zero return value
  SG:     0  Abnormal exit by signal
  TM:     0  Test ran too long
  BI:     0  Build failed
  LA:     0  Unsupported Language
  SK:  1335  Skipped

$ cat failed-tests.txt
Compiler                  gcc                                           clang
Runtime test case         pg             finstrument-fu fpatchable-fun  pg             finstrument-fu fpatchable-fun
------------------------: O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os  O0 O1 O2 O3 Os O0 O1 O2 O3 Os O0 O1 O2 O3 Os
045 report_avg_self     : OK OK OK OK OK OK OK NG OK OK OK OK OK OK OK  OK OK OK OK OK OK OK OK OK NG OK OK OK OK OK
124 exception1          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
182 thread_exit         : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK NG OK NG OK OK NG OK OK OK OK OK OK OK OK
185 exception2          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
186 exception3          : OK OK OK OK OK OK OK OK OK OK NG NG NG NG NG  OK OK OK OK OK NG NG NG NG NG NG NG NG NG NG
204 arg_dwarf4          : OK OK NG NG OK SK SK SK SK SK OK OK NG NG OK  OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK
212 noplt_libcall       : OK OK OK OK OK NZ NZ NZ NZ NZ OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK
220 trace_script        : NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG  NG NG NG NG NG NG NG NG NG NG NG NG NG NG NG
251 exception4          : OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK  OK OK OK OK OK NZ NZ NZ NZ NZ OK OK OK OK OK
255 arg_dwarf6          : OK OK OK OK OK SK SK SK SK SK OK OK OK OK OK  NG OK OK OK OK SK SK SK SK SK NG OK OK OK OK
273 agent_basic         : NZ OK OK OK OK OK OK OK OK OK OK NZ OK OK OK  OK OK OK OK OK OK OK OK OK OK OK OK OK OK OK
283 agent_time          : OK OK OK OK NG OK NG OK NG OK OK OK OK OK NG  OK OK OK OK OK OK OK OK OK OK NG OK OK OK NG
289 exception5          : OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK  OK OK OK OK OK NG NG NG NG NG OK OK OK OK OK

Please note that I didn't change perf_paranoid so some tests are skipped in this result.