Helper microbench

[paugier]

For the record, I needed that to run HPy microbenchmarks.

[hodgestar]

Left some questions, but seems helpful. Thank you.

[hodgestar]

Any thoughts on how to improve the general HPy situation here?

[hodgestar]

@paugier Some of the tests are failing. It looks unrelated but it would be nice to understand why.

[paugier]

@paugier Some of the tests are failing. It looks unrelated but it would be nice to understand why.

I fixed the job related to microbench but the CI is broken for unrelated things:

         # now recompile with universal *without* cleaning the build
        self.python('setup.py', '--hpy-abi=universal', 'install')
        # in the build/ dir, we should have 4 directories: 2 temp*, and 2 lib*
        build = self.hpy_test_project.join('build')
        temps = build.listdir('temp*')
        libs = build.listdir('lib*')
        assert len(temps) == 2
        assert len(libs) == 2
        #
        doc = self.get_docstring('hpymod')
>       assert doc == 'hpymod with HPy ABI: universal'
E       AssertionError: assert 'hpymod with HPy ABI: cpython' == 'hpymod with ...BI: universal'
E         
E         - hpymod with HPy ABI: universal
E         ?                      ^ -------
E         + hpymod with HPy ABI: cpython
E         ?                      ^^^^^^

/home/runner/work/hpy/hpy/test/hpy_devel/test_distutils.py:327: AssertionError

Note that this test calls python setup.py ... which is anyway deprecated.

By the way, there are many calls of setup.py in HPy repository and it would be better to replace them with modern alternatives.

[paugier]

For the record, the result of last commands in the README (on a machine called meige8pcpa79):

=================================== CPython ====================================
                                                     cpy                    hpy
                                        ----------------    -------------------
TestModule::test_noargs                        504.73 us       515.68 us [1.02]
TestModule::test_onearg_None                   486.32 us       484.58 us [1.00]
TestModule::test_onearg_int                    454.54 us       454.20 us [1.00]
TestModule::test_varargs                       770.54 us       460.98 us [0.60]
TestModule::test_call_with_tuple              1517.38 us      1106.61 us [0.73]
TestModule::test_call_with_tuple_and_dict      2981.37 us      2544.76 us [0.85]
TestModule::test_allocate_int                  710.28 us       701.40 us [0.99]
TestModule::test_allocate_tuple               1318.51 us      1467.16 us [1.11]
TestType::test_allocate_obj                    908.30 us      1093.94 us [1.20]
TestType::test_method_lookup                   926.17 us       960.43 us [1.04]
TestType::test_noargs                          386.56 us       393.36 us [1.02]
TestType::test_onearg_None                     416.85 us       445.93 us [1.07]
TestType::test_onearg_int                      411.62 us       419.57 us [1.02]
TestType::test_varargs                         731.76 us       425.61 us [0.58]
TestType::test_len                             430.28 us       413.93 us [0.96]
TestType::test_getitem                         413.99 us       418.32 us [1.01]
TestHeapType::test_allocate_obj_and_survive      1015.44 us      1095.10 us [1.08]
TestHeapType::test_allocate_obj_and_die       1004.35 us       999.09 us [0.99]
TestHeapType::test_method_lookup               913.01 us       917.65 us [1.01]
TestHeapType::test_noargs                      382.78 us       386.69 us [1.01]
TestHeapType::test_onearg_None                 418.65 us       414.32 us [0.99]
TestHeapType::test_onearg_int                  411.64 us       419.59 us [1.02]
TestHeapType::test_varargs                     743.31 us       425.37 us [0.57]
TestHeapType::test_len                         415.09 us       425.64 us [1.03]
TestHeapType::test_getitem                     413.62 us       418.73 us [1.01]

============================= 50 passed in 37.27s ==============================
==================================== PyPy ======================================
                                                     cpy                    hpy
                                        ----------------    -------------------
TestModule::test_noargs                        310.62 us       251.10 us [0.81]
TestModule::test_onearg_None                   332.27 us       290.19 us [0.87]
TestModule::test_onearg_int                   1924.47 us       293.98 us [0.15]
TestModule::test_varargs                       573.54 us       713.54 us [1.24]
TestModule::test_call_with_tuple              6421.06 us      3352.38 us [0.52]
TestModule::test_call_with_tuple_and_dict     14394.19 us      4473.64 us [0.31]
TestModule::test_allocate_int                 1981.56 us       476.53 us [0.24]
TestModule::test_allocate_tuple               8578.29 us      8346.12 us [0.97]
TestType::test_allocate_obj                  14106.48 us      5354.26 us [0.38]
TestType::test_method_lookup                    24.35 us        46.08 us [1.89]
TestType::test_noargs                          286.63 us       328.95 us [1.15]
TestType::test_onearg_None                     313.02 us       396.81 us [1.27]
TestType::test_onearg_int                     1563.72 us       401.30 us [0.26]
TestType::test_varargs                         564.73 us       860.63 us [1.52]
TestType::test_len                             130.50 us       207.08 us [1.59]
TestType::test_getitem                         245.93 us       300.59 us [1.22]
TestHeapType::test_allocate_obj_and_survive      1664.91 us      5185.84 us [3.11]
TestHeapType::test_allocate_obj_and_die       2867.47 us      4092.15 us [1.43]
TestHeapType::test_method_lookup                23.83 us        43.30 us [1.82]
TestHeapType::test_noargs                      312.28 us       322.34 us [1.03]
TestHeapType::test_onearg_None                 347.38 us       389.76 us [1.12]
TestHeapType::test_onearg_int                 1635.76 us       396.83 us [0.24]
TestHeapType::test_varargs                     581.25 us       893.49 us [1.54]
TestHeapType::test_len                         150.09 us       206.24 us [1.37]
TestHeapType::test_getitem                     273.73 us       324.19 us [1.18]

================== 50 passed, 1 warning in 105.45s (0:01:45) ===================
=================================== GraalPy ====================================
                                                     cpy                    hpy
                                        ----------------    -------------------
TestModule::test_noargs                                       1158.00 us 
TestModule::test_onearg_None                                   799.00 us 
TestModule::test_onearg_int                                   3618.00 us 
TestModule::test_varargs                                      1166.00 us 
TestModule::test_call_with_tuple                             12371.00 us 
TestModule::test_call_with_tuple_and_dict                      9556.00 us 
TestModule::test_allocate_int                                  648.00 us 
TestModule::test_allocate_tuple                               3750.00 us 
TestType::test_allocate_obj                                  19748.00 us 
TestType::test_method_lookup                                  2278.00 us 
TestType::test_noargs                                         2310.00 us 
TestType::test_onearg_None                                    2470.00 us 
TestType::test_onearg_int                                     1995.00 us 
TestType::test_varargs                                        1360.00 us 
TestType::test_len                                            2152.00 us 
TestType::test_getitem                                        1364.00 us 
TestHeapType::test_allocate_obj_and_survive                     20557.00 us 
TestHeapType::test_allocate_obj_and_die                      21019.00 us 
TestHeapType::test_method_lookup                               610.00 us 
TestHeapType::test_noargs                                     1985.00 us 
TestHeapType::test_onearg_None                                1473.00 us 
TestHeapType::test_onearg_int                                 2445.00 us 
TestHeapType::test_varargs                                    2307.00 us 
TestHeapType::test_len                                        1403.00 us 
TestHeapType::test_getitem                                    4555.00 us 

================ 25 passed, 25 deselected in 126.03s (0:02:06) =================

========== PyPy HPy univ / CPy native (time ratio, smaller is better) ==========
TestModule::test_noargs                       0.50
TestModule::test_onearg_None                  0.60
TestModule::test_onearg_int                   0.65
TestModule::test_varargs                      0.93
TestModule::test_call_with_tuple              2.21
TestModule::test_call_with_tuple_and_dict     1.50
TestModule::test_allocate_int                 0.67
TestModule::test_allocate_tuple               6.33
TestType::test_allocate_obj                   5.89
TestType::test_method_lookup                  0.05
TestType::test_noargs                         0.85
TestType::test_onearg_None                    0.95
TestType::test_onearg_int                     0.97
TestType::test_varargs                        1.18
TestType::test_len                            0.48
TestType::test_getitem                        0.73
TestHeapType::test_allocate_obj_and_survive   5.11
TestHeapType::test_allocate_obj_and_die       4.07
TestHeapType::test_method_lookup              0.05
TestHeapType::test_noargs                     0.84
TestHeapType::test_onearg_None                0.93
TestHeapType::test_onearg_int                 0.96
TestHeapType::test_varargs                    1.20
TestHeapType::test_len                        0.50
TestHeapType::test_getitem                    0.78

========= GraalPy HPy univ / CPy native (time ratio, smaller is better) =========
TestModule::test_noargs                       2.29
TestModule::test_onearg_None                  1.64
TestModule::test_onearg_int                   7.96
TestModule::test_varargs                      1.51
TestModule::test_call_with_tuple              8.15
TestModule::test_call_with_tuple_and_dict     3.21
TestModule::test_allocate_int                 0.91
TestModule::test_allocate_tuple               2.84
TestType::test_allocate_obj                   21.74
TestType::test_method_lookup                  2.46
TestType::test_noargs                         5.98
TestType::test_onearg_None                    5.93
TestType::test_onearg_int                     4.85
TestType::test_varargs                        1.86
TestType::test_len                            5.00
TestType::test_getitem                        3.29
TestHeapType::test_allocate_obj_and_survive   20.24
TestHeapType::test_allocate_obj_and_die       20.93
TestHeapType::test_method_lookup              0.67
TestHeapType::test_noargs                     5.19
TestHeapType::test_onearg_None                3.52
TestHeapType::test_onearg_int                 5.94
TestHeapType::test_varargs                    3.10
TestHeapType::test_len                        3.38
TestHeapType::test_getitem                    11.01

For PyPy most microbenchmarks are good but there are few known problems (see pypy/pypy#3816).

[hodgestar]

@paugier If you're happy with this now, I'll merge.

[paugier]

@paugier If you're happy with this now, I'll merge.

I happy with this state so as you want. If #487 is merged before, this one could be rebased and then merged with a green CI.

[hodgestar]

I happy with this state so as you want. If #487 is merged before, this one could be rebased and then merged with a green CI.

#487 merged. Are you happy to do the rebasing?

[paugier]

Done and CI is green!

[hodgestar]

@paugier Thanks again for this!