Future of HPy? Is the HPy project still doable? What would help?

[paugier]

After few discussions (in particular in this discuss.python.org thread), I'm more and more convinced that HPy is needed to fix the roots of Python ecosystem. In my point of view, the conclusion of the whole discussion in this thread is quite clear: the Python community needs HPy and we should not wait for anything else to really improve the situation.

Unfortunately,

• very few people seem to agree,
• the activity on HPy is currently very weak.

I open this issue to see if some actions could be taken to improve the situation and to get the point of view of people involved in HPy.

It seems to me that HPy suffers from a vicious cycle related to:

• CPython does not care too much.
• Numpy does not care too much.
• Official instances of Python (PSF, Steering Council, C API working group) are not concerned because there is no related PEP to discuss and/or it's not in their scopes.
• Package maintainers do not want to invest in a third party tool which is not well established, robust and sustainable.
• If HPy is not used in practice why would people care about HPy and try to contribute to the project?
• From outside, HPy looks quite dead.

It would be nice to think if such vicious cycle can be broken and if the HPy project can be relaunched.

The first thing that I can do personally is to write a serious text presenting the current situation and possible plans. I will try to do that but I realize that the impact of such personal opinion can only be very limited.

On the subject, I have to mention this interesting message by Stepan Sindelar. @steve-s mentioned two possibilities for HPy future. The first one is about finalizing the HPy project with new contributors. The second one is about an official alternative C API for Python designed with and by CPython core devs. The answers in the threads (and also the lack of answers) show that it would require a PEP.

A first PEP related to HPy?

It seems to me that anyway, fixing Python ecosystem would need a PEP.

• The current ecosystem is actually hostile to alternative Python implementations and even to any attempts to accelerate Python implementations (in particular CPython). We should cite PEP 733 and give more practical arguments (in particular about the lack of wheels).
• The current direction won't fix that in a reasonable time scale (something than less than 10 years).
• Python needs a project with some kind of official status that would provide what HPy provides (an alternative C API, universal ABI, debug mode, ...). I tend to think that at the end it should be HPy but a PEP like this should keep this question open.
• The PEP should describe a convincing plan to fix the ecosystem with HPy (or something like HPy) in few years.
• The PEP should mention the specific difficulties of a project like HPy if it stays just a third-party project without official status.
• The PEP could mention the second possibility described by @steve-s in its post.
• The PEP could propose that the Steering Council delegates to a dedicated working group to study the best way to fix the ecosystem and reach a satisfactory state in few years.

Current performance issues

We saw that providing HPy with all its advantages is not enough to attract people. In particular, universal wheels are not enough if we cannot show that they will be really fast on alternative implementations. Unfortunately, current results (see for example #486 (comment) and pypy/pypy#3816) are not very convincing to motivate people that the Python community should invest on HPy.

I realize that it is crazy to ask, but it would be a strong argument to be able to state that most Python programs using universal extensions (implemented in C using HPy or in Cython using its HPy backend) would be faster with PyPy than with CPython. At least, HPy universal should always be faster than cpyext on PyPy.

Therefore, it seems to me that improving the results of the HPy microbenchmarks with PyPy and GraalPy is mandatory before trying to relaunch the HPy project. I'd like to ask here if you (@mattip, @cfbolz, @antocuni?) think it could be doable, first theoretically but also taking into account availability of people that could do that in practice?

Raising funding?

It seems to me that it could be possible to raise funding for the global project of fixing Python ecosystem. We have plenty of good arguments to explain that it's important enough (to save time, money and CO₂) so that private companies and public institutions could help the project, by giving money and/or workforce.

It seems to me that it is possible to list a limited number of tasks needed for the whole project. Some are highly technical and require very specific and advanced skills (for example API design and implementation). Others are much "simpler" (maintenance tasks, support Meson or even restart a new Numpy port). I guess that finalizing the work on Cython HPy backend could be seen as an intermediate level.

Anyway, it seems to me that getting few paid people working even part time on the projects would be useful and possible.