dev call 20201105

Dev Call: 5 November 2020

Present

Florian, Tim, Antonio, Ronan, Matti, Simon

Discussion with CPython developers interested in a new GC and removing the GIL

A couple of days ago Antonio, Armin and Carl had a call with some C Python developers. They are thinking of rewriting CPython to use a GC. Their bigger problem is to use C extensions. Initially the GC will be non-moving, but later only some will be pinned. Might start by using Bohm (2x slower than PyPy's current GC).

ACTION: How can we combine forces with these CPython developers?

Larry's email

Simon emailed Larry Hastings, Mark Shannon and Victor Stinner and invited them to the dev call. Larry declined but sent a very nice detailed response via email. The upside is that we are doing all the things Larry thinks we should be doing (handles, functional API, ability to port code incrementally). The downside is that Larry didn't realise we are doing those things despite reading our documentation, so we need to improve our documentation.

ACTION: Documentation -- more user orientated.

• Zero overhead on CPython.
• 2x faster on PyPy and Graal.
• Why we are doing this.
• Examples of problems.
• How to port your C extension.
  • Link to proof of concept.
  • Make sure we test examples.
• Example of how to port your C extension.
  • Philosophy of HPy.
  • Web site.

HPy releases

Matti is doing a PyPy release that will include HPy support. It would be good to be able to point to the version of HPy this PyPy release is compatible with and some good documentation explaining HPy (see above).

ACTION: At least version and tag HPy in the repository. Look at doing a PyPI release before the next PyPy release.

Debug mode

We've always intended to have a debug mode that can be activate without recompiling and without overhead. The time has come to start implementing it.

ACTION: Antonio would like to start the debug mode.

Microbenchmarks

Antonio is working on microbenchmarks (as a way to check that we are not making silly performance mistakes).

For example, method calls on types are currently 20% slower than on CPython, probably for no good reason.

ACTION: Antonio is working on microbenchmarks.

ACTION: Investigate slow method calls on types and fix -- possibly it's related to types being on the heap?.

Packaging

People are looking at HPy already, and some of them are struggling to install HPy and try it out.

Better packaging would help people try things out and engage with HPy.

ACTION: Talk to Pierre Augier a bit more and get feedback and see if he is interested in being more involved.

Package structure -- can we organize it in a less confusing way?

• universal/src contains .h files
• devel is too nested / confusing inside
• devel/src/runtime/ctx is actually cpython only
• Should runtime/ctx files be reimplemented to use only HPy methods or should they move somewhere else.
• We should split out 1) what is cpython only, 2) what is universal only, 3) common header files, 4) shared runtime.

ACTION: Document how to build your own HPy universal.

HPy_Cast

HPy_Cast returns a pointer to the C structure for a custom Python object (i.e. the C structure that holds custom fields for instances of the type) that is valid for as long as the handle used is valid.

Florian raised the issue that this is problematic for GraalPython because a handle may exist for longer than the pointer needs to. This is potentially an issue for other implementations too.

• This requires returning a pinned pointer, and it isn't alway obvious when this pointer is no longer used.
• This can be overcome using HPy_Cast(ctx, HPy_Dup(ctx, h)), but it isn't obvious that one should do this.
• On HPy_Cast on CPython threads will see updates. On Graal this won't happen. What happens on PyPy?
• In DEBUG mode we should copy the memory so it behaves more like on Graal.
• There is already an issue for using HPy_Cast on GitHub around the difference between types with PyObject_Head and those without.

ACTION: Update the existing GitHub issue with the discussion from this call or create a new issue.

Add tests against other Pythons to the HPy CI builds

We could add Pyston, Graal or PyPy builds to the HPy CI. It would be good to be able to see if we are breaking things. The difficulty is that with Graal and PyPy we're breaking things by definition at the moment because they have their own hpy.universal implementations.

Pyston we could build for, but not sure how much immediate advantage there is.

We agreed to not add these for the moment.

Better function pointers

The C API (and HPy) currently use void * for function pointers in numerous places. On certain platforms this will possibly fail. It will be a bit painful to fix, and its unclear if the problem can be solved for the C API.

ACTION: Ask George (who filed the PR -- #23) is this is a genuine problem for Rust when calling Python C extensions, or whether it is more a theoretical issue. If it's a genuine issue, it would be good to understand the issue better.

Implementing a new hpy.universal

Each Python implementation needs to create it's own hpy.univeral implementation. hpy.tools.autogen is intended to help with this but currently it's undocumented and a bit difficult to use.

Autogen should be more officially documented and usable from other Python implementations (Graal, PyPy, Rust).

ACTION: Document how to use autogen.

ACTION: Add tests that it autogen works.

Put dev call meeting notes in GitHub wiki

We should record meeting notes so those not in attendance can read them and so that we can remember what we decided on before.

ACTION: Put meeting notes into GitHub wiki.

Visibility of GitHub issues

Some contributors receive many many GitHub notifications, so it would be easier for them to be notified of new HPy issues via IRC.

ACTION: Make GitHub issues appear on IRC.

Testing on Windows

It would be good to run CI tests on Windows and not too difficult to add. Azure Pipelines should support Windows and we can borrow the module loading code for Windows from CFFI.

However, it's low priority right now.

ACTION: Wait. :)

Topics deferred to the next dev call

• Python linting and style (flake8, black)
• C linting and style (??)
• C docstrings.
• Python docstrings.