[C API] Add `PyTupleWriter` API

[vstinner]

Feature or enhancement

Hi,

Creating a tuple with the current C API has multiple issues:

• PyTuple_SetItem() and PyTuple_SET_ITEM() modify an immutable tuple.
• PyTuple_New() creates an incomplete object: items are set to NULL. This is bad:
  • Disallow creation of incomplete/inconsistent objects capi-workgroup/problems#56
  • Avoid creating incomplete/invalid objects capi-workgroup/api-evolution#36
• PyTuple_New() tracks directly the tuple in the garbage collector. For example, gc.get_objects() gives access to the incomplete tuple. Using the tuple, like calling repr(tuple), can crash Python.
• _PyTuple_Resize() is private which is surprising for a documented API. It stays private because it has issues.
• _PyTuple_Resize() modifies an immutable tuple.
• _PyTuple_Resize() must not be used of the refcount is greater than 1: the API is fragile.

I propose adding a new efficient PyTupleWriter API: work on a temporary "writer" object, and then call Finish() on it to get the tuple.

I already proposed a similar API in 2023: _PyTupleBuilder. Since that, Python C API got the PyUnicodeWriter API and the PyBytesWriter API which are efficient writers for str and bytes objects, and the C API Working Group was created. The proposed API is now public and allocates the structure on the heap memory to hide the implementation details (the structure).

Mark Shannon asked if it would be possible to work on a list and then convert the list to a tuple, but it's less efficient. Tuples are commonly used in Python, and so creating a tuple should be efficient.

Mark Shannon also tried to initialize tuple items to None instead of NULL in PyTuple_New(). His attempt failed because of implementation issues. Also, this change only fix some of the issues that I listed, not all of them.

An alternative is to fill a C array of Python objects, call the new PyTuple_FromArray(), and then call Py_DECREF() on the array items. It requires to allocate and deallocate an array, and call Py_DECREF() on items. It can be less efficient.

Linked PRs

• gh-139888: Add PyTupleWriter C API #139891
• gh-139888: Add PyTupleWriter C API (stack flavor) #140129

[zooba]

... but why? We're adding PyTuple_FromArray already, and it's going to be just as easy for a user to fill out a native array and turn it into a tuple in one go. The majority of tuples aren't variable length, they're known at compile time (because that's what a tuple represents - a list is for dynamically sized objects).

Where's the real world benchmarks showing that this API is worth any performance or correctness benefit? We can still deprecate the bad APIs in favour of "just use a native array until you're ready".

[vstinner]

The majority of tuples aren't variable length

This API is mostly a replacement to _PyTuple_Resize(): function when the input size is not known in advance. In that case, you should allocate a few items, fill these items, allocate more items, etc.

Having to manage manually the buffer/array is non trivial and so PyTupleWriter offers a high-level API for that.

When the input size is known, there are other existing safe functions: PyTuple_FromArray() (new! I just added it), PyTuple_Pack(), Py_BuildValue(), etc.

Where's the real world benchmarks showing that this API is worth any performance or correctness benefit?

I ran a micro-benchmark: #139891 (comment)

About correctness, this API should fix bugs when it is possible to discover an incomplete tuple through the GC:

• Incomplete tuple created by PyTuple_New() and accessed via the GC can trigged a crash #59313: the main issue about the GC tracking issue
• gc.get_referrers() is inherently dangerous #39117
• segfault due to null pointer in tuple #70998
• Correct reuse argument tuple in property descriptor #68464

PySequence_Tuple() was fixed recently (Python 3.14) against such bug: commit 5a23994.

[sergey-miryanov]

What do you think about adding a PyTuple_FromSingle and PyTuple_FromPair (or PyTuple_MakeSingle and PyTuple_MakePair) (according to my measurements for pyperformance 1-size and 2-size tuples have about 80% of occurrences)?

Following is a data of occurrences of calls that can be replaced with PyTuple_FromSingle and PyTuple_FromPair:

file	function	count
_asyncmodule.c	PyTuple_New(2)	2
_collectionsmodule.c	PyTuple_Pack(1)	2
_csv.c	PyTuple_Pack(1)	1
_datetimemodule.c	PyTuple_Pack(2)	4
	PyTuple_Pack(1)	3
_elementtree.c	PyTuple_Pack(2)	4
_functoolsmodule.c	PyTuple_New(2)	2
_interpretersmodule.c	PyTuple_Pack(2)	1
_json.c	PyTuple_New(2)	1
	PyTuple_Pack(2)	1
	PyTuple_Pack(1)	1
_operator.c	PyTuple_Pack(2)	1
_pickle.c	PyTuple_Pack(2)	3
	PyTuple_New(2)	2
	PyTuple_New(1)	2
_ssl.c	PyTuple_New(2)	6
	PyTuple_Pack(2)	1
_threadmodule.c	PyTuple_New(2)	1
_tkinter.c	PyTuple_Pack(1)
arraymodule.c	PyTuple_New(2)	2
itertoolsmodule.c	PyTuple_Pack(2)	2
	PyTuple_New(2)	1
main.c	PyTuple_Pack(2)	1
overlapped.c	PyTuple_New(2)	2
posixmodule.c	PyTuple_Pack(2)	1
pyexpat.c	PyTuple_New(1)	1
selectmodule.c	PyTuple_Pack(2)	1
	PyTuple_New(2)	1
signal_module.c	PyTuple_New(2)	1
socket_module.c	PyTuple_Pack(2)	3
termios.c	PyTuple_New(2)	2
_ctypes.c	PyTuple_Pack(2)	2
stgdict.c	PyTuple_Pack(2)	1
decimal.c	PyTuple_Pack(2)	7
	PyTuple_Pack(1)	2
microprotocol.c	PyTuple_Pack(2)	2
_sre.c	PyTuple_New(2)	1
datetime.c	PyTuple_Pack(1)	2
	PyTuple_Pack(2)	1
getargs.c	PyTuple_Pack(1)	1
heaptype.c	PyTuple_Pack(2)	1
	PyTuple_Pack(1)	2
	PyTuple_New(2)	1
vectorcall_limited.c	PyTuple_New(1)	2
multibytecodec.c	PyTuple_New(2)	1
codeobject.c	PyTuple_Pack(2)	7
dictobject.c	PyTuple_Pack(2)	2
	PyTuple_New(2)	4
enumobject.c	PyTuple_Pack(2)	1
	PyTuple_New(2)	2
exceptions.c	PyTuple_Pack(2)	7
floatobject.c	PyTuple_Pack(2)	1
frameobject.c	PyTuple_Pack(2)	2
	PyTuple_Pack(1)	1
genericaliasobject.c	PyTuple_Pack(1)	2
listobject.c	PyTuple_Pack(2)	1
longobject.c	PyTuple_Pack(2)	1
	PyTuple_New(2)	2
odictobject.c	PyTuple_Pack(2)	2
	PyTuple_New(2)	1
setobject.c	PyTuple_Pack(1)	1
typeobject.c	PyTuple_Pack(2)	2
	PyTuple_Pack(1)	5
typevarobject.c	PyTuple_Pack(2)	1
	PyTuple_Pack(1)	2
unicode_format.h	PyTuple_Pack(2)	2
pegen_errors.c	PyTuple_Pack(2)	2
_warnings.c	PyTuple_Pack(2)	1
bltnmodule.c	PyTuple_Pack(2)	1
ceval.c	PyTuple_Pack(1)	1
_codegen.c	PyTuple_Pack(1)	2
compile.c	PyTuple_Pack(2)	1
crossinterp.c	PyTuple_Pack(1)	1
errors.c	PyTuple_Pack(1)	1
hamt.c	PyTuple_Pack(2)	1
marshal.c	PyTuple_Pack(2)	1
pylifecycle.c	PyTuple_Pack(2)	1
Python-tokenize.c	PyTuple_Pack(2)	1
sysmodule.c	PyTuple_Pack(1)	1
tracemalloc.c	PyTuple_New(2)	1

Is it worth to implement those methods, replace in the codebase and benchmark it?
I believe that my question interleaves with #140009 a bit.

[eendebakpt]

What do you think about adding a PyTuple_FromSingle and PyTuple_FromPair (or PyTuple_MakeSingle and PyTuple_MakePair) (according to my measurements for pyperformance 1-size and 2-size tuples have about 80% of occurrences)?

This was suggested in #118222

[vstinner]

What do you think about adding a PyTuple_FromSingle and PyTuple_FromPair (or PyTuple_MakeSingle and PyTuple_MakePair) (according to my measurements for pyperformance 1-size and 2-size tuples have about 80% of occurrences)?

Would you mind to open a separated issue for that?

[sergey-miryanov]

@eendebakpt Thanks!
@vstinner Done - #140052.