Skip to content

gh-140149: use PyBytesWriter in action_helpers.c's _build_concatenated_bytes; 3x faster bytes concat in the parser #140150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
+35 −6
Open

gh-140149: use PyBytesWriter in action_helpers.c's _build_concatenated_bytes; 3x faster bytes concat in the parser #140150

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
692a4f0
use PyBytesWriter in _build_concatenated_bytes
maurycy Oct 15, 2025
904fe91
no var read twice
maurycy Oct 15, 2025
bde6eaf
NEWS
maurycy Oct 15, 2025
e5b07d6
Update Parser/action_helpers.c
maurycy Oct 15, 2025
9955759
Update Parser/action_helpers.c
maurycy Oct 15, 2025
3d375ce
Update Misc/NEWS.d/next/Core_and_Builtins/2025-10-15-17-12-32.gh-issu…
vstinner Oct 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 31 additions & 6 deletions Parser/action_helpers.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1612,19 +1612,44 @@ _build_concatenated_bytes(Parser *p, asdl_expr_seq *strings, int lineno,
Py_ssize_t len = asdl_seq_LEN(strings);
assert(len > 0);

PyObject* res = Py_GetConstant(Py_CONSTANT_EMPTY_BYTES);

/* Bytes literals never get a kind, but just for consistency
since they are represented as Constant nodes, we'll mirror
the same behavior as unicode strings for determining the
kind. */
PyObject* kind = asdl_seq_GET(strings, 0)->v.Constant.kind;
PyObject *kind = asdl_seq_GET(strings, 0)->v.Constant.kind;

Py_ssize_t total = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bit of a meta question: How much performance change is it to not pre-calculate the length? The precalculation + memcpy makes this code quite a bit more complex. If it's only a couple percentage difference (so most the 3x is kept) the simpler code for some of these would be nice

for (Py_ssize_t i = 0; i < len; i++) {
expr_ty elem = asdl_seq_GET(strings, i);
Py_ssize_t part = PyBytes_GET_SIZE(elem->v.Constant.value);
if (part > PY_SSIZE_T_MAX - total) {
PyErr_NoMemory();
return NULL;
}
total += part;
}

PyBytesWriter *writer = PyBytesWriter_Create(total);
if (writer == NULL) {
return NULL;
}

char *out = PyBytesWriter_GetData(writer);
for (Py_ssize_t i = 0; i < len; i++) {
expr_ty elem = asdl_seq_GET(strings, i);
PyBytes_Concat(&res, elem->v.Constant.value);
Py_ssize_t part = PyBytes_GET_SIZE(elem->v.Constant.value);
if (part > 0) {
memcpy(out, PyBytes_AS_STRING(elem->v.Constant.value), part);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not using PyBytesWriter_WriteBytes here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyBytesWriter_WriteBytes() grows the buffer if needed. It's not needed since the code already computes the total size in advance.

out += part;
}
}
if (!res || _PyArena_AddPyObject(arena, res) < 0) {
Py_XDECREF(res);

PyObject *res = PyBytesWriter_FinishWithPointer(writer, out);
if (res == NULL) {
return NULL;
}
if (_PyArena_AddPyObject(arena, res) < 0) {
Py_DECREF(res);
return NULL;
}
return _PyAST_Constant(res, kind, lineno, col_offset, end_lineno, end_col_offset, p->arena);
Expand Down
Loading