function iter_DIEs is verify slow， 1200000 dies takes 45 seconds

[chall1123]

function iter_DIEs is very slow， 1200000 dies taking 45 seconds， I want to have multiple threads to process one cu per thread ，but the thread will report an error in iter_DIES, or is there any other way to speed up the traversal，I want to traversal all dies in 3 seconds

[chall1123]

[ltfish]

You are probably better off using a different library instead of pyelftools.

[chall1123]

You are probably better off using a different library instead of pyelftools.

If pyelftools is used, is there a way to speed it up? For example, when iter_DIES, only the die with tag DW_TAG_Subprogram is traversed

[ltfish]

Based on my past profiling experience, I don't think you can make it so that the DIE traversal (using pyelftools) completes within three seconds. There is definitely room for improvement though.

[sevaa]

May I see the thread initiation code? Not as a screenshot, as text.

That said, using a non-Python DWARF parser is solid advice. :)

EDIT: now that I think of it, the underlying stream (shared by all CUs' iterators) would be the source of errors. Can be solved. @eliben this goes back to what we sort of discussed, how we are losing performance by treating DWARF sections as streams instead of memory buffers that they are. Construct and its streamy design is holding us back.

[chall1123]

May I see the thread initiation code? Not as a screenshot, as text.

That said, using a non-Python DWARF parser is solid advice. :)

EDIT: now that I think of it, the underlying stream (shared by all CUs' iterators) would be the source of errors. Can be solved. @eliben this goes back to what we sort of discussed, how we are losing performance by treating DWARF sections as streams instead of memory buffers that they are. Construct and its streamy design is holding us back.

def process_cu(cu) :
for die in cu.iter_DIEs():
pass
def parallel_dwarf_parse(file_path):
with open(file_path, 'rb') as f:
thread = []
elffile = ELFFile(f)
if not elffile.has_dwarf_info():
return
dwarfInfo = elffile.get_dwarf_info()
for cu in dwarfInfo.iter_CUs():
th = threading.Thread(target=process_cu, args=(cu,))
th.start()
thread.append(th)
[t.join() for t in thread]
return

error info:

If I want to use elftools, is there any way to speed up parsing

[sevaa]

Yeah, I have repro. Thanks. OBTW, are you using the free threaded mode of Python 2.13? For a memory-only operation such as DIE parsing, I feel you could get quite a bit of performance from the free threaded mode, once pyelftools is ready for that. Also, since there is overhead to threading support, I'd prefer to not use it by default, and use the no-GIL flag (sys._is_gil_enabled()) as a trigger.

@eliben: I could provide a fix that would allow for this kind of CU-driven parallelism. One caveat though. My idea is to construct multiple lightweight streams around the info section buffer for parallel access. Right now, we don't store the info section buffer. We store a BytesIO built around it. BytesIO is a blackbox. I have no idea whether/when does it copy the underlying buffer. Storing both the BytesIO and the buffer runs the risk of consuming twice the memory. I could replace the BytesIO with a lightweight read-only stream class (I have one already) that I know for a fact doesn't duplicate, but that may break things for library users out there - in a contrived scenario where they get the stream from a pyelftools' objects and work with it assuming it was a BytesIO. Unlikely, but who knows. Is that a risk we'd be willing to take?

OBTW, I could also provide a fix as a sample as opposed to a PR - just for this case. It will be an ugly monkeypatch.

[eliben]

@sevaa it's worth trying an experiment if you have the time. It'd be interesting to look at a PR to see how the code is affected (in terms of API change), as well as the performance effects

[chall1123]

Yeah, I have repro. Thanks. OBTW, are you using the free threaded mode of Python 2.13? For a memory-only operation such as DIE parsing, I feel you could get quite a bit of performance from the free threaded mode, once pyelftools is ready for that. Also, since there is overhead to threading support, I'd prefer to not use it by default, and use the no-GIL flag (sys._is_gil_enabled()) as a trigger.

@eliben: I could provide a fix that would allow for this kind of CU-driven parallelism. One caveat though. My idea is to construct multiple lightweight streams around the info section buffer for parallel access. Right now, we don't store the info section buffer. We store a BytesIO built around it. BytesIO is a blackbox. I have no idea whether/when does it copy the underlying buffer. Storing both the BytesIO and the buffer runs the risk of consuming twice the memory. I could replace the BytesIO with a lightweight read-only stream class (I have one already) that I know for a fact doesn't duplicate, but that may break things for library users out there - in a contrived scenario where they get the stream from a pyelftools' objects and work with it assuming it was a BytesIO. Unlikely, but who knows. Is that a risk we'd be willing to take?

OBTW, I could also provide a fix as a sample as opposed to a PR - just for this case. It will be an ugly monkeypatch.

I am using Python 3.8, completely default behavior, without specifically setting the free threaded mode . Pyelftools may not be as fast as C language libraries such as libdw, but if it can support multi-threading, large binary parsing can really be much faster

[sevaa]

I can 99% promise that you won't get any performance enhancement from using threads under Python 3.8. Python notoriously has a GIL, global interpreter lock. When Python code executes on a thread (as opposed to, e. g., waiting for I/O), all other threads are stalled. There is no I/O in the DIE parsing logic - it's a completely in memory operation.

[chall1123]

I can 99% promise that you won't get any performance enhancement from using threads under Python 3.8. Python notoriously has a GIL, global interpreter lock. When Python code executes on a thread (as opposed to, e. g., waiting for I/O), all other threads are stalled. There is no I/O in the DIE parsing logic - it's a completely in memory operation.

thanks， if pyelftools support muti-threading, I can update python version

[sevaa]

Funny thing, with threads (even with handful of them) I'm getting weird overlarge memory consumption - up to 20GB in peak. The file I'm testing on consumes about 2 GB in single threaded mode (it's big). I'm getting MemoryErrors on threads and the process crashes.

A part of the blame is on pyelftools - it caches aggressively (see #586), but still. 2 GB is due to the caches, but 20 is crazy. It reproduces consistently.

I guess the experimental free threaded mode is not all it's cracked up to be. Sorry, I've tried. As a side note, running the single threaded DIE scan under the free threaded Python increases the run time by ~30%. Such is the overhead of basic thread sync in Python internals alone.

I feel there is still some performance to be squeezed out of single threaded DIE parsing performance, but we'd be lucky if we got a two-fold improvement.

[ltfish]

@sevaa This issue might be causing the memory usage behavior you saw: python/cpython#133136

[nascheme]

Hi @sevaa, could you test with the latest Python 3.14 release (currently rc1)? It includes a fix for the issue Itfish linked to above. The fix is backported to 3.13 as well but the 3.13.x release containing that fix is not out yet. If you can make an example test case, I would try running it myself.

[sevaa]

I'd rather hold on until the proper 3.14 release.

Unrelated, I've noticed that CompileUnit.iter_DIEs() is unnecessarily convoluted in terms of parse logic. Its purpose it to return the DIEs in the sequential order, but instead it relies on the logic of CompileUnit.iter_DIE_children(), which contains the more complicated logic of going over immediate children of a DIE. Especially on a siblingless DIE tree, this will contain extra work of parsing the whole subtree only to throw away the children of children.

Physically, DIEs are stored linearly. Logically, they are a tree. iter_DIEs() is expected to return a flat collection, but it does so by building a tree (out of a flat collection) and then flattening it. I feel this could be streamlined.

The downside would be that the logic of setting parents during parsing (normally done by iter_DIE_children()) would be somewhat duplicated. Since #611 seems to be stalled, I'll try to do some tests, and if we win any time, a PR.

UPDATE: at a first stab, I've got about 7% performance increase for a firehose CU/DIE scan. @eliben - worth it or not?