Skip to content

Conversation

@augelu-tng
Copy link
Contributor

@augelu-tng augelu-tng commented Nov 7, 2025

This PR replaces the usage of pathlib.Path objects with plain strings for performance improvements.
Creating the cmd graph on the tinyconfig using Path objects takes ~13 seconds:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   13.452   13.452 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/cmd_graph.py:32(build_cmd_graph)
        1    0.000    0.000   13.452   13.452 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/cmd_graph.py:34(<listcomp>)
 237524/1    0.568    0.000   13.452   13.452 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/cmd_graph.py:41(build_cmd_graph_node)
      920    0.286    0.000    7.719    0.008 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/cmd_graph.py:102(_parse_cmd_file)
  1202765    0.831    0.000    5.211    0.000 /usr/lib/python3.10/pathlib.py:569(_parse_args)
  1202765    2.775    0.000    4.120    0.000 /usr/lib/python3.10/pathlib.py:56(parse_parts)
   714104    0.275    0.000    3.835    0.000 /usr/lib/python3.10/pathlib.py:957(__new__)
   714104    0.326    0.000    3.560    0.000 /usr/lib/python3.10/pathlib.py:589(_from_parts)
   480113    0.119    0.000    2.744    0.000 /usr/lib/python3.10/pathlib.py:853(__truediv__)
   480113    0.328    0.000    2.626    0.000 /usr/lib/python3.10/pathlib.py:615(_make_child)
   475043    1.004    0.000    2.103    0.000 /usr/lib/python3.10/posixpath.py:338(normpath)
      505    0.276    0.001    1.776    0.004 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/deps_parser.py:13(parse_deps)
  921/920    0.004    0.000    1.166    0.001 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/savedcmd_parser.py:562(parse_commands)
  1429834    0.231    0.000    0.897    0.000 {built-in method posix.fspath}
   965277    0.454    0.000    0.704    0.000 /usr/lib/python3.10/pathlib.py:621(__str__)
   721113    0.151    0.000    0.678    0.000 /usr/lib/python3.10/pathlib.py:631(__fspath__)
      919    0.008    0.000    0.674    0.001 /usr/lib/python3.10/shlex.py:305(split)
    51916    0.010    0.000    0.661    0.000 /usr/lib/python3.10/shlex.py:299(__next__)
      504    0.002    0.000    0.660    0.001 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/savedcmd_parser.py:200(_parse_gcc_or_clang_command)
    51916    0.015    0.000    0.652    0.000 /usr/lib/python3.10/shlex.py:101(get_token)

A lot of time is spent in the pathlib library simply converting between str and Path objects and performing simple path operations.

With the changes in this PR the total runtime gets more than halved to <5 seconds while the generated sbom.spdx.json remains the same:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    4.726    4.726 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/cmd_graph.py:32(build_cmd_graph)
        1    0.000    0.000    4.726    4.726 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/cmd_graph.py:34(<listcomp>)
 237524/1    0.267    0.000    4.726    4.726 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/cmd_graph.py:41(build_cmd_graph_node)
      920    0.160    0.000    2.932    0.003 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/cmd_graph.py:102(_parse_cmd_file)
   475043    0.881    0.000    1.258    0.000 /usr/lib/python3.10/posixpath.py:338(normpath)
  921/920    0.004    0.000    1.198    0.001 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/savedcmd_parser.py:562(parse_commands)
      919    0.010    0.000    0.687    0.001 /usr/lib/python3.10/shlex.py:305(split)
      504    0.003    0.000    0.677    0.001 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/savedcmd_parser.py:200(_parse_gcc_or_clang_command)
    51916    0.013    0.000    0.673    0.000 /usr/lib/python3.10/shlex.py:299(__next__)
    51916    0.016    0.000    0.660    0.000 /usr/lib/python3.10/shlex.py:101(get_token)
    51916    0.540    0.000    0.644    0.000 /usr/lib/python3.10/shlex.py:133(read_token)
   476313    0.360    0.000    0.577    0.000 /usr/lib/python3.10/posixpath.py:71(join)
      923    0.003    0.000    0.473    0.001 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/savedcmd_parser.py:533(_split_commands)
     1324    0.402    0.000    0.469    0.000 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/savedcmd_parser.py:493(_find_first_top_level_command_separator)
      505    0.176    0.000    0.452    0.001 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/deps_parser.py:13(parse_deps)
      920    0.162    0.000    0.447    0.000 /home/luis/code/WSKernelSbom/KernelSbom/sbom_analysis/profiling/../../sbom/lib/sbom/cmd/cmd_file_parser.py:22(parse_cmd_file)
   975073    0.250    0.000    0.250    0.000 {method 'match' of 're.Pattern' objects}
  4149642    0.204    0.000    0.204    0.000 {method 'append' of 'list' objects}
   237516    0.093    0.000    0.177    0.000 /usr/lib/python3.10/posixpath.py:60(isabs)
   717736    0.102    0.000    0.152    0.000 /usr/lib/python3.10/posixpath.py:41(_get_sep)

Successful Testrun

@augelu-tng augelu-tng changed the title Profiling Replace Path objects with str Nov 7, 2025
@augelu-tng augelu-tng force-pushed the profiling branch 8 times, most recently from 283b664 to e0c0bda Compare November 11, 2025 10:41
@augelu-tng augelu-tng marked this pull request as ready for review November 11, 2025 10:44
@augelu-tng augelu-tng force-pushed the profiling branch 2 times, most recently from 2eeb463 to c07b8e9 Compare November 11, 2025 10:55
Signed-off-by: Luis Augenstein <[email protected]>
@maxhbr maxhbr merged commit 62cc99a into main Nov 11, 2025
2 checks passed
@augelu-tng augelu-tng deleted the profiling branch November 12, 2025 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants