Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Implement cudf.pandas Profiler using sys.monitoring #17519

Open
mroeschke opened this issue Dec 4, 2024 · 0 comments
Open

[FEA] Implement cudf.pandas Profiler using sys.monitoring #17519

mroeschke opened this issue Dec 4, 2024 · 0 comments
Labels
cudf.pandas Issues specific to cudf.pandas feature request New feature or request

Comments

@mroeschke
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Currently the cudf.pandas.Profiler uses sys.settrace to profile lines or cudf.pandas fast-slow calls. Currently the downsides of this approach is:

  1. The tracing function is called for every frame while we're only interested in certain events (line and function calls) providing unnecessary overhead otherwise.
  2. Maintaining some complexity in a monolithic tracing function.

Describe the solution you'd like
The sys.monitoring namespace, a new Python 3.12 feature, should be able to address both downsides: https://docs.python.org/3/library/sys.monitoring.html

  1. We can register a callback for specific events only (sys.monitoring.events.LINE, sys.monitoring.events.CALL)
  2. We can isolate profiling logic for each event
@contextmanager
def profile(function_profile, line_profile, fn):
    if line_profile:
        sys.monitoring.register_callback(sys.monitoring.PROFILER_ID, sys.monitoring.events.LINE, line_profiler)
    elif function_profile:
        sys.monitoring.register_callback(sys.monitoring.PROFILER_ID, sys.monitoring.events.CALL, function_profiler)

Additionally we may find opportunities to improve profiling performance by disabling the profiling by having the callback return sys.monitoring.DISABLE.

Describe alternatives you've considered
Status quo

@mroeschke mroeschke added cudf.pandas Issues specific to cudf.pandas feature request New feature or request labels Dec 4, 2024
@mroeschke mroeschke changed the title [FEA] Implemented cudf.pandas Profiler using sys.monitoring [FEA] Implement cudf.pandas Profiler using sys.monitoring Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cudf.pandas Issues specific to cudf.pandas feature request New feature or request
Projects
Status: Todo
Development

No branches or pull requests

1 participant