generated from gpauloski/python-template
-
Notifications
You must be signed in to change notification settings - Fork 6
Open
Labels
bugSomething isn't workingSomething isn't workinghelp wantedExtra attention is neededExtra attention is needed
Description
Each task created by the Engine
has a callback which will call result()
on the future to get the TaskInfo
for logging. This has the unfortunate side-effect of pulling the actual task result onto the client, which could incur some non-trivial data movement cost depending on the executor and app.
We might be able to avoid this by changing the TaskInfo
to get passed on to child tasks, and only logged if TaskFuture.result()
is called. This means that:
- A child task failing could mean we lose the
TaskInfo
of parent tasks. - A parent with multiple children could each log the parent's
TaskInfo
so we'd need to check if task IDs were already logged. - If the app never calls
result()
, theTaskInfo
for possibly many tasks in a chain do not get logged.
I think we can solve all of these.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workinghelp wantedExtra attention is neededExtra attention is needed