Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClusterTask should optionally save job stdout/stderr to member variables #26

Open
knrafto opened this issue Feb 12, 2020 · 1 comment
Open

Comments

@knrafto
Copy link
Collaborator

knrafto commented Feb 12, 2020

Right now, the job stdout/stderr goes to /dev/null, but this means it's not a drop-in replacement for ProcessTask in this case.

@nicmcd
Copy link
Owner

nicmcd commented Feb 12, 2020

I propose we revamp ProcessTask then make ClusterTask match. I do believe ClusterTask should be a drop in replacement for ProcessTask (whether or not inheritance is used). In general, when dealing with stdout and stderr, I think there are 3 main use cases:

  1. Stdout and stderr are not tracked or wanted whatsoever. Only execution and return code. In this case it would be unwise to store std{out,err} in the task anyway (memory space bloat).
  2. Stdout and stderr are tracked but kept in the task object, not written to files.
  3. Stdout and stderr are written to files.

ClusterTask has another output stream (call it run{out,err} for now) that is the output streams of the runner application that schedules the cluster job in the cluster scheduler (e.g., "srun" for Slurm, "qsub" for SGE, etc.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants