[Feature] View output of `oumi launch up` on Slurm job

[stefanwebb]

Feature request

A really useful feature would be to be able to view the output of your Slurm job from your local machine.

One workaround I am going to try is to enable Tensorboard and examine the logs on the head node. Still, I think it would be really useful to see the exact remote output so you know your job is doing okay

Motivation / references

Running a fine-tuning job

Your contribution

N/A

[taenin]

I think @wizeng23 also had a similar ask :)

[wizeng23]

I've been meaning to do OPE-322 for a while and haven't gotten around to it 🙁 It's asking to tail the results of SkyPilot jobs when we launch a job. However, supporting this for non-SkyPilot jobs will be more complex, as we'll need to tail a logs file.