Some jobs timing out - UCLA 70007

[soichih]

=>> PBS: job killed: walltime 34235 exceeded limit 34200

I think we could increase walltime, or investigate why this subject is taking longer?

[soichih]

Looks like the timeout is now set to 14 hours (for master and 1.1 branch), but it seems to be still not enough; jobs are timing out still on carbonate. It could be updated to much longer walltime (and accept a longer queue time...) or tweak the default parameters so that jobs will complete quicker by default?

This app is currently set to do "retry" of 2 on Brainlife. Another way to mitigate this might be to update various shell scripts so that they will skip processing tasks if they generated the outputs. This should allow retried jobs to only process what's left to process.

@kitchell @bcmcpher

[bcmcpher]

This is the optic radiation tracking slowly failing. There is no speeding it up by default without ignoring the key parameters. Longer queue would likely help. It should already be skipping the created files if it moves on to a retry. Brent

…

On Wed, Apr 18, 2018 at 9:50 AM, Soichi Hayashi ***@***.***> wrote: Looks like the timeout is now set to 14 hours (for master and 1.1 branch), but this it seems to be not enough.. and jobs are timing out still. It could require much longer walltime (and accept a longer queue time...) or tweak the default parameters so that jobs will complete quicker by default? This app is currently set to do "retry" of 2 on Brainlife. Another way to mitigate this might be to update various shell scripts so that it will skip processing if it already generated the output. This should allow retried jobs to only process what's left to process - instead of starting from scratch. @kitchell <https://github.com/kitchell> @bcmcpher <https://github.com/bcmcpher> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANRr_Sce26GjBLROpmLOr-kT7hM3IpD1ks5tp0SwgaJpZM4RksUw> .

[soichih]

OK.

I see some (pre?)processing being skipped but not the actual streamtrack executions.
Like > https://github.com/brain-life/app-ensembletracking/blob/master/ensembletracking.sh#L231

[soichih]

Another timeout

slurmstepd: *** JOB 17092 ON slurm2-node1 CANCELLED AT 2018-06-05T10:41:55 DUE TO TIME LIMIT ***