Workflow Debugging #20735
Description
I would love to extend the new workflow debug tab with a few things to help locate potential errors or discover the reason things failed. For some workflows this will... exercise the backend so we should cache aggressively and warn the user that these operations will take some time and the results might be inconclusive.
The three actions I can imagine are:
- Iterate through the jobs in the workflow and run the common problems check for each job and place the results in the UI in some fashion: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/webapps/galaxy/api/jobs.py#L305. This will catch certain classes of things that might be green but may have been linked up incorrectly.
- Iterate through all the datasets (or maybe all the datasets backwards from errored jobs) and run the datatype validators for all those datasets and summaries the results - Datatype Validation and Tool-based QC #7913. This will catch truncated data and data corruption we don't check for automatically and that can result in green datasets that break downstream steps in hard to detect ways.
These two can then both expanded with more in depth checks. There are notes in the common problems API implementation about other things we could check and we could always write more and more detailed datatype validators. Improving these checks helps with standalone jobs also - not just jobs in the context of workflows - they seem like high value development prospects.
- The third action I was imaging is to gather all the standard outputs, errors, and dataset peaks and feed them to some sort of LLM to ask it to look for the problem or speculate on what went wrong. It is hard to imagine a nice way to present all parent job standard outputs to help figure out why job X failed to a human but this may be a good task for an LLM.