Implement Sample Sheets

[jmchilton]

Implements #19085.

The Problem

Driving use cases - these complex data sample sheets that encode metadata not currently captured by simple list structures.

Problems with just uploading tabular files with list identifiers:

The solution (this PR):

Using Sample Sheets

New Collections from URIs using Galaxy

New Collections from URIs using External Spreadsheet Programs

Sample Sheet Collections from Existing Collections

(See slides - more of the same screenshots and description.)

Defining Sample Sheets

These are just defined in the workflow editor directly.

You can see what the sample sheet as a spreadsheet will look like - including cell validation and help with a preview from the workflow editor.

Using Sample Sheets from Workflows

There is a tool to convert it to a table so it can be used directly by workflows such as galaxyproject/iwc#581.

But it has also been baked into the Apply Rules tool so a lot of the complex operations that happen in cluster jobs in that workflow can work more efficiently, directly, and robustly using model objects.

Baking it into the Apply Rules tool also means it should be fairly straight forward to derive collections with group tags and nested collections from these inputs as part of workflows. The inputs have a lot more description and so should making using these advanced features in workflows much more straight-forward for Galaxy users.

The Model

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[jmchilton]

I'm trying to distill the numerous comments into a checklist:

• Workflow editor navigates away when clicking on template (fixed)
• Collection type input sort of sucks - sample sheet into a new input and updated issue Add a dedicated interface for collection types in Workflow input field #19049.
• Column definition validation errors -
  • frontend fixed to not produce these but…
  • Marius would still like more backend validation to prevent.
• Indicate optional is okay in columns (unaddressed)
• Descriptions in column headers (implemented)
• Optional strings not allowed (I think this is fixed but this optional logic is tricky)
• Wrong collection part IDs (dce vs had id) (fixed)
• Various issues with keyboard input and AG Grid (@jmchilton argues we should push this off - framework is better than hardcoding something for now)
• Just called nested lists in history panel (fixed - updated collection description in history panel)
• No way to see columns in history panel (unaddressed)
• Selection boxes do not do well with smaller width displays - certainly made strides with this at the cost of it being not quite as clean for larger widths.
• Selection boxes are too verbose - reviewed language and shortened it up.
• Pop-up collection selection when at that point in the wizard
• Sort collections by HID descending (done in Sort collection dialog by HID descending. #20674 follow up in Dataset Collection Dialog does not do pagination #20682)
• Incomplete job states in existing refactored code (addressed in Rework and expand client job state handling constants. #20675)
• Better use of Vue/Typescript Implement Sample Sheets  #19305 (comment) - resolved.
• POST vs Base64 GETs - addressed.
• Various issues with how we’re watching jobs - should be mostly resolved now using resource watcher stuff.
• Issues with validation in the worksheet in external apps (unaddressed).
• CData in the new tool.

[jmchilton]

In response to various issues with that landing set of selections being both too verbose and not dealing with small screen widths well - I shortened most of the options and swapped the BCardGroup from "stack" to "columns" - these seem to be the only levers the framework provides without breaking abstractions. I think it is a little better maybe.

Before:

After:

reviseoptions.mov

[mvdbeek]

This looks relevant

FAILED lib/galaxy_test/selenium/test_workflow_run.py::TestWorkflowRun::test_collection_input_sample_sheet_chipseq_example_from_list_pairs - selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <div data-v-78f582f4="" data-description="selection collection card" class="card wizard-selection-card border-primary">...</div> is not clickable at point (542, 526). Other element would receive the click: <div id="__BVID__492___BV_modal_body_" class="modal-body">...</div>

[jmchilton]

8fe4518 seems to have fixed the Selenium test - thanks for that catch Marius!

[mvdbeek]

This is huge! Thanks a lot @jmchilton!