Tracking accomplishments at #BioHackEU20

[mr-c]

PRs

• Annotate CWL 1.2 Conformance Tests #120
• Fix object_store_store_by handling #119
• Fix FieldTypeParameterValue wrapping #118
• Rev cwltool to 3.0.20201109103151 #10658
• Unique labels in string interpolation tests. #55
• expr refactor: better interface #40
• Refactor logic for workflow_step.input_type into model. #10657
• Fix ObjectUploadTarget handling in tool util staging code. #10650
• Late Monday fixes and enhancements Various fixes and enhancements #125
  • Fix running recursive conformance tests imported introduced in CWL 1.2 test suite (previous fix Update CWL populator load conformance tests for $imports in CWL 1.2. #124 had a bug).
  • When pulling workflow output CWL outputs, handle non-data/non-collection datatypes (i.e. simple values). Needed for null and string outputs of a conditional test case.
  • When evaluating expressions for workflows, walk inputs not consumed by the underlying tool also (kind of messy).
  • Add pytest marks for each test tag found in conformance_tests.yaml when generating Python test suites, allows for much more flexible testing combinations.
  • When generating test suites in Python suite, print some common problems so we can fix up the source test specifications.
  • Bring in fixes for the cwl v1.1 conformance tests file (xref Missing tags on conformance tests. common-workflow-language/cwl-v1.1#72).
• Improvements to workflow test monitoring for upstream wait_for_invocation should take in an assert_ok flag like other wait methods. galaxyproject/galaxy#10668
• Upstream test verifying structure in tool state of optional parameters API workflow test showing structure of default/optionals parameters. galaxyproject/galaxy#10669
• Fix regressions related to optional parameters and field types (Fix workflow field input defaults. #127)
• Migrate API tests to GitHub workflow Migrate API tests to GitHub workflow galaxyproject/galaxy#10674
• Re-sync galaxy.tool_util.cwl with CWL branch of Galaxy [20.01] Bump cryptography dependency to 3.2.1 galaxyproject/galaxy#10704
• CWL style workflow conditionals
  • Done for CWL in Galaxy: [WIP] Implement conditionals. #123
  • Build up tests and abstractions required to have conditionals for Galaxy workflows - so we can upstream expressions in workflows.
    • Add to format2/ga https://github.com/galaxyproject/gxformat2/pull/48/files
    • gxformat2 plumbing to allow dev releases Update project plumbing to allow dev release. galaxyproject/gxformat2#49
    • Harden, formalize 'null' handling around boolean tool parameters Harden, formalize 'null' handling around boolean tool parameters galaxyproject/galaxy#10694
    • Allow expression tools to output copies of input HDAs Allow expression tools to output copies of input HDAs. galaxyproject/galaxy#10700
    • Null handling and optional values for integers, floats, and text params Null handling and optional values for integers, floats, and text params. galaxyproject/galaxy#10703
    • Implement expression tool for picking non-null values for CWL-style merging of conditional branches in native Galaxy workflows Implement expression tool for picking non-null values for CWL-style merging of conditional branches in native Galaxy workflows galaxyproject/galaxy#10712

New Issues

• Parse creator and license information from CWL documents. #117
• Migrate Travis Tests to Github Actions #121
• Figure out plan for pickValue synchronization across steps #126
• Rework resource handling for cwltool>=3.0.20200317142916 #116
• Update CWL Branch #115: @jmchilton
  • Rebase against latest Galaxy.
  • Update commit that brings in conformance test data to bring 1.1 and 1.2 conformance tests.
  • Updated the test case generation code to generate news test classes for each version (v1.0, v1.1, and v1.2) corresponding to red and green "required" tests as well as regressions.
  • Re-run red tests and add 36 news test to green list.
  • Refactor to use new client staging stuff (resulting in Fix ObjectUploadTarget handling in tool util staging code. galaxyproject/galaxy#10650 merged by Marius)
  • Fix some small issue with a refactoring to cleanup CWL branch, caused nested workflows to break (https://github.com/galaxyproject/galaxy/pull/10657/files).
  • Various fixes for test generation, directory datatype.
  • Updated conformance suite to test case code for newer recursive imports.
  • Categorized the remaining regressions into 4 likely bugs, documented alone side the red/green list code.
  • More correct, robust should_fail handling for conformance tests (More robust should_fail handling when running CWL conformance tests. #130). Needed for conditional tests.

to do:

• What happened to the code to integrate the CWL Expresion refactoring tool?
  galaxyproject@9a8855d
  https://github.com/galaxyproject/galaxy/compare/dev...common-workflow-language:cwl-1.0-rewrite-expressions-2?expand=1

• Presenting non-zero (and ideally correct) information on the memory and disk space reservation in the CWL $(runtime) object

• recover tests that used to pass in this branch, but don't currently.

Bonus / semi-related work by team-members

• Full Python dependencies update #10660
• Update mercurial to 5.6 #73

[mr-c]

Since all unchecked items have an issue or a pull request open, and the #BioHackEU20 is over I'm going to close this tracking issue. Many thanks to @jmchilton @mvdbeek @nsoranzo for their contributions this last week!