subsample: Improve relative filepath search

[victorlin]

Description

Currently, filepaths in the augur subsample config file are required to be absolute or relative to the current working directory. The limited search of relative filepaths is limiting and has resulted in workflow-level workarounds such as nextstrain/measles#74.

Proposed solution

Search for filepaths relative to multiple directories and make it configurable.

These should be the default directories:

1. config file location
2. current working directory

Order of preference can be discussed – the order above is most intuitive when reading the config file.

Additionally, a command line argument and/or environment variable should allow extending (or overriding?) the default directories. Examples:

augur subsample \
  --config defaults/config.yaml
  --search-paths /path/to/directory-1 /path/to/directory-2

AUGUR_SEARCH_PATHS="/path/to/directory-1:/path/to/directory-2"

augur subsample \
  --config defaults/config.yaml

Names can be discussed.

Additional notes

1. Snakemake's input change detection

When augur subsample is used in a Snakemake workflow, Snakemake's input change detection no longer applies to filepaths referenced in the config. This was an unintentional side effect of simplifying the rule configuration to be more dynamic.

The functionality could potentially be brought back with a workflow-level helper function that dynamically adds inputs, but this may be tricky if filepath resolution is moved from the workflow level to within augur as proposed in this issue.

See also: nextstrain/public#31

2. Expanding to additional augur commands

With nextstrain/pathogen-repo-guide#98 in mind, the functionality of filepath resolution can apply to other augur commands as well. I think it can remove the need for workflow-level filepath resolution in resolve_config_path(). Instead, the workflow could simply run:

AUGUR_SEARCH_PATHS="/path/to/analysis-directory:/path/to/defaults-directory"

augur subsample \
  --config defaults/config.yaml

augur align \
  --reference-sequence {input.reference}

Links

• Initial motivation comes from a Slack thread on resolving relative filepaths.

[victorlin]

I have a working draft on branch victorlin/subsample-filepath-configs. Will open a PR once I properly test it in the measles workflow.