wip! workflows as programs prototyping #55
Conversation
XXX FIXME: rationale, relationship to "workflows as programs" XXX FIXME: alternatives considered (but declined) for path-in-config-value handling XXX FIXME: document layout/structure of workdir
| # Use default configuration values. Extend with Snakemake's --configfile/--config options. | ||
| configfile: os.path.join(workflow.basedir, "defaults/config.yaml") | ||
|
|
||
| # Use custom configuration from analysis directory (i.e. working dir), if any. | ||
| if os.path.exists("config.yaml"): | ||
| configfile: "config.yaml" |
There was a problem hiding this comment.
The behaviour of multiple configfiles is somewhat documented but I think will prove a tripping hazard to people.
Multiple configfiles (as per this section of code, and/or via additional --configfiles) will overwrite scalars and lists; dicts will be merged, but nested dicts won't be; there's no way to remove a dictionary key as far as I can find.
This has implications for how we structure configs, specifically around how we encode flags to skip rules or skip a particular argument.
There was a problem hiding this comment.
Nod, for sure, but "multiple config files" is already our established approach, so this is making that more accessible/standard but isn't changing approaches.
We could change approaches, though, by combining the configs ourselves outside of Snakemake's machinery. I didn't think there was appetite for that, but it's very possible.
dicts will be merged, but nested dicts won't be; there's no way to remove a dictionary key as far as I can find.
Hmm. In my recollection (and testing right now), nested dicts are merged? That is, the config merge is deep.
The merge is additive though, so yes, there is no way to remove keys.
There was a problem hiding this comment.
nested dicts are merged
Oh yeah, I wasn't formatting my YAML correctly. That they are merged will be very helpful for us.
|
I'd like to see config parsing such as this become part of our general snakemake workflows regardless of workflows as programs CLI work. |
Agreed! (ever since I started 🙃) |
See commit messages. Opening for visibility and soliciting commentary. Not for merge yet.
Checklist