v2.1.1

Fixed

• The exec component would sometimes fail on Python 3.9 because it wasn't
  catching the right exception.
• The exec component would sometimes issue an empty write to stdout/stderr
  which (at least in theory) could confuse the consumer of those streams.
• Fixed syntax errors in the gwf run and gwf status commands such
  that they now work in Python 3.8 again.

Added

• We now test correctly on all supported Python versions.

v2.1.0

This is a pretty substantial release (but entirely backwards compatible).
There's new functionality submitted by several new contributors, as well as a
new component that enables nice runtime (as in, when a target is being run by
the backend) features.

Added

• This release introduces the concept of executors which can be used to enable
  runtime behavior for targets. This means that you can now use executors to
  run your target inside Conda, Pixi, Apptainer, and Singularity environments:

  .. code-block:: python

  from gwf import Workflow
  from gwf.executors import Conda

  gwf = Workflow()

  gwf.target("Test", inputs=[], outputs=[], executor=Conda("myenv")) << """
  echo this will run inside the myenv Conda environment.
  """

  A default executor can also be specified for the workflow:

  .. code-block:: python

  from gwf import Workflow
  from gwf.executors import Conda

  gwf = Workflow(executor=Conda("myenv"))

  gwf.target("Test", inputs=[], outputs=[]) <<< """
  echo this will run inside the myenv Conda environment.
  """

  The available executors are documented :ref:here <executors>. By default,
  the Bash executor is used, so nothing will change for existing workflows.

  Executors are currently only available using the Slurm backend, but will be
  implemented for other backends in the near future.

• Writes to standard out and error inside the target run are now buffered. This
  prevents overloading network filesystems when a program is doing bad IO, such
  as printing a progress bar to standard error, which then has to be flushed to
  the filesystem very often. The buffer is written at least every 10 seconds,
  so progress indicators can still be used to check progress of a target in the
  log file, but it will not overload the filesystem.

  This functionality is also enabled by the new executor component.

• A backend for IBM Spectrum Load Sharing Facilility (LSF) was contributed by
  @gregoryleeman in PR #412 and #418.

• A backend for the Portable Batch System (PBS) was contributed in PR #419, also
  by @gregoryleeman!

• You can now specify a group for a target which enables grouping targets:

  .. code-block:: python

  from gwf import Workflow

  gwf = Workflow()

  gwf.target("UnzipPatient1", ..., group="patient1") << "..."
  gwf.target("AnalyzePatient1", ..., group="patient1") << "..."
  gwf.target("UnzipPatient2", ..., group="patient2") << "..."
  gwf.target("AnalyzePatient2", ..., group="patient2") << "..."

  You can then tell gwf status to group the targets:

  .. code-block:: shell

  $ gwf status -f grouped
  patient1 0 shouldrun 1 submitted 0 running 0 completed 0 failed 1 cancelled
  patient2 0 shouldrun 0 submitted 0 running 0 completed 0 failed 2 cancelled

  This makes it easier to monitor large workflows. The feature was contributed
  by @jakobjn in PR #414 and extended by @dansondergaard.

• The gwf run and gwf status commands will now show an identifier that
  identifies each target in the backend. Concretely, this means that Slurm job
  id's are now easily obtainable:

  .. code-block:: shell

  $ gwf run SayHello
  Submitted target SayHello (id: 53839420)
  $ gwf status SayHello

  • SayHello submitted (id: 53839420)

• Two new flags have been added to gwf run for controlling what is submitted.

  If --force is given, gwf will submit the specified targets and their
  dependencies no matter what.

  If --no-deps is given, gwf will only submit the specified targets, but
  not their dependencies. This can be combined with force to force submit
  a list of targets, but not any of their dependencies.

• Support for Python 3.13 has been added.

Changed

• Support for Python 3.7 has been dropped.

v2.0.5

Added

• Added support for Python 3.12.

v2.0.4

Fixed

• Fixed bug introduced in 2.0.3 that only affected the Slurm backend.
• Fixed the behavior of the cancel command (can now cancel multiple target, even if cancelling one of them fails).

v2.0.3

Fixed

• Fixed path validation when using named paths (thanks, Anders!)

Changed

• Updated list of known Slurm states in Slurm backend.

v2.0.2

Added

• You can now filter by "failed" status in the status command (thanks Ludvig).
• The output of gwf status -f summary is now more consistent, has colors,
  and shows all possible states instead of only non-zero ones (thanks, Jakob Grove).

Fixed

• Fixed bug where a failed target would not be included as dependency when
  submitting an upstream target.
• When accounting is enabled in the Slurm backend (on by default), gwf will
  now requests job states from both squeue and sacct, with the status
  from squeue taking precedence. This prevents strange-looking output from gwf status when a target has been submitted, but its state has not yet been
  flushed to the database by Slurm.

v2.0.1

Added

• You can now specify which parts of a workflow to touch with the gwf touch command.
  E.g. gwf touch Foo will only touch Foo and its dependencies (and their dependencies).

Fixed

• Fixed loading of entrypoints on Python 3.7.

v2.0.0

This is a major release. Many internals have been changed, specifically on the scheduler,
which is now much more robust and consistent, and to backends, which now can report whether
a target failed or not. The local backend has also been rewritten from scratch. In the
future it will be able to support e.g. target cancellation.

Added

• For supported backends (currently slurm and local), the status command can now show
  failed targets. These will be rescheduled as if they were incomplete. If accounting is
  not enabled in slurm, this functionality must be turned off with
  gwf config set backend.slurm.accounting_enabled no.

Fixed

• The configuration directory will now always be placed next to the workflow file in use.
• The status command is now consistent with the output of gwf runn.
• The scheduler will not re-run targets that are not "should run".
• Targets with only outputs will now be scheduled if an output file is missing. Otherwise,
  it will not be scheduled.
• And many, many other tiny things!

Changed

• Workflows can no longer be included into each other. All namespacing has been removed.
• Templates returning tuples are no longer supported. Templates must now return an AnonymousTarget.
• APIs for loading workflow, using the scheduler, creating and loading backends etc. have
  been changed. This only affects plugins and custom backends.
• The gwf init command has been removed. Instead, gwf will ask if a skeleton should be
  created in the current directory, if it can't find an existing workflow file.
• Spec hashing is now turned off by default since it can be confusing. It can be turned on
  with gwf config set use_spec_hashes yes.

v1.8.5

Added

• Spec hashes will now be computed automatically on the first run with spec
  hashes enabled.
• gwf touch will now also update spec hashes.