v1.0.0

:Date: September 25, 2023

New capabilities:

• libE_specs option final_gen_send returns last results to the generator (replaces final_fields). #1086
• libE_specs option reuse_output_dir allows reuse of workflow and ensemble directories. #1028 #1041
• libE_specs option calc_dir_id_width no. of digits for calc ID in output sim/gen directories. #1052 / #1066
• Added gen_num_procs and gen_num_gpus libE_specs (and persis_info) options for resourcing a generator. #1068
• Added gpu_env_fallback option to platform fields - specifies a GPU environment variable (for non-MPI usage). #1050
• New MPIExecutor submit() argument mpi_runner_type specifies an MPI runner for current call only. #1054
• Allow oversubscription when using the num_procs gen_specs["out"] option. #1058
• sim/gen_specs can use outputs in place of out to be consistent with inputs. #1075
• Executor can be obtained from libE_info (4th parameter) in user functions. #1078

Breaking changes:

• libE_specs option final_fields is removed in favor of final_gen_send. #1086
• libE_specs option kill_canceled_sims now defaults to False. #1062
• parse_args is not run automatically by Ensemble constructor.

Updates to Object Oriented Ensemble interface:

• Added parse_args as option to Ensemble constructor. #1065
• The executor can be passed as an option to the Ensemble constructor. #1078
• Better handling of Ensemble.add_random_streams and ensemble.persis_info. #1074

Output changes:

• The worker ID suffix is removed from sim/gen output directories. #1041
• Separate ensemble.log and libE_stats.txt for different workflows directories. #1027 #1041
• Defaults to four digits for sim/gen ID in output directories (adds digits on overflow). #1052 / #1066

Bug fixes:

• Resolved PETSc/OpenMPI issue (when using the Executor). #1064
• Prevent mpi4py validation running during local comms (when using OO interface). #1065

Performance changes:

• Optimize kill_cancelled_sims function. #1043 / #1063
• safe_mode defaults to False (for performance). #1053

Updates to example functions:

• Multiple regression tests and examples ported to use OO ensemble interface. #1014

Update forces examples:

• Make persistent generator the default for both simple and GPU examples (inc. updated tutorials).
• Update to object oriented interface.
• Added separate variable resources example for forces GPU.
• Rename multi_task example to multi_app.

Documentation:

• General overhaul and simplification of documentation. #992

:Note:

• Tested platforms include Linux, MacOS, Windows, and major systems such as Frontier (OLCF), Polaris, and Perlmutter (NERSC). The major system tests ran heterogeneous workflows.
• Tested Python versions: (Cpython) 3.8, 3.9, 3.10, 3.11.

:Known Issues:

• See known issues section in the documentation.

v0.10.2

:Date: July 24, 2023

• Fixes issues with workflow directories:

  • Ensure relative paths are interpreted from where libEnsemble is run. #1020
  • Create intermediate directories for workflow paths. #1017

• Fixes issue where libEnsemble pre-initialized a shared multiprocssing queue. #1026

:Note:

• Tested platforms include Linux, MacOS, Windows and major systems including Frontier (OLCF), Polaris (ALCF), Perlmutter (NERSC), Theta (ALCF) and Bebop. The major system tests ran heterogeneous workflows.

:Known issues:

• On systems using SLURM 23.02, some issues have been experienced when using mpi4py comms.
• See the known issues section in the documentation for more information (https://libensemble.readthedocs.io/en/main/known_issues.html).

v0.10.1

:Date: July 10, 2023

Hotfix for breaking changes in Pydantic.

• Pin Pydantic to version < 2.
• Minor fixes for NumPy 1.25 deprecations.

:Note:

• Tested platforms include Linux, MacOS, Windows and major systems including Frontier (OLCF) and Perlmutter (NERSC). The major system tests ran heterogeneous workflows.
• Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10, 3.11.

:Known issues:

• See known issues section in the documentation.

v0.10.0

:Date: May 26, 2023

New capabilities:

• Enhance portability and simplify the assignment of procs/GPUs to worker resources #928 / #983

  • Auto-detect GPUs across systems (inc. Nvidia, AMD, and Intel GPUs).
  • Auto-determination of GPU assignment method by MPI runner or provided platform.
  • Portable auto_assign_gpus / match_procs_to_gpus and num_gpus arguments added to the MPI executor submit.
  • Add set_to_gpus function (similar to set_to_slots).
  • Allow users to specify known systems via option or environment variable.
  • Allow users to specify their own system configurations.
  • These changes remove a number of tweaks that were needed for particular platforms.

• Resource management supports GPU and non-GPU simulations in the same ensemble. #993

  • User's can specify num_procs and num_gpus in the generator for each evaluation.

• Pydantic models are used for validating major libE input (input can be provided as classes or dictionaries). #878

• Added option to store output and ensemble directories in a workflow directory. #982

• Simplify user function interface. Valid user functions can accept <4 parameters and return <3 values. #971

• New option to parse settings from TOML. #745

• New dry_run option to libE() that checks scripts are valid and returns. #987

• Added an option to the executor submit function to pre-execute a script in the task environment. #996

Breaking changes:

• Removed old Balsam Executor. #921
• Ensemble class moved from libensemble.api to libensemble.ensemble. #1003
• Default to one resource set per simulation in dynamic scheduling mode. #996

Documentation:

• Added type hints/annotations for major modules/functions. #823
• Added Polaris Guide. #930
• Added Frontier Guide. #909
• Added PBS example scripts. #956 #930
• Streamlined and improved the readability of docs. #1004

Tests and Examples:

• Updated forces_gpu tutorial example. #956
  • Source code edit is not required for the GPU version.
  • Reports whether running on device or host.
  • Increases problem size.
  • Added versions with persistent generator and multi-task (GPU v non-GPU).
• Moved multiple tests, generators, and simulators to the community repo.
• Added ytopt example. And updated heFFTe example. #943
• Support Python 3.11 #922

:Note:

• Tested platforms include Linux, MacOS, Windows and major systems: Frontier (OLCF), Polaris (ALCF), and Perlmutter (NERSC). The major system tests ran heterogeneous workflows.
• Recent testing was also carried out on Summit (IBM Power9/LSF), but this was not possible at time of release.
• Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10, 3.11.

:Known issues:

• See known issues section in the documentation.

v0.9.3

:Date: October 13, 2022

New capabilities:

• New pair of utilities, liberegister and libesubmit (based on PSI/J), for easily preparing and launching libEnsemble workflows with local comms onto most machines and schedulers. #807
• New persistent support function to cancel sim_ids (request_cancel_sim_ids). #880
• keep_state option for persistent workers: this lets the manager know that the information being sent is intermediate. #880

Other enhancements:

• The Executor manager_poll() interface now sets consistent flags instead of literal strings. #877
• Some internal modules and the test suite now work on Windows. #869 #888
• Specifying the num_resource_sets libE_specs option instead of zero_resource_workers is now recommended except when using a fixed worker/resource mapping. Use persis_info["gen_resources"] to assign persistent generator resources (default is zero). #905
• An extraneous warning removed. #903

:Note:

• Tested platforms include Linux, MacOS, Windows, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm), Swing (A100 GPU system), Perlmutter (HPE Cray EX with A100 NVIDIA GPUs). For Perlmutter, see "Known issues" below.
• Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10.

:Known issues:

• At time of testing on Perlmutter there was an issue running concurrent applications on a node, following a recent system update. This also affects previous versions of libEnsemble, and is being investigated.
• See known issues section in the documentation.

v0.9.2

:Date: July 06, 2022

New capabilities:

• Support auto-detection of PBS node lists. #602

• Added configuration options for libE_stats.txt file. #743

• Support for spawn and forkserver multiprocessing start methods. #797

• Note that macOS no longer switches to using fork. macOS (since Python 3.8) and Windows
  default to using spawn. When using spawn, we recommend placing calling script code in
  an if __name__ == "__main__": block. The multiprocessing interface can be used to switch methods (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method).

Updates to example functions:

• Added simple dynamic sampling example. #833
• Added heFFTe example. #844
• Regression tests separated into problem examples and functionality tests. #839

:Note:

• Tested platforms include Linux, MacOS, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm), Swing (A100 GPU system), Perlmutter (HPE Cray EX with A100 NVIDIA GPUs).
• Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10.

:Known issues:

• The APOSMM generator function has been noted to operate slower than expected with the spawn multiprocessing start method. For this reason we recommend using fork with APOSMM, when using local comms (fork is the default method on Linux systems).
• See known issues section in the documentation.

v0.9.1

:Date: May 11, 2022

This release has minimal changes, but a large number of touched lines.

• Reformatted code for black compliance, including string normalization. #811, #814, #821
• Added Spock and Crusher guides. #802
• User can now set calc_status to any string (for output in libE_stats). #808
• Added a workflows community initiative file. #817

:Note:

• Tested platforms include Linux, MacOS, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm), Swing (A100 GPU system), Perlmutter (HPE Cray EX with A100 NVIDIA GPUs).
• Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10.

:Known issues:

• See known issues section in the documentation.

v0.9.0

:Date: Apr 29, 2022

Featured new capabilities:

• New Balsam Executor with multi-site capability (run user applications on remote systems). #631, #729
• Support for funcX (place user functions on remote systems). #712 / #713
• Added partial support for concurrent/futures interface. #719

(cancel(), cancelled(), done(), running(), result(), exception() and context manager)

Breaking API / helper function changes:

See "Updating for libEnsemble v0.9.0" wiki for details:
https://github.com/Libensemble/libensemble/wiki/Updating-for-libEnsemble-v0.9.0

• Scheduler options moved from alloc_specs['user'] to libE_specs. #790
• BalsamMPIExecutor is now LegacyBalsamMPIExecutor. #729
• The exit_criteria elapsed_wallclock_time has been renamed wallclock_max. #750 (with a deprecation warning)
• Clearer and consistent naming of libE-protected fields in history array. #760

Updates to example functions:

• Moved some examples to new repository - libe-community-examples (VTMOP, DEAP, DeepDriveMD). #716, #721, #726
• Updates to Tasmanian examples to include asynchronous generator example. #727 / #732
• Added multi-task, multi-fidelity optimization regression tests using ax. #717 / #720

Other functionality enhancements:

• Non-blocking option added for persistent user function receives. #752
• Added match_slots option to resource scheduler. #746

Documentation:

• Added tutorial on assigning tasks to GPUs. #768
• Refactored Executor tutorial for simplicity. #749
• Added Perlmutter guide. #728
• Added Slurm guide. #728
• Refactored examples and tutorials - added exercises. #736 / #737
• Updated history array documentation with visual workflow example. #723

:Note:

• Tested platforms include Linux, MacOS, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm), Swing (A100 GPU system), Perlmutter (HPE Cray EX with A100 NVIDIA GPUs).
• Tested Python versions: (Cpython) 3.7, 3.8, 3.9, 3.10.

:Known issues:

• OpenMPI does not work with direct MPI job launches in mpi4py comms mode,
  since it does not support nested MPI launches.
  (Either use local mode or the Balsam Executor.)
• See known issues section in the documentation for more issues.

v0.8.0

:Date: Oct 20, 2021

Featured new capabilities:

• Variable resource workers (dynamic reassignment of resources to workers). #643
• Alternative libE interface. An Ensemble object is created and can be parameterized by a YAML file. #645
• Improved support classes/functions for alloc/gen/sims and executors.
• Many new example generator/simulators and workflows.

Breaking API / helper function changes:

See "Updating for libEnsemble v0.8.0" wiki for details:
https://github.com/Libensemble/libensemble/wiki/Updating-for-libEnsemble-v0.8.0

• Resources management is now independent of the executor. #345
• The 'persis_in' field has been added to gen_specs/sim_specs (instead of being hard-coded in alloc funcs). #626 / #670
• alloc support module is now a class. #643 / #656
• gen_support module is replaced by Persistent Worker support module (now a class). #609 / #671
• Remove libE_ prefix from the logger. #608
• avail_worker_ids function should specify EVAL_GEN_TAG or EVAL_SIM_TAG instead of True. #615 #643
• Pass libE_info to allocation functions (allows more flexibility for user and efficiency improvements). #672
• 'given_back' is now a protected libEnsemble field in the manager's history array. #651
• Several name changes to functions and parameters (See the wiki above for details). #529 / #659

Updates to example functions:

• Suite of distributed optimization methods for minimizing sums of convex functions. #647 / #649. Methods include:

  • primal-dual sliding (https://arxiv.org/pdf/2101.00143).
  • N-agent, or distributed gradient descent w/ gradient tracking (https://arxiv.org/abs/1908.11444).
  • proximal sliding (https://arxiv.org/abs/1406.0919).

• Added batched construction for Tasmanian example. #644

• Added Tasmanian dependency to Spack package. spack/spack#25762

• Added VTMOP source code and example usage. #676

• Added a multi-fidelity persistent_gp regression test. #683 / #684

• Added a DeepDriveMD inspired machine learning/molecular dynamics workflow. #630

• Created a persistent sim example. #614 / #615

• Added an example where cancellations are given from the alloc func. #677

Other functionality changes:

• A helper function for generic task polling loop has been added. #572 / #612
• Break main loop now happens when sim_max is returned rather than given out. #624
• Enable a final communication with gen. #620 / #628
• Logging updates - includes timestamps, enhanced debug logging, and libEnsemble version. #629 / #674

:Note:

• Tested platforms include Linux, MacOS, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm), Swing (A100 GPU system).
• Tested Python versions: (Cpython) 3.6, 3.7, 3.8, 3.9, 3.10*.

*A reduced set of tests were run for python 3.10 due to some unavailable test dependencies at time of release.

:Known issues:

• OpenMPI does not work with direct MPI job launches in mpi4py comms mode, since it does not support nested MPI launches (either use local mode or the Balsam Executor).
• See known issues section in the documentation for more issues.

v0.7.2

:Date: May 03, 2021

API additions:

• Active receive option added that allows irregular manager/worker communication patterns. (#527 / #595)
• A mechanism is added for the cancellation/killing of previously issued evaluations. (#528 / #595 / #596)
• A submit function is added in the base Executor class that runs a serial application locally. (#531 / #595)
• Added libEnsemble history array protected fields: returned_time, last_given_time, and last_gen_time. (#590)
• Updated libE_specs options (mpi_comm and profile). (#547 / #548)
• Explicit seeding of random streams in add_unique_random_streams() is now possible. (#542 / #545)

Updates to example functions:

• Added Surmise calibration generator function and two examples (regression tests). (#595)

Other changes:

• Better support for uneven worker to node distribution (including at sub-node level). (#591 / #600)
• Fixed crash when running on Windows. (#534)
• Fixed crash when running with empty persis_info. (#571 / #578)
• Error handling has been made more robust. (#592)
• Improve H0 processing (esp. for pre-generated, but not evaluated points). (#536 / #537)
• A global sim_id is now given, rather than a local count, in libE_stats.txt. Also a global gen count is given. (#587, #588)
• Added support for Python 3.9. (#532) / Removed support for Python 3.5. (#562)
• Improve SLURM nodelist detection (more robust). (#560)
• Add check that user does not change protected history fields (Disable via libE_specs['safe_mode'] = False). (#541)
• Added print_fields.py script for better interrogating the output history files. (#558)
• In examples, is_master changed to is_manager to be consistent with manager/worker nomenclature. (#524)

Documentation:

• Added tutorial Borehole Calibration with Selective Simulation Cancellation. (#581 / #595)

:Note:

• Tested platforms include Linux, MacOS, Theta (Cray XC40/Cobalt), Summit (IBM Power9/LSF), Bebop (Cray CS400/Slurm).
• Tested Python versions: (Cpython) 3.6, 3.7, 3.8, 3.9.

:Known issues:

• OpenMPI does not work with direct MPI job launches in mpi4py comms mode, since it does not support nested MPI launches
  (Either use local mode or Balsam Executor).
• See known issues section in the documentation for more issues.