Releases: InstituteforDiseaseModeling/idmtools
Releases · InstituteforDiseaseModeling/idmtools
v2.2.1
Hot Fix Release for idmtools: Dependency Update
📝 idmtools_platform_container
Replaced previous ##2541 wrong dependency on idmtools with more modular dependency on idmtools_platform_general.
📝 idmtools_platform_slurm
Replaced previous wrong dependency on idmtools with more modular dependency on idmtools_platform_general.
Note:
Users can still install related platform packages using extras with the following commands:
pip install idmtools[container] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
pip install idmtools[slurm] --index-url=https://packages.idmod.org/api/pypi/pypi-production/simple
This ensures that all required dependencies for the container or slurm platforms are included automatically.
v2.2.0
Release notes for idmtools 2.2.0
🛠 idmtools Code Refactor
- Refactored SlurmPlatform to inherit from FilePlatform (unified code base for many idmtools platforms)
- Refactored flatten method — bug fixed and performance improved (used in AnalyzeManager, PlatformAnalysis, etc.)
- Code cleanup: removed unused functions and legacy code
✨ New Features
- Simulation filtering by tags, status and more:
experiment.get_simulations_by_tags(...)
suite.get_simulations_by_tags(...)
platform.filter_simulations_by_tags(...) - Added new methods to get children objects:
suite.get_experiments()
experiment.get_simulations() - Added
get_directory()method for Suite, Experiment, and Simulation - Added
get_tags()method for Suite, Experiment, and Simulation - Added support for alias SLURM_CLUSTER for SlurmPlatform
- Improved dev_script to work around pip "Access Denied" issues on Windows
- Removed task_type tag from simulations to reduce tag clutter in the COMPS UI
- Added bump version Github Actions script
- Changed setup.py to pyproject.toml for package build
- Updated Github Actions test scripts to allow user pick different python version to tests
- Updated Github Actions test scripts to allow user pick specific package to test
- Added Github Actions build scripts to allow auto trigger build new packages after version bump
- Update devscript for code coverage and Github Actions
- Updated Github Actions scripts to use pypython.toml to build packages
🐞 Bug Fixes
- Fixed case-sensitivity issue in get_files() on Windows
- Fixed incorrect log level in dev_script
- Resolved duplicate Experiment entries in Suite metadata
- Fixed display issue in process/thread logging output
Change log:
Bugs
- #2478 - bootstrap.py didn't initialize logging well
- #2207 - Fix AnalyzeManager: output message
- #2502 - AnalyzeManager not able to retrieve assets files
- #2503 - get_files failed with COMPS Simulation
Core
Developer/Test
- #2472 - Test calibration with new changes
- #2474 - Update dev scripts
- #2476 - Update unit tests
- #2484 - Remove slurm utils GA related scripts
- #2490 - Test AnalyzeManager and PlatformAnalysis
Documentation
- #2475 - Update idmtools documents
- #2535 - Intermittent Sphinx Build Failures Triggered by PUML Rendering
Feature Request
- #2493 - Epic: Add tag-based filtering and other utility enhancements
- #2494 - Reallocate JobHistory utility and we may use it in other platforms
- #2498 - Add get_directory methods for Suite\Experiment\Simulations
- #2499 - Add get_simulations() and get_experiments()
- #2500 - Add simulation filtering with tags
- #2504 - get_files is case-sensitive on files names
- #2510 - Reduce UI tags space used by idmtools task_type tag in Simulation/Experiment
Other
- #2477 - Refactor Slurm Platform: prepare new idmtools 2.2.0 release
- #2491 - Work with EMOD team to test before the release
Platforms
- #2424 - Epic: Refactor SLURMPlatform to reduce the duplicate coding
- #2465 - Update tests and GHA for SLURM platform change
- #2466 - Make SlurmPlatform inherit from FilePlatform
- #2467 - Refactor FilePlatform
- #2468 - Remove bridged and remote utility
- #2483 - Remove mode related functions
- #2488 - Consolidate FileExperiment, SlurmExperiment, etc.
- #2489 - Refactor flatten_item method
Support
- #2482 - bootstrap.py may encounter "Access is denied." issue
Full Changelog: v2.1.0...v2.2.0
v2.1.0
Release notes for idmtools 2.1.0
The release of idmtools 2.1.0 introduces some key enhancements for workflow efficiency and platform support, including:
Platforms:
- Added SlurmPlatform support for fully running scripts as SLURM jobs on compute nodes. (i.e.: run_on_slurm=True on Platform)
- Fixed ContainerPlatform compatibility issues on MacBook.
Features:
- ArmSimulationBuilder now allows reusing SweepArm.
- AssetCollection introduces get_files_by_id for file retrieval.
- Added a custom devcontainer for GitHub CodeSpaces.
Bug Fixes:
- Resolved CommandLine quote handling for options with spaces.
- Provide appropriate "Developer mode" message while running container platform in Windows
- Fixed pip install issues related to package indexes.
Documentation:
- Added a README for MacBook support.
- Fixed build issues with idm-buildtools.
Change log:
Bugs
- #2420 - Fix Macbook issues to support ContainerPlatform
- #2413 - CommandLine.add_option adds extra double quotes around strings with spaces, making file paths inaccessible
- #2422 - Not providing the appropriate message while running container platform on Windows when Developer Mode is off
- #2432 - idmtools_platform_comps/ssmt_work_items/work_order.py field error cause doc build fail
Documentation
- #2431 - Fix document build issue with idm-buildtools
Feature Request
- #2426 - Arm simulation builder does not throw an appropriate error message if configured wrong
- #2445 - Fix pip install issue which depends on index-url or extra-index-url
- #2450 - Create devcontainer for CodeSpaces
Platforms
- #2421 - Run script as a SLURM job
- #1574 - Can not download AssetCollection files with platform.get_files_by_id
User Experience
- #2415 - Write doc to summarize ContainerPlatform support in Macbook
v2.0.2
Release notes for idmtools 2.0.2
The release of idmtools 2.0.2 represents a feature release, focusing on MPI support in Platforms and some bug fixes.
Highlights:
Key features and changes in this release include:
-
MPI support in several idmtools platforms:
SlurmPlatform
ContainerPlatform
ProcessPlatform -
Fixed assets copy issue with relative symlinks. (Set developer mode for Windows: https://learn.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development)
-
Fixed major performance in Container and Slurm platforms.
-
Fixed idmtools cache issue.
-
Removed direct srun call in sbatch.
-
Changed the default 'max_running_jobs' in SlurmPlatform from 1 to 100
-
Bug fixes
Change log:
Bugs
- #2391 - Random fail with large number of simulations with NoneType of platform
- #2393 - Calibra may fail with ContainerPlatform
- #2395 - Slurm: Do not need to delete suite if job_directory is not exist or suite not exist anymore
- #2399 - docker container can become unusable in linux
Documentation
- #2386 - jupyter lab version fix
Feature Request
- #2377 - Need to change github actions to only build container docker image when there are related changes
- #2172 - Adjust the default 'max_running_jobs' in SlurmPlatform
- #2379 - Container Platform needs to support multiple processes
- #2396 - Assets duplication
Platforms
- #2388 - Performance issue with get_item in slurm platform when there are lot of items in job directory
- #2389 - Performance issue with get_item in container platform when there are lot of items in job directory
- #2247 - Investigation: run each simulation on SLURM with multi cores
Release/Packaging
- #2400 - Add script for generating changelog by project id and release version
User Experience
- #2387 - Investigation multiple warnings for binding in NU cluster for singularity
- #2145 - slurm platform -- unable to run from compute nodes as expected
All commits from 2.0.1 to 2.0.2: #2411
v2.0.1
v2.0.0
Release Notes for idmtools 2.0.0
The release of idmtools 2.0.0 represents a major update, featuring several important enhancements such as a newly configurable directory structure for Suites, Experiments, and Simulations, a restructured Platform Factory, and enhanced logging capabilities.
New Job Directory Options with Suite/Experiment Prefix Names
The newly introduced Suite/Experiment/Simulation directory structure enhances experiment tracking by incorporating descriptive prefixes. The default layout is now:
{prefix_suite_name}_{suite_id}/{prefix_experiment_name}_{experiment_id}/{simulation_id}
Users can revert to a simplified ID-only structure ({suite_id}/{experiment_id}/{simulation_id}) by setting name_directory = False in idmtools.ini.
Platform Factory Refactoring
The Platform Factory has been simplified by eliminating the confusing missing_ok parameter and ensuring uniformity of parameters among INI files, platform aliases, and user inputs. Moreover, logging enhancements contribute to a clearer and more user-friendly interface.
Highlights
Key features and changes in this release include:
- Configurable Directory Structure: Users now have the option to select either named folders or UUIDs for Suite/Experiment/Simulation directories.
- Simplified API: The
runmethod has been streamlined by removing thewait_on_doneparameter, leaving onlywait_until_donefor job execution management. - Platform Type Case-Insensitivity: Platform types are now insensitive to case variations.
- Platform Creation: Platforms can be generated via an INI file, pre-defined platform Alias, or Platform Factory. Additionally, users can create individual concrete platform classes to leverage the availability of input parameters.
Change log
Platform
- #1655 - Support 'missing_ok' argument to Platform class without generating warnings
- #2313 - idmtools.ini 'type' parameter should not be case sensitive
- #2327 - When submit experiment with specific platform(instead of factory platform), it requires pass platform to run function
- #2171 - Experiment is missing add_simulation and add_simulations methods
- #2028 - iplatform non-descriptive error message: invalid type
- #2357 - Need to make sure assets exist when retrieving assets
- #2366 - Add max folder/files path length check for Windows
Bugs
- #2355 - Cache issue when deleting suite
- #2342 - idmtools cli still contains some command for Local Platform
- #2325 - If not use factory platform, it will throw warning
Feature requests
- #2143 - Readable/meaningful suite/experiment/simulation output directory names
- #2352 - Refactor SlurmPlatform item path (user readable)
- #2037 - experiment should only need one wait_on_done method
Documentation
- #2345 - Improve document (add more details to Container Platform)
v1.7.11
Release notes for idmtools 1.7.11
The idmtools 1.7.11 release is a feature update that introduces a new container platform and maintains compatibility across other platforms with no breaking changes.
The ContainerPlatform is a new platform designed to streamline the execution of experiments and simulations within Docker containers. It offers a fully equipped environment with all the necessary tools and dependencies, ensuring smooth integration and execution of computational tasks.
Highlights
Highlights of this release include:
-
New features:
- Introduced ContainerPlatform for running experiments and simulations within Docker containers.
- Provided default Docker images based on RockyLinux 9.2 and Debian 12, with automatic pulling of these images when using ContainerPlatform.
- Enabled support for custom Docker images, allowing users to run experiments and simulations with their own configurations.
- Added CLI commands for managing and monitoring experiments and simulations on ContainerPlatform:
idmtools container jobs [<container-id>] [-l <limit>] [-n <next>]: List running jobs.idmtools container status <item-id> [-c <container_id>] [-l <limit>] [--verbose/--no-verbose]: Check the status of an experiment or simulation.idmtools container cancel <item-id> [-c <container_id>]: Cancel a job.idmtools container history [<container-id>] [-l <limit>] [-n <next>]: View experiment history.idmtools container --all --help: View all container platform CLI commands.
- Added
containerandslurmpip install groups streamlined installation of related packages. - Implemented the ability to add extra Python packages to the container platform on the fly or via CLI commands.
- Facilitated viewing experiment/simulation results either within the container or on the host machine.
- Provided default and custom binding mounts to the container.
- Included examples/tests to demonstrate running experiments with ContainerPlatform.
-
idmtools improvements:
- Removed support for Python 3.7 in idmtools.
- Added a new platform alias for Nibbler.
- Resolved the scheduling issue on the COMPS platform.
- Corrected default values for the COMPSPlatform's environment and URL fields.
-
Documentation improvements:
- Introduced new documentation for ContainerPlatform.
- Added new docstrings for the platform factory and platform classes.
- Fixed the broken build issue on Windows and GitHub actions.
- Updated existing documentation with improved structure and more detailed information.
Change log
Platform
- #2253 - ContainerPlatform: Design Container Platform
- #2254 - ContainerPlatform: Task list
- #2266 - ContainerPlatform: Platform implementation (basic class)
- #2267 - ContainerPlatform: Handle Platform alias
- #2268 - ContainerPlatform: Handle file directory binding
- #2269 - ContainerPlatform: Modify required to File/Process Platform
- #2270 - ContainerPlatform: Container Utilities
- #2271 - ContainerPlatform: Handle Windows/Linux differences
- #2272 - ContainerPlatform: Implement CLI Status check
- #2273 - ContainerPlatform: Run AnalyzeManager
- #2274 - ContainerPlatform: Consider Calibra
- #2275 - ContainerPlatform: Consider Container Cleanup
- #2276 - ContainerPlatform: Consider Cancel experiment/simulation
- #2277 - ContainerPlatform: Consider Pre and Post run (ep4)
- #2278 - ContainerPlatform: Install extra packages required for ep4
- #2279 - ContainerPlatform: Collect EMOD requirements
- #2280 - ContainerPlatform: Build docker image
- #2281 - ContainerPlatform: Upload image to hub (idm artifactory)
- #2320 - ContainerPlatform: user select container to use
- #2331 - Container Platform: Code cleanup
Bugs
- #2318 - add_schdule_config should not hard code for num_cores and node_group
- #2306 - idmtoolscore experiment run can wait long time until timeout in certain case
- #2298 - idmtools package should not include the tests folder
- #2251 - local doc build (make html) errors out and does not complete
- #2169 - For dry run, we better not to display 'check status message
- #2208 - SlurmPlatform: experiment's metadata.json file has suite_id is null
Feature Request
- #2261- Consider changes to new COMPS scheduling support
- #2328 - COMPSPlatform should set default values for endpoint and environment set to production instead of staging
- #2326 - SlurmPlatform should have default mode to 'Local' instead of None
- #2319 - Add new alias for COMPS new environment: Nibbler
- #2317 - idmtools should stop support python3.7
- #2337 - Add container, slurm options in idmtools setup for customized installation
Developer/Test
- #2282 - ContainerPlatform: Test image
- #2283 - ContainerPlatform: Test platform on docker container
- #2284 - ContainerPlatform: Test utilities
- #2285 - ContainerPlatform: Test Windows/Linux
- #2286 - ContainerPlatform: Test CLI
- #2287 - ContainerPlatform: Test AnalyzeManager
- #2288 - ContainerPlatform: Test Calibration (Calibra)
- #2289 - ContainerPlatform: Write unit tests
Documentation
v1.7.10
What's Changed
- Fix pygit2 by @shchen-idmod in #2158
- Assetization workflow for generating SIF image should track using Asset ID instead of AC ID by @shchen-idmod
#2032 - Remove local platform from idmtools repo by @shchen-idmod in #2160
- Improved Simulation builders (fixed several issues) by @ZDu-IDM in #2184
- Refactor idmtools Simulation Builders structure. by @ZDu-IDM in #2203
- Refactor Simulation Builders to remove duplicated code by @ZDu-IDM in #2196
- Combine two add sweeping functions into one @ZDu-IDM in #2195
- Sweeping function allow parameters with default values @ZDu-IDM #2194
- Builder function add_multiple_parameter_sweep_definition doesn't support function with single dictionary parameter @ZDu-IDM #2192
- SimulationBuilder assume input values object has len defined @ZDu-IDM #2183
- Refactor ArmSimulationBuilder and move general functionality to a base class @ZDu-IDM #2168
- ArmSimulationBuilder needs rasie exception for call to add_sweep_definition and add_multiple_parameter_sweep_definition @ZDu-IDM #2167
- SweepArm should support add_multiple_parameter_sweep_definition @ZDu-IDM #2166
- ArmSimulationBuilder display simulation count incorrect @ZDu-IDM #2165
- Give deprecating warning for duplicate wait_on_done function by @shchen-idmod in #2173
- Fix circular dependency for core and cli tests and move comps related test in core to comps test by @shchen-idmod in #2175
- Add download asset and use asset id in generate singularity builder instead of assetcollection by @shchen-idmod in #2210
- Fix bug for save_as for content type and add unittest by @shchen-idmod in #2213
- Update idmtools to work with python 3.12 by @shchen-idmod in #2215
- unregister plugin in test_hooks.py to avoid affecting other test in p… by @shchen-idmod in #2226
- Update idm-buildtools requirement from ~=1.0.3 to ~=1.0.5 by @dependabot in #2227
- Update junitparser requirement from ~=3.1.1 to ~=3.1.2 by @dependabot in #2228
- Update coverage requirement from <6.6,>=5.3 to >=5.3,<7.5 by @dependabot in #2229
- Update flake8 requirement from ~=6.0.0 to ~=7.0.0 by @dependabot in #2230
- Update pytest-xdist requirement from ~=3.3 to ~=3.5 by @dependabot in #2231
- fix hook run with serial mark by @shchen-idmod in #2235
- Update docker requirement from <6.1.0,>=4.3.1 to >=4.3.1,<7.1.0 by @dependabot in #2177
- Update junitparser requirement from ~=2.8.0 to ~=3.1.1 by @dependabot in #2178
- Update jinja2 requirement from ~=3.1.2 to ~=3.1.3 by @dependabot in #2176
- Update packaging requirement from <22.0,>=20.4 to >=20.4,<24.0 by @dependabot in #2181
- Update readthedocs-sphinx-search requirement from ~=0.3.1 to ~=0.3.2 by @dependabot in #2182
- Update click requirement from ~=8.1.3 to ~=8.1.7 by @dependabot in #2180
- Update pytest requirement from ~=7.2.0 to ~=8.0.0 by @dependabot in #2179
- Revert "Update pytest requirement from ~=7.2.0 to ~=8.0.0" by @shchen-idmod in #2185
- Update pytest-timeout requirement from ~=2.1.0 to ~=2.3.1 by @dependabot in #2204
- Update cookiecutter requirement from ~=2.1.1 to ~=2.6.0 by @dependabot in #2199
- Update jupyterlab requirement from ~=4.0.2 to ~=4.1.5 by @dependabot in #2212
- update examples for platform from Belegost to CALCULON by @shchen-idmod in #2225
- Update nbsphinx requirement from ~=0.9.2 to ~=0.9.3 by @dependabot in #2221
- Update pygithub requirement from ~=1.57 to ~=2.3 by @dependabot in #2223
- Update packaging requirement from ~=23.2 to ~=24.0 by @dependabot in #2222
- Update pluggy requirement from ~=1.2 to ~=1.4 by @dependabot in #2220
- Update yaspin requirement from <2.4.0,>=1.2.0 to >=1.2.0,<3.1.0 by @dependabot in #2189
Full Changelog: v1.7.9...v1.7.10
Contributors
ZDu-IDM, dependabot, and shchen-idmod
Assets 2
v1.7.9
## What's Changed
- First PR to fix some bugs by @ZDu-IDM in #2149
- Add hook tests to comps, file, slurm platforms by @shchen-idmod in #2150
- Remove Bayesian tests by @shchen-idmod in #2134
- Read the Docs Sphinx updates by @JSchripsema-IDM in #2142
- docs/gitignores by @bryanressler-idmod in #2141
- changed default search behavior to search only across this doc project by @JSchripsema-IDM in #2151
- add metadata for python3.11 to all setup.py package by @shchen-idmod in #2108
- Bump json5 from 1.0.1 to 1.0.2 in /idmtools_platform_local/idmtools_webui by @dependabot in #1985
- Bump markupsafe from 2.1.1 to 2.1.3 by @dependabot in #2076
- Update pytest-xdist requirement from ~=3.1 to ~=3.3 by @dependabot in #2072
- Update natsort requirement from ~=8.2.0 to ~=8.4.0 by @dependabot in #2085
- Bump crypto-js from 3.1.9-1 to 3.3.0 in /idmtools_platform_local/idmtools_webui by @dependabot in #2081
- Bump pipreqs from 0.4.11 to 0.4.12 in /examples by @dependabot in #2099
- Bump @hapi/hoek from 8.5.0 to 8.5.1 in /idmtools_platform_local/idmtools_webui by @dependabot in #2046
- Update sqlalchemy requirement from ~=1.4.45 to ~=2.0.18 by @dependabot in #2111
- Bump sphinx-copybutton from 0.5.1 to 0.5.2 by @dependabot in #2117
- Update psycopg2-binary requirement from ~=2.9.5 to ~=2.9.6 by @dependabot in #2115
- Update more-itertools requirement from ~=9.0.0 to ~=9.1.0 by @dependabot in #2113
- Update flask requirement from ~=2.2.2 to ~=2.3.2 by @dependabot in #2116
- Update pluggy requirement from ~=1.0.0 to ~=1.2.0 by @dependabot in #2114
- change branch related from master to main by @shchen-idmod in #2123
- Added Jenkinsfile to main branch by @shchen-idmod in #2127
- Merge Jenkins and main branch change to release branch by @shchen-idmod in #2128
- Merge Jenkins and main branch changes to dev branch by @shchen-idmod in #2129
- Update allure-pytest requirement from <2.13,>=2.8.34 to >=2.8.34,<2.14 by @dependabot in #2118
- bump version to 1.7.9 by @shchen-idmod in #2153
New Contributors
- @bryanressler-idmod made their first contribution in #2141
Full Changelog: v1.7.8...v1.7.9
Contributors
ZDu-IDM, JSchripsema-IDM, and 3 other contributors
Assets 2
v1.7.8
What's Changed
=====
1.7.8
Additional Changes
#2100 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2100>_ - Setup.py does not conform to newest pip in python requires#2101 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2101>_ - Deprecate 3.6 references from idmtools#2102 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2102>_ - Doc fix
Bugs
#2083 <https://github.com/InstituteforDiseaseModeling/idmtools/issues/2083>_ - python11 issue with dataclasses
Full Changelog: v1.7.7...v1.7.8