Merge pull request #177 from precice/micro-manager-v0.7.0

Release v0.7.0

Author: IshaanDesai

.github/dependabot.yml

@@ -0,0 +1,12 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
+
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      # Check for updates to GitHub Actions every week
+      interval: "weekly"

.github/workflows/pythonpublish.yml

@@ -0,0 +1,41 @@
+name: Upload Python Package
+
+on:
+  push:
+    tags:
+      - v*
+
+jobs:
+  build:
+    name: Build package
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.x'
+      - run: pip install --upgrade build
+      - name: Build package
+        run: pyproject-build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist
+          overwrite: true
+          if-no-files-found: error
+          retention-days: 1
+
+  publish:
+    name: Upload release to PyPI
+    needs: build
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write
+    steps:
+      - name: Download package
+        uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist
+      - name: Publish package distributions to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

.github/workflows/release.yml

@@ -1,5 +1,24 @@
 # Micro Manager changelog
 
+## v0.7.0
+
+- Share similarity distance matrix between ranks on a compute node https://github.com/precice/micro-manager/pull/176
+- Use booleans instead of strings `"True"` and `"False"` in the configuration https://github.com/precice/micro-manager/pull/175
+- Renaming and using a newer workflow for publishing according to the trusted publishing in PyPI https://github.com/precice/micro-manager/pull/173
+- Add config options for adaptivity metrics and memory usage output to allow for different levels https://github.com/precice/micro-manager/pull/172
+- Fix bug in adaptivity computation when an active simulation with associations is deactivated https://github.com/precice/micro-manager/pull/171
+- Properly handle micro simulation initialization for lazy initialization https://github.com/precice/micro-manager/pull/169
+- Delete the simulation object when the simulation is deactivated https://github.com/precice/micro-manager/pull/167
+- Remove float32 data type restriction for adaptivity data [commit](https://github.com/precice/micro-manager/commit/bfa44ff4d3432c6ac0f3b1311274308d2ec9c2a4)
+- Trigger adaptivity when all the adaptivity data is the same https://github.com/precice/micro-manager/pull/170
+- Add configuration option to control frequency of adaptivity computation https://github.com/precice/micro-manager/pull/168
+- Remove checkpointing of adaptivity and fix output of memory usage https://github.com/precice/micro-manager/pull/166
+- Performance improvements: restricting data types, in-place modifications https://github.com/precice/micro-manager/pull/162
+- Handle adaptivity case when deactivation and activation happens in the same time window https://github.com/precice/micro-manager/pull/165
+- Add command line input argument to set log file https://github.com/precice/micro-manager/pull/163
+- Fix adaptivity metrics logging and add logging documentation https://github.com/precice/micro-manager/pull/160
+- Checkpoint lazily created simulation only if a checkpoint is necessary https://github.com/precice/micro-manager/pull/161
+
 ## v0.6.0
 
 - Add functionality for lazy creation and initialization of micro simulations https://github.com/precice/micro-manager/pull/117

CHANGELOG.md

@@ -5,12 +5,10 @@
 </a>
 
 <a style="text-decoration: none" href="https://pypi.org/project/micro-manager-precice/" target="_blank">
-    <img src="https://github.com/precice/micro-manager/actions/workflows/pythonpublish.yml/badge.svg" alt="Upload Python Package">
+    <img src="https://github.com/precice/micro-manager/actions/workflows/release.yml/badge.svg" alt="Upload Python Package">
 </a>
 
-[![DOI](https://joss.theoj.org/papers/10.21105/joss.05842/status.svg)](https://doi.org/10.21105/joss.05842)
-
-A tool to facilitate solving two-scale (macro-micro) coupled problems using the coupling library [preCICE](https://precice.org/).
+A tool to create and manage a large number of (micro) simulations and couple them to a (macro) simulation using the coupling library [preCICE](https://precice.org/).
 
 The main documentation is rendered on the [preCICE website](https://precice.org/tooling-micro-manager-overview.html).
 

README.md

@@ -7,7 +7,7 @@ summary: Provide a JSON file to configure the Micro Manager.
 
 {% note %} In the preCICE XML configuration the Micro Manager is a participant with the name `Micro-Manager`. {% endnote %}
 
-The Micro Manager is configured with a JSON file. An example configuration file is
+The Micro Manager is configured with a JSON file. An example configuration file looks like
 
 ```json
 {
@@ -23,16 +23,23 @@ The Micro Manager is configured with a JSON file. An example configuration file
         "micro_dt": 1.0
     },
     "diagnostics": {
-      "output_micro_sim_solve_time": "True"
+      "output_micro_sim_solve_time": true
     }
 }
 ```
 
 This example configuration file is in [`examples/micro-manager-config.json`](https://github.com/precice/micro-manager/tree/develop/examples/micro-manager-config.json).
 
-The path to the file containing the Python importable micro simulation class is specified in the `micro_file_name` parameter. If the file is not in the working directory, give the relative path.
+## Micro Manager Configuration
 
-There are three main sections in the configuration file, the `coupling_params`, the `simulation_params` and the optional `diagnostics`.
+Parameter | Description
+--- | ---
+`micro_file_name` | Path to the file containing the Python importable micro simulation class. If the file is not in the working directory, give the relative path from the directory where the Micro Manager is executed.
+`output_directory` | Path to output directory for logging and performance metrics. Directory is created if not existing already.
+`memory_usage_output_type` | Set to either `local`, `global`, or `all`. `local` outputs rank-wise peak memory usage. `global` outputs global averaged peak memory usage. `all` outputs both local and global levels. All output is to a CSV file with the peak memory usage (RSS) in every time window, in MBs.
+`memory_usage_output_n` | Frequency of output of memory usage (integer which is number of time windows).
+
+Apart from the base settings, there are three main sections in the configuration file, [coupling parameters](#coupling-parameters), [simulation parameters](#simulation-parameters), and [diagnostics](#diagnostics).
 
 ## Coupling Parameters
 
@@ -55,9 +62,9 @@ Adaptivity parameters | See section on [adaptivity](#adaptivity). By default, ad
 ## Diagnostics
 
 Parameter | Description
---- | ---
+--- | ---H
 `data_from_micro_sims` | A Python dictionary with the names of the data from the micro simulation to be written to VTK files as keys and `"scalar"` or `"vector"` as values.
-`output_micro_sim_solve_time` | If `True`, the Micro Manager writes the wall clock time of the `solve()` function of each micro simulation.
+`output_micro_sim_solve_time` | If `true`, the Micro Manager writes the wall clock time of the `solve()` function of each micro simulation.
 `micro_output_n`|  Frequency of calling the optional output functionality of the micro simulation in terms of number of time steps. If not given, `micro_sim.output()` is called every time step.
 
 ### Adding diagnostics in the preCICE XML configuration
@@ -109,18 +116,21 @@ The Micro Manager can adaptively control micro simulations. The adaptivity strat
 
 All the adaptivity parameters are chosen from the second publication.
 
-To turn on adaptivity, set `"adaptivity": True` in `simulation_params`. Then under `adaptivity_settings` set the following variables:
+To turn on adaptivity, set `"adaptivity": true` in `simulation_params`. Then under `adaptivity_settings` set the following variables:
 
 Parameter | Description
 --- | ---
 `type` | Set to either `local` or `global`. The type of adaptivity matters when the Micro Manager is run in parallel. `local` means comparing micro simulations within a local partitioned domain for similarity. `global` means comparing micro simulations from all partitions, so over the entire domain.
 `data` | List of names of data which are to be used to calculate if micro-simulations are similar or not. For example `["temperature", "porosity"]`.
+`adaptivity_every_n_time_windows` | Frequency of adaptivity computation (integer which is number of time windows).
+`output_type` | Set to either `local`, `global`, or `all`. `local` outputs rank-wise adaptivity metrics. `global` outputs global averaged metrics. `all` outputs both local and global metrics.
+`output_n` | Frequency of output of adaptivity metrics (integer which is number of time windows).
 `history_param` | History parameter $$ \Lambda $$, set as $$ \Lambda >= 0 $$.
 `coarsening_constant` | Coarsening constant $$ C_c $$, set as $$ 0 =< C_c < 1 $$.
 `refining_constant` | Refining constant $$ C_r $$, set as $$ 0 =< C_r < 1 $$.
-`every_implicit_iteration` | If True, adaptivity is calculated in every implicit iteration. <br> If False, adaptivity is calculated once at the start of the time window and then reused in every implicit time iteration.
-`similarity_measure`| Similarity measure to be used for adaptivity. Can be either `L1`, `L2`, `L1rel` or `L2rel`. By default, `L1` is used. The `rel` variants calculate the respective relative norms. This parameter is *optional*.
-`lazy_initialization` | Set to `True` to lazily create and initialize micro simulations. If selected, micro simulation objects are created only when the micro simulation is activated for the first time. Default: `False`.
+`every_implicit_iteration` | If `true`, adaptivity is calculated in every implicit iteration. <br> If False, adaptivity is calculated once at the start of the time window and then reused in every implicit time iteration. Default `false`.
+`similarity_measure`| Similarity measure to be used for adaptivity. Can be either `L1`, `L2`, `L1rel` or `L2rel`. By default, `L1` is used. The `rel` variants calculate the respective relative norms. This parameter is *optional*. Default `L1`.
+`lazy_initialization` | Set to `true` to lazily create and initialize micro simulations. If selected, micro simulation objects are created only when the micro simulation is activated for the first time. Default: `false`.
 
 The primary tuning parameters for adaptivity are the history parameter $$ \Lambda $$, the coarsening constant $$ C_c $$, and the refining constant $$ C_r $$. Their effects can be interpreted as:
 
@@ -133,15 +143,15 @@ Example of adaptivity configuration is
 ```json
 "simulation_params": {
     "macro_domain_bounds": [0, 1, 0, 1, 0, 1],
-    "adaptivity": "True",
+    "adaptivity": true,
     "adaptivity_settings" {
         "type": "local",
         "data": ["temperature", "porosity"],
         "history_param": 0.5,
         "coarsening_constant": 0.3,
         "refining_constant": 0.4,
-        "every_implicit_iteration": "True",
-        "lazy_initialization": "True"
+        "every_implicit_iteration": true,
+        "lazy_initialization": true
     }
 }
 ```
@@ -175,9 +185,9 @@ The Micro Manager uses the output functionality of preCICE, hence these data set
 ## Interpolate a crashed micro simulation
 
 If the optional dependency `sklearn` is installed, the Micro Manager will derive the output of a crashed micro simulation by interpolating outputs from similar simulations. To enable this, set
-`"interpolate_crash": "True"` in the `simulation_params` section of the configuration file.
+`"interpolate_crash": true` in the `simulation_params` section of the configuration file.
 
-For more details on the interpolation see the [crash handling documentation](tooling-micro-manager-running.html/#what-happens-when-a-micro-simulation-crashes).
+For more details on the interpolation see the [crash handling documentation](tooling-micro-manager-running.html#what-happens-when-a-micro-simulation-crashes).
 
 ## Next step
 

docs/configuration.md

@@ -7,23 +7,25 @@ summary: Install the Micro Manager by running `pip install --user micro-manager-
 
 ## Get the latest Micro Manager release
 
-The Micro Manager can be installed using `pip`. Make sure [preCICE](installation-overview.html) is installed before installing the Micro Manager. The Micro Manager is compatible with preCICE version [2.3.0](https://github.com/precice/precice/releases/tag/v2.3.0) and higher.
+### Option 1: Install using pip
 
-### Option 1: Install from PyPI
-
-The Micro Manager package has the name [micro-manager-precice](https://pypi.org/project/micro-manager-precice/) on PyPI. To install `micro-manager-precice`, run
+The Micro Manager package on PyPI is [micro-manager-precice](https://pypi.org/project/micro-manager-precice/). To install, run
 
 ```bash
-pip install --user micro-manager-precice
+pip install micro-manager-precice
 ```
 
-Unless already installed, the dependencies will be installed by `pip` during the installation procedure. To enable [crash handling by interpolation](tooling-micro-manager-running.html/#what-happens-when-a-micro-simulation-crashes), the optional dependency `sklearn` is required. To install `micro-manager-precice` with `sklearn`, run
+To enable [crash handling by interpolation](tooling-micro-manager-running.html/#what-happens-when-a-micro-simulation-crashes), the optional dependency sklearn is required. To install with sklearn, run
 
 ```bash
-pip install --user micro-manager-precice[sklearn]
+pip install micro-manager-precice[sklearn]
 ```
 
-preCICE itself needs to be installed separately. If you encounter problems in the direct installation, see the [dependencies section](#required-dependencies) and [optional dependency section](#optional-dependencies) below.
+To use the Micro Manager for [snapshot computation](tooling-micro-manager-snapshot-configuration.html), the optional dependency h5py is required. To install with h5py, run
+
+```bash
+pip install micro-manager-precice[snapshot]
+```
 
 ### Option 2: Install manually
 
@@ -32,10 +34,10 @@ preCICE itself needs to be installed separately. If you encounter problems in th
 Ensure that the following dependencies are installed:
 
 * Python 3
-* [preCICE](installation-overview.html) [v2.3.0](https://github.com/precice/precice/releases/tag/v2.3.0) or higher
 * [pyprecice: Python language bindings for preCICE](installation-bindings-python.html)
 * [numpy](https://numpy.org/install/)
 * [mpi4py](https://mpi4py.readthedocs.io/en/stable/install.html)
+* [psutil](https://psutil.readthedocs.io/en/latest/)
 
 #### Optional dependencies
 
@@ -53,7 +55,7 @@ git clone https://github.com/precice/micro-manager.git
 To install using `pip`, go to the directory `micro-manager/` and run
 
 ```bash
-pip install --user .
+pip install .
 ```
 
 Adding optional dependencies works as above by adding them after the dot, e.g. `.[sklearn]`.

docs/installation.md

@@ -0,0 +1,63 @@
+---
+title: Logging in the Micro Manager
+permalink: tooling-micro-manager-logging.html
+keywords: tooling, macro-micro, two-scale
+summary: The Micro Manager logs relevant information and adaptivity metrics.
+---
+
+The Micro Manager uses the Python [logging](https://docs.python.org/3/library/logging.html) functionality. The format is:
+
+```bash
+(<rank>) <date and time> <part/functionality of Micro Manager> <log level> <message>
+```
+
+For example
+
+```bash
+(0) 04/17/2025 02:54:02 PM - micro_manager.micro_manager - INFO - Time window 1 converged.
+```
+
+The information (`INFO` level) message `Time window 1 converged.` from the file `micro_manager/micro_manager.py` is logged by rank `0` at `02:54:02 PM` on `04/17/2025`.
+
+## Parsing log output
+
+By default, the Micro Manager parses the log output to the terminal (`sys.out`). A log file, for example `micro-manager.log`, can be passed as a command line input in the following way
+
+```bash
+micro-manager-precice micro-manager-precice.json micro-manager.log
+```
+
+## Logging adaptivity metrics
+
+If the Micro Manager is run with adaptivity, rank-wise and global metrics are written to CSV files. By default, the files are created in the working directory. To create the files in a specific folder, provide the folder path via the configuration parameter `output_dir`. More information is in the [configuration section](tooling-micro-manager-configuration.html).
+
+The following global metrics are logged to the file `adaptivity-metrics-global.csv`:
+
+- Time window at which the metrics are logged
+- Average number of active simulations
+- Average number of inactive simulations
+- Maximum number of active simulations
+- Maximum number of inactive simulations
+
+The CSV file heading is `n,avg active,avg inactive,max active,max inactive`.
+
+The following local metrics are logged to the file `adaptivity-metrics-`rank`.csv`:
+
+for local adaptivity:
+
+- Time window at which the metrics are logged
+- Number of active simulations
+- Number of inactive simulations
+
+The CSV file heading is `n,n active,n inactive`.
+
+for global adaptivity:
+
+- Time window at which the metrics are logged
+- Number of active simulations
+- Number of inactive simulations
+- Ranks to which inactive simulations on this rank are associated
+
+The CSV file heading is `n,n active,n inactive,assoc ranks`.
+
+To set the output interval of adaptivity metrics, set `output_n` in the [adaptivity configuration](tooling-micro-manager-configuration.html#adaptivity).

docs/logging.md

@@ -97,6 +97,11 @@ The `solve()` function should have the following signature:
     This will create a shared library `micro_dummy.so` which can be directly imported in Python.
     For more information on compiling C++ libraries, see the [pybind11 documentation](https://pybind11.readthedocs.io/en/stable/compiling.html).
 
+There are more examples which demonstrate how a micro simulation written in C++ is converted to a Python-importable class
+
+1. [DuMuX-based micro simulation](https://github.com/precice/tutorials/blob/v202404.0/two-scale-heat-conduction/micro-dumux/appl/micro_sim.cpp) in the two-scale heat conduction tutorial.
+2. [pyFANS](https://github.com/DataAnalyticsEngineering/FANS/tree/develop/pyfans).
+
 ## Initializing micro simulations
 
 Micro simulations can be initialized before the actual coupling starts. To initialize a micro simulation, define an `initialize()` function in the code. The Micro Manager calls the initialize function for every micro simulation. If the macro simulation writes initial data to preCICE, the Micro Manager attempts to pass it to the micro simulation. If the `initialize()` function does not have input parameters, the initial data will not be passed. The `initialize()` function can return data to the Micro Manager. This data is only relevant to compute the adaptivity before the coupling starts. Therefore, if the `initialize()` functions returns data, it must be the data expected by the adaptivity.