Merge pull request #2717 from ax3l/release-0.4.0-rc4

Release 0.4.0-rc4

Author: psychocoderHPC

CHANGELOG.md

@@ -135,7 +135,7 @@ and Sebastian Hahn for contributing to this release!
    - add a workaround for MSVC bug with capturing `constexpr` #2522
    - compile time string #2532
    - `Vector`: add method `remove<...>()` #2602
-   - add support for more cpu alpaka accelerators #2603
+   - add support for more cpu alpaka accelerators #2603 #2701
    - Vector `sumOfComponents` #2609
    - `math::CT::max` improvement #2612
  - plugins:
@@ -155,10 +155,14 @@ and Sebastian Hahn for contributing to this release!
    - Python:
      - Energy Histogram Reader #2209 #2658
      - Phase Space Reader #2334 #2634 #2679
-     - Move SliceField Module #2354
+     - Move SliceField Module & add Python3 support #2354 #2718
      - Multi-Iteration Energy Histogram #2508
-     - MPL Visualization modules #2484
-     - migrated documentation to Sphinx manual #2172
+     - MPL Visualization modules #2484 #2728
+     - migrated documentation to Sphinx manual #2172 #2726 #2738
+     - shorter python imports for postprocessing tools #2727
+     - fix energy histogram deprecation warning #2729
+     - `data`: base class for readers #2730
+     - `param_parser` for JSON parameter files #2719
  - tools:
    - Tool: New Version #2080
    - Changelog & Left-Overs from 0.3.0 #2120
@@ -215,6 +219,8 @@ and Sebastian Hahn for contributing to this release!
    - `pmacc::math::Size_t<0>::create()` in Visual Studio #2513
    - fix V100 deadlock #2600
    - fix missing include #2608
+   - fix gameOfLife #2700
+   - Boost template aliases: fix older CUDA workaround #2706
  - plugins:
    - energy fields: fix reduce #2112
    - background fields: fix restart `GUARD` #2139
@@ -251,12 +257,15 @@ and Sebastian Hahn for contributing to this release!
  - new example: Foil (LCT) TNSA #2008
  - adjust LWFA setup for 8 GPUs #2480
  - `picongpu --version` #2147
- - add Internal Alpaka & cupla #2179 #2345
- - add alpaka dependency #2205 #2328 #2346 #2590 #2501 #2626 #2648 #2684
+ - add internal Alpaka & cupla #2179 #2345
+ - add alpaka dependency #2205 #2328 #2346 #2590 #2501 #2626 #2648 #2684 #2717
  - Update mallocMC to `2.3.0crp` #2350 #2629
- - Update cuda_memtest #2356
- - Examples: remove unused loaders #2247
- - examples: Update `species.param` #2474
+ - cuda_memtest:
+   - update #2356 #2724
+   - usage on hypnos #2722
+ - Examples:
+   - remove unused loaders #2247
+   - update `species.param` #2474
  - Bunch: no `precision.param` #2329
  - Travis:
    - stages #2341
@@ -385,6 +394,7 @@ and Sebastian Hahn for contributing to this release!
      - EnergyHistogram: Remove Detector Filter #2465
      - ISAAC: unify the usage of period #2455
      - add filter support to phase space plugin #2425
+     - Resource Plugin: `fix boost::core::swap` #2721
    - tools:
      - Python: Fix Scripts PEP8 #2028
      - Prepare for Python Modules #2058
@@ -398,14 +408,17 @@ and Sebastian Hahn for contributing to this release!
      - Group parameters in LWFA example #2417
      - Python Tools (PS, Histo): Filter Aware #2431
      - Clearer conversion functions for Parameter values between UI scale and internal scale #2432
-     - tbg: Add content of -o arg to env #2499
+     - tbg:
+       - add content of -o arg to env #2499
+       - better handling of missing egetopt error message #2712
    - Format speciesAttributes.param #2087
    - Reduce # photons in Bremsstrahlung example #1979
    - TBG: .tpl no `_profile` suffix #2244
    - Default Inputs: C++11 Using for Typedef #2315
    - Examples: C++11 Using for Typedef #2314
    - LWFA Example: Restore a0=8.0 #2324
    - add support for CUDA9 `__shfl_snyc` #2333
+   - add support for CUDA10 #2732
    - Update cuda_memtest: no cuBLAS #2401
    - Examples: Init of Particles per Cell #2412
    - Travis: Image Updates #2435
@@ -420,14 +433,16 @@ and Sebastian Hahn for contributing to this release!
    - Refactor Laser Profiles to Functors #2587
    - Params: float_X Constants to Literals #2625
  - documentation:
+   - new subtitle #2734
    - Lockstep Programming Model #2026 #2064
    - `IdxConfig` append documentation #2022
    - `multiMask`: Refactor Documentation #2119
    - `CtxArray` #2390
-   - Update openPMD Post-Processing #2322
+   - Update openPMD Post-Processing #2322 #2733
    - Checkpoints Backends #2387
    - Plugins:
-     - HDF5: fix links & lists #2313
+     - HDF5: fix links, lists & MPI hints #2313 #2711
+     - typo in libSplash install #2735
      - External dependencies #2175
      - Multi & CPU #2423
      - Update PS & Energy Histo #2427
@@ -468,13 +483,12 @@ and Sebastian Hahn for contributing to this release!
    - Doxygen: How to Build HTML #2134
    - Badge: Docs #2144
    - CMake 3.7.0 #2181
-   - Boost 1.62.0 #2182
+   - Boost (1.62.0-) 1.65.1 - 1.68.0 #2182 #2707 #2713
    - Bash Subshells: `cmd` to $(cmd) #2187
    - Boost Transient Deps: date_time, chrono, atomic #2195
    - Install Docs: CUDA is optional #2199
    - Fix broken links #2200
    - PIConGPU Logo: More Platforms #2190
-   - Profiles for Titan & Taurus #2201
    - Repo Structure #2218
    - Document KNL GCC -march #2252
    - Streamline Install #2256
@@ -484,13 +498,16 @@ and Sebastian Hahn for contributing to this release!
    - Added an example profile and tpl file for taurus' KNL #2270
    - Profile: Draco (MPCDF) #2308
    - $PIC_EXAMPLES #2327
-   - CUDA 8.0.61 on Taurus #2337
-   - Link KNL Profile #2339
+   - Profiles for Titan & Taurus #2201
+   - Taurus:
+     - CUDA 8.0.61 #2337
+     - Link KNL Profile #2339
+     - SCS5 Update #2667
    - Move ParaView Profile #2353
    - Spack: Own GitHub Org #2358
    - LWFA Example: Improve Ranges #2360
    - fix spelling mistake in checkpoint #2372
-   - Spack Install: Clarify #2373
+   - Spack Install: Clarify #2373 #2720
    - Probe Pusher #2379
    - CI/Deps: CUDA 8.0 #2420
    - Piz Daint (CSCS):
@@ -503,9 +520,10 @@ and Sebastian Hahn for contributing to this release!
    - Hypnos (HZDR): New Modules #2521 #2661
    - Hypnos: PNGwriter 0.6.0 #2166
    - Hypnos & Taurus: Profile Examples Per Queue #2249
+   - Hemera: tbg templates #2723
    - Community Map #2445
    - License Header: Update 2018 #2448
-   - Docker: Nvidia-Docker 2.0 #2462
+   - Docker: Nvidia-Docker 2.0 #2462 #2557
    - Hide Double ToC #2463
    - Param Docs: Title Only #2466
    - New Developers #2487

INSTALL.rst

@@ -35,7 +35,7 @@ gcc
 
   - CUDA 8.0: Use gcc 4.9 - 5.3
   - CUDA 9.0 - 9.1: Use gcc 4.9 - 5.5
-  - CUDA 9.2: Use gcc 4.9 - 7
+  - CUDA 9.2 - 10.0: Use gcc 4.9 - 7
 - *note:* be sure to build all libraries/dependencies with the *same* gcc version
 - *Debian/Ubuntu:*
 
@@ -89,8 +89,8 @@ zlib
 
 boost
 """""
-- 1.62.0 - 1.67.0 (``program_options``, ``regex`` , ``filesystem``, ``system``, ``math``, ``serialization`` and header-only libs, optional: ``fiber`` with ``context``, ``thread``, ``chrono``, ``atomic``, ``date_time``)
-- *note:* for CUDA 9 support, use boost 1.65.1 or newer
+- 1.62.0 - 1.68.0 (``program_options``, ``regex`` , ``filesystem``, ``system``, ``math``, ``serialization`` and header-only libs, optional: ``fiber`` with ``context``, ``thread``, ``chrono``, ``atomic``, ``date_time``)
+- *note:* for CUDA 9+ support, use boost 1.65.1 or newer
 - *Debian/Ubuntu:* ``sudo apt-get install libboost-program-options-dev libboost-regex-dev libboost-filesystem-dev libboost-system-dev libboost-thread-dev libboost-chrono-dev libboost-atomic-dev libboost-date-time-dev libboost-math-dev libboost-serialization-dev libboost-fiber-dev libboost-context-dev``
 - *Arch Linux:* ``sudo pacman --sync boost``
 - *Spack:* ``spack install boost``
@@ -155,7 +155,7 @@ Optional Libraries
 
 CUDA
 """"
-- `8.0 - 9.2 <https://developer.nvidia.com/cuda-downloads>`_
+- `8.0 - 10.0 <https://developer.nvidia.com/cuda-downloads>`_
 - required if you want to run on Nvidia GPUs
 - *Debian/Ubuntu:* ``sudo apt-get install nvidia-cuda-toolkit``
 - *Arch Linux:* ``sudo pacman --sync cuda``
@@ -206,7 +206,7 @@ libSplash
   - ``mkdir -p ~/src ~/build ~/lib``
   - ``git clone https://github.com/ComputationalRadiationPhysics/libSplash.git ~/src/splash/``
   - ``cd ~/build``
-  - ``cmake -DCMAKE_INSTALL_PREFIX=$HOME/lib/splash -DSplash_USE_MPI=ON -DSplash_USE_PARALLE=ON ~/src/splash``
+  - ``cmake -DCMAKE_INSTALL_PREFIX=$HOME/lib/splash -DSplash_USE_MPI=ON -DSplash_USE_PARALLEL=ON ~/src/splash``
   - ``make install``
 
 - *environment:* (assumes install from source in ``$HOME/lib/splash``)
@@ -234,6 +234,7 @@ HDF5
   - ``make``
   - *optional:* ``make test``
   - ``make install``
+  - If you encounter errors related to linking MPI during ``./configure``, you might try setting the compiler manually via ``./configure --enable-parallel --enable-shared --prefix $HOME/lib/hdf5/ CC=mpicc CXX=mpic++``.
 - *environment:* (assumes install from source in ``$HOME/lib/hdf5``)
 
   - ``export HDF5_ROOT=$HOME/lib/hdf5``

README.md

@@ -1,5 +1,5 @@
-PIConGPU - A Many GPGPU PIC Code
-================================================================
+PIConGPU - Particle-in-Cell Simulations for the Exascale Era
+============================================================
 
 [![Code Status master](https://img.shields.io/travis/ComputationalRadiationPhysics/picongpu/master.svg?label=master)](https://travis-ci.org/ComputationalRadiationPhysics/picongpu/branches)
 [![Code Status dev](https://img.shields.io/travis/ComputationalRadiationPhysics/picongpu/dev.svg?label=dev)](https://travis-ci.org/ComputationalRadiationPhysics/picongpu/branches)
@@ -15,8 +15,8 @@ PIConGPU - A Many GPGPU PIC Code
 Introduction
 ------------
 
-PIConGPU is a fully relativistic, many
-[GPGPU](http://en.wikipedia.org/wiki/Graphics_processing_unit),
+PIConGPU is a fully relativistic,
+[manycore](https://en.wikipedia.org/wiki/Manycore_processor),
 3D3V particle-in-cell ([PIC](http://en.wikipedia.org/wiki/Particle-in-cell))
 code. The Particle-in-Cell algorithm is a central tool in plasma physics.
 It describes the dynamics of a plasma by computing the motion of
@@ -59,9 +59,10 @@ a wide range of tools and diagnostics, e.g.:
   extreme I/O scalability and massively parallel online-analysis
 - 2D and 3D live view and diagnostics tools
 - a large selection of extensible
-  [online-plugins](https://github.com/ComputationalRadiationPhysics/picongpu/wiki/PIConGPU-Plugins)
+  [online-plugins](http://picongpu.readthedocs.io/en/latest/usage/plugins.html)
 
-Todays GPUs provide a computational performance of several
+As one of our supported compute platforms, GPUs provide a computational
+performance of several
 [TFLOP/s](http://en.wikipedia.org/wiki/FLOPS) at considerable lower invest and
 maintenance costs compared to multi CPU-based compute architectures of similar
 performance. The latest high-performance systems

docs/requirements.txt

@@ -1,8 +1,9 @@
 sphinx_rtd_theme>=0.3.1
 recommonmark
-sphinx
+sphinx==1.7
 breathe>=4.5
 sphinxcontrib.programoutput
+sphinxcontrib-napoleon>=0.7
 pygments
 # generate plots
 matplotlib

docs/source/conf.py

@@ -19,8 +19,9 @@
 import os
 import subprocess
 from recommonmark.parser import CommonMarkParser
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import sys
+python_libs = os.path.abspath('../../lib/python')
+sys.path.insert(0, python_libs)
 
 
 # -- General configuration ------------------------------------------------
@@ -38,13 +39,17 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = ['sphinx.ext.mathjax',
+              'sphinx.ext.napoleon',
               'breathe',
               'sphinxcontrib.programoutput',
               'matplotlib.sphinxext.plot_directive']
 
 if not on_rtd:
     extensions.append('sphinx.ext.githubpages')
 
+# napoleon autodoc config
+napoleon_include_init_with_doc = True
+
 # breathe config
 breathe_projects = {'PIConGPU': '../xml'}
 breathe_default_project = 'PIConGPU'
@@ -96,7 +101,7 @@
 # The short X.Y version.
 version = u'0.4.0'
 # The full version, including alpha/beta/rc tags.
-release = u'0.4.0rc3'
+release = u'0.4.0rc4'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/source/dev/py_postprocessing.rst

@@ -0,0 +1,69 @@
+.. _development-pytools:
+
+Python Postprocessing Tool Structure
+====================================
+
+Each plugin should implement at least the following Python classes.
+
+1. A data reader class responsible for loading the data from the simulation directory
+2. A visualizer class that outputs a matplotlib plot
+
+The repository directory for PIConGPU Python modules for plugins is ``lib/python/picongpu/plugins/``.
+
+Data Reader
+~~~~~~~~~~~
+
+The data readers should reside in the ``lib/python/picongpu/plugins/data`` directory.
+There is a base class in ``base_reader.py`` defining the interface of a reader.
+Each reader class should derive from this class and needs to implement the following interface functions:
+
+.. autoclass:: picongpu.plugins.data.base_reader.DataReader
+   :members:
+   :private-members:
+
+To shorten the import statements for the readers, please also add an entry in the ``__init__.py`` file of the ``data`` directory.
+
+Visualizer
+~~~~~~~~~~
+
+The visualizers should reside in the ``lib/python/picongpu/plugins/plot_mpl/`` directory.
+The module names should end on ``_visualizer.py`` and the class name should only be ``Visualizer``.
+
+To shorten the import statements for the visualizers, please also add an entry in the ``__init__.py`` file of the ``plot_mpl`` directory.
+
+There is a base class for visualization found in ``base_visualizer.py`` which already handles the plotting logic.
+It uses the data reader classes for accessing the data.
+After getting the data, it ensures that (for performance reasons) a matplotlib artist is created only for the first plot and later only gets updated with fresh data.
+
+.. autoclass:: picongpu.plugins.plot_mpl.base_visualizer.Visualizer
+   :members:
+   :private-members:
+
+The complete implementation logic of the ``visualize`` function is pretty simple.
+
+.. code:: python
+
+   def visualize(self, **kwargs):
+        self.data = self.data_reader.get(**kwargs)
+        if self.plt_obj is None:
+            self._create_plt_obj()
+        else:
+            self._update_plt_obj()
+
+All new plugins should derive from this class.
+
+When implementing a new visualizer you have to perform the following steps:
+
+1. Let your visualizer class inherit from the ``Visualizer`` class in ``base visualizer.py``.
+
+2. Implement the ``_create_data_reader(self, run_directory)`` function.
+This function should return a data reader object (see above) for this plugin's data.
+
+3. Implement the ``_create_plt_obj(self)`` function.
+This function needs to access the plotting data from the ``self.data`` member (this is the data structure as returned by the data readers ``.get(...)`` function, create some kind of matplotlib artist by storing it in the ``self.plt_obj`` member variable and set up other plotting details (e.g. a colorbar).
+
+4. Implement the ``_update_plt_obj(self)`` function.
+This is called only after a valid ``self.plt_obj`` was created.
+It updates the matplotlib artist with new data.
+Therefore it again needs to access the plotting data from the ``self.data`` member and call the data update API for the matplotlib artist (normally via ``.set_data(...)``.
+

docs/source/index.rst

@@ -7,9 +7,9 @@
 
   .. image:: ../logo/pic_logo.pdf
 
-*A particle-in-cell code for GPGPUs*
+*Particle-in-Cell Simulations for the Exascale Era*
 
-PIConGPU is a fully relativistic, many GPGPU, 3D3V particle-in-cell (PIC) code.
+PIConGPU is a fully relativistic, manycore, 3D3V particle-in-cell (PIC) code.
 The PIC algorithm is a central tool in plasma physics.
 It describes the dynamics of a plasma by computing the motion of electrons and ions in the plasma based on Maxwell's equations.
 
@@ -113,6 +113,7 @@ Development
    dev/clangtools
    dev/picongpu
    dev/pmacc
+   dev/py_postprocessing
    dev/doxyindex
 
 ********************