Merge branch 'dev' into features/#250-CTS-load-areas

AUTHORS.rst

@@ -2,4 +2,4 @@
 Authors
 =======
 
-* Guido Pleßmann, Ilka Cußman, Stephan Günther - https://github.com/openego/eGon-data
+* Guido Pleßmann, Ilka Cußman, Stephan Günther, Jonathan Amme, Julian Endres - https://github.com/openego/eGon-data

CHANGELOG.rst

@@ -33,6 +33,14 @@ Added
   returns the current configuration settings. See `PR #159`_ for more
   details.
 
+* You can now use tasks which are not part of a ``Dataset``, i.e. which are
+  unversioned, as dependencies of a dataset. See `PR #318`_ for more
+  details.
+
+* You can now force the tasks of a ``Dataset`` to be always executed by
+  giving the version of the ``Dataset`` a ``".dev"`` suffix. See `PR
+  #318`_ for more details.
+
 * OSM data import as done in open_ego
   `#1 <https://github.com/openego/eGon-data/issues/1>`_
   which was updated to the latest long-term data set of the 2021-01-01 in
@@ -65,20 +73,26 @@ Added
   `#9 <https://github.com/openego/eGon-data/issues/9>`_
 * Add hydro and biomass power plants eGon2035
   `#127 <https://github.com/openego/eGon-data/issues/127>`_
-* Creation of the ehv/hv grid model with osmTGmod, see 
-  `issue #4 <https://github.com/openego/eGon-data/issues/4>`_ and 
+* Creation of the ehv/hv grid model with osmTGmod, see
+  `issue #4 <https://github.com/openego/eGon-data/issues/4>`_ and
   `PR #164 <https://github.com/openego/eGon-data/pull/164>`_
 * Identification of medium-voltage grid districts
-  `#10 <https://github.com/openego/eGon-data/pull/10>`_  
+  `#10 <https://github.com/openego/eGon-data/pull/10>`_
 * Distribute electrical demands of households to zensus cells
   `#181 <https://github.com/openego/eGon-data/issues/181>`_
 * Distribute electrical demands of cts to zensus cells
   `#210 <https://github.com/openego/eGon-data/issues/210>`_
 * Include industrial sites' download, import and merge
   `#117 <https://github.com/openego/eGon-data/issues/117>`_
-* Integrate scenario table with parameters for each sector 
+* Integrate scenario table with parameters for each sector
   `#177 <https://github.com/openego/eGon-data/issues/177>`_
-* Extract landuse areas from OSM 
+* The volume of the docker container for the PostgreSQL database
+  is saved in the project directory under `docker/database-data`.
+  The current user (`$USER`) is owner of the volume.
+  Containers created prior to this change will fail when using the
+  changed code. The container needs to be re-created.
+  `#228 <https://github.com/openego/eGon-data/issues/228>`_
+* Extract landuse areas from OSM
   `#214 <https://github.com/openego/eGon-data/issues/214>`_
 * Integrate weather data and renewable feedin timeseries
   `#19 <https://github.com/openego/eGon-data/issues/19>`_
@@ -88,6 +102,57 @@ Added
   `#109 <https://github.com/openego/eGon-data/issues/109>`_
 * Assign voltage level and bus_id to power plants
   `#15 <https://github.com/openego/eGon-data/issues/15>`_
+* Integrate solar rooftop for etrago tables
+  `#255 <https://github.com/openego/eGon-data/issues/255>`_
+* Integrate gas bus and link tables
+  `#198 <https://github.com/openego/eGon-data/issues/198>`_
+* Integrate data bundle
+  `#272 <https://github.com/openego/eGon-data/issues/272>`_
+* Add household electricity demand time series, mapping of
+  demand profiles to census cells and aggregated household
+  electricity demand time series at MV grid district level
+  `#256 <https://github.com/openego/eGon-data/issues/256>`_
+* Integrate power-to-gas installation potential links
+  `#293 <https://github.com/openego/eGon-data/issues/293>`_
+* Integrate distribution of wind onshore and pv ground mounted generation
+  `#146 <https://github.com/openego/eGon-data/issues/146>`_
+* Integrate dynamic line rating potentials
+  `#72 <https://github.com/openego/eGon-data/issues/72>`_
+* Integrate gas voronoi polygons
+  `#308 <https://github.com/openego/eGon-data/issues/308>`_
+* Integrate supply strategies for individual and district heating
+  `#232 <https://github.com/openego/eGon-data/issues/232>`_
+* Integrate gas production
+  `#321 <https://github.com/openego/eGon-data/issues/321>`_
+* Integrate industrial time series creation
+  `#237 <https://github.com/openego/eGon-data/issues/237>`_
+* Merge electrical loads per bus and export to etrago tables
+  `#328 <https://github.com/openego/eGon-data/issues/328>`_
+* Insert industial gas demand
+  `#321 <https://github.com/openego/eGon-data/issues/358>`_
+* Integrate existing CHP and extdended CHP > 10MW_el
+  `#266 <https://github.com/openego/eGon-data/issues/266>`_
+* Add random seed to CLI parameters
+  `#351 <https://github.com/openego/eGon-data/issues/351>`_
+* Extend zensus by a combined table with all cells where
+  there's either building, apartment or population data
+  `#359 <https://github.com/openego/eGon-data/issues/359>`_
+* Include allocation of pumped hydro units
+  `#332 <https://github.com/openego/eGon-data/issues/332>`_
+* Add example metadata for OSM, VG250 and Zensus VG250.
+  Add metadata templates for licences, context and some helper
+  functions. Extend docs on how to create metadata for tables.
+  `#139 <https://github.com/openego/eGon-data/issues/139>`_
+* Integrate DSM potentials for CTS and industry
+  `#259 <https://github.com/openego/eGon-data/issues/259>`_
+* Assign weather cell id to weather dependant power plants
+  `#330 <https://github.com/openego/eGon-data/issues/330>`_
+* Distribute wind offshore capacities
+  `#329 <https://github.com/openego/eGon-data/issues/329>`_
+* Add CH4 storages
+  `#405 <https://github.com/openego/eGon-data/issues/405>`_
+* Include allocation of conventional (non CHP) power plants
+  `#392 <https://github.com/openego/eGon-data/issues/392>`_
 
 .. _PR #159: https://github.com/openego/eGon-data/pull/159
 
@@ -122,6 +187,58 @@ Changed
   `#196 <https://github.com/openego/eGon-data/issues/196>`_
 * Update installation of demandregio's disaggregator
   `#202 <https://github.com/openego/eGon-data/issues/202>`_
+* Update etrago tables
+  `#243 <https://github.com/openego/eGon-data/issues/243>`_and
+  `#285 <https://github.com/openego/eGon-data/issues/285>`_
+* Migrate VG250 to datasets
+  `#283 <https://github.com/openego/eGon-data/issues/283>`_
+* Allow configuring the airflow port
+  `#281 <https://github.com/openego/eGon-data/issues/281>`_
+* Migrate mastr, mv_grid_districts and re_potential_areas to datasets
+  `#297 <https://github.com/openego/eGon-data/issues/297>`_
+* Migrate industrial sites to datasets
+  `#237 <https://github.com/openego/eGon-data/issues/237>`_
+* Rename etrago tables from e.g. egon_pf_hv_bus to egon_etrago bus etc.
+  `#334 <https://github.com/openego/eGon-data/issues/334>`_
+* Move functions used by multiple datasets
+  `#323 <https://github.com/openego/eGon-data/issues/323>`_
+* Migrate scenario tables to datasets
+  `#309 <https://github.com/openego/eGon-data/issues/309>`_
+* Migrate weather data and power plants to datasets
+  `#314 <https://github.com/openego/eGon-data/issues/314>`_
+* Create and fill table for CTS electricity demand per bus
+  `#326 <https://github.com/openego/eGon-data/issues/326>`_
+* Migrate osmTGmod to datasets
+  `#305 <https://github.com/openego/eGon-data/issues/305>`_
+* Filter osm landuse areas, rename industrial sites tables and update load curve function
+  `#378 <https://github.com/openego/eGon-data/issues/378>`_
+* Remove version columns from eTraGo tables and related code
+  `#384 <https://github.com/openego/eGon-data/issues/384>`_
+* Remove country column from scenario capacities table
+  `#391 <https://github.com/openego/eGon-data/issues/391>`_
+* Update version of zenodo download
+  `#397 <https://github.com/openego/eGon-data/issues/397>`_
+* Rename columns gid to id
+  `#169 <https://github.com/openego/eGon-data/issues/169>`_
+* Remove upper version limit of pandas
+  `#383 <https://github.com/openego/eGon-data/issues/383>`_
+* Use random seed from CLI parameters for CHP and society prognosis functions
+  `#351 <https://github.com/openego/eGon-data/issues/351>`_
+* Changed demand.egon_schmidt_industrial_sites - table and merged table (industrial_sites)
+  `#423 <https://github.com/openego/eGon-data/issues/423>`_
+* Replace 'gas' carrier with 'CH4' and 'H2' carriers
+  `#436 <https://github.com/openego/eGon-data/issues/436>`_
+* Adjust file path for industrial sites import
+  `#397 <https://github.com/openego/eGon-data/issues/418>`_
+* Rename columns subst_id to bus_id
+  `#335 <https://github.com/openego/eGon-data/issues/335>`_
+* Apply black and isort for all python scripts
+  `#463 <https://github.com/openego/eGon-data/issues/463>`_
+* Update deposit id for zenodo download
+  `#397 <https://github.com/openego/eGon-data/issues/498>`_
+* Migrate dlr script to datasets
+  `#508 <https://github.com/openego/eGon-data/issues/508>`_
+
 
 Bug fixes
 ---------
@@ -139,7 +256,56 @@ Bug fixes
   `#204 <https://github.com/openego/eGon-data/issues/204>`_
 * Replace wrong table name in SQL function used in substation extraction
   `#236 <https://github.com/openego/eGon-data/issues/236>`_
-* Fix osmtgmod for osm data from 2021 by updating substation in Garenfeld 
+* Fix osmtgmod for osm data from 2021 by updating substation in Garenfeld and set srid
   `#241 <https://github.com/openego/eGon-data/issues/241>`_
+  `#258 <https://github.com/openego/eGon-data/issues/258>`_
 * Adjust format of voltage levels in hvmv substation
   `#248 <https://github.com/openego/eGon-data/issues/248>`_
+* Change order of osmtgmod tasks
+  `#253 <https://github.com/openego/eGon-data/issues/253>`_
+* Fix missing municipalities
+  `#279 <https://github.com/openego/eGon-data/issues/279>`_
+* Fix import of hydro power plants
+  `#270 <https://github.com/openego/eGon-data/issues/270>`_
+* Fix path to osm-file for osmtgmod_osm_import
+  `#258 <https://github.com/openego/eGon-data/issues/258>`_
+* Fix conflicting docker containers by setting a project name
+  `#289 <https://github.com/openego/eGon-data/issues/289>`_
+* Update task insert-nep-data for pandas version 1.3.0
+  `#322 <https://github.com/openego/eGon-data/issues/322>`_
+* Fix versioning conflict with mv_grid_districts
+  `#340 <https://github.com/openego/eGon-data/issues/340>`_
+* Set current working directory as java's temp dir when executing osmosis
+  `#344 <https://github.com/openego/eGon-data/issues/344>`_
+* Fix border gas voronoi polygons which had no bus_id
+  `#362 <https://github.com/openego/eGon-data/issues/362>`_
+* Add dependency from WeatherData to Vg250
+  `#387 <https://github.com/openego/eGon-data/issues/387>`_
+* Fix unnecessary columns in normal mode for inserting the gas production
+  `#387 <https://github.com/openego/eGon-data/issues/390>`_
+* Add xlrd and openpyxl to installation setup
+  `#400 <https://github.com/openego/eGon-data/issues/400>`_
+* Store files of OSM, zensus and VG250 in working dir
+  `#341 <https://github.com/openego/eGon-data/issues/341>`_
+* Remove hard-coded slashes in file paths to ensure Windows compatibility
+  `#398 <https://github.com/openego/eGon-data/issues/398>`_
+* Add missing dependency in pipeline.py
+  `#412 <https://github.com/openego/eGon-data/issues/412>`_
+* Add prefix egon to MV grid district tables
+  `#349 <https://github.com/openego/eGon-data/issues/349>`_
+* Bump MV grid district version no
+  `#432 <https://github.com/openego/eGon-data/issues/432>`_
+* Add curl to prerequisites in the docs
+  `#440 <https://github.com/openego/eGon-data/issues/440>`_
+* Replace NAN by 0 to avoid empty p_set column in DB
+  `#414 <https://github.com/openego/eGon-data/issues/414>`_
+* Exchange bus 0 and bus 1 in Power-to-H2 links
+  `#458 <https://github.com/openego/eGon-data/issues/458>`_
+* Fix missing cts demands for eGon2035
+  `#511 <https://github.com/openego/eGon-data/issues/511>`_
+* Add `data_bundle` to `industrial_sites` task dependencies
+  `#468 <https://github.com/openego/eGon-data/issues/468>`_
+* Lift `geopandas` minimum requirement to `0.10.0`
+  `#504 <https://github.com/openego/eGon-data/issues/504>`_
+* Use inbuilt `datetime` package instead of `pandas.datetime`
+  `#516 <https://github.com/openego/eGon-data/issues/516>`_

CONTRIBUTING.rst

@@ -189,10 +189,15 @@ Before requesting a review, please
 1. Include passing tests (run ``tox``). [#tox-note]_
 2. Let the workflow run in :ref:`Test mode` once from scratch to verify
    successful execution
-3. Update documentation when there's new API, functionality etc.
-4. Add a note to ``CHANGELOG.rst`` about the changes and refer to the
+3. Make sure that your changes are tested in integration with other
+   tasks and on a complete run at least once by merging them into the
+   `continuous-integration/run-everything-over-the-weekend`_ branch.
+   This branch will regularly be checked out and tested on a complete
+   workflow run on friday evening.
+4. Update documentation when there's new API, functionality etc.
+5. Add a note to ``CHANGELOG.rst`` about the changes and refer to the
    corresponding Github issue.
-5. Add yourself to ``AUTHORS.rst``.
+6. Add yourself to ``AUTHORS.rst``.
 
 .. [#tox-note]
     If you don't have all the necessary Python versions available locally
@@ -202,6 +207,7 @@ Before requesting a review, please
     It will be slower though ...
 
 .. _run the tests: https://github.com/openego/eGon-data/actions?query=workflow%3A%22Tests%2C+code+style+%26+coverage%22
+.. _continuous-integration/run-everything-over-the-weekend: https://github.com/openego/eGon-data/tree/continuous-integration/run-everything-over-the-weekend
 
 
 When requesting reviews, please keep in mind it might be a significant effort
@@ -269,6 +275,105 @@ be saved locally, please use `CWD` to store the data. This is achieved by using
   filepath = Path(".") / "filename.csv"
   urlretrieve("https://url/to/file", filepath)
 
+Add metadata
+------------
+
+Add a metadata for every dataset you create for describing data with
+machine-readable information. Adhere to the OEP Metadata v1.4.1, you can
+follow
+`the example <https://github.com/OpenEnergyPlatform/oemetadata/blob/develop/metadata/latest/example.json>`_
+to understand how the fields are used. Field are described in detail in the
+`Open Energy Metadata Description`_.
+
+You can obtain the metadata string from a table you created in SQL via
+
+.. code-block:: sql
+
+  SELECT obj_description('<SCHEMA>.<TABLE>'::regclass);
+
+Alternatively, you can write the table comment directly to a JSON file by
+
+.. code-block:: bash
+
+  psql -h <HOST> -p <PORT> -d <DB> -U <USER> -c "\COPY (SELECT obj_description('<SCHEMA>.<TABLE>'::regclass)) TO '/PATH/TO/FILE.json';"
+
+For bulk export of all DB's table comments you can use `this script
+<https://gist.github.com/nesnoj/86145999eca8182f43c2bca36bcc984f>`_.
+Please verify that your metadata string is in compliance with the OEP Metadata
+standard version 1.4.1 using the `OMI tool
+<https://github.com/OpenEnergyPlatform/omi>`_ (tool is shipped with eGon-data):
+
+.. code-block:: bash
+
+  omi translate -f oep-v1.4 metadata_file.json
+
+If your metadata string is correct, OMI puts the keys in the correct order and
+prints the full string (use `-o` option for export).
+
+You may omit the fields `id` and `publicationDate` in your string as it will be
+automatically set at the end of the pipeline but you're required to set them to
+some value for a complete validation with OMI. For datasets published on the
+OEP `id` will be the URL which points to the table, it will follow the pattern
+`https://openenergy-platform.org/dataedit/view/SCHEMA/TABLE`.
+
+For previous discussions on metadata, you may want to check
+`PR 176 <https://github.com/openego/eGon-data/pull/176>`_.
+
+Helpers
+^^^^^^^
+
+There are some **licence templates** provided in :py:mod:`egon.data.metadata`
+you can make use of for fields 11.4 and 12 of the
+`Open Energy Metadata Description`_. Also, there's a template for the
+**metaMetadata** (field 16).
+
+There are some functions to quickly generate a template for the
+**resource fields** (field 14.6.1 in `Open Energy Metadata Description`_) from
+a SQLA table class or a DB table. This might be especially helpful if your
+table has plenty of columns.
+
+* From SQLA table class:
+  :py:func:`egon.data.metadata.generate_resource_fields_from_sqla_model`
+* From database table:
+  :py:func:`egon.data.metadata.generate_resource_fields_from_db_table`
+
+Sources
+^^^^^^^
+
+The **sources** (field 11) are the most important parts of the metadata which
+need to be filled manually. You may also add references to tables in eGon-data
+(e.g. from an upstream task) so you don't have to list all original sources
+again. Make sure you include all upstream attribution requirements.
+
+The following example uses various input datasets whose attribution must be
+retained:
+
+.. code-block:: python
+
+  "sources": [
+      {
+          "title": "eGo^n - Medium voltage grid districts",
+          "description": (
+              "Medium-voltage grid districts describe the area supplied by "
+              "one MV grid. Medium-voltage grid districts are defined by one "
+              "polygon that represents the supply area. Each MV grid district "
+              "is connected to the HV grid via a single substation."
+          ),
+          "path": "https://openenergy-platform.org/dataedit/view/"
+                  "grid/egon_mv_grid_district", # "id" in the source dataset
+          "licenses": [
+              license_odbl(attribution=
+                  "© OpenStreetMap contributors, 2021; "
+                  "© Statistische Ämter des Bundes und der Länder, 2014; "
+                  "© Statistisches Bundesamt, Wiesbaden 2015; "
+                  "(Daten verändert)"
+              )
+          ]
+      },
+      # more sources...
+  ]
+
+.. _Open Energy Metadata Description: https://github.com/OpenEnergyPlatform/oemetadata/blob/develop/metadata/v141/metadata_key_description.md
 
 Adjusting test mode data
 ------------------------
@@ -295,7 +400,7 @@ How to document Python scripts
 
 Use docstrings to document your Python code. Note that PEP 8 also
 contains a `section <PEP8-docstrings_>`_ on docstrings and that there is
-a whole `PEP <PEP257_>`_ dedicated to docstring convetions. Try to
+a whole `PEP <PEP257_>`_ dedicated to docstring conventions. Try to
 adhere to both of them.
 Additionally every Python script needs to contain a header describing
 the general functionality and objective and including information on