feat: update electricity demand profiles

[tgilon]

Closes #6.

Changes proposed in this Pull Request

This PR suggests to add options in PyPSA-Eur to use the custom electricity demand profiles of TYNDP.

The feature can be tested with:

make tyndp

Tasks

• Archive ENTSO-E data on Zenodo by @daniel-rdt
• Create switches in the configuration file by @tgilon
• Implement raw data processing in separate script by @daniel-rdt
• Check whether UK or GB should be used
• Define bidding zones as geo data in build_shapes
  • [Electricity map topology data](https://github.com/electricitymaps/electricitymaps-contrib/blob/master DATA_SOURCES.md), using https://www.naturalearthdata.com/downloads/10m-cultural-vectors/
  • https://github.com/EnergieID/entsoe-py/tree/master/entsoe/geo/geojson
• Add parameters in add_electricity and prepare_sector_networkby @tgilon
• Add news rules to documentation
• Address nodes vs countries in electricity demand raw data
• Process data through build_electricity_demand_base
• Deactivate Heating technologies (biomass_boilers and oil_boilers) in prepare_sector_network add_biomass
• Deactivate industrial electricity with new config switch and low temp heat load in prepare_sector_network add_industry based on heating sector switch
• Fix issue with missing Luxembourg pop_layout and low-voltage nodes for LUF1 and LUG1
• Fix errors in post processing plotting routines due to missing heat sector
• Check for linear interpolation in build_electricity_demand
• Check for manual adjustments in build_electricity_demand (should not be allowed)
• Validate make tyndp
• Validate that electricity demand in output network corresponds to original data (xref: feat: share testing notebook for electricity demand profiles #44)
• Generalize using planning_horizons wildcard

Workflow

1. retrieve_tyndp_bundle
   • Update outputs to add electricity demand data
2. (new) clean_tyndp_data
   • Prepare the demand data input for build_electricity_demand (electricity) and prepare_sector_network (other carriers - will be covered later)
3. build_electricity_demand and (new) build_electricity_demand_tyndp
   • By default, data/electricity_demand_raw.csv (OPSD data) and data/load_synthetic_raw.csv (demand calculator) are used.
   • electricity_demand.csv from PyPSA-Eur is a CSV file with hourly resolution data as rows and countries as columns. TYNDP electricity demand profiles have climate years as columns and tabs for each node. Electricity demand data varies across planning years (hence the need for the planning_horizons wildcard for TYNDP demand-related rules).
   • Use of manual adjustments is disabled (as we don't want to alter TYNDP data).
   • Data is given per bidding zones. We want to keep this granularity and will modify the subsequent build_electricity_demand_base rule to work with it.
4. build_electricity_demand_base and (new) build_electricity_demand_base_tyndp
   • Can be skipped for TYNDP data, no upsampling needed
   • Introduction of planning_horizons wildcard in the TYNDP case
5. cluster_network
   • Disable cluster option in TYNDP case
   • Make resources/electricity_demand_base_s.nc an optional input as electrical load is used for clustering
6. add_electricity
   • Optional load attached to the network if OPSD load is used
   • If TYNDP load data is used, no load is attached as it relies on planning_horizons wildcard. Attached in prepare_sector_network.
7. prepare_sector_network
   • Electricity
     • Electrical load attached to the network in the TYNDP case (for a given planning_horizons).
     • Distribution losses are disabled. TYNDP electricity demand includes T&D losses (Methodology report, p.21). Therefore, there is no additional modelling of the losses.
   • Transport:
     • Electric transport demand is treated in Implementation of prosumer, EV, and wholesale market nodes #5
     • Oil and H2 transport demand can be disabled using shares in the configuration file.
     • Always enable distribution as TYNDP model both prosumer and street nodes
     • Only passenger cars are considered, to be addressed by Implementation of prosumer, EV, and wholesale market nodes #5
   • Heating:
     • Considering the fact that most of the heat demand is included in the provided profiles (except HHP), the heating sector is deactivated in the configuration file
     • Electrical heat demand is not removed from the load in the TYNDP case as heat is not explicitly modelled.
     • District heating demand is disabled as it is calculated exogenously using the supply tool.
     • Oil boilers are disabled in the config when using TYNDP load
     • As HHP will not be extended, it makes sense to only add existing in add_existing_baseyear. To be addressed by Implementation of hybrid heat pumps & removal of other technologies #3
     • Waste heat ? To be addressed by Implementation of hybrid heat pumps & removal of other technologies #3
   • Biomass:
     • Biomass boilers are disabled in the config when using TYNDP load (see Heating sector).
   • Industry:
     • Shipping and aviation are included over here (e-kerosene, e-diesel)
     • Synthetic fuels modelling is covered in Separation of synthetic fuel carriers and integration of their demand #4
     • Low temperatrue industrial heat demand is disabled with the Heating sector switch in TYNDP case as heat is not explicitly modelled. A new dependency is added that ensures low temperature heat industrial heat is only enabled when heating sector is enabled.
     • A new configuration switch use_industry_load is introduced for modelling Industrial electricity demand explicitly. This is disabled for TYNDP load.
8. add_existing_baseyear
   • Check existing technologies added
   • Add existing HHP ? Will be addressed by Implementation of hybrid heat pumps & removal of other technologies #3

Checklist

• I tested my contribution locally and it works as intended.
• Code and workflow changes are sufficiently documented.
• Changed dependencies are added to envs/environment.yaml.
• Changes in configuration options are added in config/config.default.yaml.
• Changes in configuration options are documented in doc/configtables/*.csv.
• Changes in configuration options are added in config/test/*.yaml.
• Sources of newly added data are documented in doc/data_sources.rst.
• A release note doc/release_notes.rst is added.

[daniel-rdt]

Thanks @tgilon! Good catch with the UK / GB load! I also validated the load again on a yearly total load per country basis and it passes the assertion locally. Should we maybe introduce an assertion unit test for this into the workflow based on your validation notebook?

[coroa]

Looks good in general, a few small improvements and questions, i do still have :)

[coroa]

Re: Heating demand. It would be better if we can deactivate the heating demand with some heating specific settings (since this can then also avoid calculating all the heat pump profiles and temperature dependent heat demand inputs).

[coroa]

Maybe we need to add a sort of validate config step during which incompatible settings are raising. Out of scope for this PR.

[tgilon]

Should we maybe introduce an assertion unit test for this into the workflow based on your validation notebook?

I thought about it. Since we need the output of prepare_sector_network, I haven't been able to identify a useful test.

[coroa]

So, nothing here is blocking anymore in my opinion.

I would like that the load_source != "tyndp" fences vanish in favour of existing or new heat sector config settings, which at the moment should be only documented to be co-dependent. It's fine with me if you want to address this in a follow up PR.

And before the final tyndp release we are aiming for some sort of config validation, which complains if incompatible settings are chosen.

[tgilon]

After discussing with @coroa and @daniel-rdt, I will use the sector:heating rather than the specific load_source parameter. This avoids using electricity related configurations in prepare_sector_network. However, as this introduces configuration dependencies, an issue has been opened #43 to document dependencies.

[tgilon]

@coroa @daniel-rdt Here is a summary of what I did:

• biomass_boilers and oil_boilers are existing config. I've disabled them.
• I've removed the warning for the electricity distribution grid as this will be addressed by the config validation.
• I introduced a new config for the industrial load: sector:use_industry_load. This is similar to the heating question.

All of this is documented in #43.

[tgilon]

All the requests and comments have been addressed.

I have also re-validated the results. The PR is available here: #44 and an updated notebook can be found here.