Revert "Revert "Finishes adding Interconnects, Integrates REeDS Balan…

…cing Authority Shapes, Interface Transmission Limits (ITLs).""
PyPSA · Apr 8, 2024 · a4f1b03 · a4f1b03
1 parent eb6e96f
commit a4f1b03
Show file tree

Hide file tree

Showing 98 changed files with 899,371 additions and 31,604 deletions.
diff --git a/CITATION.cff b/CITATION.cff
@@ -11,6 +11,10 @@ authors:
 - family-names: "Barnes"
   given-names: "Trevor"
   orchid: https://orcid.org/0000-0003-2458-2968
-- family-names: "Inês Lima"
-  given-names: "Azevedo"
+- family-names: "Frysztacki"
+  given-names: "Martha"
+- family-names: "Hofmann"
+  given-names: "Fabian"
+- family-names: "Azevedo"
+  given-names: "Inês L."
 doi: 10.5281/zenodo.10815964
diff --git a/README.md b/README.md
@@ -7,7 +7,7 @@
 PyPSA-USA is an open-source power systems model of the bulk transmission systems in the United States. This workflow draws from the work of [pypsa-eur](https://pypsa-eur.readthedocs.io/en/latest/index.html) to build a highly configurable power systems model that can be used for capacity expansion modeling, production cost simulation, and power flow analysis.
 
 Example 500 Node Western Interconnection Network:
-![pypsa-usa Base Network](https://github.com/PyPSA/pypsa-usa/blob/master/workflow/repo_data/network_500.jpg)
+![pypsa-usa Base Network](https://pypsa-usa.readthedocs.io/en/latest/_images/PyPSA-USA_network.jpg)
 
 # Contributing
 

diff --git a/docs/source/_static/PyPSA-USA_network.png b/docs/source/_static/PyPSA-USA_network.png
diff --git a/docs/source/about-install.md b/docs/source/about-install.md
@@ -1,10 +1,10 @@
 # Installation
 
-## Step 1: Clone Github Repository
+## Step 1: Set up github ssh-key
 
-```{note}
-If it your first time cloning a repository with a submodule, you will need to set up your git with an ssh-key by following these [directions](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent).
-```
+If it your first time cloning a **repository submodule through ssh**, you will need to set up your git with an ssh-key by following these [directions](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent).
+
+## Step 2: Clone Github Repository
 
 Clone this repository and all submodules
 
@@ -23,8 +23,7 @@ $ git submodule update
 ```
 
 
-
-## Step 2: Create Conda Environment
+## Step 3: Create Conda Environment
 
 PyPSA-USA uses conda/mamba to manage project dependencies. You can download and install mamba following the [instructions](https://mamba.readthedocs.io/en/latest/mamba-installation.html). Follow links for mambaforge installation. There are two ways to install mamba, the first (recommended) method will start with a fresh install, meaning if you have previously installed conda environments, you will need to recreate these conda envs. If you already have conda installed and do not wish to install mamba, you can follow the same set of instructions replacing any `mamba` with `conda`
 
@@ -38,7 +37,7 @@ $ mamba activate pypsa-usa
 You also have the option to use miniconda. Download [Miniconda](https://docs.conda.io/en/latest/miniconda.html) following their [instructions](https://docs.conda.io/en/latest/miniconda.html).
 
 
-## Step 3: Install a Solver
+## Step 4: Install a Solver
 
 PyPSA-USA uses an external solver to solve the optimization problem formulated in the workflow. After you install your solver and confirm it is accessible within your conda environment, update your solving configuration to match your solver of choice.
 You can download and install several free options here:

diff --git a/docs/source/about-introduction.md b/docs/source/about-introduction.md
@@ -1,7 +1,21 @@
 (introduction)=
 # Introduction
 
-PyPSA-USA is an open-source bulk power system planning model for the United-States. PyPSA-USA is built on the Python for Power System Analysis (pypsa) platform, and leverages much of the work of [PyPSA-EUR](https://pypsa-eur.readthedocs.io/en/latest/index.html) throughout the workflow. For some introduction to the pypsa modeling workflow- please see the [pypsa-eur](https://youtu.be/ty47YU1_eeQ?si=Cz90jWcN1xk1Eq4i) introductory video.
+```{warning}
+**This model is under active development. If you would like to chat about using the model please don't hesitate to reach out to [email protected] and [email protected] for support**
+```
+
+PyPSA-USA is an open-source energy system dataset of the United States energy system with continental US coverage.
+
+You're might be here because you are interested in running planning or operations models of the United States energy system. You also might be asking yourself- "why would I want to use PyPSA-USA instead of the plethora of other energy system models available?"
+
+PyPSA-USA provides you with a toolkit to customize the **data** of energy system model with ease. Through configuration file you can control the spatial, temporal, and operational resolution of your energy system model with access to cleaned and prepared historical and forecasted data. This means, you can build a model of **ERCOT, WECC, or the Eastern interconnection**, where the transmission network is clustered to **N# of user defined nodes**, which can respect the boundaries of **balancing areas, states, or REeDs GIS Shapes**, using **historical EIA-930 demand data years 2018-2023** OR **NREL EFS forcasted demand [2030, 2040, 2050]**, with **historical daily/monthly fuel prices from ISOs/EIA [choice of year]**, AND imported capital cost projections from the **NREL Annual Technology Baseline**.
+
+You can create data model- and export to use in your own homebrewed optimization model via csv tables, or xarray netCDF model.
+
+Beyond creating a data model, PyPSA-USA also provides an interface for running capacity expansion planning and operational simulation models with DC power flow with the Python for Power System Analysis package. You can run expansion planning exercises which integrate regional and national policy constraints like RPS standards, emissions standards, PRMs, and more.
+
+PyPSA-USA builds on and leverages the work of [PyPSA-EUR](https://pypsa-eur.readthedocs.io/en/latest/index.html) developed by TU Berlin. PyPSA-USA is actively developed by the [INES Research Group](https://ines.stanford.edu) at Stanford University and the [ΔE+ Research Group](https://www.sfu.ca/see/research/delta-e.html) at Simon Fraser University.
 
 (workflow)=
 ## Workflow

diff --git a/docs/source/config-configuration.md b/docs/source/config-configuration.md
@@ -1,8 +1,6 @@
 (config)=
 # Configuration
 
-**This workflow is currently only being tested for the `western` and  `texas` interconnection wildcards.**
-
 (network_cf)=
 ## Pre-set Configuration Options
 
@@ -15,13 +13,13 @@ Most users will leave this as `pypsa-usa`.
 | Configuration Options: | PyPSA-USA | ADS2032(lite) |
 |:----------:|:----------:|:----------:|
 | Transmission | TAMU/BE | TAMU/BE |
-| Thermal Generators | EIA860, WECC-ADS, CEC Plexos | WECC-ADS |
+| Thermal Generators | EIA860, WECC-ADS | WECC-ADS |
 | Renewable Time-Series | Atlite | WECC-ADS |
 | Hydro Time-Series | Breakthrough (temp) | WECC-ADS |
 | Demand | EIA930 | WECC-ADS |
 | Historical Demand | 2019-2023 | - |
 | Future Demand | NREL EFS | WECC ADS 2032 |
-| Interconnections Supported | WECC + ERCOT | WECC |
+| Interconnections Supported | western, texas, eastern, usa | WECC |
 | Capital Costs | NREL-ATB | NREL-ATB |
 | Fuel Costs | CAISO/EIA | CAISO/EIA |
 | Purpose[^+] | CEM, PCS | PCS |
@@ -230,13 +228,11 @@ If using the `{opts}` wildcard to reduce emissions, the user must put in a `co2b
 (clustering_cf)=
 ## `clustering`
 
-Minimum Number of clusters:
-```bash
-Eastern: TBD
-Western: 30
-Texas: TBD
-```
+When clustering `aggregation_zones` defines the region boundaries which will be respected through the clustering process; State boarders, balancing authority regions, or REeDs shapes. This feature is important for imposing constraints (`opts`) which are defined over specific regions. For example, the data included in the model on interface transfer capacities are prepared for REeDs shapes but not states and BA regions. Moving forward we plan to use REeDs shapes as our default however we will maintain States and BA regions as well.
+
+Each clustering and interconnection option will have a different number of minimum nodes which can be clustered to, an error will be thrown in `cluster_network` notifying you of that number if you have selected a value too low.
 
+Cleaned and labeled REeDs Shapes are pulled from this github repository: https://github.com/pandaanson/NYU-law-work
 
 ```{eval-rst}
 .. literalinclude:: ../../workflow/config/config.default.yaml

diff --git a/docs/source/configtables/clustering.csv b/docs/source/configtables/clustering.csv
@@ -6,6 +6,6 @@ feature,str," {'solar+onwind-time', 'solar+onwind-cap', 'solar-time', 'solar-cap
 cluster_network:,,,
 algorithm,str,{'kmeans'},
 feature,str," {'solar+onwind-time', 'solar+onwind-cap', 'solar-time', 'solar-cap', 'solar+offwind-cap'}",For HAC clustering.
-aggregation_zones,str,"{'balancing_area', 'state'}",Boundaries of GIS shapes that are to be respected in clustering. Retain if you would like to analyze expansion within a given zone.
+aggregation_zones,str,"{'balancing_area', 'state', 'reeds_zone'}",Boundaries of GIS shapes that are to be respected in clustering. Retain if you would like to analyze expansion within a given zone.
 aggregation_strategies:,,,
 table --> {key},str,"{'mean','max','min',etc}","Specifiy the method of aggregating fields within the generators, buses tables. "
diff --git a/docs/source/configtables/lines.csv b/docs/source/configtables/lines.csv
@@ -5,6 +5,7 @@ s_nom_max,MW,float,Global upper limit for the maximum capacity of each extendabl
 max_extension,MW,float,Upper limit for the extended capacity of each extendable line.
 length_factor,--,float,Correction factor to account for the fact that buses are *not* connected by lines through air-line distance.
 under_construction,--,"One of {'zero': set capacity to zero, 'remove': remove completely, 'keep': keep with full capacity}",Specifies how to handle lines which are currently under construction.
+interface_transmission_limits, --, true or false, Activate the Interface Transmission Limits (ITL) zones limits.
 dynamic_line_rating,,,
 #NAME?,bool,true or false,Whether to take dynamic line rating into account
 #NAME?,--,Should be a folder listed in the configuration ``atlite: cutouts:`` (e.g. 'europe-2013-era5') or reference an existing folder in the directory ``cutouts``. Source module must be ERA5.,Specifies the directory where the relevant weather data is stored.

diff --git a/docs/source/index.md b/docs/source/index.md
@@ -3,16 +3,23 @@
 % You can adapt this file completely to your liking, but it should at least
 % contain the root `toctree` directive.
 
-# PyPSA USA
+# PyPSA-USA
 
 ```{warning}
-**This model is under active development. The western interconnection is stable, however you are likely to find bugs in the workflow as we continue to develop the model. Please file [github issues](https://github.com/PyPSA/pypsa-usa/issues) or email [email protected] or [email protected] for support**
+**This model is under active development. If you would like to chat about using the model please don't hesitate to reach out to [email protected] and [email protected] for support**
 ```
 
-PyPSA-USA is an open-source power systems model of the transmission system in the United States. It is a highly configurable power systems model that can be used for capacity expansion modeling, production cost simulation, and power flow analysis. This workflow draws from the work of [pypsa-eur](https://pypsa-eur.readthedocs.io/en/latest/index.html) and [pypsa-meets-earth](https://pypsa-earth.readthedocs.io/en/latest/) projects.
+PyPSA-USA is an open-source energy system dataset of the United States energy system with continental US coverage.
+
+PyPSA-USA provides you with a toolkit to customize the **data** and **policy constraints** of energy system planning model with ease. Through configuration file you can control the spatial, temporal, and operational resolution of your energy system model with access to cleaned and prepared historical and forecasted data. This means, you can build a model of **ERCOT, WECC, or the Eastern interconnection**, where the transmission network is clustered to **N# of user defined nodes**, which can respect the boundaries of **balancing areas, states, or REeDs GIS Shapes**, using **historical EIA-930 demand data years 2018-2023** OR **NREL EFS forcasted demand [2030, 2040, 2050]**, with **historical daily/monthly fuel prices from ISOs/EIA [choice of year]**, AND imported capital cost projections from the **NREL Annual Technology Baseline**.
+
+You can create and export the power system data model to use in your own homebrewed optimization model OR use the built-in PyPSA-USA optimization features to layer on additional policy and operational constraints. For planning studies, we've integrated data on regional Renewable Portfolio Standards (RPS), emissions constraints, and other state-level policy constraints. We're actively building this model so more features are on the way!
+
+PyPSA-USA builds on and leverages the work of [PyPSA-EUR](https://pypsa-eur.readthedocs.io/en/latest/index.html) developed by TU Berlin. PyPSA-USA is actively developed by the [INES Research Group](https://ines.stanford.edu) at Stanford University and the [ΔE+ Research Group](https://www.sfu.ca/see/research/delta-e.html) at Simon Fraser University.
+
 
 % update to be a url
-![WECC-grid](_static/WECC.jpg)
+![PyPSA-USA_Network](_static/PyPSA-USA_network.png)
 
 <!-- ```{include} ../../README.md
 :relative-images:

diff --git a/docs/source/rules-simplify.md b/docs/source/rules-simplify.md
@@ -9,7 +9,7 @@
 (cluster)=
 ## Rule `cluster_network`
 ```{eval-rst}
-.. automodule:: cluster_network_eur
+.. automodule:: cluster_network
 ```
 
 (extra_components)=

diff --git a/workflow/Snakefile b/workflow/Snakefile
@@ -57,10 +57,11 @@ FIGURES_VALIDATE = [
     "production_deviation_bar.pdf",
     "val_bar_regional_emissions.pdf",
     "val_generator_data_panel.pdf",
-    "val_heatmap_curtailment.pdf",
+    # "val_heatmap_curtailment.pdf",
     "val_heatmap_capacity_factor.pdf",
     "val_box_region_lmps.pdf",
     "val_map_load_shedding.pdf",
+    "val_generator_stack.pdf",
 ]
 
 

diff --git a/workflow/config/config.default.yaml b/workflow/config/config.default.yaml
@@ -10,9 +10,9 @@ run:
 
 # docs :
 scenario:
-  interconnect: western #"usa|texas|western|eastern"
-  clusters: [100]
-  opts: [RCo2L-SAFER-RPS]
+  interconnect: ['usa'] #"usa|texas|western|eastern"
+  clusters: [150]
+  opts: [Ep-Co2L0.2]
   ll: [v1.0]
   scope: "total" # "urban", "rural", or "total"
   sector: "" # G
@@ -36,7 +36,7 @@ snapshots:
 
 # docs :
 electricity:
-  conventional_carriers: [nuclear, oil, OCGT, CCGT, coal, geothermal] # Choose the conventional plant types to include in network
+  conventional_carriers: [nuclear, oil, OCGT, CCGT, coal, geothermal, biomass] # Choose the conventional plant types to include in network
   renewable_carriers: [onwind, offwind, offwind_floating, solar, hydro] # Choose the renewable plant types to include in network
   voltage_simplified: 230 #Voltage level to simplify network to in rule "simplify network"
   co2limit: 1.4728e+9 # 0.8 * 1.841e+9
@@ -50,6 +50,8 @@ electricity:
   agg_p_nom_limits: 'config/policy_constraints/agg_p_nom_minmax.csv'
   portfolio_standards: 'config/policy_constraints/portfolio_standards.csv'
   SAFE_regional_reservemargins: 'config/policy_constraints/SAFE_regional_prm.csv'
+  transmission_interface_limits: 'config/policy_constraints/transmission_interface_limits.csv'
+
 
   operational_reserve:
     activate: false
@@ -184,6 +186,8 @@ lines:
   s_nom_max: .inf
   length_factor: 1.25
   under_construction: 'zero' # 'zero': set capacity to zero, 'remove': remove, 'keep': with full capacity
+  interface_transmission_limits: true
+
 
 # docs :
 links:
@@ -241,9 +245,9 @@ clustering:
     algorithm: kmeans # choose from: [hac, kmeans]
     feature: solar+onwind-time # only for hac. choose from: [solar+onwind-time, solar+onwind-cap, solar-time, solar-cap, solar+offwind-cap] etc.
   cluster_network:
-    algorithm: hac # choose from: [hac, kmeans]
+    algorithm: kmeans # choose from: [hac, kmeans]
     feature: solar+onwind-time
-    aggregation_zones: 'state' # [balancing_area, state]
+    aggregation_zones: 'reeds_zone' # [balancing_area, state, reeds_zone]
     exclude_carriers: []
     consider_efficiency_classes: false
   aggregation_strategies:
@@ -305,7 +309,7 @@ solving:
     track_iterations: false
     min_iterations: 4
     max_iterations: 6
-    transmission_losses: 0
+    transmission_losses: 2
     linearized_unit_commitment: true
     horizon: 8760
     assign_all_duals: true

diff --git a/workflow/config/policy_constraints/transmission_interface_limits.csv b/workflow/config/policy_constraints/transmission_interface_limits.csv
@@ -0,0 +1,8 @@
+interface,region_1,region_2,flow_12,flow_21
+p10|p11,"CISO-SCE, IID",CISO-SDGE,1998.433,4701.644
+p10|p13,"CISO-SCE, IID","CISO-VEA, NEVP",8751.271,8049.93
+p10|p27,"CISO-SCE, IID",WALC,472.535,811
+p10|p28,"CISO-SCE, IID",Arizona,4899.847,4424.758
+p6+p8 | p9,PACW,CISO-PGAE,5100,5100
+P9|P10,CISO-PGAE,CISO-SCE,9328.34,6133.425
+P9|P12,CISO-PGAE,NEVP,297.86,436.768
diff --git a/workflow/config/tests/config.test_simple.yaml b/workflow/config/tests/config.test_simple.yaml
@@ -10,8 +10,8 @@ run:
 
 # docs :
 scenario:
-  interconnect: [western] #"usa|texas|western|eastern"
-  clusters: [30]
+  interconnect: [eastern] #"usa|texas|western|eastern"
+  clusters: [20]
   opts: [Co2L1.0-6H-RCo2L-SAFER-RPS]
   ll: [v1.0]
   scope: "total" # "urban", "rural", or "total"
@@ -29,12 +29,12 @@ countries: [US]
 
 snapshots:
   start: "2019-01-01"
-  end: "2019-01-14"
+  end: "2019-01-07"
   inclusive: 'left'
 
 # docs :
 electricity:
-  conventional_carriers: [nuclear, oil, OCGT, CCGT, coal, geothermal] # Choose the conventional plant types to include in network
+  conventional_carriers: [nuclear, oil, OCGT, CCGT, coal, geothermal, biomass] # Choose the conventional plant types to include in network
   renewable_carriers: [onwind, offwind, offwind_floating, solar, hydro] # Choose the renewable plant types to include in network
   voltage_simplified: 230 #Voltage level to simplify network to in rule "simplify network"
   co2limit: 1.4728e+9 # 0.8 * 1.841e+9
@@ -48,6 +48,8 @@ electricity:
   agg_p_nom_limits: 'config/policy_constraints/agg_p_nom_minmax.csv'
   portfolio_standards: 'config/policy_constraints/portfolio_standards.csv'
   SAFE_regional_reservemargins: 'config/policy_constraints/SAFE_regional_prm.csv'
+  transmission_interface_limits: 'config/policy_constraints/transmission_interface_limits.csv'
+
 
   operational_reserve:
     activate: false
@@ -76,7 +78,7 @@ electricity:
 # docs :
 conventional:
   unit_commitment: false
-  dynamic_fuel_price: true
+  dynamic_fuel_price: false
 
 # docs :
 renewable:
@@ -169,7 +171,7 @@ renewable:
 
 # docs :
 lines:
-  types: # All temporary values, need to be updated
+  types: # remove, not used
     115.: "Al/St 240/40 2-bundle 220.0"
     138.: "Al/St 240/40 2-bundle 220.0"
     161.: "Al/St 240/40 2-bundle 220.0"
@@ -181,6 +183,7 @@ lines:
   s_nom_max: .inf
   length_factor: 1.25
   under_construction: 'zero' # 'zero': set capacity to zero, 'remove': remove, 'keep': with full capacity
+  interface_transmission_limits: true
 
 # docs :
 links:
@@ -239,9 +242,9 @@ clustering:
     algorithm: hac # choose from: [hac, kmeans]
     feature: solar+onwind-time # only for hac. choose from: [solar+onwind-time, solar+onwind-cap, solar-time, solar-cap, solar+offwind-cap] etc.
   cluster_network:
-    algorithm: hac
+    algorithm: kmeans
     feature: solar+onwind-time
-    aggregation_zones: 'state' # [balancing_area, state]
+    aggregation_zones: 'reeds_zone' # [balancing_area, state, reeds_zone]
     exclude_carriers: []
     consider_efficiency_classes: false
   aggregation_strategies: