Metadata migration for EIA923 #4422

cmgosnell · 2025-07-15T15:45:32Z

Overview

Closes #4400

What problem does this address?

What did you change?

Documentation

Make sure to update relevant aspects of the documentation:

Update the release notes: reference the PR and related issues.
Update relevant Data Source jinja templates (see docs/data_sources/templates).
Update relevant table or source description metadata (see src/metadata).
Review and update any other aspects of the documentation that might be affected by this PR.

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

If updating analyses or data processing functions: make sure to update row count expectations in dbt tests.
Run make pytest-coverage locally to ensure that the merge queue will accept your PR.
Review the PR yourself and call out any questions or issues you have.
For minor ETL changes or data additions, once make pytest-coverage passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and run relevant data validation tests using pytest and --live-dbs.
For bigger ETL or data changes run the full ETL locally and then run the data validations using make pytest-validate.
Alternatively, run the build-deploy-pudl GitHub Action manually.

Lightly-revise table name data stubs for use in templating. Includes minimal seed data for testing. Includes new 'label' attribute for DataSource metadata.

…ased

…orner cases

…on metadata in type checks.

…adata

…migrated tables

…play if empty

…e location

(we're not adding any new libraries so this should be fine?)

cmgosnell · 2025-07-15T16:13:58Z

docs/release_notes.rst

+Documentation
+^^^^^^^^^^^^^
+* Migrated table description metadata into new format for EIA 923. See :issue:`4400`


we should probably add a note in here before the dataset specific migrations about the whole migration.

Agreed! Also might be a good idea to highlight the table name changes specifically.

krivard · 2025-07-15T19:00:15Z

Hmmm sphinx can't find your references for _core_eia923__monthly_cooling_system_information; are the docs still using the old name?

… front wattt

cmgosnell · 2025-07-16T15:19:54Z

Hmmm sphinx can't find your references for _core_eia923__monthly_cooling_system_information; are the docs still using the old name?

I've learned that in order to :ref: a table name that starts with an underscore you need to put an i in front of it?!?

cmgosnell · 2025-07-16T15:21:00Z

docs/dev/build_docs.rst

+
+    `make docs-build` will build and the delete all of the rst files via
+    ``cleanup_rsts`` and ``cleanup_csv_dir`` in ``docs/conf.py``. If you want to
+    preserve them for a one-off build, you can comment out that step in
+    ``docs/conf.py``.


i added this guy to hopefully help debug warnings like this

We might also want to point folks at the autoapi_keep_files variable in conf.py for debugging problems in any of the docstrings, since that's separate from our custom cleanup functions.

I think the note above "If you create a new module, the corresponding documentation file will also need to be checked in to version control." is incorrect, and dates from a time when we had all these generated files checked into git.

cmgosnell · 2025-07-16T15:21:27Z

docs/release_notes.rst

-  :ref:`i_core_eia923__fgd_operation_maintenance` and
+  :ref:`i_core_eia923__yearly_fgd_operation_maintenance` and


this is how i learned it needs to start with an i_core..

I have never heard of this? But I trust?

aesharpe

Not done! Just publishing this bit because I need to switch computers.

aesharpe · 2025-07-17T18:00:55Z

docs/release_notes.rst

+Documentation
+^^^^^^^^^^^^^
+* Migrated table description metadata into new format for EIA 923. See :issue:`4400`


Agreed! Also might be a good idea to highlight the table name changes specifically.

aesharpe · 2025-07-17T18:01:27Z

docs/release_notes.rst

-  :ref:`i_core_eia923__fgd_operation_maintenance` and
+  :ref:`i_core_eia923__yearly_fgd_operation_maintenance` and


I have never heard of this? But I trust?

aesharpe · 2025-07-17T18:09:28Z

src/pudl/metadata/resources/allocate_gen_fuel.py

-                f"{KNOWN_DRAWBACKS_DESCRIPTION}"
-            ),
+            "description": {
+                "additional_summary_text": "of Downscaled Net Generation and Fuel Consumption.",


I don't think we need these to be all uppercased bc it's a description, not a title (same goes for all the rest below).

Also, I'm not sure what "downscaled" means in this context. It might make more sense to just use the first sentence of the old description: "estimated net generation and fuel consumption associated with each combination of generator, energy source, and prime mover."

We might also want to have "estimated" in here somewhere

aesharpe · 2025-07-17T18:10:58Z

src/pudl/metadata/resources/allocate_gen_fuel.py

-            ),
+            "description": {
+                "additional_summary_text": "of Downscaled Net Generation and Fuel Consumption.",
+                "additional_source_text": "(Schedule 3)",


I don't know that this needs to be in parenthesis here. We should decide on a standard.

With the parenthesis, this displays as

EIA Form 923 -- Power Plant Operations Report (Schedule 3)

Which I do like best out of the alternatives,

EIA Form 923 -- Power Plant Operations Report Schedule 3

EIA Form 923 -- Power Plant Operations Report , Schedule 3

(with the space between Report and the comma, unfortunately)

This is a good point. There are some cases where there is more than just a schedule to add to the source, but I think we can probably put it all in parenthesis.

aesharpe · 2025-07-17T18:14:54Z

src/pudl/metadata/resources/allocate_gen_fuel.py

+                    "reported in the EIA-923 generation fuel are allocated to individual "
+                    "generators. Then, these allocations are aggregated to unique generator, "
+                    f"prime mover, and energy source code combinations. "
+                    f"{KNOWN_DRAWBACKS_DESCRIPTION}"


Should any of these known drawbacks be usage warnings?

aesharpe · 2025-07-17T18:30:34Z

src/pudl/metadata/resources/allocate_gen_fuel.py

+                "additional_details_text": (
+                    f"{freq.title()} estimated net generation and fuel consumption by generator. "
+                    "Based on allocating net electricity generation and fuel consumption reported "
+                    "in the EIA-923 generation and generation_fuel tables to individual generators."


We should probably have some way of indicating whether a table has been altered significantly by us. For instance, the estimated net generation is calculated by us, not EIA. We could mention this in additional_source_text or in additional_details_text maybe? Or maybe elaborate on the estimated_values usage warning detail text?

aesharpe

Okay Part 2: I didn't go into too much detail for the sake of this being a migration vs. editing pass, but one high level thought I have is this:

In my opinion, the additionally_summary_text should read more like a topic sentence and less like a title. Some of the ones you have in here feel like they need just a little bit more information in them--I think one thing you could do is take the first sentence of the additional_details_text and use that instead, or something similar. What do you think?

aesharpe · 2025-07-18T06:33:25Z

src/pudl/metadata/resources/allocate_gen_fuel.py

-            ),
+            "description": {
+                "additional_summary_text": "of Downscaled Net Generation and Fuel Consumption.",
+                "additional_source_text": "(Schedule 3)",


This is a good point. There are some cases where there is more than just a schedule to add to the source, but I think we can probably put it all in parenthesis.

aesharpe · 2025-07-18T06:38:58Z

src/pudl/metadata/resources/allocate_gen_fuel.py

-                f"{KNOWN_DRAWBACKS_DESCRIPTION}"
-            ),
+            "description": {
+                "additional_summary_text": "of Downscaled Net Generation and Fuel Consumption.",


We might also want to have "estimated" in here somewhere

aesharpe · 2025-07-18T06:40:26Z

src/pudl/metadata/resources/eia923.py

-
-Reports annual information about flue gas desulfurization systems at generation facilities,
+    "_core_eia923__yearly_fgd_operation_maintenance": {
+        "additional_summary_text": "FGD Operation & Maintenance.",


I feel like we should mostly avoid acronyms when possible. I can't even remember what this means.... 😅

aesharpe · 2025-07-18T06:41:36Z

src/pudl/metadata/resources/eia923.py

+        "additional_source_text": "(Schedule 8C)",
+        "usage_warnings": ["irregular_years"],
+        "additional_details_text": (
+            """Reports annual information about flue gas desulfurization (FGD) systems at generation facilities,


This additional_details_text bit feels redundant with the additional_summary_text section.

aesharpe · 2025-07-18T06:52:22Z

src/pudl/metadata/resources/eia923.py

 Note that a small number of respondents only report annual fuel consumption and net
 generation, and all of it is reported in December."""


This feels redundant with the usage warning. If we aren't going to provide any more insight as to why this is happening, I think we can remove this from this section and just keep it in the usage warnings for now. That goes for all the tables where this comes up too.

aesharpe · 2025-07-18T06:56:07Z

src/pudl/metadata/resources/eia923.py

-
-This table is produced during the transformation of fuel delivery data, in order to
+        "description": {
+            "additional_summary_text": "coal mines reporting deliveries in the Fuel Receipts and Costs.",


"In the fuel receipts and costs" feels weird, did you mean to add the word table after this?

aesharpe · 2025-07-18T06:57:27Z

src/pudl/metadata/resources/eia923.py

+        # + "\n\nThis table exists for naming consistency. While it is technically "
+        # "aggregated by month, it ends up being identical to the "
+        # "``out_eia923__generation`` table from which it is derived.",


Why did you comment this out vs. remove it?

i flagged this in the spreadsheet. i think we should delete these non-aggregated tables. its harder now to append text to portions of the shared dictionary so i commented it out instead of trying to incorporate it when imo we should no be publishing this table. But i will convert this to a TODO instead of just comment it out.

cmgosnell and others added 30 commits April 1, 2025 16:36

wip first very draft of table name descriptor

665f5e0

slight clean up of wip table name metadata

768aba8

Incorporate metadata checker and table name data extractor.

de3c249

Lightly-revise table name data stubs for use in templating. Includes minimal seed data for testing. Includes new 'label' attribute for DataSource metadata.

Resource metadata: cosmetic adjustments for preview wizard

af30f63

Adjust templates for wizard

8569c2e

split shared text from prompts

a06081e

Merge remote-tracking branch 'origin/main' into metadata-table-name-b…

8f3c0b3

…ased

Remove outdated 861 variance

06295bb

Add table description rendering step

e63301e

Merge branch 'main' into metadata-table-name-based

712f432

Add usage warnings

140f912

Add command line access to default table description templates; fix c…

75ef045

…orner cases

Add table description build phase; support structured table descripti…

2bb7a78

…on metadata in type checks.

Fully-decomposed metadata build

6f5d8ee

Update core_eia923__monthly_boiler_fuel with new decomposed table met…

94eb8c6

…adata

Adapt new metadata build for previews

9282bf8

Add support for quarterly; make descriptions render acceptably for un…

b051359

…migrated tables

Merge branch 'main' into metadata-table-name-based

14b99ce

Revert sample migrations

6ab7d7e

Handle usage warnings defaults at build time; skip usage warnings dis…

76156a1

…play if empty

Permit the sphinx build to override default table description templat…

9dbda65

…e location

Try moving the table description template into the primary source tree

7abc19e

Merge branch 'main' into metadata-table-name-based

12405a8

Drop docs_dir args that are no longer needed

15daaa1

Add docstrings to description rendering code

4f41679

Refactor with more sensible layout, naming, and documentation

b4e3333

Add missing files

34ceba6

replace lockfiles with copies from main.

090920d

(we're not adding any new libraries so this should be fine?)

Fix typos :D

4f09a33

Merge branch 'main' into metadata-table-name-based

6c2ab20

Kathryn Mazaitis and others added 6 commits July 9, 2025 13:49

Fix rst in docstrings

7304150

Permit nonstandard table types if you have additional_summary_text set

66ee0b4

typo

601765d

Fix docs

dcd3fce

Merge branch 'main' into metadata-table-name-based

51e7d4f

Metadata migration for EIA923

2192f8c

github-project-automation bot added this to Catalyst Megaproject Jul 15, 2025

github-project-automation bot moved this to New in Catalyst Megaproject Jul 15, 2025

cmgosnell self-assigned this Jul 15, 2025

cmgosnell linked an issue Jul 15, 2025 that may be closed by this pull request

Migrate resource description metadata for eia923 (26 tables, 3011 words) #4400

Open

cmgosnell moved this from New to In progress in Catalyst Megaproject Jul 15, 2025

cmgosnell added eia923 Anything having to do with EIA Form 923 docs Documentation for users and contributors. labels Jul 15, 2025

cmgosnell added 2 commits July 15, 2025 11:47

Merge branch 'metadata-table-name-based' into metadata-eia923

a1d38b5

add release notes

c4b7d8e

cmgosnell commented Jul 15, 2025

View reviewed changes

Base automatically changed from metadata-table-name-based to main July 15, 2025 20:10

cmgosnell added 5 commits July 16, 2025 09:28

Merge branch 'main' into metadata-eia923

e0d7cba

fix alembic migration

db74093

ah add back in the lock files where did they go how did they go away?

0bb4744

fix release notes refs... apparently the _core tables need a i in the…

f5e3f0a

… front wattt

remove the accedential line break in no-pk explaination

a67391b

cmgosnell commented Jul 16, 2025

View reviewed changes

cmgosnell moved this from In progress to In review in Catalyst Megaproject Jul 17, 2025

Merge branch 'main' into metadata-eia923

da87416

aesharpe reviewed Jul 17, 2025

View reviewed changes

aesharpe reviewed Jul 18, 2025

View reviewed changes

		:ref:`i_core_eia923__fgd_operation_maintenance` and
		:ref:`i_core_eia923__yearly_fgd_operation_maintenance` and

		Note that a small number of respondents only report annual fuel consumption and net
		generation, and all of it is reported in December."""

Uh oh!

Metadata migration for EIA923 #4422

Are you sure you want to change the base?

Metadata migration for EIA923 #4422

Conversation

cmgosnell commented Jul 15, 2025

Overview

What problem does this address?

What did you change?

Documentation

Testing

To-do list

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krivard commented Jul 15, 2025

Uh oh!

cmgosnell commented Jul 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aesharpe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aesharpe Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aesharpe left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aesharpe Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aesharpe Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aesharpe Jul 17, 2025 •

edited

Loading

aesharpe left a comment •

edited

Loading

aesharpe Jul 18, 2025 •

edited

Loading

aesharpe Jul 18, 2025 •

edited

Loading