-
-
Notifications
You must be signed in to change notification settings - Fork 123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Transform PHMSA company data #4005
Open
e-belfer
wants to merge
70
commits into
main
Choose a base branch
from
phmsa-company-transform
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
70 commits
Select commit
Hold shift + click to select a range
d41717d
Update phmsagas DOI and start transformation
ba27b7e
Merge remote-tracking branch 'upstream/main' into issue-3770-transfor…
b40071e
Starting data transformation
90fd277
Update notebook and change etl_fast phmsagas years
c5406b1
Add 2023 package data columns for new phmsagas run
9fc3f45
Added documentation
c3e2c67
Add troubleshooting to index
1bfb71d
Update troubleshooting
7760e9a
Add helpers
e23bd61
Temp add change
b5c7acd
Update column mappings
035712e
Merge remote-tracking branch 'upstream/main' into issue-3770-transfor…
5d7d00f
Merge remote-tracking branch 'upstream/main' into issue-3770-transfor…
9518e83
Update notebook and add draft transform script
418cd55
Merge remote-tracking branch 'upstream/main' into issue-3770-transfor…
66b63b2
Remove old files and cleanup helpers
cfc62d4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 3225680
Resolved merge conflicts
f276746
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] ac67b39
Resolve merge conflicts
57209e9
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 6b2747e
Remove list of columns
d6bb6ea
Remove '.0' logic
1769487
Updates in response to comments
405cc1c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 78cf151
Clean up documentation and logic
0c69a84
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 044dcd5
Use bulk series str ops
d8be474
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] c5aa116
Reorder transformations
aa3ac9c
Remove .0 substring from phone numbers
e3ec14e
Remove temp dev logic
e84f348
Merge remote-tracking branch 'upstream/main' into issue-3770-transfor…
7f94d13
Cleanup notebook
f1ba3dc
Merge remote-tracking branch 'upstream/main' into issue-3770-transfor…
cb6c767
Make updates per PR feedback
05cc8ce
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 1d3db43
Merge remote-tracking branch 'upstream/main' into issue-3770-transfor…
seeess1 2979526
Merge remote-tracking branch 'upstream/main' into issue-3770-transfor…
seeess1 9f57d77
Cleanup method description
seeess1 94d4d5d
Merge branch 'issue-3770-transform-phmsagas-data' of https://github.c…
seeess1 6f504e5
Update inits, classes, and fields
seeess1 759f1e6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] f1fa7a3
Deduplication and test updates
seeess1 2ae5dfb
Merge branch 'issue-3770-transform-phmsagas-data' of https://github.c…
seeess1 d5f5ffe
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 62a821c
Merge branch 'main' into phmsa-company-transform
e-belfer 875da71
Extract new PHMSA data, fix state enum, add alembic migration
e-belfer 06ae880
Merge branch 'phmsa-company-transform' of https://github.com/catalyst…
e-belfer 9ac7858
Address ruff failures and unit test failure, move analyzing code to n…
e-belfer 520c287
Get asset checks to run
e-belfer 95e9a5e
Merge branch 'main' into phmsa-company-transform
e-belfer 50ca79a
Merge branch 'main' into phmsa-company-transform
e-belfer 3cfd6fc
Update release notes
e-belfer 6bda226
Merge branch 'main' into phmsa-company-transform
e-belfer 669520e
Fix release notes, rebase migration, update resource metadata
e-belfer 7e66272
Add state encoding to new tables
e-belfer afee877
Add territories to enums
e-belfer c7cbbae
Update blocking test on fast ETL
e-belfer 98578fc
Remove state encoding
e-belfer 6d0d510
Merge branch 'main' into phmsa-company-transform
e-belfer c0567c5
Decapitalize notes, fix report_date
e-belfer 55181bd
Merge branch 'phmsa-company-transform' of https://github.com/catalyst…
e-belfer 5aa6d1a
Merge branch 'main' into phmsa-company-transform
e-belfer eb8a62b
Merge branch 'main' into phmsa-company-transform
zaneselvans 2bdaa54
Merge branch 'main' into phmsa-company-transform
zaneselvans e9c27fe
Add dbt models and row counts for phmsa gas distributon operators
zaneselvans 7a2e787
Add tests for phone number standardization function
zaneselvans eda9524
Add richer field descriptions and some minor transform tweaks.
zaneselvans 5b61a81
Merge branch 'main' into phmsa-company-transform
zaneselvans File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
44 changes: 44 additions & 0 deletions
44
dbt/models/phmsagas/core_phmsagas__yearly_distribution_operators/schema.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
version: 2 | ||
sources: | ||
- name: pudl | ||
tables: | ||
- name: core_phmsagas__yearly_distribution_operators | ||
data_tests: | ||
- check_row_counts_per_partition: | ||
table_name: core_phmsagas__yearly_distribution_operators | ||
partition_column: report_year | ||
columns: | ||
- name: report_date | ||
- name: report_number | ||
- name: report_submission_type | ||
- name: report_year | ||
- name: operator_id_phmsa | ||
- name: operator_name_phmsa | ||
- name: office_address_street | ||
- name: office_address_city | ||
- name: office_address_state | ||
- name: office_address_zip | ||
- name: office_address_county | ||
- name: headquarters_address_street | ||
- name: headquarters_address_city | ||
- name: headquarters_address_state | ||
- name: headquarters_address_zip | ||
- name: headquarters_address_county | ||
- name: excavation_damage_excavation_practices | ||
- name: excavation_damage_locating_practices | ||
- name: excavation_damage_one_call_notification | ||
- name: excavation_damage_other | ||
- name: excavation_damage_total | ||
- name: excavation_tickets | ||
- name: services_efv_in_system | ||
- name: services_efv_installed | ||
- name: services_shutoff_valve_in_system | ||
- name: services_shutoff_valve_installed | ||
- name: federal_land_leaks_repaired_or_scheduled | ||
- name: percent_unaccounted_for_gas | ||
- name: additional_information | ||
- name: preparer_email | ||
- name: preparer_fax | ||
- name: preparer_name | ||
- name: preparer_phone | ||
- name: preparer_title |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3693,3 +3693,34 @@ _out_eia__yearly_heat_rate_by_unit,2021,1954 | |
_out_eia__yearly_heat_rate_by_unit,2022,1886 | ||
_out_eia__yearly_heat_rate_by_unit,2023,1865 | ||
_out_ferc1__yearly_plants_utilities,,7887 | ||
core_phmsagas__yearly_distribution_operators,1990,1504 | ||
core_phmsagas__yearly_distribution_operators,1991,1569 | ||
core_phmsagas__yearly_distribution_operators,1992,1545 | ||
core_phmsagas__yearly_distribution_operators,1993,1570 | ||
core_phmsagas__yearly_distribution_operators,1998,1464 | ||
core_phmsagas__yearly_distribution_operators,1999,1461 | ||
core_phmsagas__yearly_distribution_operators,2000,1446 | ||
core_phmsagas__yearly_distribution_operators,2001,1440 | ||
core_phmsagas__yearly_distribution_operators,2002,1423 | ||
core_phmsagas__yearly_distribution_operators,2003,1428 | ||
core_phmsagas__yearly_distribution_operators,2004,1523 | ||
core_phmsagas__yearly_distribution_operators,2005,1522 | ||
core_phmsagas__yearly_distribution_operators,2006,1518 | ||
core_phmsagas__yearly_distribution_operators,2007,1502 | ||
core_phmsagas__yearly_distribution_operators,2008,1476 | ||
core_phmsagas__yearly_distribution_operators,2009,1449 | ||
core_phmsagas__yearly_distribution_operators,2010,1437 | ||
core_phmsagas__yearly_distribution_operators,2011,1462 | ||
core_phmsagas__yearly_distribution_operators,2012,1477 | ||
core_phmsagas__yearly_distribution_operators,2013,1492 | ||
core_phmsagas__yearly_distribution_operators,2014,1494 | ||
core_phmsagas__yearly_distribution_operators,2015,1491 | ||
core_phmsagas__yearly_distribution_operators,2016,1487 | ||
core_phmsagas__yearly_distribution_operators,2017,1498 | ||
core_phmsagas__yearly_distribution_operators,2018,1489 | ||
core_phmsagas__yearly_distribution_operators,2019,1478 | ||
core_phmsagas__yearly_distribution_operators,2020,1458 | ||
core_phmsagas__yearly_distribution_operators,2021,1443 | ||
core_phmsagas__yearly_distribution_operators,2022,1447 | ||
core_phmsagas__yearly_distribution_operators,2023,1438 | ||
core_phmsagas__yearly_distribution_operators,3900,6211 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Man if we still have natural gas distribution systems in the year 3900 something is very wrong. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added using the
devtoosl/dbt_helper.py
script.