Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GFT tweaks for 24v3 draft 1 review #1196

Merged
merged 3 commits into from
Oct 22, 2024
Merged

GFT tweaks for 24v3 draft 1 review #1196

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eeae751
only export non-null lot geoms
fvankrieken Oct 15, 2024
1dec604
hack for repeating gft builds
fvankrieken Oct 15, 2024
1ab6bc9
include both raw polygons and lots for shadow open spaces
fvankrieken Oct 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions dcpy/lifecycle/builds/plan.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -341,30 +341,32 @@ def _cli_wrapper_repeat_recipe(
),
):
product_key: publishing.BuildKey | publishing.DraftKey | publishing.PublishKey

product_label = (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on just passing along a --product-label CLI option?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like that would make sense, though then we'd need to have this sort of logic in the repeat gha - still need to refer to both formattings in our code at the moment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I guess I'm angling for a way to avoid hardcoding these product overrides, which feels like it's going in the wrong direction. I'm fine if we just want to punt though until we have a strong feeling about where's best for this type of thing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. Yeah I feel like a CLI option isn't really great either, in that it still adds some unneeded complexity. But it's 100% a lesser evil than what's in this PR

Maybe we should just have some sort of minimal ProductPaths object? And then dcpy can rely on pyproject.toml (or .dcpy file, etc) that defines the path to product folder, DO folder name, db build name, etc. OR for each product, we have a standard spot to look, (i.e. the product folder) and store the additional metadata there. This seems like this could belong in a recipe actually. Though then you would need the recipe to run an upload step.

Long way of saying I think we should punt the issue. I can make an issue for this

"db-green-fast-track" if product == "green_fast_track" else f"db-{product}"
)
match product_type:
case "build":
product_key = publishing.BuildKey(
product=f"db-{product}", build=version_or_build
product=product_label, build=version_or_build
)
case "draft":
if draft_revision_number is None:
raise ValueError(
"For repeating builds of 'draft' type, need to provide draft revision number"
)
draft_revision = publishing.get_draft_revision_label(
product=f"db-{product}",
product=product_label,
version=version_or_build,
revision_num=draft_revision_number,
)
product_key = publishing.DraftKey(
product=f"db-{product}",
product=product_label,
version=version_or_build,
revision=draft_revision,
)
case "publish":
product_key = publishing.PublishKey(
product=f"db-{product}", version=version_or_build
product=product_label, version=version_or_build
)
case _:
raise ValueError(
Expand Down
1 change: 1 addition & 0 deletions products/green_fast_track/models/product/source/air_quality/source__cats_permit_lots.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ SELECT
variable_id,
lot_geom
FROM {{ ref("int_spatial__cats_permits") }}
WHERE lot_geom IS NOT NULL
1 change: 1 addition & 0 deletions products/green_fast_track/models/product/source/air_quality/source__state_facility_lots.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ SELECT
variable_id,
lot_geom
FROM {{ ref("int_spatial__state_facility") }}
WHERE lot_geom IS NOT NULL
1 change: 1 addition & 0 deletions products/green_fast_track/models/product/source/air_quality/source__title_v_permit_lots.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ SELECT
variable_id,
lot_geom
FROM {{ ref("int_spatial__title_v_permit") }}
WHERE lot_geom IS NOT NULL
1 change: 1 addition & 0 deletions ...st_track/models/product/source/historic_resources/source__historic_resources_adj_lots.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ SELECT
variable_id,
lot_geom
FROM {{ ref("int_spatial__historic_resources_adj") }}
WHERE lot_geom IS NOT NULL
1 change: 1 addition & 0 deletions ...n_fast_track/models/product/source/historic_resources/source__historic_resources_lots.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ SELECT
variable_id,
lot_geom
FROM {{ ref("int_spatial__historic_resources") }}
WHERE lot_geom IS NOT NULL
11 changes: 9 additions & 2 deletions products/green_fast_track/models/product/source/shadow/source__shadow_open_spaces_polys.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
SELECT
variable_type,
variable_id,
raw_geom
raw_geom AS geom
FROM {{ ref("int_spatial__shadow_open_spaces") }}
WHERE ST_GEOMETRYTYPE(raw_geom) = 'ST_MultiPolygon'
WHERE ST_GEOMETRYTYPE(raw_geom) = 'ST_MultiPolygon' AND lot_geom IS NULL
UNION ALL
SELECT
variable_type,
variable_id,
lot_geom AS geom
FROM {{ ref("int_spatial__shadow_open_spaces") }}
WHERE lot_geom IS NOT NULL
Loading