GFT tweaks for 24v3 draft 1 review

[fvankrieken]

Tweaks for #1004 (comment)

rebuild here

Next draft is in QA, but I think it makes sense to get this merged in the meantime and follow-up as needed.

Changes are

1. making sure that when we export both points and lots source dataset outputs that the lots only contain rows where the geom is not null
2. make sure that for open spaces/shadow polys, we're exporting both poly raw geoms and points that were joined to lots.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.60%. Comparing base (8ea544b) to head (1ab6bc9).
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1196   +/-   ##
=======================================
  Coverage   68.60%   68.60%           
=======================================
  Files         109      109           
  Lines        5771     5771           
  Branches      634      634           
=======================================
  Hits         3959     3959           
  Misses       1685     1685           
  Partials      127      127           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[alexrichey]

Thoughts on just passing along a --product-label CLI option?

[fvankrieken]

Something like that would make sense, though then we'd need to have this sort of logic in the repeat gha - still need to refer to both formattings in our code at the moment

[alexrichey]

yeah, I guess I'm angling for a way to avoid hardcoding these product overrides, which feels like it's going in the wrong direction. I'm fine if we just want to punt though until we have a strong feeling about where's best for this type of thing.

[fvankrieken]

Cool. Yeah I feel like a CLI option isn't really great either, in that it still adds some unneeded complexity. But it's 100% a lesser evil than what's in this PR

Maybe we should just have some sort of minimal ProductPaths object? And then dcpy can rely on pyproject.toml (or .dcpy file, etc) that defines the path to product folder, DO folder name, db build name, etc. OR for each product, we have a standard spot to look, (i.e. the product folder) and store the additional metadata there. This seems like this could belong in a recipe actually. Though then you would need the recipe to run an upload step.

Long way of saying I think we should punt the issue. I can make an issue for this

[alexrichey]

@fvankrieken Do we need to do this for all lot types? e.g. air_quality/source__industrial_lots? or is it just that these were problematic somehow?

[fvankrieken]

@fvankrieken Do we need to do this for all lot types? e.g. air_quality/source__industrial_lots? or is it just that these were problematic somehow?

We need to do this for every dataset that starts out as points and attempts to join to lots (so final geom column contains both points and polys). Industrial lots has pluto as a source, so every row is a lot from the start

[alexrichey]

Open question about hard-coding overrides. Seems less than ideal, but I'm fine if we to address that later.