Skip to content

[1pt] PR: Add timestamp to post-proc log and fix centroid zero value threshold columns #1570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: dev
Choose a base branch
from

Conversation

ZahraGhahremani
Copy link
Contributor

@ZahraGhahremani ZahraGhahremani commented Jun 20, 2025

This PR fixes issue #1560 and #1544.

This covers a fix for adding a date/time stamp to a post proc log. When you first run pipeline, it does make the post proc log file with a date/time stamp. When you run just post processing a second time, you now have two post proc logs to compare which was the desired effect. However, if you run fim-pipeline again with the overwrite flag, the first post proc log file disappears. This is also desired. When you run fim-pipeline with overwrite, it removes the entire log folder when it starts.

For the centroid file, zero values will no longer show up in the threshold columns. Those records are dropped if threshold values are in place.

Changes

  • fim_post_processing.sh: Added timestamp to post-processing.log file.
  • src/heal_bridges_osm.py: Removed bridge points with threshold_hand = 0.

Testing

Deployment Plan (For developer use)

How does the changes affect the product?

  • Code only?
  • If applicable, has a deployment plan be created with the deployment person/team?
  • Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
  • If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
  • Require new pre-clip set?
  • Has new or updated python packages?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

  • Informative and human-readable title, using the format: [_pt] PR: <description>
  • Links are provided if this PR resolves an issue, or depends on another other PR
  • If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
  • Changes are limited to a single goal (no scope creep)
  • The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
  • pre-commit hooks were run locally
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • CHANGELOG updated with template version number, e.g. 4.x.x.x
  • Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

  • Update CHANGELOG with latest version number and merge date
  • Update the Citation.cff file to reflect the latest version number in the CHANGELOG
  • If applicable, update README with major alterations

@ZahraGhahremani ZahraGhahremani self-assigned this Jun 20, 2025
@ZahraGhahremani ZahraGhahremani changed the title Add timestamp tp post-processing log [1pt] PR: Add timestamp to post-processing log Jun 20, 2025
@ZahraGhahremani ZahraGhahremani marked this pull request as ready for review June 20, 2025 23:55
“RobHanna-NOAA” added 2 commits June 26, 2025 23:46
RobHanna-NOAA
RobHanna-NOAA previously approved these changes Jun 27, 2025
Copy link
Contributor

@RobHanna-NOAA RobHanna-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

I did a couple of test on the post proc log and they were fine. Note: The file now has a date/time in the file name which is perfect. When you re-run just post processing, you now have the two post proc files so can compare info, as hoped. One byproduct, and it makes perfect sense, is that if you fully re-run that pipeline with the overwrite flag, it removes all log files/folders to the post proc file disappears. This too is perfectly fine.

For the centroid fix, it too is good. In the issue card, we had a HUC and some bridge records in error. I ran pipeline on that 01010008 HUC, then check the centroids file. There are no longer any zeros in any of the threshold columns as hoped.

@RobHanna-NOAA RobHanna-NOAA changed the title [1pt] PR: Add timestamp to post-processing log [1pt] PR: Add timestamp to post-proc log and fix centroid zero value threshold columns Jun 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[5pt] Post Processing log and duration [5pt] Empty threshold data columns in bridge data
3 participants