Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce max table expiration days from 10,000 to 9,985 #833

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

sean-rose
Copy link
Contributor

Tables can have a maximum of 10,000 partitions, but when partitions expire they apparently continue to count toward the table's partition quota for up to 14 days afterwards (7 days of time travel and 7 days of fail-safe), per the BigQuery docs:

When a partition expires, BigQuery deletes that partition. The partition data is retained in accordance with time travel and fail-safe policies, and can be charged for, depending on your billing model. Until then, the partition counts for purposes of table quotas.

And I subtracted one additional day just to make sure we don't hit the table partition limit.

This goes along with mozilla/probe-scraper#840.


Checklist for reviewer:

For glean changes:

  • Update templates/include/glean/CHANGELOG.md

For modifications to schemas in restricted namespaces (see CODEOWNERS):

Tables can have a maximum of 10,000 partitions, but when partitions expire they apparently continue to count toward the table's partition quota for up to 14 days afterwards (7 days of time travel and 7 days of fail-safe), and I subtracted one additional day just to make sure we don't hit the table partition limit.
@auto-assign auto-assign bot requested a review from akkomar November 27, 2024 19:43
@dataops-ci-bot
Copy link

Integration report for "Reduce max table expiration days from 10,000 to 9,985."

bq_schema_86480854-d3cedf27.diff

No content detected.

compact_schema_86480854-d3cedf27.diff

No content detected.

@sean-rose
Copy link
Contributor Author

Note that there's some discussion in the associated probe-scraper PR about whether this is worth doing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants