Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the WRITE_TRUNCATE disposition to upload to BigQuery #742

Open
nacnudus opened this issue Feb 11, 2025 · 0 comments
Open

Use the WRITE_TRUNCATE disposition to upload to BigQuery #742

nacnudus opened this issue Feb 11, 2025 · 0 comments

Comments

@nacnudus
Copy link
Contributor

  1. So that the tables are instantly updated with new data, rather than being empty for a few minutes while new data is uploaded.
  2. To avoid false alarms when a table is detected as being empty, but is about to be repopulated.
  3. To mitigate failures to refresh the data, which is currently catastrophic, because not only is the data not refreshed, but the existing data is wiped.

See the diagram for the current update process, which does a TRUNCATE first, and then a bq load: #736 (comment)

Background to why it was developed in the current way:

Since #517, there is every reason to use those JSON schemas in the bq load command. Doing so would allow us to use replace=true (which will delete the existing schema), because the schema would immediately be replaced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant