Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SmartSurvey data to BigQuery #748

Open
nacnudus opened this issue Feb 14, 2025 · 0 comments
Open

Add SmartSurvey data to BigQuery #748

nacnudus opened this issue Feb 14, 2025 · 0 comments

Comments

@nacnudus
Copy link
Contributor

Currently, https://github.com/alphagov/govuk-feedback-processing/blob/main/src/services/smart_survey_service.py requests survey responses from the SmartSurvey API endpoint, and is scheduled to run daily. It is implemented as a Cloud Run Function, and handles its own retries and backoffs.

Options:

  1. Scheduler -> Pub/Sub -> Function. Not so easy to handle backoff when backfilling several dates.
  2. Scheduler -> Pub/Sub -> Workflow. This might get expensive, because the Workflow would charge per API request.
  3. Scheduler -> Pub/Sub -> Workflow (to generate tasks) -> Tasks -> Function (triggered by http). Retries/backoff could be implemented in the task, rather than the function. Would the Workflow charge per task?
  4. Scheduler -> Pub/Sub -> Function (to generate tasks) -> Tasks -> Function (triggered by https). Retries/backoff could be implemented in the task, rather than the function. Cheaper than Workflows.
  5. Use Cloud Composer, which apparently has more sensible pricing now (isn't always on?)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant