Skip to content

Refresh FeedSearch view asynchronically #1172

@davidgamez

Description

@davidgamez

Describe the problem

The FeedSearch view is getting larger and more expensive to refresh. However, it's still a viable option for searching, as results are milliseconds away. This refresh operation is called for every single function that changes the data. As we are updating the datasets daily and the data is processed by at least four independent functions, the refresh function can potentially be called 1000 times in a day.

Proposed solution

  • Create a function to refresh the view. Make sure we still use the CONCURRENTLY command to avoid table locks
  • Replace the synchronous call to refresh the materialized view with a cloud task that executes the implemented function. Use the de-duplication mechanism to avoid multiple refreshes at the same time, see
  • Use the Bouncing mechanism(scheduling time) to delay the update for at least 5 or more minutes(TBD how much time after the event is triggered)
  • Consider scheduling a refresh task a few hours after batch datasets are triggered to make sure all data is updated daily in case a raise condition is reached

Alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions