-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Labels
Description
Describe the problem
The FeedSearch view is getting larger and more expensive to refresh. However, it's still a viable option for searching, as results are milliseconds away. This refresh operation is called for every single function that changes the data. As we are updating the datasets daily and the data is processed by at least four independent functions, the refresh function can potentially be called 1000 times in a day.
Proposed solution
- Create a function to refresh the view. Make sure we still use the
CONCURRENTLY
command to avoid table locks - Replace the synchronous call to refresh the materialized view with a cloud task that executes the implemented function. Use the de-duplication mechanism to avoid multiple refreshes at the same time, see
- Use the Bouncing mechanism(scheduling time) to delay the update for at least 5 or more minutes(TBD how much time after the event is triggered)
- Consider scheduling a refresh task a few hours after batch datasets are triggered to make sure all data is updated daily in case a raise condition is reached
Alternatives you've considered
No response
Additional context
No response