feat(slo): Delete stale summary data

[kdelemme]

Fix #210049

Summary

This PR introduce a new route for purging stale summary documents.
The API is flexible to allow usage as bulk action when a list of SLO ids is specified or as a global action when no list is provided. We default to the stale threshold configured in the slo settings when not specified, and we enforce a greater value when specified, except when force is used.

The modal loads with the default stale threshold settings applied, and the user must confirm the override when entering a lower value.

The status API returns some information about the task, like the number of deleted documents, its completion state and some timing information like how long it took and when it started. It is currently not in used...

I've also cleanup the purge rollup data actions to remove some duplication. Also added some unit and integration tests for the purge instances flow.

[elasticmachine]

Pinging @elastic/actionable-obs-team (Team:actionable-obs)

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[mgiota]

If there are millions of documents to delete, slices: auto could overwhelm ES. I was looking into throttling and slicing in the official documentation . Have you considered using these optimization techniques?

[kdelemme]

If there are millions of documents to delete, slices: auto could overwhelm ES

I don't think so. where do you get this from?
slices auto will use the best settings based on the number of shards available. We should use this as trying to set it ourselves when we don't know what is the customer shard settings will be more complex.

[mgiota]

slices: auto caught my attention. I searched a bit in the official documentation to check what it means and what it does and here's what I got, Delete by query supports sliced scroll to parallelize the delete process. This can improve efficiency and provide a convenient way to break the request down into smaller parts.. I searched for scroll_size in the codebase and I found a few references, for example here used together with max_docs.

I am fine using it as is until we notice any performance issues. Just wanted to investigate if there are more performant ways for the deleteByQuery.

[dmlemeshko]

x-pack/solutions/observability/test/api_integration_deployment_agnostic/services/slo_api.ts changes LGTM

[mgiota]

I tried to delete one SLO and here's what I got. When I checked the Ignore purge policy restrictions checkbox or I selected 30 days, I didn't get any error.

After I successfully purged the data (I selected 30 days), I expected the SLO to not appear as stale anymore, but it appears as stale. Is it expected? Can we have a tooltip that explains what purge does behind the scenes?

[kdelemme]

@mgiota You've tested the "purge rollup data" action, not the "delete stale instance". This one is in the "Actions" menu (big blue button on top right) in the header of the SLO Management page. I can't take a screenshot right now since i'm in a middle of something in another branch.

[mgiota]

@kdelemme Yep thanks! I figured it out, I clicked the Purge stale instances actions and it worked as expected. The stale SLO didn't appear as stale anymore in the SLO overview page, but it is still present in the SLO Management page. Is it expected?

[kdelemme]

@mgiota

The stale SLO didn't appear as stale anymore in the SLO overview page, but it is still present in the SLO Management page. Is it expected?

Yes since the SLO Management page shows only the SLO definitions, regardless of the number of instances and their state (stale or not). When we purge stale instances, we delete all SLO Instances that have not been updated for X time, but their SLO Definition stays.