[Cloud Security] CDR Data View versioning and migration logic

[opauloh]

Motivation

We changed the CSP Misconfiguration index pattern as part of this PR changes, however, we discovered that long running environments still reference the deprecated Data View (with stale index patterns like logs-*_latest_misconfigurations_cdr,logs-cloud_security_posture.findings_latest-default) while the latest index pattern is now security_solution-cloud_security_posture.misconfiguration_latest-*.

Because the CSP initialization don't migrate the Data View ID, existing spaces/environments keep the stale Data View indefinitely unless manually deleted. This affects:

• “no data” checks that rely on the Data View.
• Available fields in Data Table visualizations being limited to the old Data View.

Creating a new Data View identifier (security_solution_cdr_latest_misconfigurations_v2) will ensures all upgraded environments get the right index pattern and mapping configuration without requiring users to delete or manually fix the old Data View.

Removing the old index pattern will also lead to a better the experience to avoid any confusions about displaying 2 dataviews (old and new) for the users .

Definition of done

• Introduce a new constant (e.g., CDR_MISCONFIGURATIONS_DATA_VIEW_ID_PREFIX_V1) and update the CDR_MISCONFIGURATIONS_DATA_VIEW_ID_PREFIX constant with the v2 suffix (security_solution_cdr_latest_misconfigurations_v2).
• The new data view (security_solution_cdr_latest_misconfigurations_v2) is created if missing whenever the Findings page is accessed.
• The old dataview (security_solution_cdr_latest_misconfigurations) is removed if present (once during the CSP initialization).
• Add a lightweight guideline: whenever we have changes to the CSP data view, we create a new Data View ID.
• Backport to all applicable 8.x and 9.x branches that can still receive an update (8.18., 8.19., 9.0., 9.1.)

Related tasks/epics

• CDR Data View #188625 (comment)

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[maxcold]

When we discussed the migration logic previously one of the concerns was that users can add modifications to the data view as it's available for modification. Then, deleting the old data view means losing these modifications. Ofc the questions is if we really want to allow users to add modifications, but probably it's a must for Cross cluster search support. @kfirpeled do you remember something else regarding data view migration?

[opauloh]

Yes there's a trade-off between automating the Dataview modification vs not losing users modification.

An alternative perhaps is to always ensure the security_solution-*.misconfiguration_latest index pattern is included, that would allow for user customization of adding other indices with the limitation that wouldn't be possible to remove our default index pattern.

---

The Security Solution dataview formalizes this by allowing users to configure their custom index patterns in Kibana advanced settings, that got appended to the minimum required index pattern (.alerts-security.alerts-space-2).

---

I don't think without the telemetry of how many users actually customizes their Misconfiguration Dataview we can make an informed decision to put effort on adding the advanced setting, but we can follow the same strategy of ensuring the required index pattern is there, without removing user's customizations.

[niros1]

The old dataview (security_solution_cdr_latest_misconfigurations) is removed if present (once during the CSP initialization).
what does it means CSP Init? when a user access a page or when the integration installed?

regarding users modifications, can we deny user modifications on our dataview?

why BP only to 9.X?

[opauloh]

The old dataview (security_solution_cdr_latest_misconfigurations) is removed if present (once during the CSP initialization).

what does it means CSP Init? when a user access a page or when the integration installed?

The full CSP Initialization happens whenever Kibana is started/restarted and also is triggered whenever the integration is installed. It is not being triggered whenever the page is accessed.

We have, however, a method that is executed in every CSP server request, that intercepts an API call to create a DataView when it's missing.

The suggestion is that we keep creating the Dataview when missing during initialization on the API interception, but any removal should happen only in the initialization (Kibana restart or integration installation)

regarding users modifications, can we deny user modifications on our dataview?

That's a good call, at this moment, we don't have internal DataViews, so all Dataviews are allowed by default to be modified by the user. The good news is, I just found out we have however a PR coming soon targeting 9.2 that will introduce the concept of Managed Dataview, where allow us to define a dataview that users won't be able to modify it, but never modify the original managed dataview.

I think we should create an additional ticket so once Managed DataViews are supported, we explore how to switch to managed dataviews, so we don't need to worry about user modifying the dataviews.

why BP only to 9.X?

Good call, I thought the new index pattern was introduced with the namespace Epic on 9.0, but it was actually introduced here on 8.16, so we actually should backport the fix to 8.x and 9.x, I updated the description accordingly.

[maxcold]

@niros1 mind that I think this ticket is quite urgent as without the fix we might end up with incidents and SDHs from users whos Findings page just stop working at some point. No Data check are looking at the old indexes which don't get new data and if users have ILP policies to clean them up or clean up manually, the Findings page will be in "no data" state while there is data in the new index pattern

[niros1]

@maxcold thank for bring it up. @opauloh lets priorities it over Asset Inventory / Entity Store items, talk with me if needed.
I guess it better doing it before 9.2 rollout...

[maxcold]

btw it also worth doing for vulnerabilitie,s not only for misconfigs

[opauloh]

Update:

I just pushed the PR to migrate dataviews. I've also added the logic to Vulnerabilities following the same format so we have a consistent migration system.

I also created a follow up ticket to adopt the managed dataviews, the reasons why doing it in separate is that the migration ticket we are going to backport to 8.18, 8.19 and 9.1 which doesn't have managed dataview support. The managed dataview ticket we can backport to 9.2