Validate mapping ECS compliance in system test

[flash1293]

Integrations rely on ecs@mappings to make sure ECS fields are mapped correctly.
However, it's possible fields will get with the wrong type in case they are sent with the wrong type in the incoming JSON document (e.g. sending a boolean value as a string "true" will cause the field to be mapped as keyword even though it's specified as boolean in ECS).

To catch these problems early, part of the integration test should be a validation of the generated mappings for all data streams, making sure all fields are actually mapped in compliance with ECS.

While this won't rule out potential mapping issues completely, it should make it much easier to catch these issues during development.

This is part of https://github.com/elastic/observability-dev/issues/3967

cc @zmoog @jsoriano

[jsoriano]

Some validations on field types already happen, but not for all types, and it can be disabled for numeric and string fields with numeric_keyword_fields and string_number_fields.

[flash1293]

and it can be disabled for numeric and string fields with numeric_keyword_fields and string_number_fields.

Does this validation happen on the source or on the mappings the system ends up with? With ecs@mappings that's not necessarily the same thing.

[jsoriano]

Does this validation happen on the source or on the mappings the system ends up with? With ecs@mappings that's not necessarily the same thing.

For every field in the stored document (fields taken from _source or fields depending on the case), it checks that it has a definition and that the type of the value matches with the type of the definition.
It looks for the definitions in the ones included in the package or in ECS for packages including ECS mappings.

We are considering to make validations also based on the mappings the data streams end up with after running system tests (related internal discussion in https://github.com/elastic/ingest-dev/issues/3935), but we don't do anything with it now.

[flash1293]

@jsoriano @kpollich is this still something that's on your roadmap?

[jsoriano]

@jsoriano @kpollich is this still something that's on your roadmap?

@mrodm is working on a revamp on how we do validations #2214, I think it includes some changes to try to address better the ECS cases.

[flash1293]

@mrodm is your PR also addressing the ecs mappings defined as dynamic templates coming from ecs@mappings?

[mrodm]

@mrodm is your PR also addressing the ecs mappings defined as dynamic templates coming from ecs@mappings?

@flash1293 currently, that PR is just validating the actual mappings created under properties, checking whether or not there is a corresponding mapping in the preview too. If those mappings come from dynamic templates those are not validated yet. It needs to be validated/ensured that there is a definition in the dynamic templates for them (checking path_match, path_unmatch, ...).

The support to validate those mappings that could come from dynamic templates is intended to be done as part of #2207

[jsoriano]

I think that after the validations added in recent versions of elastic-package, we are already achieving this. I am going to close this issue.

@flash1293 please reopen if you find some case that we are not properly catching. Thanks!