[Entity Store] Add Upsert Entity API (#234454)

## Summary

Add Upsert Entity API which reflects changes made via the API directly
in the final entities index.

#### What is implemented
- Update documents
  - Allowed fields:
    - `entity.attributes.*`
    -  `entity.lifecyle.*`
    - `entity.behavior.*`
- Force update documents


#### Added ES Assets:

- Component Template `security_${type}_default-updates@platform`
- Index Template
`entities_v1_updates_security_${type}_default_index_template`
- Index `.entities.v1.updates.security_${type}_default`

#### What is not implemented
- Create
- ILM Policy to delete update documents

#### How to test

Ingest entities and run in the dev console:
```
PUT kbn:/api/entity_store/entities/generic
{
    "entity": {
        "id": "<ID>",
        "attributes": {
          "StorageClass": "hot"
        }
    }
}
```


### How it works

Before explaining the API itself, a refresher on the entity store

<details>
<summary> Entity Store Diagram </summary>

```mermaid
flowchart TB
    subgraph Main Flow
        A[(.logs*)] ~~~~ B[Transform]
        B ---> |Fetches raw entity data| A
        B ---> | Sends Aggregated Data | G{Ingest Pipeline}
        G --> | Combines new and old data and stores it| C[(.entity.v1.latest*)]
    end

    G -.-> | Fetches data older than transform retention policy| D[(.enrich-index-entities)]

    subgraph Retention Policy Flow
        direction LR 
        E((Kibana Task)) -->|trigger every hour| F[Enrich Policy Entities]
        F -.->| Fetches most upto date entities| C
        F --->| Stores data | D
    end
 ```

</details>

Entity store works based on a Transform which has a look back period of X hours (current 3h). That means data older than look period won't be retained. To solve that an Enrich Policy is set in place that takes hourly snapshots of the current state of the entity store and makes it available to, via ingest pipeline, enrich entity updates and make sure that we have data older than look back period present. Awesome.

This adds complexity to this feature. The goal is add an api that once called reflects data changes immediately in the latest index. A few things were considered:

- ❌ Add a new document to an update index to be picked up by the transform. 
  - That doesn't satisfy the requirement because changes will be made available only after a transform finishes its run
- ❌  Perform update by query in the latest index. 
  - That works great if the entity in the latest index doesn't get any other update via the transform - what we can't guarantee of course.
  
So the arrived solution was to both perform update by query in the latest index and publish an update document to be picked up by the transform, this way we get the best of both worlds.

- So first Update by query on `.entities.v1.latest.security_$TYPE_default` (update made via painless)
- Indexes a new document on `.entities.v1.updates.security_$TYPE_default` to be picked up by the transform.
```mermaid
flowchart LR
    A[User] -->|PUT /api/entity_store/entities/$TYPE| B[Kibana]
B --> |update by query| C[(.entities.v1.latest.security_$TYPE_default)]
B --> |create new doc| D[(.entities.v1.updates.security_$TYPE_default)]
```

We have considered adding a priority mechanism to the update index so we would make sure that documents published to it would be picked up. First we found out that we don't need to make sure a document is seen by the transform. By its definition, transforms process every document - it doesn't have any mechanism to drop documents in case processing is taking too long. Second, we can't do it because the aggregations we run on already sort to find latest values, and sort on multiple fields is not possible. 

### Fields and Schema

Prior to this PR non generic entities (`user`, `host`, and `service`) had no exposure to concepts defined in the proposed `entity.*` ECS Schema. We had to address this to be able to make changes to `entity.attributes`, `entity.lifecyle` and `entity.behavior` fields.

[The current direction](elastic/ecs#2513) is that `entity.*` fields will be nested under `user`, `host`,`service` and `generic` for data input and the latest index, with the final entities, would have a root `entity.*` field set.

In other words, there is a difference between entity data input location and entity data output location.
The document
```json
{ "user": { "entity": { "id" : "romulo", "type": "aws-user" } } }
```
Will be represented in the latest index as 
```json
{ "entity": { "id" : "romulo", "type": "aws-user" } } 
```

Because of the current direction of the discussion we decided to go towards there already. Therefore this PR contains changes to the entity definitions themselves adding entity fields that uses data source `{TYPE}.entity.*` and as destination `entity.*` (`x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/entity_definitions/entity_descriptions/common.ts`).

That also posed another question, what will be the input like? Will it accept entity "input" or entity "output" format?

I had decided to stay close to "output" format, therefore accept `entity.*` json fields and would be applied to the entity store. The reason behind it is simplicity of API. I believe that having a inconsistent placement for `entity` in the api isn't a great experience, therefore always accepting 
```json
{ "entity": { "id" : "romulo", "type": "aws-user" } } 
```
is better imo.

**That's contradictory to the input via logs however**. Curious to hear people's opinion.

There is another problem that further deviates the API from any ECS definition (input or output). For fields under `entity.attributes`, `entity.lifecyle` and `entity.behavior` we decided to define them on ECS. And because they are "custom fields" product would like them to have a `Capital_snake_case` format, which is not a traditional and developing with TS in such a case is not really allowed at the moment. To curb that, the api expose those fields as `snake_case` and before storing convert them to `Capital_snake_case`. That was the best way I found while still having field definition on OpenAPI spec.

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Mark Hopkin <[email protected]>

Author: 3 people

oas_docs/output/kibana.serverless.yaml

@@ -12223,6 +12223,51 @@ paths:
       x-metaTags:
         - content: Kibana, Elastic Cloud Serverless
           name: product_name
+  /api/entity_store/entities/{entityType}:
+    put:
+      description: |
+        Update or create an entity in Entity Store.
+        If the specified entity already exists, it is updated with the provided values.  If the entity does not exist, a new one is created. By default, only the following fields can be updated: * `entity.attributes.*` * `entity.lifecycle.*` * `entity.behavior.*` To update other fields, set the `force` query parameter to `true`. > info > Some fields always retain the first observed value. Updates to these fields will not appear in the final index.
+        > Due to technical limitations, not all updates are guaranteed to appear in the final list of observed values.
+      operationId: UpsertEntity
+      parameters:
+        - in: path
+          name: entityType
+          required: true
+          schema:
+            $ref: '#/components/schemas/Security_Entity_Analytics_API_EntityType'
+        - in: query
+          name: force
+          required: false
+          schema:
+            default: false
+            type: boolean
+      requestBody:
+        content:
+          application/json:
+            schema:
+              $ref: '#/components/schemas/Security_Entity_Analytics_API_Entity'
+        description: Schema for the updating a single entity
+        required: true
+      responses:
+        '200':
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Security_Entity_Analytics_API_Entity'
+          description: Entity updated or created
+        '403':
+          description: Operation on a restricted field
+        '404':
+          description: Entity not found
+        '503':
+          description: Operation on an uninitialized Engine or in a cluster without CRUD API Enabled
+      summary: Upsert an entity in Entity Store
+      tags:
+        - Security Entity Analytics API
+      x-metaTags:
+        - content: Kibana, Elastic Cloud Serverless
+          name: product_name
   /api/entity_store/entities/list:
     get:
       description: List entities records, paging, sorting and filtering as needed.
@@ -71577,6 +71622,7 @@ components:
         - entity_engine
         - entity_definition
         - index
+        - data_stream
         - component_template
         - index_template
         - ingest_pipeline
@@ -71682,6 +71728,7 @@ components:
         - status
         - fieldHistoryLength
     Security_Entity_Analytics_API_EngineMetadata:
+      additionalProperties: false
       type: object
       properties:
         Type:
@@ -71736,6 +71783,40 @@ components:
       required:
         - has_all_required
         - privileges
+    Security_Entity_Analytics_API_EntityField:
+      additionalProperties: false
+      type: object
+      properties:
+        attributes:
+          additionalProperties: false
+          type: object
+          properties:
+            privileged:
+              type: boolean
+        behavior:
+          additionalProperties: false
+          type: object
+        EngineMetadata:
+          $ref: '#/components/schemas/Security_Entity_Analytics_API_EngineMetadata'
+        id:
+          type: string
+        lifecycle:
+          additionalProperties: false
+          type: object
+          properties:
+            first_seen:
+              format: date-time
+              type: string
+        name:
+          type: string
+        source:
+          type: string
+        sub_type:
+          type: string
+        type:
+          type: string
+      required:
+        - id
     Security_Entity_Analytics_API_EntityRiskLevels:
       enum:
         - Unknown
@@ -71819,74 +71900,50 @@ components:
         - generic
       type: string
     Security_Entity_Analytics_API_GenericEntity:
+      additionalProperties: false
       type: object
       properties:
         '@timestamp':
           format: date-time
           type: string
         asset:
+          additionalProperties: false
           type: object
           properties:
             criticality:
               $ref: '#/components/schemas/Security_Entity_Analytics_API_AssetCriticalityLevel'
           required:
             - criticality
         entity:
-          type: object
-          properties:
-            category:
-              type: string
-            EngineMetadata:
-              $ref: '#/components/schemas/Security_Entity_Analytics_API_EngineMetadata'
-            id:
-              type: string
-            name:
-              type: string
-            source:
-              type: string
-            type:
-              type: string
-          required:
-            - id
-            - name
-            - type
+          $ref: '#/components/schemas/Security_Entity_Analytics_API_EntityField'
       required:
         - entity
     Security_Entity_Analytics_API_HostEntity:
+      additionalProperties: false
       type: object
       properties:
         '@timestamp':
           format: date-time
           type: string
         asset:
+          additionalProperties: false
           type: object
           properties:
             criticality:
               $ref: '#/components/schemas/Security_Entity_Analytics_API_AssetCriticalityLevel'
           required:
             - criticality
         entity:
-          type: object
-          properties:
-            EngineMetadata:
-              $ref: '#/components/schemas/Security_Entity_Analytics_API_EngineMetadata'
-            name:
-              type: string
-            source:
-              type: string
-            type:
-              type: string
-          required:
-            - name
-            - source
-            - type
+          $ref: '#/components/schemas/Security_Entity_Analytics_API_EntityField'
         event:
+          additionalProperties: false
           type: object
           properties:
             ingested:
               format: date-time
               type: string
         host:
+          additionalProperties: false
           type: object
           properties:
             architecture:
@@ -71897,6 +71954,8 @@ components:
               items:
                 type: string
               type: array
+            entity:
+              $ref: '#/components/schemas/Security_Entity_Analytics_API_EntityField'
             hostname:
               items:
                 type: string
@@ -71924,7 +71983,6 @@ components:
           required:
             - name
       required:
-        - host
         - entity
     Security_Entity_Analytics_API_IdField:
       enum:
@@ -72118,50 +72176,42 @@ components:
         - description
         - category
     Security_Entity_Analytics_API_ServiceEntity:
+      additionalProperties: false
       type: object
       properties:
         '@timestamp':
           format: date-time
           type: string
         asset:
+          additionalProperties: false
           type: object
           properties:
             criticality:
               $ref: '#/components/schemas/Security_Entity_Analytics_API_AssetCriticalityLevel'
           required:
             - criticality
         entity:
-          type: object
-          properties:
-            EngineMetadata:
-              $ref: '#/components/schemas/Security_Entity_Analytics_API_EngineMetadata'
-            name:
-              type: string
-            source:
-              type: string
-            type:
-              type: string
-          required:
-            - name
-            - source
-            - type
+          $ref: '#/components/schemas/Security_Entity_Analytics_API_EntityField'
         event:
+          additionalProperties: false
           type: object
           properties:
             ingested:
               format: date-time
               type: string
         service:
+          additionalProperties: false
           type: object
           properties:
+            entity:
+              $ref: '#/components/schemas/Security_Entity_Analytics_API_EntityField'
             name:
               type: string
             risk:
               $ref: '#/components/schemas/Security_Entity_Analytics_API_EntityRiskScoreRecord'
           required:
             - name
       required:
-        - service
         - entity
     Security_Entity_Analytics_API_StoreStatus:
       enum:
@@ -72237,40 +72287,31 @@ components:
         - exponential_avg_documents_indexed
         - exponential_avg_documents_processed
     Security_Entity_Analytics_API_UserEntity:
+      additionalProperties: false
       type: object
       properties:
         '@timestamp':
           format: date-time
           type: string
         asset:
+          additionalProperties: false
           type: object
           properties:
             criticality:
               $ref: '#/components/schemas/Security_Entity_Analytics_API_AssetCriticalityLevel'
           required:
             - criticality
         entity:
-          type: object
-          properties:
-            EngineMetadata:
-              $ref: '#/components/schemas/Security_Entity_Analytics_API_EngineMetadata'
-            name:
-              type: string
-            source:
-              type: string
-            type:
-              type: string
-          required:
-            - name
-            - source
-            - type
+          $ref: '#/components/schemas/Security_Entity_Analytics_API_EntityField'
         event:
+          additionalProperties: false
           type: object
           properties:
             ingested:
               format: date-time
               type: string
         user:
+          additionalProperties: false
           type: object
           properties:
             domain:
@@ -72297,14 +72338,14 @@ components:
               type: string
             risk:
               $ref: '#/components/schemas/Security_Entity_Analytics_API_EntityRiskScoreRecord'
+              additionalProperties: false
             roles:
               items:
                 type: string
               type: array
           required:
             - name
       required:
-        - user
         - entity
     Security_Entity_Analytics_API_UserName:
       type: object