[Security Solution][AI Assistant] Update ConversationSummary schema and interface (elastic#13657) (elastic#232288)

## Summary

Epic: elastic/security-team#12768
Meta: elastic/security-team#13657
RFC: [internal
link](https://docs.google.com/document/d/13jAJ5Q3_At_zAuwKjvpYehnM5uzKJSwZIZ1aJJNAf5k)

With these changes we update the conversation summary schema to
accommodate new fields to allow conversation summarization and past
conversation search. Also, as part of these changes, the OLD (unused)
summary fields are removed from the APIs.

### Mapping changes

The conversations index mapping already has a `summary` field which is
an object that looks like:

<details>
  <summary><b>OLD</b> summary schema</summary>
  
```json
"summary": {
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "confidence": {
      "type": "keyword"
    },
    "content": {
      "type": "text"
    },
    "public": {
      "type": "boolean"
    }
  }
}
```

</details>

To be able to summarize conversations and semantically search through
existing summaries, the new fields (`semantic_content` and
`summarized_message_ids`) are added into the mapping:

<details>
  <summary><b>Updated</b> summary schema</summary>
  

```json
"summary": {
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "confidence": {
      "type": "keyword"
    },
    "content": {
      "type": "text"
    },
    "public": {
      "type": "boolean"
    },
    "semantic_content": {
      "type": "semantic_text",
      "inference_id": ".elser-2-elasticsearch"
    },
    "summarized_message_ids": {
      "type": "keyword",
      "array": true
    }
  }
}
```

</details>

### New fields description

`semantic_content` field will be used to store conversation summary and
allows semantical search through the ELSER v2 or E5 models.

`summarized_message_ids` field will contain a list of all messages that
are summarized and part of the summary stored within the
`semantic_content` field.

### Legacy fields and API interface changes

There are bunch of fields that were never used and won't be supported or
used in future - `summary.confidence`, `summary.content` and
`summary.public`. After discussion with @YulNaumenko and
@elastic/security-generative-ai, this fields will be marked as legacy on
the mappings level for compatibility with the installed indices and will
be removed on the API level. Previously, we allowed to update
`summary.confidence`, `summary.content` and `summary.public` fields via
API calls and never used in kibana UI.

**NOTE**: Thanks @spong to pointing to [this
cluster](https://overview.elastic-cloud.com/app/dashboards#/view/serverless-api-services-http-requests-overview?_g=h@6558260)
to see the API usage in production. It shows that within last 90 days,
the update conversation API (the only way for users to update
conversations and potentially add a summary to it) was used only 41
times which looks low and I believe negligible.

<img width="1144" height="423" alt="Screenshot 2025-08-20 at 10 50 35"
src="https://github.com/user-attachments/assets/6cb8e1a2-4d9d-44d2-8e66-2de6d8ac74e2"
/>

From now on, the conversation will have next summary fields on the **API
level**:

```typescript
interface ConversationSummary {
  /**
   * The timestamp summary was updated.
   */
  timestamp: string;

  /**
   * Summary text of the conversation over time.
   */
  semanticContent?: string;

  /**
   * The list of summarized messages.
   */
  summarizedMessageIds?: string[];
}
```

### Testing

To test, you can use next API calls:

<details>
  <summary><b>Fetch</b> all existing conversations</summary>
  
This call will fetch all existing conversation. Good for overview of
existing conversations and verifying expected summary values.
```curl
curl --location 'http://localhost:5601/sbb/api/security_ai_assistant/current_user/conversations/_find' \
--header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 2023-10-31'
```

</details>

<details>
  <summary><b>Update</b> a conversation</summary>

This call will update a conversation and add/update a summary.  

```curl
curl --location --request PUT 'http://localhost:5601/sbb/api/security_ai_assistant/current_user/conversations/{{CONVERSATION_ID}}' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 2023-10-31' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
--data '{
    "id": "a565baa8-5566-47b2-ab69-807248b2fc46",
    "summary": {
        "semanticContent": "Very nice demo semantic content."
    }
}'
```

</details>

<details>
  <summary><b>Bulk Update</b> existing conversation(s)</summary>

This call will update a conversation and add/update a summary.  

```curl
curl --location 'http://localhost:5601/sbb/internal/elastic_assistant/current_user/conversations/_bulk_action' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: Kibana' \
--header 'kbn-version: 9.2.0' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
--data '{
  "update":
    [
        {
            "id": "{{CONVERSATION_ID}}",
            "summary": {
                "semanticContent": "Very nice demo semantic content."
            }
        }
    ]
}'
```

</details>

Some test cases:
1. Check that if not updated, a new conversation does not have a summary
2. Check that `summary` contains expected value after it has been
updated via one of the above APIs
3. Check that we do not return legacy fields (`summary.confidence`,
`summary.content` and `summary.public`) even if you add a document with
those fields set. You can set legacy fields, either via DevTools or via
update APIs from above in previous kibana version.

---------

Co-authored-by: kibanamachine <[email protected]>

Author: 2 people

.buildkite/ftr_security_serverless_configs.yml

@@ -108,6 +108,8 @@ enabled:
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/genai/attack_discovery/schedules/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/genai/attack_discovery/schedules/basic_license_essentials_tier/configs/ess.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/genai/attack_discovery/schedules/basic_license_essentials_tier/configs/serverless.config.ts
+  - x-pack/solutions/security/test/security_solution_api_integration/test_suites/genai/conversations/trial_license_complete_tier/configs/ess.config.ts
+  - x-pack/solutions/security/test/security_solution_api_integration/test_suites/genai/conversations/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/risk_engine/basic_license_essentials_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/entity_analytics/entity_store/trial_license_complete_tier/configs/serverless.config.ts

oas_docs/output/kibana.serverless.yaml

@@ -61559,14 +61559,6 @@ components:
         - insights
       example: assistant
       type: string
-    Security_AI_Assistant_API_ConversationConfidence:
-      description: The conversation confidence.
-      enum:
-        - low
-        - medium
-        - high
-      example: high
-      type: string
     Security_AI_Assistant_API_ConversationCreateProps:
       type: object
       properties:
@@ -61651,24 +61643,28 @@ components:
         - namespace
         - category
     Security_AI_Assistant_API_ConversationSummary:
+      allOf:
+        - $ref: '#/components/schemas/Security_AI_Assistant_API_ConversationSummaryBase'
+        - type: object
+          properties:
+            timestamp:
+              $ref: '#/components/schemas/Security_AI_Assistant_API_NonEmptyTimestamp'
+              description: The timestamp summary was updated.
+              example: '2025-04-30T16:00:00Z'
+          required:
+            - timestamp
+    Security_AI_Assistant_API_ConversationSummaryBase:
       type: object
       properties:
-        confidence:
-          $ref: '#/components/schemas/Security_AI_Assistant_API_ConversationConfidence'
-          description: How confident you are about this being a correct and useful learning.
-          example: high
-        content:
+        semanticContent:
           description: Summary text of the conversation over time.
           example: This conversation covered how to configure the Security AI Assistant.
           type: string
-        public:
-          description: Define if summary is marked as publicly available.
-          example: true
-          type: boolean
-        timestamp:
-          $ref: '#/components/schemas/Security_AI_Assistant_API_NonEmptyTimestamp'
-          description: The timestamp summary was updated.
-          example: '2025-04-30T16:00:00Z'
+        summarizedMessageIds:
+          description: The list of summarized messages.
+          items:
+            $ref: '#/components/schemas/Security_AI_Assistant_API_NonEmptyString'
+          type: array
     Security_AI_Assistant_API_ConversationUpdateProps:
       type: object
       properties:
@@ -61692,7 +61688,7 @@ components:
         replacements:
           $ref: '#/components/schemas/Security_AI_Assistant_API_Replacements'
         summary:
-          $ref: '#/components/schemas/Security_AI_Assistant_API_ConversationSummary'
+          $ref: '#/components/schemas/Security_AI_Assistant_API_ConversationSummaryBase'
         title:
           description: The conversation title.
           example: Updated Security AI Assistant Setup

oas_docs/output/kibana.yaml

@@ -74108,14 +74108,6 @@ components:
         - insights
       example: assistant
       type: string
-    Security_AI_Assistant_API_ConversationConfidence:
-      description: The conversation confidence.
-      enum:
-        - low
-        - medium
-        - high
-      example: high
-      type: string
     Security_AI_Assistant_API_ConversationCreateProps:
       type: object
       properties:
@@ -74200,24 +74192,28 @@ components:
         - namespace
         - category
     Security_AI_Assistant_API_ConversationSummary:
+      allOf:
+        - $ref: '#/components/schemas/Security_AI_Assistant_API_ConversationSummaryBase'
+        - type: object
+          properties:
+            timestamp:
+              $ref: '#/components/schemas/Security_AI_Assistant_API_NonEmptyTimestamp'
+              description: The timestamp summary was updated.
+              example: '2025-04-30T16:00:00Z'
+          required:
+            - timestamp
+    Security_AI_Assistant_API_ConversationSummaryBase:
       type: object
       properties:
-        confidence:
-          $ref: '#/components/schemas/Security_AI_Assistant_API_ConversationConfidence'
-          description: How confident you are about this being a correct and useful learning.
-          example: high
-        content:
+        semanticContent:
           description: Summary text of the conversation over time.
           example: This conversation covered how to configure the Security AI Assistant.
           type: string
-        public:
-          description: Define if summary is marked as publicly available.
-          example: true
-          type: boolean
-        timestamp:
-          $ref: '#/components/schemas/Security_AI_Assistant_API_NonEmptyTimestamp'
-          description: The timestamp summary was updated.
-          example: '2025-04-30T16:00:00Z'
+        summarizedMessageIds:
+          description: The list of summarized messages.
+          items:
+            $ref: '#/components/schemas/Security_AI_Assistant_API_NonEmptyString'
+          type: array
     Security_AI_Assistant_API_ConversationUpdateProps:
       type: object
       properties:
@@ -74241,7 +74237,7 @@ components:
         replacements:
           $ref: '#/components/schemas/Security_AI_Assistant_API_Replacements'
         summary:
-          $ref: '#/components/schemas/Security_AI_Assistant_API_ConversationSummary'
+          $ref: '#/components/schemas/Security_AI_Assistant_API_ConversationSummaryBase'
         title:
           description: The conversation title.
           example: Updated Security AI Assistant Setup

x-pack/platform/packages/shared/kbn-elastic-assistant-common/docs/openapi/ess/elastic_assistant_api_2023_10_31.bundled.schema.yaml

@@ -1959,14 +1959,6 @@ components:
         - insights
       example: assistant
       type: string
-    ConversationConfidence:
-      description: The conversation confidence.
-      enum:
-        - low
-        - medium
-        - high
-      example: high
-      type: string
     ConversationCreateProps:
       type: object
       properties:
@@ -2051,28 +2043,30 @@ components:
         - namespace
         - category
     ConversationSummary:
+      allOf:
+        - $ref: '#/components/schemas/ConversationSummaryBase'
+        - type: object
+          properties:
+            timestamp:
+              $ref: '#/components/schemas/NonEmptyTimestamp'
+              description: The timestamp summary was updated.
+              example: '2025-04-30T16:00:00Z'
+          required:
+            - timestamp
+    ConversationSummaryBase:
       type: object
       properties:
-        confidence:
-          $ref: '#/components/schemas/ConversationConfidence'
-          description: >-
-            How confident you are about this being a correct and useful
-            learning.
-          example: high
-        content:
+        semanticContent:
           description: Summary text of the conversation over time.
           example: >-
             This conversation covered how to configure the Security AI
             Assistant.
           type: string
-        public:
-          description: Define if summary is marked as publicly available.
-          example: true
-          type: boolean
-        timestamp:
-          $ref: '#/components/schemas/NonEmptyTimestamp'
-          description: The timestamp summary was updated.
-          example: '2025-04-30T16:00:00Z'
+        summarizedMessageIds:
+          description: The list of summarized messages.
+          items:
+            $ref: '#/components/schemas/NonEmptyString'
+          type: array
     ConversationUpdateProps:
       type: object
       properties:
@@ -2096,7 +2090,7 @@ components:
         replacements:
           $ref: '#/components/schemas/Replacements'
         summary:
-          $ref: '#/components/schemas/ConversationSummary'
+          $ref: '#/components/schemas/ConversationSummaryBase'
         title:
           description: The conversation title.
           example: Updated Security AI Assistant Setup

x-pack/platform/packages/shared/kbn-elastic-assistant-common/docs/openapi/serverless/elastic_assistant_api_2023_10_31.bundled.schema.yaml

@@ -1959,14 +1959,6 @@ components:
         - insights
       example: assistant
       type: string
-    ConversationConfidence:
-      description: The conversation confidence.
-      enum:
-        - low
-        - medium
-        - high
-      example: high
-      type: string
     ConversationCreateProps:
       type: object
       properties:
@@ -2051,28 +2043,30 @@ components:
         - namespace
         - category
     ConversationSummary:
+      allOf:
+        - $ref: '#/components/schemas/ConversationSummaryBase'
+        - type: object
+          properties:
+            timestamp:
+              $ref: '#/components/schemas/NonEmptyTimestamp'
+              description: The timestamp summary was updated.
+              example: '2025-04-30T16:00:00Z'
+          required:
+            - timestamp
+    ConversationSummaryBase:
       type: object
       properties:
-        confidence:
-          $ref: '#/components/schemas/ConversationConfidence'
-          description: >-
-            How confident you are about this being a correct and useful
-            learning.
-          example: high
-        content:
+        semanticContent:
           description: Summary text of the conversation over time.
           example: >-
             This conversation covered how to configure the Security AI
             Assistant.
           type: string
-        public:
-          description: Define if summary is marked as publicly available.
-          example: true
-          type: boolean
-        timestamp:
-          $ref: '#/components/schemas/NonEmptyTimestamp'
-          description: The timestamp summary was updated.
-          example: '2025-04-30T16:00:00Z'
+        summarizedMessageIds:
+          description: The list of summarized messages.
+          items:
+            $ref: '#/components/schemas/NonEmptyString'
+          type: array
     ConversationUpdateProps:
       type: object
       properties:
@@ -2096,7 +2090,7 @@ components:
         replacements:
           $ref: '#/components/schemas/Replacements'
         summary:
-          $ref: '#/components/schemas/ConversationSummary'
+          $ref: '#/components/schemas/ConversationSummaryBase'
         title:
           description: The conversation title.
           example: Updated Security AI Assistant Setup

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/schemas/conversations/common_attributes.gen.ts

@@ -298,26 +298,28 @@ export const ApiConfig = z.object({
   model: z.string().optional(),
 });
 
-export type ConversationSummary = z.infer<typeof ConversationSummary>;
-export const ConversationSummary = z.object({
+export type ConversationSummaryBase = z.infer<typeof ConversationSummaryBase>;
+export const ConversationSummaryBase = z.object({
   /**
    * Summary text of the conversation over time.
    */
-  content: z.string().optional(),
-  /**
-   * The timestamp summary was updated.
-   */
-  timestamp: NonEmptyTimestamp.optional(),
+  semanticContent: z.string().optional(),
   /**
-   * Define if summary is marked as publicly available.
+   * The list of summarized messages.
    */
-  public: z.boolean().optional(),
-  /**
-   * How confident you are about this being a correct and useful learning.
-   */
-  confidence: ConversationConfidence.optional(),
+  summarizedMessageIds: z.array(NonEmptyString).optional(),
 });
 
+export type ConversationSummary = z.infer<typeof ConversationSummary>;
+export const ConversationSummary = ConversationSummaryBase.merge(
+  z.object({
+    /**
+     * The timestamp summary was updated.
+     */
+    timestamp: NonEmptyTimestamp,
+  })
+);
+
 export type ErrorSchema = z.infer<typeof ErrorSchema>;
 export const ErrorSchema = z
   .object({
@@ -389,7 +391,7 @@ export const ConversationUpdateProps = z.object({
    * LLM API configuration.
    */
   apiConfig: ApiConfig.optional(),
-  summary: ConversationSummary.optional(),
+  summary: ConversationSummaryBase.optional(),
   /**
    * Exclude from last conversation storage.
    */

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/schemas/conversations/common_attributes.schema.yaml

@@ -306,25 +306,30 @@ components:
           description: Model
           example: 'gpt-4'
 
-    ConversationSummary:
+    ConversationSummaryBase:
       type: object
       properties:
-        content:
+        semanticContent:
           type: string
           description: Summary text of the conversation over time.
           example: 'This conversation covered how to configure the Security AI Assistant.'
-        timestamp:
-          $ref: '../common_attributes.schema.yaml#/components/schemas/NonEmptyTimestamp'
-          description: The timestamp summary was updated.
-          example: '2025-04-30T16:00:00Z'
-        public:
-          type: boolean
-          description: Define if summary is marked as publicly available.
-          example: true
-        confidence:
-          $ref: '#/components/schemas/ConversationConfidence'
-          description: How confident you are about this being a correct and useful learning.
-          example: 'high'
+        summarizedMessageIds:
+          type: array
+          description: The list of summarized messages.
+          items:
+            $ref: '../common_attributes.schema.yaml#/components/schemas/NonEmptyString'
+
+    ConversationSummary:
+      allOf:
+        - $ref: '#/components/schemas/ConversationSummaryBase'
+        - type: object
+          required:
+            - timestamp
+          properties:
+            timestamp:
+              $ref: '../common_attributes.schema.yaml#/components/schemas/NonEmptyTimestamp'
+              description: The timestamp summary was updated.
+              example: '2025-04-30T16:00:00Z'
 
     ErrorSchema:
       type: object
@@ -426,7 +431,7 @@ components:
           $ref: '#/components/schemas/ApiConfig'
           description: LLM API configuration.
         summary:
-          $ref: '#/components/schemas/ConversationSummary'
+          $ref: '#/components/schemas/ConversationSummaryBase'
         excludeFromLastConversationStorage:
           description: Exclude from last conversation storage.
           type: boolean