[Security Solution][AI Assistant] Update ConversationSummary schema and interface (#13657)

[e40pud]

Summary

Epic: https://github.com/elastic/security-team/issues/12768
Meta: https://github.com/elastic/security-team/issues/13657
RFC: internal link

With these changes we update the conversation summary schema to accommodate new fields to allow conversation summarization and past conversation search. Also, as part of these changes, the OLD (unused) summary fields are removed from the APIs.

Mapping changes

The conversations index mapping already has a summary field which is an object that looks like:

OLD summary schema

"summary": {
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "confidence": {
      "type": "keyword"
    },
    "content": {
      "type": "text"
    },
    "public": {
      "type": "boolean"
    }
  }
}

To be able to summarize conversations and semantically search through existing summaries, the new fields (semantic_content and summarized_message_ids) are added into the mapping:

Updated summary schema

"summary": {
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "confidence": {
      "type": "keyword"
    },
    "content": {
      "type": "text"
    },
    "public": {
      "type": "boolean"
    },
    "semantic_content": {
      "type": "semantic_text",
      "inference_id": ".elser-2-elasticsearch"
    },
    "summarized_message_ids": {
      "type": "keyword",
      "array": true
    }
  }
}

New fields description

semantic_content field will be used to store conversation summary and allows semantical search through the ELSER v2 or E5 models.

summarized_message_ids field will contain a list of all messages that are summarized and part of the summary stored within the semantic_content field.

Legacy fields and API interface changes

There are bunch of fields that were never used and won't be supported or used in future - summary.confidence, summary.content and summary.public. After discussion with @YulNaumenko and @elastic/security-generative-ai, this fields will be marked as legacy on the mappings level for compatibility with the installed indices and will be removed on the API level. Previously, we allowed to update summary.confidence, summary.content and summary.public fields via API calls and never used in kibana UI.

NOTE: Thanks @spong to pointing to this cluster to see the API usage in production. It shows that within last 90 days, the update conversation API (the only way for users to update conversations and potentially add a summary to it) was used only 41 times which looks low and I believe negligible.

From now on, the conversation will have next summary fields on the API level:

interface ConversationSummary {
  /**
   * The timestamp summary was updated.
   */
  timestamp: string;

  /**
   * Summary text of the conversation over time.
   */
  semanticContent?: string;

  /**
   * The list of summarized messages.
   */
  summarizedMessageIds?: string[];
}

Testing

To test, you can use next API calls:

Fetch all existing conversations

This call will fetch all existing conversation. Good for overview of existing conversations and verifying expected summary values.

curl --location 'http://localhost:5601/sbb/api/security_ai_assistant/current_user/conversations/_find' \
--header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 2023-10-31'

Update a conversation

This call will update a conversation and add/update a summary.

curl --location --request PUT 'http://localhost:5601/sbb/api/security_ai_assistant/current_user/conversations/{{CONVERSATION_ID}}' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 2023-10-31' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
--data '{
    "id": "a565baa8-5566-47b2-ab69-807248b2fc46",
    "summary": {
        "semanticContent": "Very nice demo semantic content."
    }
}'

Bulk Update existing conversation(s)

This call will update a conversation and add/update a summary.

curl --location 'http://localhost:5601/sbb/internal/elastic_assistant/current_user/conversations/_bulk_action' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: Kibana' \
--header 'kbn-version: 9.2.0' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
--data '{
  "update":
    [
        {
            "id": "{{CONVERSATION_ID}}",
            "summary": {
                "semanticContent": "Very nice demo semantic content."
            }
        }
    ]
}'

Some test cases:

1. Check that if not updated, a new conversation does not have a summary
2. Check that summary contains expected value after it has been updated via one of the above APIs
3. Check that we do not return legacy fields (summary.confidence, summary.content and summary.public) even if you add a document with those fields set. You can set legacy fields, either via DevTools or via update APIs from above in previous kibana version.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[e40pud]

cc @pgayvallet keeping you in the loop for the summarization related changes.

[spong]

Not applicable to us now since we're deprecating summary.content, but wanted to mention that the common pattern I've seen for when wanting to add embeddings for an existing field you can use copy_to or multi-fields, that way inserts and search stay the same.

[jbudz]

.buildkite/ftr_security_serverless_configs.yml LGTM

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: edb8957

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant-common	676	678	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	10.4MB	10.4MB	+9.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant-common	790	792	+2

ESLint disabled line counts

id	before	after	diff
elasticAssistant	42	41	-1

Total ESLint disabled count

id	before	after	diff
elasticAssistant	50	49	-1

History

• 💔 Build #333057 failed 10e6682
• 💔 Build #332702 failed 4d4d45e
• 💛 Build #331369 was flaky 9845be4
• 💚 Build #331069 succeeded a189843
• 💔 Build #330988 failed 7053dde
• 💔 Build #330920 failed ffa5447

cc @e40pud

[stephmilovic]

Wondering why did you remove this retry logic? Sometimes there can be a conflict when updating the title at the same time as appending a message, for example.

[e40pud]

Found this bulk update API option that we can use https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk#operation-bulk-body-application-json-update-retry_on_conflict. Will update bulk updates accordingly.

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 232288 locally
cc: @e40pud

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 232288 locally
cc: @e40pud