diff --git a/semantic-kernel/Frameworks/agent/TOC.yml b/semantic-kernel/Frameworks/agent/TOC.yml index c85787ea..2d6709d8 100644 --- a/semantic-kernel/Frameworks/agent/TOC.yml +++ b/semantic-kernel/Frameworks/agent/TOC.yml @@ -6,6 +6,8 @@ href: chat-completion-agent.md - name: OpenAI Assistant Agent href: assistant-agent.md +- name: Azure AI Agent + href: azure-ai-agent.md - name: Agent Collaboration href: agent-chat.md - name: Create an Agent from a Template diff --git a/semantic-kernel/Frameworks/agent/agent-chat.md b/semantic-kernel/Frameworks/agent/agent-chat.md index 5ada4e7c..34d1a18a 100644 --- a/semantic-kernel/Frameworks/agent/agent-chat.md +++ b/semantic-kernel/Frameworks/agent/agent-chat.md @@ -24,8 +24,8 @@ Detailed API documentation related to this discussion is available at: ::: zone pivot="programming-language-python" -- [`agent_chat`](/python/api/semantic-kernel/semantic_kernel.agents.group_chat.agent_chat) -- [`agent_group_chat`](/python/api/semantic-kernel/semantic_kernel.agents.group_chat.agent_group_chat) +- [`AgentChat`](/python/api/semantic-kernel/semantic_kernel.agents.group_chat.agent_chat.agentchat) +- [`AgentGroupChat`](/python/api/semantic-kernel/semantic_kernel.agents.group_chat.agent_group_chat.agentgroupchat) ::: zone-end diff --git a/semantic-kernel/Frameworks/agent/agent-templates.md b/semantic-kernel/Frameworks/agent/agent-templates.md index 6813d6f9..b8d0a4ca 100644 --- a/semantic-kernel/Frameworks/agent/agent-templates.md +++ b/semantic-kernel/Frameworks/agent/agent-templates.md @@ -76,10 +76,8 @@ ChatCompletionAgent agent = ::: zone pivot="programming-language-python" ```python -kernel = Kernel() - agent = ChatCompletionAgent( - kernel=kernel, + service=AzureChatCompletion(), # or other supported AI Services name="StoryTeller", instructions="Tell a story about {{$topic}} that is {{$length}} sentences long.", arguments=KernelArguments(topic="Dog", length="2"), @@ -100,24 +98,34 @@ Templated instructions are especially powerful when working with an [`OpenAIAssi ::: zone pivot="programming-language-csharp" ```csharp // Retrieve an existing assistant definition by identifier -OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.RetrieveAsync( - this.GetClientProvider(), - "", - new Kernel(), - new KernelArguments() - { - { "topic", "Dog" }, - { "length", "3" }, - }); +AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri("")); +AssistantClient assistantClient = client.GetAssistantClient(); +Assistant assistant = await client.GetAssistantAsync(); +OpenAIAssistantAgent agent = new(assistant, assistantClient, new KernelPromptTemplateFactory(), PromptTemplateConfig.SemanticKernelTemplateFormat) +{ + Arguments = new KernelArguments() + { + { "topic", "Dog" }, + { "length", "3" }, + } +} ``` ::: zone-end ::: zone pivot="programming-language-python" ```python -agent = await OpenAIAssistantAgent.retrieve( - id=, - kernel=Kernel(), +# Create the client using Azure OpenAI resources and configuration +client, model = AzureAssistantAgent.setup_resources() + +# Retrieve the assistant definition from the server based on the assistant ID +definition = await client.beta.assistants.retrieve( + assistant_id="your-assistant-id", +) + +# Create the AzureAssistantAgent instance using the client and the assistant definition +agent = AzureAssistantAgent( + client=client, + definition=definition, arguments=KernelArguments(topic="Dog", length="3"), ) ``` @@ -130,10 +138,9 @@ agent = await OpenAIAssistantAgent.retrieve( ::: zone-end -## Agent Definition from a _Prompt Template_ - -The same _Prompt Template Config_ used to create a _Kernel Prompt Function_ can also be leveraged to define an agent. This allows for a unified approach in managing both prompts and agents, promoting consistency and reuse across different components. By externalizing agent definitions from the codebase, this method simplifies the management of multiple agents, making them easier to update and maintain without requiring changes to the underlying logic. This separation also enhances flexibility, enabling developers to modify agent behavior or introduce new agents by simply updating the configuration, rather than adjusting the code itself. +## Agent Definition from a Prompt Template +The same Prompt Template Config used to create a Kernel Prompt Function can also be leveraged to define an agent. This allows for a unified approach in managing both prompts and agents, promoting consistency and reuse across different components. By externalizing agent definitions from the codebase, this method simplifies the management of multiple agents, making them easier to update and maintain without requiring changes to the underlying logic. This separation also enhances flexibility, enabling developers to modify agent behavior or introduce new agents by simply updating the configuration, rather than adjusting the code itself. #### YAML Template ```yaml @@ -192,7 +199,7 @@ data = yaml.safe_load(generate_story_yaml) prompt_template_config = PromptTemplateConfig(**data) agent = ChatCompletionAgent( - kernel=_create_kernel_with_chat_completion(), + service=AzureChatCompletion(), # or other supported AI services prompt_template_config=prompt_template_config, arguments=KernelArguments(topic="Dog", length="3"), ) @@ -249,10 +256,8 @@ await foreach (ChatMessageContent response in agent.InvokeAsync(chat, overrideAr ::: zone pivot="programming-language-python" ```python -kernel = Kernel() - agent = ChatCompletionAgent( - kernel=kernel, + service=AzureChatCompletion(), name="StoryTeller", instructions="Tell a story about {{$topic}} that is {{$length}} sentences long.", arguments=KernelArguments(topic="Dog", length="2"), diff --git a/semantic-kernel/Frameworks/agent/assistant-agent.md b/semantic-kernel/Frameworks/agent/assistant-agent.md index 17ee3199..c6d8ca77 100644 --- a/semantic-kernel/Frameworks/agent/assistant-agent.md +++ b/semantic-kernel/Frameworks/agent/assistant-agent.md @@ -8,7 +8,7 @@ ms.author: crickman ms.date: 09/13/2024 ms.service: semantic-kernel --- -# Exploring the _Semantic Kernel_ `OpenAIAssistantAgent` +# Exploring the Semantic Kernel `OpenAIAssistantAgent` > [!IMPORTANT] > This feature is in the release candidate stage. Features at this stage are nearly complete and generally stable, though they may undergo minor refinements or optimizations before reaching full general availability. @@ -17,15 +17,13 @@ Detailed API documentation related to this discussion is available at: ::: zone pivot="programming-language-csharp" - [`OpenAIAssistantAgent`](/dotnet/api/microsoft.semantickernel.agents.openai.openaiassistantagent) -- [`OpenAIAssistantDefinition`](/dotnet/api/microsoft.semantickernel.agents.openai.openaiassistantdefinition) -- [`OpenAIClientProvider`](/dotnet/api/microsoft.semantickernel.agents.openai.openaiclientprovider) ::: zone-end ::: zone pivot="programming-language-python" -- [`azure_assistant_agent`](/python/api/semantic-kernel/semantic_kernel.agents.open_ai.azure_assistant_agent) -- [`open_ai_assistant_agent`](/python/api/semantic-kernel/semantic_kernel.agents.open_ai.open_ai_assistant_agent) +- [`AzureAssistantAgent`](/python/api/semantic-kernel/semantic_kernel.agents.open_ai.azure_assistant_agent.azureassistantagent) +- [`OpenAIAssistantAgent`](/python/api/semantic-kernel/semantic_kernel.agents.open_ai.open_ai_assistant_agent.openaiassistantagent) ::: zone-end @@ -45,21 +43,55 @@ The _OpenAI Assistant API_ is a specialized interface designed for more advanced - [Assistant API in Azure](/azure/ai-services/openai/assistants-quickstart) +## Preparing Your Development Environment + +To proceed with developing an `OpenAIAIAssistantAgent`, configure your development environment with the appropriate packages. + +::: zone pivot="programming-language-csharp" + +Add the `Microsoft.SemanticKernel.Agents.OpenAI` package to your project: + +```pwsh +dotnet add package Microsoft.SemanticKernel.Agents.AzureAI --prerelease +``` + +You may also want to include the `Azure.Identity` package: + +```pwsh +dotnet add package Azure.Identity +``` +::: zone-end + +::: zone pivot="programming-language-python" + +Install the `semantic-kernel` package with the optional _Azure_ dependencies: + +```bash +pip install semantic-kernel[azure] +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + + ## Creating an `OpenAIAssistantAgent` Creating an `OpenAIAssistant` requires invoking a remote service, which is handled asynchronously. To manage this, the `OpenAIAssistantAgent` is instantiated through a static factory method, ensuring the process occurs in a non-blocking manner. This method abstracts the complexity of the asynchronous call, returning a promise or future once the assistant is fully initialized and ready for use. ::: zone pivot="programming-language-csharp" ```csharp -OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.CreateAsync( - OpenAIClientProvider.ForAzureOpenAI(/*<...service configuration>*/), - new OpenAIAssistantDefinition("") - { - Name = "", - Instructions = "", - }, - new Kernel()); +AssistantClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(...).GetAssistantClient(); +Assistant assistant = + await this.AssistantClient.CreateAssistantAsync( + "", + "", + instructions: ""); +OpenAIAssistantAgent agent = new(assistant, client); ``` ::: zone-end @@ -119,11 +151,9 @@ Once created, the identifier of the assistant may be access via its identifier. For .NET, the agent identifier is exposed as a `string` via the property defined by any agent. ```csharp -OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.RetrieveAsync( - OpenAIClientProvider.ForAzureOpenAI(/*<...service configuration>*/), - "", - new Kernel()); +AssistantClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(...).GetAssistantClient(); +Assistant assistant = await this.AssistantClient.GetAssistantAsync(""); +OpenAIAssistantAgent agent = new(assistant, client); ``` ::: zone-end @@ -220,20 +250,18 @@ await agent.delete_thread(thread_id) ## Deleting an `OpenAIAssistantAgent` -Since the assistant's definition is stored remotely, it supports the capability to self-delete. This enables the agent to be removed from the system when it is no longer needed. +Since the assistant's definition is stored remotely, it will persist if not deleted. +Deleting an assistant definition may be performed directly with the `AssistantClient`. -> Note: Attempting to use an agent instance after being deleted results in an exception. +> Note: Attempting to use an agent instance after being deleted will result in a service exception. ::: zone pivot="programming-language-csharp" For .NET, the agent identifier is exposed as a `string` via the [`Agent.Id`](/dotnet/api/microsoft.semantickernel.agents.agent.id) property defined by any agent. ```csharp -// Perform the deletion -await agent.DeleteAsync(); - -// Inspect whether an agent has been deleted -bool isDeleted = agent.IsDeleted(); +AssistantClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(...).GetAssistantClient(); +Assistant assistant = await this.AssistantClient.DeleteAssistantAsync(""); ``` ::: zone-end @@ -261,5 +289,5 @@ For an end-to-end example for a `OpenAIAssistantAgent`, see: > [!div class="nextstepaction"] -> [Agent Collaboration in `AgentChat`](./agent-chat.md) +> [Exploring the Azure AI Agent](./azure-ai-agent.md) diff --git a/semantic-kernel/Frameworks/agent/azure-ai-agent.md b/semantic-kernel/Frameworks/agent/azure-ai-agent.md new file mode 100644 index 00000000..6d6eabd7 --- /dev/null +++ b/semantic-kernel/Frameworks/agent/azure-ai-agent.md @@ -0,0 +1,671 @@ +--- +title: Exploring the Semantic Kernel Azure AI Agent Agent +description: An exploration of the definition, behaviors, and usage patterns for an Azure AI Agent +zone_pivot_groups: programming-languages +author: moonbox3 +ms.topic: tutorial +ms.author: evmattso +ms.date: 03/05/2025 +ms.service: semantic-kernel +--- +# Exploring the Semantic Kernel `AzureAIAgent` + +> [!IMPORTANT] +> This feature is in the experimental stage. Features at this stage are still under development and subject to change before advancing to the preview or release candidate stage. + +Detailed API documentation related to this discussion is available at: + +::: zone pivot="programming-language-csharp" + +- [`AzureAIAgent`](/dotnet/api/microsoft.semantickernel.agents.azureai) + +::: zone-end + +::: zone pivot="programming-language-python" + +- [`AzureAIAgent`](/python/api/semantic-kernel/semantic_kernel.agents.azure_ai.azure_ai_agent.azureaiagent) + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## What is an `AzureAIAgent`? + +An `AzureAIAgent` is a specialized agent within the Semantic Kernel framework, designed to provide advanced conversational capabilities with seamless tool integration. It automates tool calling, eliminating the need for manual parsing and invocation. The agent also securely manages conversation history using threads, reducing the overhead of maintaining state. Additionally, the `AzureAIAgent` supports a variety of built-in tools, including file retrieval, code execution, and data interaction via Bing, Azure AI Search, Azure Functions, and OpenAPI. + +To use an `AzureAIAgent`, an Azure AI Foundry Project must be utilized. The following articles provide an overview of the Azure AI Foundry, how to create and configure a project, and the agent service: + +- [What is Azure AI Foundry?](/azure/ai-foundry/what-is-ai-foundry) +- [The Azure AI Foundry SDK](/azure/ai-foundry/how-to/develop/sdk-overview) +- [What is Azure AI Agent Service](/azure/ai-services/agents/overview) +- [Quickstart: Create a new agent](/azure/ai-services/agents/quickstart) + + +## Preparing Your Development Environment + +To proceed with developing an `AzureAIAgent`, configure your development environment with the appropriate packages. + +::: zone pivot="programming-language-csharp" + +Add the `Microsoft.SemanticKernel.Agents.AzureAI` package to your project: + +```pwsh +dotnet add package Microsoft.SemanticKernel.Agents.AzureAI --prerelease +``` + +You may also want to include the `Azure.Identity` package: + +```pwsh +dotnet add package Azure.Identity +``` + +::: zone-end + +::: zone pivot="programming-language-python" + +Install the `semantic-kernel` package with the optional Azure dependencies: + +```bash +pip install semantic-kernel[azure] +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + + +## Configuring the AI Project Client + +Accessing an `AzureAIAgent` first requires the creation of a project client that is configured for a specific Foundry Project, most commonly by providing a connection string ([The Azure AI Foundry SDK: Getting Started with Projects](/azure/ai-foundry/how-to/develop/sdk-overview#get-started-with-projects)). + +::: zone pivot="programming-language-csharp" + +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +``` + +The `AgentsClient` may be accessed from the `AIProjectClient`: + +```c# +AgentsClient agentsClient = client.GetAgentsClient(); +``` + +::: zone-end + +::: zone pivot="programming-language-python" + +Modify your the `.env` file in the root directory to include: + +```bash +AZURE_AI_AGENT_PROJECT_CONNECTION_STRING = "" +AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME = "" +``` + +or + +```bash +AZURE_AI_AGENT_ENDPOINT = "" +AZURE_AI_AGENT_SUBSCRIPTION_ID = "" +AZURE_AI_AGENT_RESOURCE_GROUP_NAME = "" +AZURE_AI_AGENT_PROJECT_NAME = "" +AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME = "" +``` + +Once the configuration is defined, the client may be created: + +```python +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, +): + # Your operational code here +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Creating an `AzureAIAgent` + +To create an `AzureAIAgent`, you start by configuring and initializing the agent project through the Azure AI service and then integrate it with Semantic Kernel: + +::: zone pivot="programming-language-csharp" + +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +// 1. Define an agent on the Azure AI agent service +Agent definition = agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: ""); + +// 2. Create a Semantic Kernel agent based on the agent definition +AzureAIAgent agent = new(definition, agentsClient); +``` +::: zone-end + +::: zone pivot="programming-language-python" + +```python +from azure.identity.aio import DefaultAzureCredential +from semantic_kernel.agents.azure_ai import AzureAIAgent, AzureAIAgentSettings + +ai_agent_settings = AzureAIAgentSettings.create() + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, +): + # 1. Define an agent on the Azure AI agent service + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + name="", + instructions="", + ) + + # 2. Create a Semantic Kernel agent based on the agent definition + agent = AzureAIAgent( + client=client, + definition=agent_definition, + ) +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Interacting with an `AzureAIAgent` + +Interaction with the `AzureAIAgent` is straightforward. The agent maintains the conversation history automatically using a thread: + +::: zone pivot="programming-language-csharp" +```c# +AgentThread thread = await agentsClient.CreateThreadAsync(); +try +{ + ChatMessageContent message = new(AuthorRole.User, ""); + await agent.AddChatMessageAsync(threadId, message); + await foreach (ChatMessageContent response in agent.InvokeAsync(thread.Id)) + { + Console.WriteLine(response.Content); + } +} +finally +{ + await this.AgentsClient.DeleteThreadAsync(thread.Id); + await this.AgentsClient.DeleteAgentAsync(agent.Id); +} +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +USER_INPUTS = ["Hello", "What's your name?"] + +thread = await client.agents.create_thread() + +try: + for user_input in USER_INPUTS: + await agent.add_chat_message(thread_id=thread.id, message=user_input) + response = await agent.get_response(thread_id=thread.id) + print(response) +finally: + await client.agents.delete_thread(thread.id) +``` + +Optionally, an agent may be invoked as: + +```python +for user_input in USER_INPUTS: + await agent.add_chat_message(thread_id=thread.id, message=user_input) + async for content in agent.invoke(thread_id=thread.id): + print(content.content) +``` + +::: zone-end + +An agent may also produce a streamed response: + +::: zone pivot="programming-language-csharp" +```c# +ChatMessageContent message = new(AuthorRole.User, ""); +await agent.AddChatMessageAsync(threadId, message); +await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(thread.Id)) +{ + Console.Write(response.Content); +} +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +for user_input in USER_INPUTS: + await agent.add_chat_message(thread_id=thread.id, message=user_input) + async for content in agent.invoke_stream(thread_id=thread.id): + print(content.content, end="", flush=True) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Using Plugins with an `AzureAIAgent` + +Semantic Kernel supports extending an `AzureAIAgent` with custom plugins for enhanced functionality: + +::: zone pivot="programming-language-csharp" +```c# +Plugin plugin = KernelPluginFactory.CreateFromType(); +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +Agent definition = agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: ""); + +AzureAIAgent agent = new(definition, agentsClient, plugins: [plugin]); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +from semantic_kernel.functions import kernel_function + +class SamplePlugin: + @kernel_function(description="Provides sample data.") + def get_data(self) -> str: + return "Sample data" + +ai_agent_settings = AzureAIAgentSettings.create() + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + ) + + agent = AzureAIAgent( + client=client, + definition=agent_definition, + plugins=[SamplePlugin()] + ) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Advanced Features + +An `AzureAIAgent` can leverage advanced tools such as: + +- [Code Interpreter](#code-interpreter) +- [File Search](#file-search) +- [OpenAPI integration](#openapi-integration) +- [Azure AI Search integration](#azureai-search-integration) + +### Code Interpreter + +Code Interpreter allows the agents to write and run Python code in a sandboxed execution environment ([Azure AI Agent Service Code Interpreter](/azure/ai-services/agents/how-to/tools/code-interpreter)). + +::: zone pivot="programming-language-csharp" +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +Agent definition = agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: "", + tools: [new CodeInterpreterToolDefinition()], + toolResources: + new() + { + CodeInterpreter = new() + { + FileIds = { ... }, + } + })); + +AzureAIAgent agent = new(definition, agentsClient); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +from azure.ai.projects.models import CodeInterpreterTool + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + code_interpreter = CodeInterpreterTool() + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + tools=code_interpreter.definitions, + tool_resources=code_interpreter.resources, + ) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### File Search + +File search augments agents with knowledge from outside its model ([Azure AI Agent Service File Search Tool](/azure/ai-services/agents/how-to/tools/file-search)). + +::: zone pivot="programming-language-csharp" + +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +Agent definition = agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: "", + tools: [new FileSearchToolDefinition()], + toolResources: + new() + { + FileSearch = new() + { + VectorStoreIds = { ... }, + } + })); + +AzureAIAgent agent = new(definition, agentsClient); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +from azure.ai.projects.models import FileSearchTool + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + file_search = FileSearchTool(vector_store_ids=[vector_store.id]) + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + tools=file_search.definitions, + tool_resources=file_search.resources, + ) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### OpenAPI Integration + +Connects your agent to an external API ([How to use Azure AI Agent Service with OpenAPI Specified Tools](/azure/ai-services/agents/how-to/tools/openapi-spec)). + +::: zone pivot="programming-language-csharp" +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +string apiJsonSpecification = ...; // An Open API JSON specification + +Agent definition = agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: "", + tools: [ + new OpenApiToolDefinition( + "", + "", + BinaryData.FromString(apiJsonSpecification), + new OpenApiAnonymousAuthDetails()) + ], +); + +AzureAIAgent agent = new(definition, agentsClient); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +from azure.ai.projects.models import OpenApiTool, OpenApiAnonymousAuthDetails + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + openapi_spec_file_path = "sample/filepath/..." + with open(os.path.join(openapi_spec_file_path, "spec_one.json")) as file_one: + openapi_spec_one = json.loads(file_one.read()) + with open(os.path.join(openapi_spec_file_path, "spec_two.json")) as file_two: + openapi_spec_two = json.loads(file_two.read()) + + # Note that connection or managed identity auth setup requires additional setup in Azure + auth = OpenApiAnonymousAuthDetails() + openapi_tool_one = OpenApiTool( + name="", + spec=openapi_spec_one, + description="", + auth=auth, + ) + openapi_tool_two = OpenApiTool( + name="", + spec=openapi_spec_two, + description="", + auth=auth, + ) + + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + tools=openapi_tool_one.definitions + openapi_tool_two.definitions, + ) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### AzureAI Search Integration + +Use an existing Azure AI Search index with with your agent ([Use an existing AI Search index](/azure/ai-services/agents/how-to/tools/azure-ai-search)). + +::: zone pivot="programming-language-csharp" +```c# +AIProjectClient client = AzureAIAgent.CreateAzureAIClient("", new AzureCliCredential()); +AgentsClient agentsClient = client.GetAgentsClient(); + +ConnectionsClient cxnClient = client.GetConnectionsClient(); +ListConnectionsResponse searchConnections = await cxnClient.GetConnectionsAsync(AzureAIP.ConnectionType.AzureAISearch); +ConnectionResponse searchConnection = searchConnections.Value[0]; + +Agent definition = agentsClient.CreateAgentAsync( + "", + name: "", + description: "", + instructions: "", + tools: [new AzureAIP.AzureAISearchToolDefinition()], + toolResources: new() + { + AzureAISearch = new() + { + IndexList = { new AzureAIP.IndexResource(searchConnection.Id, "") } + } + }); + +AzureAIAgent agent = new(definition, agentsClient); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +from azure.ai.projects.models import AzureAISearchTool, ConnectionType + +async with ( + DefaultAzureCredential() as creds, + AzureAIAgent.create_client(credential=creds) as client, + ): + conn_list = await client.connections.list() + + ai_search_conn_id = "" + for conn in conn_list: + if conn.connection_type == ConnectionType.AZURE_AI_SEARCH: + ai_search_conn_id = conn.id + break + + ai_search = AzureAISearchTool( + index_connection_id=ai_search_conn_id, + index_name=AZURE_AI_SEARCH_INDEX_NAME, + ) + + agent_definition = await client.agents.create_agent( + model=ai_agent_settings.model_deployment_name, + instructions="Answer questions using your index.", + tools=ai_search.definitions, + tool_resources=ai_search.resources, + headers={"x-ms-enable-preview": "true"}, + ) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +### Retrieving an Existing `AzureAIAgent` + +An existing agent can be retrieved and reused by specifying its assistant ID: + +::: zone pivot="programming-language-csharp" + +```c# +Agent definition = agentsClient.GetAgentAsync(""); +AzureAIAgent agent = new(definition, agentsClient); +``` + +::: zone-end + +::: zone pivot="programming-language-python" +```python +agent_definition = await client.agents.get_agent(assistant_id="your-agent-id") +agent = AzureAIAgent(client=client, definition=agent_definition) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +## Deleting an `AzureAIAgent` + +Agents and their associated threads can be deleted when no longer needed: + +::: zone pivot="programming-language-csharp" + +```c# +await agentsClient.DeleteThreadAsync(thread.Id); +await agentsClient.DeleteAgentAsync(agent.Id); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +await client.agents.delete_thread(thread.id) +await client.agents.delete_agent(agent.id) +``` +::: zone-end + +If working with a vector store or files, they may be deleted as well: + +::: zone pivot="programming-language-csharp" +```c# +await agentsClient.DeleteVectorStoreAsync(""); +await agentsClient.DeleteFileAsync(""); +``` +::: zone-end + +::: zone pivot="programming-language-python" +```python +await client.agents.delete_file(file_id=file.id) +await client.agents.delete_vector_store(vector_store_id=vector_store.id) +``` +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +> More information on the _file search_ tool is described in the [Azure AI Agent Service file search tool](/azure/ai-services/agents/how-to/tools/file-search) article. + +## How-To + +For practical examples of using an `AzureAIAgent`, see our code samples on GitHub: + +::: zone pivot="programming-language-csharp" + +- [Getting Started with Azure AI Agents](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/GettingStartedWithAgents/AzureAIAgent) +- [Advanced Azure AI Agent Code Samples](https://github.com/microsoft/semantic-kernel/tree/main/dotnet/samples/Concepts/Agents) + +::: zone-end + +::: zone pivot="programming-language-python" + +- [Getting Started with Azure AI Agents](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/getting_started_with_agents/azure_ai_agent) +- [Advanced Azure AI Agent Code Samples](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/concepts/agents/azure_ai_agent) + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + +> [!div class="nextstepaction"] +> [Agent Collaboration in Agent Chat](./agent-chat.md) \ No newline at end of file diff --git a/semantic-kernel/Frameworks/agent/chat-completion-agent.md b/semantic-kernel/Frameworks/agent/chat-completion-agent.md index 0257948f..6ec5ff91 100644 --- a/semantic-kernel/Frameworks/agent/chat-completion-agent.md +++ b/semantic-kernel/Frameworks/agent/chat-completion-agent.md @@ -1,5 +1,5 @@ --- -title: Exploring the Semantic Kernel Chat Completion Agent +title: Exploring the Semantic Kernel ChatCompletionAgent description: An exploration of the definition, behaviors, and usage patterns for a Chat Completion Agent zone_pivot_groups: programming-languages author: crickman @@ -8,7 +8,7 @@ ms.author: crickman ms.date: 09/13/2024 ms.service: semantic-kernel --- -# Exploring the _Semantic Kernel_ Chat Completion Agent +# Exploring the Semantic Kernel `ChatCompletionAgent` > [!IMPORTANT] > This feature is in the release candidate stage. Features at this stage are nearly complete and generally stable, though they may undergo minor refinements or optimizations before reaching full general availability. @@ -25,8 +25,7 @@ Detailed API documentation related to this discussion is available at: ::: zone pivot="programming-language-python" -- [`chat_completion_agent`](/python/api/semantic-kernel/semantic_kernel.agents.chat_completion.chat_completion_agent) -- [`chat_completion_client_base`](/python/api/semantic-kernel/semantic_kernel.connectors.ai.chat_completion_client_base) +- [`ChatCompletionAgent`](/python/api/semantic-kernel/semantic_kernel.agents.chat_completion.chat_completion_agent.chatcompletionagent) ::: zone-end @@ -37,7 +36,7 @@ Detailed API documentation related to this discussion is available at: ::: zone-end -## Chat Completion in _Semantic Kernel_ +## Chat Completion in Semantic Kernel [_Chat Completion_](../../concepts/ai-services/chat-completion/index.md) is fundamentally a protocol for a chat-based interaction with an AI model where the chat-history maintained and presented to the model with each request. _Semantic Kernel_ [AI services](../../concepts/ai-services/index.md) offer a unified framework for integrating the chat-completion capabilities of various AI models. @@ -49,7 +48,7 @@ For .NET, _chat-completion_ AI Services are based on the [`IChatCompletionServic For .NET, some of AI services that support models with chat-completion include: -Model|_Semantic Kernel_ AI Service +Model|Semantic Kernel AI Service --|-- Azure OpenAI|[`Microsoft.SemanticKernel.Connectors.AzureOpenAI`](/dotnet/api/microsoft.semantickernel.connectors.azureopenai) Gemini|[`Microsoft.SemanticKernel.Connectors.Google`](/dotnet/api/microsoft.semantickernel.connectors.google) @@ -62,8 +61,8 @@ Onnx|[`Microsoft.SemanticKernel.Connectors.Onnx`](/dotnet/api/microsoft.semantic ::: zone pivot="programming-language-python" -- [`azure_chat_completion`](/python/api/semantic-kernel/semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion) -- [`open_ai_chat_completion`](/python/api/semantic-kernel/semantic_kernel.connectors.ai.open_ai.services.open_ai_chat_completion) +- [`AzureChatCompletion`](/python/api/semantic-kernel/semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion.azurechatcompletion) +- [`OpenAIChatCompletion`](/python/api/semantic-kernel/semantic_kernel.connectors.ai.open_ai.services.open_ai_chat_completion.openaichatcompletion) ::: zone-end @@ -74,9 +73,40 @@ Onnx|[`Microsoft.SemanticKernel.Connectors.Onnx`](/dotnet/api/microsoft.semantic ::: zone-end -## Creating a Chat Completion Agent +## Preparing Your Development Environment -A _chat completion agent_ is fundamentally based on an [AI services](../../concepts/ai-services/index.md). As such, creating an _chat completion agent_ starts with creating a [`Kernel`](../../concepts/kernel.md) instance that contains one or more chat-completion services and then instantiating the agent with a reference to that [`Kernel`](../../concepts/kernel.md) instance. +To proceed with developing an `AzureAIAgent`, configure your development environment with the appropriate packages. + +::: zone pivot="programming-language-csharp" + +Add the `Microsoft.SemanticKernel.Agents.Core` package to your project: + +```pwsh +dotnet add package Microsoft.SemanticKernel.Agents.Core --prerelease +``` + +::: zone-end + +::: zone pivot="programming-language-python" + +Install the `semantic-kernel` package: + +```bash +pip install semantic-kernel +``` + +::: zone-end + +::: zone pivot="programming-language-java" + +> Agents are currently unavailable in Java. + +::: zone-end + + +## Creating a `ChatCompletionAgent` + +A `ChatCompletionAgent` is fundamentally based on an [AI services](../../concepts/ai-services/index.md). As such, creating a `ChatCompletionAgent` starts with creating a [`Kernel`](../../concepts/kernel.md) instance that contains one or more chat-completion services and then instantiating the agent with a reference to that [`Kernel`](../../concepts/kernel.md) instance. ::: zone pivot="programming-language-csharp" ```csharp @@ -99,20 +129,35 @@ ChatCompletionAgent agent = ::: zone-end ::: zone pivot="programming-language-python" +There are two ways to create a `ChatCompletionAgent`: + +### 1. By providing the chat completion service directly: + ```python -# Define the Kernel +# Create the agent by directly providing the chat completion service +agent = ChatCompletionAgent( + service=AzureChatCompletion(), # your chat completion service instance + name="", + instructions="", +) +``` +### 2. By creating a Kernel first, adding the service to it, then providing the kernel: + +```python +# Define the kernel kernel = Kernel() -# Add the AzureChatCompletion AI Service to the Kernel +# Add the chat completion service to the kernel kernel.add_service(AzureChatCompletion()) -# Create the agent +# Create the agent using the kernel agent = ChatCompletionAgent( kernel=kernel, name="", instructions="", ) ``` +The first method is useful when you already have a chat completion service ready. The second method is beneficial when you need a kernel that manages multiple services or additional functionalities. ::: zone-end ::: zone pivot="programming-language-java" @@ -124,9 +169,9 @@ agent = ChatCompletionAgent( ## AI Service Selection -No different from using _Semantic Kernel_ [AI services](../../concepts/ai-services/index.md) directly, a `ChatCompletionAgent` supports the specification of a _service-selector_. A _service-selector_ identifies which [AI service](../../concepts/ai-services/index.md) to target when the [`Kernel`](../../concepts/kernel.md) contains more than one. +No different from using Semantic Kernel [AI services](../../concepts/ai-services/index.md) directly, a `ChatCompletionAgent` supports the specification of a service-selector. A service-selector identifies which [AI service](../../concepts/ai-services/index.md) to target when the [`Kernel`](../../concepts/kernel.md) contains more than one. -> Note: If multiple [AI services](../../concepts/ai-services/index.md) are present and no _service-selector_ is provided, the same _default_ logic is applied for the agent that you'd find when using an [AI services](../../concepts/ai-services/index.md) outside of the `Agent Framework` +> Note: If multiple [AI services](../../concepts/ai-services/index.md) are present and no service-selector is provided, the same default logic is applied for the agent that you'd find when using an [AI services](../../concepts/ai-services/index.md) outside of the `Agent Framework` ::: zone pivot="programming-language-csharp" ```csharp @@ -190,7 +235,7 @@ agent = ChatCompletionAgent( ::: zone pivot="programming-language-csharp" -Conversing with your `ChatCompletionAgent` is based on a `ChatHistory` instance, no different from interacting with a _Chat Completion_ [AI service](../../concepts/ai-services/index.md). +Conversing with your `ChatCompletionAgent` is based on a `ChatHistory` instance, no different from interacting with a Chat Completion [AI service](../../concepts/ai-services/index.md). ```csharp // Define agent @@ -224,8 +269,7 @@ agent = ChatCompletionAgent(...) chat = ChatHistory() # Add the user message -chat.add_message(ChatMessageContent(role=AuthorRole.USER, content=input)) - +chat.add_user_message(user_input) # Generate the agent response response = await agent.get_response(chat) # response is a `ChatMessageContent` object @@ -240,7 +284,7 @@ agent = ChatCompletionAgent(...) chat = ChatHistory() # Add the user message -chat.add_user_message(ChatMessageContent(role=AuthorRole.USER, content=input)) +chat.add_user_message(user_input) # Generate the agent response(s) async for response in agent.invoke(chat): @@ -281,4 +325,4 @@ For an end-to-end example for a `ChatCompletionAgent`, see: > [!div class="nextstepaction"] -> [Exploring `OpenAIAssistantAgent`](./assistant-agent.md) +> [Exploring the OpenAI Assistant Agent](./assistant-agent.md) diff --git a/semantic-kernel/Frameworks/agent/examples/example-agent-collaboration.md b/semantic-kernel/Frameworks/agent/examples/example-agent-collaboration.md index 4329e493..083d3841 100644 --- a/semantic-kernel/Frameworks/agent/examples/example-agent-collaboration.md +++ b/semantic-kernel/Frameworks/agent/examples/example-agent-collaboration.md @@ -862,8 +862,6 @@ Try using these suggested inputs: 8. its good, but is it ready for my college professor? ```csharp -// Copyright (c) Microsoft. All rights reserved. - using System; using System.ComponentModel; using System.Diagnostics; @@ -1131,8 +1129,6 @@ You can try using one of the suggested inputs. As the agent chat begins, the age > You can reference any file by providing `@`. To reference the "WomensSuffrage" text from above, download it [here](https://github.com/microsoft/semantic-kernel/blob/main/python/samples/learn_resources/resources/WomensSuffrage.txt) and place it in your current working directory. You can then reference it with `@WomensSuffrage.txt`. ```python -# Copyright (c) Microsoft. All rights reserved. - import asyncio import os diff --git a/semantic-kernel/Frameworks/agent/examples/example-assistant-code.md b/semantic-kernel/Frameworks/agent/examples/example-assistant-code.md index be6fdc4d..0ce0dfcc 100644 --- a/semantic-kernel/Frameworks/agent/examples/example-assistant-code.md +++ b/semantic-kernel/Frameworks/agent/examples/example-assistant-code.md @@ -208,13 +208,12 @@ The full example code is provided in the [Final](#final) section. Refer to that Prior to creating an `OpenAIAssistantAgent`, ensure the configuration settings are available and prepare the file resources. -Instantiate the `Settings` class referenced in the previous [Configuration](#configuration) section. Use the settings to also create an `OpenAIClientProvider` that will be used for the [Agent Definition](#agent-definition) as well as file-upload. +Instantiate the `Settings` class referenced in the previous [Configuration](#configuration) section. Use the settings to also create an `AzureOpenAIClient` that will be used for the [Agent Definition](#agent-definition) as well as file-upload. ```csharp Settings settings = new(); -OpenAIClientProvider clientProvider = - OpenAIClientProvider.ForAzureOpenAI(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); +AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); ``` ::: zone-end @@ -226,11 +225,11 @@ OpenAIClientProvider clientProvider = ::: zone pivot="programming-language-csharp" -Use the `OpenAIClientProvider` to access an `OpenAIFileClient` and upload the two data-files described in the previous [Configuration](#configuration) section, preserving the _File Reference_ for final clean-up. +Use the `AzureOpenAIClient` to access an `OpenAIFileClient` and upload the two data-files described in the previous [Configuration](#configuration) section, preserving the _File Reference_ for final clean-up. ```csharp Console.WriteLine("Uploading files..."); -OpenAIFileClient fileClient = clientProvider.Client.GetOpenAIFileClient(); +OpenAIFileClient fileClient = client.GetOpenAIFileClient(); OpenAIFile fileDataCountryDetail = await fileClient.UploadFileAsync("PopulationByAdmin1.csv", FileUploadPurpose.Assistants); OpenAIFile fileDataCountryList = await fileClient.UploadFileAsync("PopulationByCountry.csv", FileUploadPurpose.Assistants); ``` @@ -303,27 +302,27 @@ We first set up the Azure OpenAI resources to obtain the client and model. Next, ::: zone pivot="programming-language-csharp" -We are now ready to instantiate an `OpenAIAssistantAgent`. The agent is configured with its target model, _Instructions_, and the _Code Interpreter_ tool enabled. Additionally, we explicitly associate the two data files with the _Code Interpreter_ tool. +We are now ready to instantiate an `OpenAIAssistantAgent` by first creating an assistant definition. The assistant is configured with its target model, _Instructions_, and the _Code Interpreter_ tool enabled. Additionally, we explicitly associate the two data files with the _Code Interpreter_ tool. ```csharp Console.WriteLine("Defining agent..."); -OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.CreateAsync( - clientProvider, - new OpenAIAssistantDefinition(settings.AzureOpenAI.ChatModelDeployment) - { - Name = "SampleAssistantAgent", - Instructions = - """ - Analyze the available data to provide an answer to the user's question. - Always format response using markdown. - Always include a numerical index that starts at 1 for any lists or tables. - Always sort lists in ascending order. - """, - EnableCodeInterpreter = true, - CodeInterpreterFileIds = [fileDataCountryList.Id, fileDataCountryDetail.Id], - }, - new Kernel()); +AssistantClient assistantClient = client.GetAssistantClient(); + Assistant assistant = + await assistantClient.CreateAssistantAsync( + settings.AzureOpenAI.ChatModelDeployment, + name: "SampleAssistantAgent", + instructions: + """ + Analyze the available data to provide an answer to the user's question. + Always format response using markdown. + Always include a numerical index that starts at 1 for any lists or tables. + Always sort lists in ascending order. + """, + enableCodeInterpreter: true, + codeInterpreterFileIds: [fileDataCountryList.Id, fileDataCountryDetail.Id]); + +// Create agent +OpenAIAssistantAgent agent = new(assistant, assistantClient); ``` ::: zone-end @@ -355,7 +354,7 @@ Let's also ensure the resources are removed at the end of execution to minimize ::: zone pivot="programming-language-csharp" ```csharp Console.WriteLine("Creating thread..."); -string threadId = await agent.CreateThreadAsync(); +AssistantThread thread = await assistantClient.CreateThreadAsync(); Console.WriteLine("Ready!"); @@ -374,8 +373,8 @@ finally Console.WriteLine("Cleaning-up..."); await Task.WhenAll( [ - agent.DeleteThreadAsync(threadId), - agent.DeleteAsync(), + assistantClient.DeleteThreadAsync(thread.Id), + assistantClient.DeleteAssistantAsync(assistant.Id), fileClient.DeleteFileAsync(fileDataCountryList.Id), fileClient.DeleteFileAsync(fileDataCountryDetail.Id), ]); @@ -424,7 +423,7 @@ if (input.Trim().Equals("EXIT", StringComparison.OrdinalIgnoreCase)) break; } -await agent.AddChatMessageAsync(threadId, new ChatMessageContent(AuthorRole.User, input)); +await agent.AddChatMessageAsync(thread.Id, new ChatMessageContent(AuthorRole.User, input)); Console.WriteLine(); ``` @@ -542,7 +541,7 @@ To generate an `Agent` response to user input, invoke the agent by specifying th ::: zone pivot="programming-language-csharp" ```csharp bool isCode = false; -await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(threadId)) +await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(thread.Id)) { if (isCode != (response.Metadata?.ContainsKey(OpenAIAssistantAgent.CodeInterpreterMetadataKey) ?? false)) { @@ -604,17 +603,19 @@ Try using these suggested inputs: ::: zone pivot="programming-language-csharp" ```csharp +using Azure.AI.OpenAI; +using Azure.Identity; +using Microsoft.SemanticKernel; +using Microsoft.SemanticKernel.Agents.OpenAI; +using Microsoft.SemanticKernel.ChatCompletion; +using OpenAI.Assistants; +using OpenAI.Files; using System; using System.Collections.Generic; using System.Diagnostics; using System.IO; using System.Linq; using System.Threading.Tasks; -using Azure.Identity; -using Microsoft.SemanticKernel; -using Microsoft.SemanticKernel.Agents.OpenAI; -using Microsoft.SemanticKernel.ChatCompletion; -using OpenAI.Files; namespace AgentsSample; @@ -625,35 +626,39 @@ public static class Program // Load configuration from environment variables or user secrets. Settings settings = new(); - OpenAIClientProvider clientProvider = - OpenAIClientProvider.ForAzureOpenAI(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); + // Initialize the clients + AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); + //OpenAIClient client = OpenAIAssistantAgent.CreateOpenAIClient(new ApiKeyCredential(settings.OpenAI.ApiKey))); + AssistantClient assistantClient = client.GetAssistantClient(); + OpenAIFileClient fileClient = client.GetOpenAIFileClient(); + // Upload files Console.WriteLine("Uploading files..."); - OpenAIFileClient fileClient = clientProvider.Client.GetOpenAIFileClient(); OpenAIFile fileDataCountryDetail = await fileClient.UploadFileAsync("PopulationByAdmin1.csv", FileUploadPurpose.Assistants); OpenAIFile fileDataCountryList = await fileClient.UploadFileAsync("PopulationByCountry.csv", FileUploadPurpose.Assistants); - Console.WriteLine("Defining agent..."); - OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.CreateAsync( - clientProvider, - new OpenAIAssistantDefinition(settings.AzureOpenAI.ChatModelDeployment) - { - Name = "SampleAssistantAgent", - Instructions = + // Define assistant + Console.WriteLine("Defining assistant..."); + Assistant assistant = + await assistantClient.CreateAssistantAsync( + settings.AzureOpenAI.ChatModelDeployment, + name: "SampleAssistantAgent", + instructions: """ Analyze the available data to provide an answer to the user's question. Always format response using markdown. Always include a numerical index that starts at 1 for any lists or tables. Always sort lists in ascending order. """, - EnableCodeInterpreter = true, - CodeInterpreterFileIds = [fileDataCountryList.Id, fileDataCountryDetail.Id], - }, - new Kernel()); + enableCodeInterpreter: true, + codeInterpreterFileIds: [fileDataCountryList.Id, fileDataCountryDetail.Id]); + // Create agent + OpenAIAssistantAgent agent = new(assistant, assistantClient); + + // Create the conversation thread Console.WriteLine("Creating thread..."); - string threadId = await agent.CreateThreadAsync(); + AssistantThread thread = await assistantClient.CreateThreadAsync(); Console.WriteLine("Ready!"); @@ -676,12 +681,12 @@ public static class Program break; } - await agent.AddChatMessageAsync(threadId, new ChatMessageContent(AuthorRole.User, input)); + await agent.AddChatMessageAsync(thread.Id, new ChatMessageContent(AuthorRole.User, input)); Console.WriteLine(); bool isCode = false; - await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(threadId)) + await foreach (StreamingChatMessageContent response in agent.InvokeStreamingAsync(thread.Id)) { if (isCode != (response.Metadata?.ContainsKey(OpenAIAssistantAgent.CodeInterpreterMetadataKey) ?? false)) { @@ -709,8 +714,8 @@ public static class Program Console.WriteLine("Cleaning-up..."); await Task.WhenAll( [ - agent.DeleteThreadAsync(threadId), - agent.DeleteAsync(), + assistantClient.DeleteThreadAsync(thread.Id), + assistantClient.DeleteAssistantAsync(assistant.Id), fileClient.DeleteFileAsync(fileDataCountryList.Id), fileClient.DeleteFileAsync(fileDataCountryDetail.Id), ]); @@ -761,8 +766,6 @@ public static class Program ::: zone pivot="programming-language-python" ```python -# Copyright (c) Microsoft. All rights reserved. - import asyncio import logging import os diff --git a/semantic-kernel/Frameworks/agent/examples/example-assistant-search.md b/semantic-kernel/Frameworks/agent/examples/example-assistant-search.md index 001395a6..81bcc550 100644 --- a/semantic-kernel/Frameworks/agent/examples/example-assistant-search.md +++ b/semantic-kernel/Frameworks/agent/examples/example-assistant-search.md @@ -209,16 +209,12 @@ Prior to creating an `OpenAIAssistantAgent`, ensure the configuration settings a ::: zone pivot="programming-language-csharp" -Instantiate the `Settings` class referenced in the previous [Configuration](#configuration) section. Use the settings to also create an `OpenAIClientProvider` that will be used for the [Agent Definition](#agent-definition) as well as file-upload and the creation of a `VectorStore`. +Instantiate the `Settings` class referenced in the previous [Configuration](#configuration) section. Use the settings to also create an `AzureOpenAIClient` that will be used for the [Agent Definition](#agent-definition) as well as file-upload and the creation of a `VectorStore`. ```csharp - Settings settings = new(); -OpenAIClientProvider clientProvider = - OpenAIClientProvider.ForAzureOpenAI( - new AzureCliCredential(), - new Uri(settings.AzureOpenAI.Endpoint)); +AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); ``` ::: zone-end @@ -242,11 +238,11 @@ Now create an empty _Vector Store for use with the _File Search_ tool: ::: zone pivot="programming-language-csharp" -Use the `OpenAIClientProvider` to access a `VectorStoreClient` and create a `VectorStore`. +Use the `AzureOpenAIClient` to access a `VectorStoreClient` and create a `VectorStore`. ```csharp Console.WriteLine("Creating store..."); -VectorStoreClient storeClient = clientProvider.Client.GetVectorStoreClient(); +VectorStoreClient storeClient = client.GetVectorStoreClient(); CreateVectorStoreOperation operation = await storeClient.CreateVectorStoreAsync(waitUntilCompleted: true); string storeId = operation.VectorStoreId; ``` @@ -317,7 +313,7 @@ Now upload those files and add them to the _Vector Store_ by using the previousl Dictionary fileReferences = []; Console.WriteLine("Uploading files..."); -OpenAIFileClient fileClient = clientProvider.Client.GetOpenAIFileClient(); +OpenAIFileClient fileClient = client.GetOpenAIFileClient(); foreach (string fileName in _fileNames) { OpenAIFile fileInfo = await fileClient.UploadFileAsync(fileName, FileUploadPurpose.Assistants); @@ -339,27 +335,26 @@ We are now ready to instantiate an `OpenAIAssistantAgent`. The agent is configur ::: zone pivot="programming-language-csharp" -We will utilize the `OpenAIClientProvider` again as part of creating the `OpenAIAssistantAgent`: +We will utilize the `AzureOpenAIClient` again as part of creating the `OpenAIAssistantAgent`: ```csharp -Console.WriteLine("Defining agent..."); -OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.CreateAsync( - clientProvider, - new OpenAIAssistantDefinition(settings.AzureOpenAI.ChatModelDeployment) - { - Name = "SampleAssistantAgent", - Instructions = +Console.WriteLine("Defining assistant..."); +Assistant assistant = + await assistantClient.CreateAssistantAsync( + settings.AzureOpenAI.ChatModelDeployment, + name: "SampleAssistantAgent", + instructions: """ The document store contains the text of fictional stories. Always analyze the document store to provide an answer to the user's question. Never rely on your knowledge of stories not included in the document store. Always format response using markdown. """, - EnableFileSearch = true, - VectorStoreId = storeId, - }, - new Kernel()); + enableFileSearch: true, + vectorStoreId: storeId); + +// Create agent +OpenAIAssistantAgent agent = new(assistant, assistantClient); ``` ::: zone-end @@ -402,7 +397,7 @@ Let's also ensure the resources are removed at the end of execution to minimize ::: zone pivot="programming-language-csharp" ```csharp Console.WriteLine("Creating thread..."); -string threadId = await agent.CreateThreadAsync(); +AssistantThread thread = await assistantClient.CreateThreadAsync(); Console.WriteLine("Ready!"); @@ -420,8 +415,8 @@ finally Console.WriteLine("Cleaning-up..."); await Task.WhenAll( [ - agent.DeleteThreadAsync(threadId), - agent.DeleteAsync(), + assistantClient.DeleteThreadAsync(thread.Id), + assistantClient.DeleteAssistantAsync(assistant.Id), storeClient.DeleteVectorStoreAsync(storeId), ..fileReferences.Select(fileReference => fileClient.DeleteFileAsync(fileReference.Key)) ]); @@ -471,7 +466,7 @@ if (input.Trim().Equals("EXIT", StringComparison.OrdinalIgnoreCase)) break; } -await agent.AddChatMessageAsync(threadId, new ChatMessageContent(AuthorRole.User, input)); +await agent.AddChatMessageAsync(thread.Id, new ChatMessageContent(AuthorRole.User, input)); Console.WriteLine(); ``` ::: zone-end @@ -524,7 +519,7 @@ To generate an `Agent` response to user input, invoke the agent by specifying th ::: zone pivot="programming-language-csharp" ```csharp List footnotes = []; -await foreach (StreamingChatMessageContent chunk in agent.InvokeStreamingAsync(threadId)) +await foreach (StreamingChatMessageContent chunk in agent.InvokeStreamingAsync(thread.Id)) { // Capture annotations for footnotes footnotes.AddRange(chunk.Items.OfType()); @@ -585,16 +580,19 @@ Try using these suggested inputs: ::: zone pivot="programming-language-csharp" ```csharp -using System; -using System.Collections.Generic; -using System.Linq; -using System.Threading.Tasks; +using Azure.AI.OpenAI; using Azure.Identity; using Microsoft.SemanticKernel; +using Microsoft.SemanticKernel.Agents; using Microsoft.SemanticKernel.Agents.OpenAI; using Microsoft.SemanticKernel.ChatCompletion; +using OpenAI.Assistants; using OpenAI.Files; using OpenAI.VectorStores; +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; namespace AgentsSample; @@ -616,21 +614,21 @@ public static class Program // Load configuration from environment variables or user secrets. Settings settings = new(); - OpenAIClientProvider clientProvider = - OpenAIClientProvider.ForAzureOpenAI( - new AzureCliCredential(), - new Uri(settings.AzureOpenAI.Endpoint)); + // Initialize the clients + AzureOpenAIClient client = OpenAIAssistantAgent.CreateAzureOpenAIClient(new AzureCliCredential(), new Uri(settings.AzureOpenAI.Endpoint)); + //OpenAIClient client = OpenAIAssistantAgent.CreateOpenAIClient(new ApiKeyCredential(settings.OpenAI.ApiKey))); + AssistantClient assistantClient = client.GetAssistantClient(); + OpenAIFileClient fileClient = client.GetOpenAIFileClient(); + VectorStoreClient storeClient = client.GetVectorStoreClient(); + // Create the vector store Console.WriteLine("Creating store..."); - VectorStoreClient storeClient = clientProvider.Client.GetVectorStoreClient(); CreateVectorStoreOperation operation = await storeClient.CreateVectorStoreAsync(waitUntilCompleted: true); string storeId = operation.VectorStoreId; - // Retain file references. - Dictionary fileReferences = []; - + // Upload files and retain file references. Console.WriteLine("Uploading files..."); - OpenAIFileClient fileClient = clientProvider.Client.GetOpenAIFileClient(); + Dictionary fileReferences = []; foreach (string fileName in _fileNames) { OpenAIFile fileInfo = await fileClient.UploadFileAsync(fileName, FileUploadPurpose.Assistants); @@ -638,28 +636,28 @@ public static class Program fileReferences.Add(fileInfo.Id, fileInfo); } - - Console.WriteLine("Defining agent..."); - OpenAIAssistantAgent agent = - await OpenAIAssistantAgent.CreateAsync( - clientProvider, - new OpenAIAssistantDefinition(settings.AzureOpenAI.ChatModelDeployment) - { - Name = "SampleAssistantAgent", - Instructions = + // Define assistant + Console.WriteLine("Defining assistant..."); + Assistant assistant = + await assistantClient.CreateAssistantAsync( + settings.AzureOpenAI.ChatModelDeployment, + name: "SampleAssistantAgent", + instructions: """ The document store contains the text of fictional stories. Always analyze the document store to provide an answer to the user's question. Never rely on your knowledge of stories not included in the document store. Always format response using markdown. """, - EnableFileSearch = true, - VectorStoreId = storeId, - }, - new Kernel()); + enableFileSearch: true, + vectorStoreId: storeId); + // Create agent + OpenAIAssistantAgent agent = new(assistant, assistantClient); + + // Create the conversation thread Console.WriteLine("Creating thread..."); - string threadId = await agent.CreateThreadAsync(); + AssistantThread thread = await assistantClient.CreateThreadAsync(); Console.WriteLine("Ready!"); @@ -681,11 +679,11 @@ public static class Program break; } - await agent.AddChatMessageAsync(threadId, new ChatMessageContent(AuthorRole.User, input)); + await agent.AddChatMessageAsync(thread.Id, new ChatMessageContent(AuthorRole.User, input)); Console.WriteLine(); List footnotes = []; - await foreach (StreamingChatMessageContent chunk in agent.InvokeStreamingAsync(threadId)) + await foreach (StreamingChatMessageContent chunk in agent.InvokeStreamingAsync(thread.Id)) { // Capture annotations for footnotes footnotes.AddRange(chunk.Items.OfType()); @@ -713,8 +711,8 @@ public static class Program Console.WriteLine("Cleaning-up..."); await Task.WhenAll( [ - agent.DeleteThreadAsync(threadId), - agent.DeleteAsync(), + assistantClient.DeleteThreadAsync(thread.Id), + assistantClient.DeleteAssistantAsync(assistant.Id), storeClient.DeleteVectorStoreAsync(storeId), ..fileReferences.Select(fileReference => fileClient.DeleteFileAsync(fileReference.Key)) ]); @@ -729,8 +727,6 @@ public static class Program ::: zone pivot="programming-language-python" ```python -# Copyright (c) Microsoft. All rights reserved. - import asyncio import os diff --git a/semantic-kernel/Frameworks/agent/index.md b/semantic-kernel/Frameworks/agent/index.md index 749207bb..165fee79 100644 --- a/semantic-kernel/Frameworks/agent/index.md +++ b/semantic-kernel/Frameworks/agent/index.md @@ -17,6 +17,11 @@ The Semantic Kernel Agent Framework provides a platform within the Semantic Kern ## What is an AI agent? +![alt text](../../media/agentSKdocs.png) +![alt text](../../media/agentSKdocs2.png) +![alt text](../../media/agentSKdocs3.png) +![alt text](../../media/agentSKdocs4.png) + An **AI agent** is a software entity designed to perform tasks autonomously or semi-autonomously by recieving input, processing information, and taking actions to achieve specific goals. Agents can send and receive messages, generating responses using a combination of models, tools, human inputs, or other customizable components. diff --git a/semantic-kernel/concepts/ai-services/TOC.yml b/semantic-kernel/concepts/ai-services/TOC.yml index 8d24a54b..fd35b379 100644 --- a/semantic-kernel/concepts/ai-services/TOC.yml +++ b/semantic-kernel/concepts/ai-services/TOC.yml @@ -6,4 +6,6 @@ - name: Embedding generation href: embedding-generation/TOC.yml - name: AI Integrations - href: integrations.md \ No newline at end of file + href: integrations.md +- name: Realtime + href: realtime.md \ No newline at end of file diff --git a/semantic-kernel/concepts/ai-services/index.md b/semantic-kernel/concepts/ai-services/index.md index 3e473aec..9ce5d2fd 100644 --- a/semantic-kernel/concepts/ai-services/index.md +++ b/semantic-kernel/concepts/ai-services/index.md @@ -14,21 +14,23 @@ One of the main features of Semantic Kernel is its ability to add different AI s Within Semantic Kernel, there are interfaces for the most popular AI tasks. In the table below, you can see the services that are supported by each of the SDKs. -| Services | C# | Python | Java | Notes | -|-----------------------------------|:----:|:------:|:----:|-------| -| [Chat completion](./chat-completion/index.md) | ✅ | ✅ | ✅ | -| Text generation | ✅ | ✅ | ✅ | -| Embedding generation (Experimental) | ✅ | ✅ | ✅ | -| Text-to-image (Experimental) | ✅ | ✅ | ❌ | -| Image-to-text (Experimental) | ✅ | ❌ | ❌ | -| Text-to-audio (Experimental) | ✅ | ✅ | ❌ | -| Audio-to-text (Experimental) | ✅ | ✅ | ❌ | +| Services | C# | Python | Java | Notes | +| --------------------------------------------- | :---: | :----: | :---: | ----- | +| [Chat completion](./chat-completion/index.md) | ✅ | ✅ | ✅ | +| Text generation | ✅ | ✅ | ✅ | +| Embedding generation (Experimental) | ✅ | ✅ | ✅ | +| Text-to-image (Experimental) | ✅ | ✅ | ❌ | +| Image-to-text (Experimental) | ✅ | ❌ | ❌ | +| Text-to-audio (Experimental) | ✅ | ✅ | ❌ | +| Audio-to-text (Experimental) | ✅ | ✅ | ❌ | +| [Realtime](./realtime.md) (Experimental) | ❌ | ✅ | ❌ | > [!TIP] > In most scenarios, you will only need to add chat completion to your kernel, but to support multi-modal AI, you can add any of the above services to your kernel. ## Next steps + To learn more about each of the services, please refer to the specific articles for each service type. In each of the articles we provide sample code for adding the service to the kernel across multiple AI service providers. > [!div class="nextstepaction"] -> [Learn about chat completion](./chat-completion/index.md) \ No newline at end of file +> [Learn about chat completion](./chat-completion/index.md) diff --git a/semantic-kernel/concepts/ai-services/integrations.md b/semantic-kernel/concepts/ai-services/integrations.md index a8d34ec5..d30a6984 100644 --- a/semantic-kernel/concepts/ai-services/integrations.md +++ b/semantic-kernel/concepts/ai-services/integrations.md @@ -18,21 +18,22 @@ With the available AI connectors, developers can easily build AI agents with swa ### AI Services -| Services | C# | Python | Java | Notes | -|-----------------------------------|:----:|:------:|:----:|-------| -| Text Generation | ✅ | ✅ | ✅ | Example: Text-Davinci-003 | -| Chat Completion | ✅ | ✅ | ✅ | Example: GPT4, Chat-GPT | -| Text Embeddings (Experimental) | ✅ | ✅ | ✅ | Example: Text-Embeddings-Ada-002 | -| Text to Image (Experimental) | ✅ | ✅ | ❌ | Example: Dall-E | -| Image to Text (Experimental) | ✅ | ❌ | ❌ | Example: Pix2Struct | -| Text to Audio (Experimental) | ✅ | ✅ | ❌ | Example: Text-to-speech | -| Audio to Text (Experimental) | ✅ | ✅ | ❌ | Example: Whisper | +| Services | C# | Python | Java | Notes | +| ------------------------------ | :---: | :----: | :---: | -------------------------------- | +| Text Generation | ✅ | ✅ | ✅ | Example: Text-Davinci-003 | +| Chat Completion | ✅ | ✅ | ✅ | Example: GPT4, Chat-GPT | +| Text Embeddings (Experimental) | ✅ | ✅ | ✅ | Example: Text-Embeddings-Ada-002 | +| Text to Image (Experimental) | ✅ | ✅ | ❌ | Example: Dall-E | +| Image to Text (Experimental) | ✅ | ❌ | ❌ | Example: Pix2Struct | +| Text to Audio (Experimental) | ✅ | ✅ | ❌ | Example: Text-to-speech | +| Audio to Text (Experimental) | ✅ | ✅ | ❌ | Example: Whisper | +| Realtime (Experimental) | ❌ | ✅ | ❌ | Example: gpt-4o-realtime-preview | ## Additional plugins If you want to extend the functionality of your AI agent, you can use plugins to integrate with other Microsoft services. Here are some of the plugins that are available for Semantic Kernel: -| Plugin | C# | Python | Java | Description | -| ---------- | :-: | :----: | :--: | ----------- | -| Logic Apps | ✅ | ✅ | ✅ | Build workflows within Logic Apps using its available connectors and import them as plugins in Semantic Kernel. [Learn more](../plugins/adding-logic-apps-as-plugins.md). | -| Azure Container Apps Dynamic Sessions | ✅ | ✅ | ❌ | With dynamic sessions, you can recreate the Code Interpreter experience from the Assistants API by effortlessly spinning up Python containers where AI agents can execute Python code. [Learn more](/azure/container-apps/sessions). | +| Plugin | C# | Python | Java | Description | +| ------------------------------------- | :---: | :----: | :---: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Logic Apps | ✅ | ✅ | ✅ | Build workflows within Logic Apps using its available connectors and import them as plugins in Semantic Kernel. [Learn more](../plugins/adding-logic-apps-as-plugins.md). | +| Azure Container Apps Dynamic Sessions | ✅ | ✅ | ❌ | With dynamic sessions, you can recreate the Code Interpreter experience from the Assistants API by effortlessly spinning up Python containers where AI agents can execute Python code. [Learn more](/azure/container-apps/sessions). | diff --git a/semantic-kernel/concepts/ai-services/realtime.md b/semantic-kernel/concepts/ai-services/realtime.md new file mode 100644 index 00000000..dd559e91 --- /dev/null +++ b/semantic-kernel/concepts/ai-services/realtime.md @@ -0,0 +1,189 @@ +--- +title: Realtime AI Integrations for Semantic Kernel +description: Learn about realtime multi-modal AI integrations available in Semantic Kernel. +author: eavanvalkenburg +ms.topic: conceptual +ms.author: edvan +ms.date: 02/26/2025 +ms.service: semantic-kernel +--- + +# Realtime Multi-modal APIs + +The first realtime API integration for Semantic Kernel has been added, it is currently only available in Python and considered experimental. This is because the underlying services are still being developed and are subject to changes and we might need to make breaking changes to the API in Semantic Kernel as we learn from customers how to use this and as we add other providers of these kinds of models and APIs. + +## Realtime Client abstraction + +To support different realtime APIs from different vendors, using different protocols, a new client abstraction has been added to the kernel. This client is used to connect to the realtime service and send and receive messages. +The client is responsible for handling the connection to the service, sending messages, and receiving messages. The client is also responsible for handling any errors that occur during the connection or message sending/receiving process. Considering the way these models work, they can be considered agents more than regular chat completions, therefore they also take instructions, rather than a system message, they keep their own internal state and can be invoked to do work on our behalf. +### Realtime API + +Any realtime client implements the following methods: + +| Method | Description | +| ---------------- | ------------------------------------------------------------------------------------------------------------------ | +| `create_session` | Creates a new session | +| `update_session` | Updates an existing session | +| `delete_session` | Deletes an existing session | +| `receive` | This is a asynchronous generator method that listens for messages from the service and yields them as they arrive. | +| `send` | Sends a message to the service | + +### Python implementations + +The python version of Semantic Kernel currently supports the following realtime clients: + +| Client | Protocol | Modalities | Function calling enabled | Description | +| ------ | --------- | ------------ | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| OpenAI | Websocket | Text & Audio | Yes | The OpenAI Realtime API is a websocket based api that allows you to send and receive messages in realtime, this connector uses the OpenAI Python package to connect and receive and send messages. | +| OpenAI | WebRTC | Text & Audio | Yes | The OpenAI Realtime API is a WebRTC based api that allows you to send and receive messages in realtime, it needs a webRTC compatible audio track at session creation time. | +| Azure | Websocket | Text & Audio | Yes | The Azure Realtime API is a websocket based api that allows you to send and receive messages in realtime, this uses the same package as the OpenAI websocket connector. | + +## Getting started + +To get started with the Realtime API, you need to install the `semantic-kernel` package with the `realtime` extra. + +```bash +pip install semantic-kernel[realtime] +``` + +Depending on how you want to handle audio, you might need additional packages to interface with speakers and microphones, like `pyaudio` or `sounddevice`. + +### Websocket clients + +Then you can create a kernel and add the realtime client to it, this shows how to do that with a AzureRealtimeWebsocket connection, you can replace AzureRealtimeWebsocket with OpenAIRealtimeWebsocket without any further changes. + +```python +from semantic_kernel.connectors.ai.open_ai import ( + AzureRealtimeWebsocket, + AzureRealtimeExecutionSettings, + ListenEvents, +) +from semantic_kernel.contents import RealtimeAudioEvent, RealtimeTextEvent + +# this will use environment variables to get the api key, endpoint, api version and deployment name. +realtime_client = AzureRealtimeWebsocket() +settings = AzureRealtimeExecutionSettings(voice='alloy') +async with realtime_client(settings=settings, create_response=True): + async for event in realtime_client.receive(): + match event: + # receiving a piece of audio (and send it to a undefined audio player) + case RealtimeAudioEvent(): + await audio_player.add_audio(event.audio) + # receiving a piece of audio transcript + case RealtimeTextEvent(): + # Semantic Kernel parses the transcript to a TextContent object captured in a RealtimeTextEvent + print(event.text.text, end="") + case _: + # OpenAI Specific events + if event.service_type == ListenEvents.SESSION_UPDATED: + print("Session updated") + if event.service_type == ListenEvents.RESPONSE_CREATED: + print("\nMosscap (transcript): ", end="") +``` + +There are two important things to note, the first is that the `realtime_client` is an async context manager, this means that you can use it in an async function and use `async with` to create the session. +The second is that the `receive` method is an async generator, this means that you can use it in a for loop to receive messages as they arrive. + +### WebRTC client + +The setup of a WebRTC connection is a bit more complex and so we need a extra parameter when creating the client. This parameter, `audio_track` needs to be a object that implements the `MediaStreamTrack` protocol of the `aiortc` package, this is also demonstrated in the samples that are linked below. + +To create a client that uses WebRTC, you would do the following: + +```python +from semantic_kernel.connectors.ai.open_ai import ( + ListenEvents, + OpenAIRealtimeExecutionSettings, + OpenAIRealtimeWebRTC, +) +from aiortc.mediastreams import MediaStreamTrack + +class AudioRecorderWebRTC(MediaStreamTrack): + # implement the MediaStreamTrack methods. + +realtime_client = OpenAIRealtimeWebRTC(audio_track=AudioRecorderWebRTC()) +# Create the settings for the session +settings = OpenAIRealtimeExecutionSettings( + instructions=""" +You are a chat bot. Your name is Mosscap and +you have one goal: figure out what people need. +Your full name, should you need to know it, is +Splendid Speckled Mosscap. You communicate +effectively, but you tend to answer with long +flowery prose. +""", + voice="shimmer", +) +audio_player = AudioPlayer +async with realtime_client(settings=settings, create_response=True): + async for event in realtime_client.receive(): + match event.event_type: + # receiving a piece of audio (and send it to a undefined audio player) + case "audio": + await audio_player.add_audio(event.audio) + case "text": + # the model returns both audio and transcript of the audio, which we will print + print(event.text.text, end="") + case "service": + # OpenAI Specific events + if event.service_type == ListenEvents.SESSION_UPDATED: + print("Session updated") + if event.service_type == ListenEvents.RESPONSE_CREATED: + print("\nMosscap (transcript): ", end="") +``` + +Both of these samples receive the audio as RealtimeAudioEvent and then they pass that to a unspecified audio_player object. + +### Audio output callback + +Next to this we have a parameter called `audio_output_callback` on the `receive` method and on the class creation. This callback will be called first before any further handling of the audio and gets a `numpy` array of the audio data, instead of it being parsed into AudioContent and returned as a RealtimeAudioEvent that you can then handle, which is what happens above. This has shown to give smoother audio output because there is less overhead between the audio data coming in and it being given to the player. + +This example shows how to define and use the `audio_output_callback`: + +```python +from semantic_kernel.connectors.ai.open_ai import ( + ListenEvents, + OpenAIRealtimeExecutionSettings, + OpenAIRealtimeWebRTC, +) +from aiortc.mediastreams import MediaStreamTrack + +class AudioRecorderWebRTC(MediaStreamTrack): + # implement the MediaStreamTrack methods. + +class AudioPlayer: + async def play_audio(self, content: np.ndarray): + # implement the audio player + +realtime_client = OpenAIRealtimeWebRTC(audio_track=AudioRecorderWebRTC()) +# Create the settings for the session +settings = OpenAIRealtimeExecutionSettings( + instructions=""" +You are a chat bot. Your name is Mosscap and +you have one goal: figure out what people need. +Your full name, should you need to know it, is +Splendid Speckled Mosscap. You communicate +effectively, but you tend to answer with long +flowery prose. +""", + voice="shimmer", +) +audio_player = AudioPlayer +async with realtime_client(settings=settings, create_response=True): + async for event in realtime_client.receive(audio_output_callback=audio_player.play_audio): + match event.event_type: + # no need to handle case: "audio" + case "text": + # the model returns both audio and transcript of the audio, which we will print + print(event.text.text, end="") + case "service": + # OpenAI Specific events + if event.service_type == ListenEvents.SESSION_UPDATED: + print("Session updated") + if event.service_type == ListenEvents.RESPONSE_CREATED: + print("\nMosscap (transcript): ", end="") +``` + +### Samples + +There are four samples in [our repo](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/concepts/realtime), they cover both the basics using both websockets and WebRTC, as well as a more complex setup including function calling. Finally there is a more [complex demo](https://github.com/microsoft/semantic-kernel/tree/main/python/samples/demos/call_automation) that uses [Azure Communication Services](/azure/communication-services/) to allow you to call your Semantic Kernel enhanced realtime API. diff --git a/semantic-kernel/concepts/plugins/adding-native-plugins.md b/semantic-kernel/concepts/plugins/adding-native-plugins.md index 68d3dcba..f2f8be4e 100644 --- a/semantic-kernel/concepts/plugins/adding-native-plugins.md +++ b/semantic-kernel/concepts/plugins/adding-native-plugins.md @@ -18,6 +18,7 @@ Behind the scenes, Semantic Kernel will then use the descriptions you provide, a ## Providing the LLM with the right information When authoring a plugin, you need to provide the AI agent with the right information to understand the capabilities of the plugin and its functions. This includes: + - The name of the plugin - The names of the functions - The descriptions of the functions @@ -34,6 +35,7 @@ Below, we'll walk through the two different ways of providing your AI agent with The easiest way to create a native plugin is to start with a class and then add methods annotated with the `KernelFunction` attribute. It is also recommended to liberally use the `Description` annotation to provide the AI agent with the necessary information to understand the function. ::: zone pivot="programming-language-csharp" + ```csharp public class LightsPlugin { @@ -73,9 +75,11 @@ public class LightsPlugin } } ``` + ::: zone-end ::: zone pivot="programming-language-python" + ```python from typing import List, Optional, Annotated @@ -102,6 +106,7 @@ class LightsPlugin: return light return None ``` + ::: zone-end ::: zone pivot="programming-language-java" @@ -110,10 +115,10 @@ class LightsPlugin: ::: zone-end +::: zone pivot="programming-language-csharp" > [!TIP] > Because the LLMs are predominantly trained on Python code, it is recommended to use snake_case for function names and parameters (even if you're using C# or Java). This will help the AI agent better understand the function and its parameters. -::: zone pivot="programming-language-csharp" > [!TIP] > Your functions can specify `Kernel`, `KernelArguments`, `ILoggerFactory`, `ILogger`, `IAIServiceSelector`, `CultureInfo`, `IFormatProvider`, `CancellationToken` as parameters and these will not be advertised to the LLM and will be automatically set when the function is called. > If you rely on `KernelArguments` instead of explicit input arguments then your code will be responsible for performing type conversions. @@ -122,6 +127,7 @@ class LightsPlugin: If your function has a complex object as an input variable, Semantic Kernel will also generate a schema for that object and pass it to the AI agent. Similar to functions, you should provide `Description` annotations for properties that are non-obvious to the AI. Below is the definition for the `LightState` class and the `Brightness` enum. ::: zone pivot="programming-language-csharp" + ```csharp using System.Text.Json.Serialization; @@ -152,19 +158,28 @@ public enum Brightness High } ``` + ::: zone-end ::: zone pivot="programming-language-python" + ```python +from enum import Enum from typing import TypedDict +class Brightness(Enum): + LOW = "low" + MEDIUM = "medium" + HIGH = "high" + class LightModel(TypedDict): id: int name: str is_on: bool | None - brightness: int | None - hex: str | None + brightness: Brightness | None + color: Annotated[str | None, "The color of the light with a hex code (ensure you include the # symbol)"] ``` + ::: zone-end ::: zone pivot="programming-language-java" @@ -176,12 +191,29 @@ class LightModel(TypedDict): > [!NOTE] > While this is a "fun" example, it does a good job showing just how complex a plugin's parameters can be. In this single case, we have a complex object with _four_ different types of properties: an integer, string, boolean, and enum. Semantic Kernel's value is that it can automatically generate the schema for this object and pass it to the AI agent and marshal the parameters generated by the AI agent into the correct object. +::: zone pivot="programming-language-csharp" + Once you're done authoring your plugin class, you can add it to the kernel using the `AddFromType<>` or `AddFromObject` methods. +::: zone-end + +::: zone pivot="programming-language-python" + +Once you're done authoring your plugin class, you can add it to the kernel using the `add_plugin` method. + +::: zone-end + +::: zone pivot="programming-language-java" + +Once you're done authoring your plugin class, you can add it to the kernel using the `AddFromType<>` or `AddFromObject` methods. + +::: zone-end + > [!TIP] > When creating a function, always ask yourself "how can I give the AI additional help to use this function?" This can include using specific input types (avoid strings where possible), providing descriptions, and examples. ::: zone pivot="programming-language-csharp" + #### Adding a plugin using the `AddFromObject` method The `AddFromObject` method allows you to add an instance of the plugin class directly to the plugin collection in case you want to directly control how the plugin is constructed. @@ -325,9 +357,11 @@ builder.Services.AddTransient((serviceProvider)=> { return new Kernel(serviceProvider, pluginCollection); }); ``` + ::: zone-end ::: zone pivot="programming-language-python" + #### Adding a plugin using the `add_plugin` method The `add_plugin` method allows you to add a plugin instance to the kernel. Below is an example of how you can construct the `LightsPlugin` class and add it to the kernel. @@ -349,8 +383,8 @@ lights_plugin = LightsPlugin(lights) # Add the plugin to the kernel kernel.add_plugin(lights_plugin) ``` -::: zone-end +::: zone-end ::: zone pivot="programming-language-java" @@ -497,7 +531,47 @@ This approach eliminates the need to manually provide and update the return type ::: zone-end +::: zone pivot="programming-language-python" + +### Providing more details about the functions + +When creating a plugin in Python, you can provide additional information about the functions in the `kernel_function` decorator. This information will be used by the AI agent to understand the functions better. + +```python +from typing import List, Optional, Annotated + +class LightsPlugin: + def __init__(self, lights: List[LightModel]): + self._lights = lights + + @kernel_function(name="GetLights", description="Gets a list of lights and their current state") + async def get_lights(self) -> List[LightModel]: + """Gets a list of lights and their current state.""" + return self._lights + + @kernel_function(name="ChangeState", description="Changes the state of the light") + async def change_state( + self, + change_state: LightModel + ) -> Optional[LightModel]: + """Changes the state of the light.""" + for light in self._lights: + if light["id"] == change_state["id"]: + light["is_on"] = change_state.get("is_on", light["is_on"]) + light["brightness"] = change_state.get("brightness", light["brightness"]) + light["hex"] = change_state.get("hex", light["hex"]) + return light + return None +``` + +The sample above shows how to override the function name and provide a description for the function. By default, the function name is the name of the function and the description is empty. If the function name is descriptive enough, you won't need a description, which will save you tokens. However, if the function behavior is not obvious from the name, you should provide a description for the AI. + +Because the LLMs are predominantly trained on Python code, it is recommended to use function names that follow the [Python naming conventions](https://peps.python.org/pep-0008/#function-and-variable-names), which means you rarely need to override the function names if you follow the conventions in your Python code. + +::: zone-end + ## Next steps + Now that you know how to create a plugin, you can now learn how to use them with your AI agent. Depending on the type of functions you've added to your plugins, there are different patterns you should follow. For retrieval functions, refer to the [using retrieval functions](./using-data-retrieval-functions-for-rag.md) article. For task automation functions, refer to the [using task automation functions](./using-task-automation-functions.md) article. > [!div class="nextstepaction"] diff --git a/semantic-kernel/concepts/vector-store-connectors/TOC.yml b/semantic-kernel/concepts/vector-store-connectors/TOC.yml index b7cfa81b..85f04702 100644 --- a/semantic-kernel/concepts/vector-store-connectors/TOC.yml +++ b/semantic-kernel/concepts/vector-store-connectors/TOC.yml @@ -12,6 +12,8 @@ href: embedding-generation.md - name: Vector Search href: vector-search.md +- name: Hybrid Search + href: hybrid-search.md - name: Serialization of data models href: serialization.md - name: Legacy Memory Stores diff --git a/semantic-kernel/concepts/vector-store-connectors/embedding-generation.md b/semantic-kernel/concepts/vector-store-connectors/embedding-generation.md index 736d5d59..f3769bec 100644 --- a/semantic-kernel/concepts/vector-store-connectors/embedding-generation.md +++ b/semantic-kernel/concepts/vector-store-connectors/embedding-generation.md @@ -99,11 +99,12 @@ public async Task GenerateEmbeddingsAndSearchAsync( await textEmbeddingGenerationService.GenerateEmbeddingAsync(descriptionText); // Search using the already generated embedding. - List> searchResult = await collection.VectorizedSearchAsync(searchEmbedding).ToListAsync(); + VectorSearchResults searchResult = await collection.VectorizedSearchAsync(searchEmbedding); + List> resultItems = await searchResult.Results.ToListAsync(); // Print the first search result. - Console.WriteLine("Score for first result: " + searchResult.FirstOrDefault()?.Score); - Console.WriteLine("Hotel description for first result: " + searchResult.FirstOrDefault()?.Record.Description); + Console.WriteLine("Score for first result: " + resultItems.FirstOrDefault()?.Score); + Console.WriteLine("Hotel description for first result: " + resultItems.FirstOrDefault()?.Record.Description); } ``` diff --git a/semantic-kernel/concepts/vector-store-connectors/how-to/build-your-own-connector.md b/semantic-kernel/concepts/vector-store-connectors/how-to/build-your-own-connector.md index fd95fc28..9f6faeaf 100644 --- a/semantic-kernel/concepts/vector-store-connectors/how-to/build-your-own-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/how-to/build-your-own-connector.md @@ -107,26 +107,31 @@ There may be cases where the database doesn't support excluding vectors in which returning them is acceptable. 1.9 *`IVectorizedSearch.VectorizedSearchAsync`* implementations should also -respect the `IncludeVectors` option provided via `VectorSearchOptions` where possible. +respect the `IncludeVectors` option provided via `VectorSearchOptions` where possible. 1.10 *`IVectorizedSearch.VectorizedSearchAsync`* implementations should simulate -the `Top` and `Skip` functionality requested via `VectorSearchOptions` if the database +the `Top` and `Skip` functionality requested via `VectorSearchOptions` if the database does not support this natively. To simulate this behavior, the implementation should fetch a number of results equal to Top + Skip, and then skip the first Skip number of results before returning the remaining results. 1.11 *`IVectorizedSearch.VectorizedSearchAsync`* implementations should ignore -the `IncludeTotalCount` option provided via `VectorSearchOptions` if the database +the `IncludeTotalCount` option provided via `VectorSearchOptions` if the database does not support this natively. -1.12 *`IVectorizedSearch.VectorizedSearchAsync`* implementations should default -to the first vector if the `VectorPropertyName` option was not provided via `VectorSearchOptions`. -If a user does provide this value, the expected name should be the property name from the data model -and not any customized name that the property may be stored under in the database. E.g. let's say -the user has a data model property called `TextEmbedding` and they decorated the property with a -`JsonPropertyNameAttribute` that indicates that it should be serialized as `text_embedding`. Assuming -that the database is json based, it means that the property should be stored in the database with the -name `text_embedding`. When specifying the `VectorPropertyName` option, the user should always provide +1.12 *`IVectorizedSearch.VectorizedSearchAsync`* implementations should not require +`VectorPropertyName` or `VectorProperty` to be specified if only one vector exists on the data model. +In this case that single vector should automatically become the search target. If no vector or +multiple vectors exists on the data model, and no `VectorPropertyName` or `VectorProperty` is provided +the search method should throw. + +When using `VectorPropertyName`, if a user does provide this value, the expected name should be the +property name from the data model and not any customized name that the property may be stored under +in the database. E.g. let's say the user has a data model property called `TextEmbedding` and they +decorated the property with a `JsonPropertyNameAttribute` that indicates that it should be serialized +as `text_embedding`. Assuming that the database is json based, it means that the property should be +stored in the database with the name `text_embedding`. + When specifying the `VectorPropertyName` option, the user should always provide `TextEmbedding` as the value. This is to ensure that where different connectors are used with the same data model, the user can always use the same property names, even though the storage name of the property may be different. @@ -299,7 +304,12 @@ To support this scenario, the connector must fulfil the following requirements: - Use the `VectorStoreRecordDefinition` to create collections / indexes. - Avoid doing reflection on the data model if a custom mapper and `VectorStoreRecordDefinition` is supplied -### 10. Support Vector Store Record Collection factory +### 10. Support Vector Store Record Collection factory (Deprecated) + +> [!IMPORTANT] +> Support for Vector Store Record Collection factories is now deprecated. The recommended pattern is to unseal +> the VectorStore class and make the `GetCollection` method virtual so that it can be overridden by developers +> who require custom construction of collections. The `IVectorStore.GetCollection` method can be used to create instances of `IVectorStoreRecordCollection`. Some connectors however may allow or require users to provide additional configuration options @@ -404,7 +414,7 @@ into small enough batches already so that parallel requests will succeed without E.g. here is an example where batching is simulated with requests happening in parallel. ```csharp -public Task DeleteBatchAsync(IEnumerable keys, DeleteRecordOptions? options = default, CancellationToken cancellationToken = default) +public Task DeleteBatchAsync(IEnumerable keys, CancellationToken cancellationToken = default) { if (keys == null) { @@ -412,7 +422,7 @@ public Task DeleteBatchAsync(IEnumerable keys, DeleteRecordOptions? opti } // Remove records in parallel. - var tasks = keys.Select(key => this.DeleteAsync(key, options, cancellationToken)); + var tasks = keys.Select(key => this.DeleteAsync(key, cancellationToken)); return Task.WhenAll(tasks); } ``` @@ -422,13 +432,14 @@ client so that it may send the entire set in one request. ## Recommended common patterns and pratices +1. Keep `IVectorStore` and `IVectorStoreRecordCollection` implementations unsealed with virtual methods, so that developers can inherit and override if needed. 1. Always use options classes for optional settings with smart defaults. 1. Keep required parameters on the main signature and move optional parameters to options. Here is an example of an `IVectorStoreRecordCollection` constructor following this pattern. ```csharp -public sealed class MyDBVectorStoreRecordCollection : IVectorStoreRecordCollection +public class MyDBVectorStoreRecordCollection : IVectorStoreRecordCollection { public MyDBVectorStoreRecordCollection(MyDBClient myDBClient, string collectionName, MyDBVectorStoreRecordCollectionOptions? options = default) { @@ -437,13 +448,18 @@ public sealed class MyDBVectorStoreRecordCollection : IVectorStoreRecor ... } -public sealed class MyDBVectorStoreRecordCollectionOptions +public class MyDBVectorStoreRecordCollectionOptions { public VectorStoreRecordDefinition? VectorStoreRecordDefinition { get; init; } = null; public IVectorStoreRecordMapper? MyDbRecordCustomMapper { get; init; } = null; } ``` +## SDK Changes + +Please also see the following articles for a history of changes to the SDK and therefore implementation requirements: + +1. [Vector Store Changes March 2025](../../../support/migration/vectorstore-march-2025.md) ::: zone-end ::: zone pivot="programming-language-python" diff --git a/semantic-kernel/concepts/vector-store-connectors/how-to/vector-store-custom-mapper.md b/semantic-kernel/concepts/vector-store-connectors/how-to/vector-store-custom-mapper.md index 81deabad..8cd2a2d5 100644 --- a/semantic-kernel/concepts/vector-store-connectors/how-to/vector-store-custom-mapper.md +++ b/semantic-kernel/concepts/vector-store-connectors/how-to/vector-store-custom-mapper.md @@ -12,6 +12,10 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. +> [!WARNING] +> Support for custom mappers may be deprecated in future, since filtering and target property selection is not able to target the top level mapped types. + +::: zone pivot="programming-language-csharp" In this how to, we will show how you can replace the default mapper for a vector store record collection with your own mapper. @@ -37,19 +41,11 @@ All Vector Store connector implementations allow you to provide a custom mapper. The underlying data stores of each Vector Store connector have different ways of storing data. Therefore what you are mapping to on the storage side may differ for each connector. -::: zone pivot="programming-language-csharp" E.g. if using the Qdrant connector, the storage type is a `PointStruct` class provided by the Qdrant SDK. If using the Redis JSON connector, the storage type is a `string` key and a `JsonNode`, while if using a JSON HashSet connector, the storage type is a `string` key and a `HashEntry` array. -::: zone-end -::: zone pivot="programming-language-python" -::: zone-end -::: zone pivot="programming-language-java" -::: zone-end If you want to do custom mapping, and you want to use multiple connector types, you will therefore need to implement a mapper for each connector type. -::: zone pivot="programming-language-csharp" - ## Creating the data model Our first step is to create a data model. In this case we will not annotate the data model with attributes, since we will provide a separate record definition @@ -204,18 +200,15 @@ When using `IVectorStore` to get `IVectorStoreRecordCollection` object instances the `GetCollection` method. This is because custom mappers differ for each Vector Store type, and would make it impossible to use `IVectorStore` to communicate with any vector store implementation. -It is however possible to provide a factory when constructing a Vector Store implementation. This can be used to customize `IVectorStoreRecordCollection` -instances as they are created. +It is however possible to override the default implementation of `GetCollection` and provide your own custom implementation of the vector store. -Here is an example of such a factory, which checks if `CreateCollection` was called with the product definition and data type, and if so -injects the custom mapper and switches on named vectors mode. +Here is an example where we inherit from the `QdrantVectorStore` and override the `GetCollection` method to do the custom construction. ```csharp -public class QdrantCollectionFactory(VectorStoreRecordDefinition productDefinition) : IQdrantVectorStoreRecordCollectionFactory +private sealed class QdrantCustomVectorStore(QdrantClient qdrantClient, VectorStoreRecordDefinition productDefinition) + : QdrantVectorStore(qdrantClient) { - public IVectorStoreRecordCollection CreateVectorStoreRecordCollection(QdrantClient qdrantClient, string name, VectorStoreRecordDefinition? vectorStoreRecordDefinition) - where TKey : notnull - where TRecord : class + public override IVectorStoreRecordCollection GetCollection(string name, VectorStoreRecordDefinition? vectorStoreRecordDefinition = null) { // If the record definition is the product definition and the record type is the product data // model, inject the custom mapper into the collection options. @@ -233,39 +226,30 @@ public class QdrantCollectionFactory(VectorStoreRecordDefinition productDefiniti return customCollection!; } - // Otherwise, just create a standard collection with the default mapper. - var collection = new QdrantVectorStoreRecordCollection( - qdrantClient, - name, - new() - { - VectorStoreRecordDefinition = vectorStoreRecordDefinition - }) as IVectorStoreRecordCollection; - return collection!; + // Otherwise, just create a standard collection. + return base.GetCollection(name, vectorStoreRecordDefinition); } } ``` -To use the collection factory, pass it to the Vector Store when constructing it, or when registering it with the dependency injection container. +To use the replacement vector store, register it with your dependency injection container or use just use it directly as you would a regular `QdrantVectorStore`. ```csharp // When registering with the dependency injection container on the kernel builder. -kernelBuilder.AddQdrantVectorStore( - "localhost", - options: new() +kernelBuilder.Services.AddTransient( + (sp) => { - VectorStoreCollectionFactory = new QdrantCollectionFactory(productDefinition) + return new QdrantCustomVectorStore( + new QdrantClient("localhost"), + productDefinition); }); ``` ```csharp // When constructing the Vector Store instance directly. -var vectorStore = new QdrantVectorStore( +var vectorStore = new QdrantCustomVectorStore( new QdrantClient("localhost"), - new() - { - VectorStoreCollectionFactory = new QdrantCollectionFactory(productDefinition) - }); + productDefinition); ``` Now you can use the vector store as normal to get a collection. diff --git a/semantic-kernel/concepts/vector-store-connectors/hybrid-search.md b/semantic-kernel/concepts/vector-store-connectors/hybrid-search.md new file mode 100644 index 00000000..f962d91c --- /dev/null +++ b/semantic-kernel/concepts/vector-store-connectors/hybrid-search.md @@ -0,0 +1,270 @@ +--- +title: Hybrid search using Semantic Kernel Vector Store connectors (Preview) +description: Describes the different options you can use when doing a hybrid search using Semantic Kernel vector store connectors. +zone_pivot_groups: programming-languages +author: westey-m +ms.topic: conceptual +ms.author: westey +ms.date: 03/06/2025 +ms.service: semantic-kernel +--- +# Hybrid search using Semantic Kernel Vector Store connectors (Preview) + +> [!WARNING] +> The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. + +::: zone pivot="programming-language-csharp" + +Semantic Kernel provides hybrid search capabilities as part of its Vector Store abstractions. This supports filtering and many other options, which this article will explain in more detail. + +Currently the type of hybrid search supported is based on a vector search, plus a keyword search, both of which are executed in parallel, after which a union of the two result sets +are returned. Sparse vector based hybrid search is not currently supported. + +To execute a hybrid search, your database schema needs to have a vector field and a string field with full text search capabilities enabled. +If you are creating a collection using the Semantic Kernel vector storage connectors, make sure to enable the `IsFullTextSearchable` option +on the string field that you want to target for the keyword search. + +> [!TIP] +> For more information on how to enable `IsFullTextSearchable` refer to [VectorStoreRecordDataAttribute parameters](./defining-your-data-model.md#vectorstorerecorddataattribute-parameters) or [VectorStoreRecordDataProperty configuration settings](./schema-with-record-definition.md#vectorstorerecorddataproperty-configuration-settings) + +## Hybrid Search + +The `HybridSearchAsync` method allows searching using a vector and an `ICollection` of string keywords. It also takes an optional `HybridSearchOptions` class as input. +This method is available on the following interface: + +1. `IKeywordHybridSearch` + +Only connectors for databases that currently support vector plus keyword hybrid search are implementing this interface. + +Assuming you have a collection that already contains data, you can easily do a hybrid search on it. Here is an example using Qdrant. + +```csharp +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +// Placeholder embedding generation method. +async Task> GenerateEmbeddingAsync(string textToVectorize) +{ + // your logic here +} + +// Create a Qdrant VectorStore object and choose an existing collection that already contains records. +IVectorStore vectorStore = new QdrantVectorStore(new QdrantClient("localhost")); +IKeywordHybridSearch collection = (IKeywordHybridSearch)vectorStore.GetCollection("skhotels"); + +// Generate a vector for your search text, using your chosen embedding generation implementation. +ReadOnlyMemory searchVector = await GenerateEmbeddingAsync("I'm looking for a hotel where customer happiness is the priority."); + +// Do the search, passing an options object with a Top value to limit resulst to the single top match. +var searchResult = await collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], new() { Top = 1 }); + +// Inspect the returned hotel. +await foreach (var record in searchResult.Results) +{ + Console.WriteLine("Found hotel description: " + record.Record.Description); + Console.WriteLine("Found record score: " + record.Score); +} +``` + +> [!TIP] +> For more information on how to generate embeddings see [embedding generation](./embedding-generation.md). + +## Supported Vector Types + +`HybridSearchAsync` takes a generic type as the vector parameter. +The types of vectors supported by each data store vary. +See [the documentation for each connector](./out-of-the-box-connectors/index.md) for the list of supported vector types. + +It is also important for the search vector type to match the target vector that is being searched, e.g. if you have two vectors +on the same record with different vector types, make sure that the search vector you supply matches the type of the specific vector +you are targeting. +See [VectorProperty and AdditionalProperty](#vectorproperty-and-additionalproperty) for how to pick a target vector if you have more than one per record. + +## Hybrid Search Options + +The following options can be provided using the `HybridSearchOptions` class. + +### VectorProperty and AdditionalProperty + +The `VectorProperty` and `AdditionalProperty` options can be used to specify the vector property and full text search property to target during the search. + +If no `VectorProperty` is provided and the data model contains only one vector, that vector will be used. +If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method will throw. + +If no `AdditionalProperty` is provided and the data model contains only one full text search property, that property will be used. +If the data model contains no full text search property or multiple full text search properties and `AdditionalProperty` is not provided, the search method will throw. + +```csharp +using Microsoft.SemanticKernel.Connectors.Qdrant; +using Microsoft.Extensions.VectorData; +using Qdrant.Client; + +var vectorStore = new QdrantVectorStore(new QdrantClient("localhost")); +var collection = (IKeywordHybridSearch)vectorStore.GetCollection("skproducts"); + +// Create the hybrid search options and indicate that we want +// to search the DescriptionEmbedding vector property and the +// Description full text search property. +var hybridSearchOptions = new HybridSearchOptions +{ + VectorProperty = r => r.DescriptionEmbedding, + AdditionalProperty = r => r.Description +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = await collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], hybridSearchOptions); + +public sealed class Product +{ + [VectorStoreRecordKey] + public int Key { get; set; } + + [VectorStoreRecordData(IsFullTextSearchable = true)] + public string Name { get; set; } + + [VectorStoreRecordData(IsFullTextSearchable = true)] + public string Description { get; set; } + + [VectorStoreRecordData] + public List FeatureList { get; set; } + + [VectorStoreRecordVector(1536)] + public ReadOnlyMemory DescriptionEmbedding { get; set; } + + [VectorStoreRecordVector(1536)] + public ReadOnlyMemory FeatureListEmbedding { get; set; } +} +``` + +### Top and Skip + +The `Top` and `Skip` options allow you to limit the number of results to the Top n results and +to skip a number of results from the top of the resultset. +Top and Skip can be used to do paging if you wish to retrieve a large number of results using separate calls. + +```csharp +// Create the vector search options and indicate that we want to skip the first 40 results and then get the next 20. +var hybridSearchOptions = new HybridSearchOptions +{ + Top = 20, + Skip = 40 +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = await collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult.Results) +{ + Console.WriteLine(result.Record.Description); +} +``` + +The default values for `Top` is 3 and `Skip` is 0. + +### IncludeVectors + +The `IncludeVectors` option allows you to specify whether you wish to return vectors in the search results. +If `false`, the vector properties on the returned model will be left null. +Using `false` can significantly reduce the amount of data retrieved from the vector store during search, +making searches more efficient. + +The default value for `IncludeVectors` is `false`. + +```csharp +// Create the hybrid search options and indicate that we want to include vectors in the search results. +var hybridSearchOptions = new HybridSearchOptions +{ + IncludeVectors = true +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = await collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult.Results) +{ + Console.WriteLine(result.Record.FeatureList); +} +``` + +### Filter + +The vector search filter option can be used to provide a filter for filtering the records in the chosen collection +before applying the vector search. + +This has multiple benefits: + +- Reduce latency and processing cost, since only records remaining after filtering need to be compared with the search vector and therefore fewer vector comparisons have to be done. +- Limit the resultset for e.g. access control purposes, by excluding data that the user shouldn't have access to. + +Note that in order for fields to be used for filtering, many vector stores require those fields to be indexed first. +Some vector stores will allow filtering using any field, but may optionally allow indexing to improve filtering performance. + +If creating a collection via the Semantic Kernel vector store abstractions and you wish to enable filtering on a field, +set the `IsFilterable` property to true when defining your data model or when creating your record definition. + +> [!TIP] +> For more information on how to set the `IsFilterable` property, refer to [VectorStoreRecordDataAttribute parameters](./defining-your-data-model.md#vectorstorerecorddataattribute-parameters) or [VectorStoreRecordDataProperty configuration settings](./schema-with-record-definition.md#vectorstorerecorddataproperty-configuration-settings). + +Filters are expressed using LINQ expressions based on the type of the data model. +The set of LINQ expressions supported will vary depending on the functionality supported +by each database, but all databases support a broad base of common expressions, e.g. equals, +not equals, and, or, etc. + +```csharp +// Create the hybrid search options and set the filter on the options. +var hybridSearchOptions = new HybridSearchOptions +{ + Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") +}; + +// This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. +var searchResult = await collection.HybridSearchAsync(searchVector, ["happiness", "hotel", "customer"], hybridSearchOptions); + +// Iterate over the search results. +await foreach (var result in searchResult.Results) +{ + Console.WriteLine(result.Record.Definition); +} + +sealed class Glossary +{ + [VectorStoreRecordKey] + public ulong Key { get; set; } + + // Category is marked as filterable, since we want to filter using this property. + [VectorStoreRecordData(IsFilterable = true)] + public string Category { get; set; } + + // Tags is marked as filterable, since we want to filter using this property. + [VectorStoreRecordData(IsFilterable = true)] + public List Tags { get; set; } + + [VectorStoreRecordData] + public string Term { get; set; } + + [VectorStoreRecordData(IsFullTextSearchable = true)] + public string Definition { get; set; } + + [VectorStoreRecordVector(1536)] + public ReadOnlyMemory DefinitionEmbedding { get; set; } +} +``` + +::: zone-end +::: zone pivot="programming-language-python" + +## Coming soon + +More information coming soon. + +::: zone-end +::: zone pivot="programming-language-java" + +## Coming soon + +More information coming soon. + +::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml index b4c46306..161e8b58 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/TOC.yml @@ -6,10 +6,14 @@ href: azure-cosmosdb-mongodb-connector.md - name: Azure CosmosDB NoSQL connector href: azure-cosmosdb-nosql-connector.md +- name: Chroma connector + href: chroma-connector.md - name: Couchbase connector href: couchbase-connector.md - name: Elasticsearch connector href: elasticsearch-connector.md +- name: Faiss connector + href: faiss-connector.md - name: In-memory connector href: inmemory-connector.md - name: JDBC connector diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md index 305ce851..869374b0 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-ai-search-connector.md @@ -17,27 +17,65 @@ ms.service: semantic-kernel The Azure AI Search Vector Store connector can be used to access and manage data in Azure AI Search. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Azure AI Search Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| -| Supported vector property types | ReadOnlyMemory\ | -| Supported index types |
  • Hnsw
  • Flat
| -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | Yes | -| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure AI Search Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| +| Supported vector property types | ReadOnlyMemory\ | +| Supported index types |
  • Hnsw
  • Flat
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure AI Search Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • numpy array
| +| Supported index types |
  • Hnsw
  • Flat
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
  • Hamming
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-java" + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure AI Search Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| +| Supported vector property types | ReadOnlyMemory\ | +| Supported index types |
  • Hnsw
  • Flat
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | + +::: zone-end ## Limitations Notable Azure AI Search connector functionality limitations. -| Feature Area | Workaround | -|--------------------------------------------------------------------------------------| -----------------------------------------------------------------------------------------------| -| Configuring full text search analyzers during collection creation is not supported. | Use the Azure AI Search Client SDK directly for collection creation | +| Feature Area | Workaround | +| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------- | +| Configuring full text search analyzers during collection creation is not supported. | Use the Azure AI Search Client SDK directly for collection creation | ::: zone pivot="programming-language-csharp" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md index 46fbbf96..6af3c29d 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-mongodb-connector.md @@ -13,30 +13,54 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. -::: zone pivot="programming-language-csharp" - ## Overview The Azure CosmosDB MongoDB Vector Store connector can be used to access and manage data in Azure CosmosDB MongoDB (vCore). The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| -| Supported index types |
  • Hnsw
  • IvfFlat
| -| Supported distance functions |
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | No | -| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| +| Supported index types |
  • Hnsw
  • IvfFlat
| +| Supported distance functions |
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | No | + +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure Cosmos DB MongoDB (vCore) Collection + Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • ndarray
| +| Supported index types |
  • Hnsw
  • IvfFlat
| +| Supported distance functions |
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
  • AnyTagsEqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | + +::: zone-end +::: zone pivot="programming-language-java" +More info coming soon. +::: zone-end ## Limitations This connector is compatible with Azure Cosmos DB MongoDB (vCore) and is *not* designed to be compatible with Azure Cosmos DB MongoDB (RU). +::: zone pivot="programming-language-csharp" + ## Getting started Add the Azure CosmosDB MongoDB Vector Store connector NuGet package to your project. @@ -165,9 +189,52 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Getting started -More info coming soon. +Add the Azure CosmosDB MongoDB Vector Store dependencies to your environment. Because the Azure CosmosDB MongoDB connector is built on the MongoDB Atlas connector and uses the same client as that one, you need to install with these extras: + +```bash +pip install semantic-kernel[azure, mongo] +``` + +You can then create the vector store. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBforMongoDBStore + +# If the right environment settings are set, namely AZURE_COSMOS_DB_MONGODB_CONNECTION_STRING and optionally AZURE_COSMOS_DB_MONGODB_DATABASE_NAME, this is enough to create the Store: +store = AzureCosmosDBforMongoDBStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +from pymongo import AsyncMongoClient +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBforMongoDBStore + +client = AsyncMongoClient(...) +store = AzureCosmosDBforMongoDBStore(mongo_client=client) +``` + +When a client is passed in, Semantic Kernel will not close the connection for you, so you need to ensure to close it, for instance with a `async with` statement. + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBforMongoDBCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = AzureCosmosDBforMongoDBCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +Since the Azure CosmosDB for MongoDB connector needs a simple dict with the fields corresponding to the index as the input, the serialization is quite easy, it only uses a predetermined key `_id`, so we replace the key of the data model with that if it is not already `_id`. + +For more details on this concept see the [serialization documentation](./../serialization.md). ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md index 6e84f353..4b9807ce 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/azure-cosmosdb-nosql-connector.md @@ -19,19 +19,20 @@ ms.service: semantic-kernel The Azure CosmosDB NoSQL Vector Store connector can be used to access and manage data in Azure CosmosDB NoSQL. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Azure Cosmos DB NoSQL Container | -| Supported key property types |
  • string
  • AzureCosmosDBNoSQLCompositeKey
| -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| -| Supported index types |
  • Flat
  • QuantizedFlat
  • DiskAnn
| -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | Yes | -| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure Cosmos DB NoSQL Container | +| Supported key property types |
  • string
  • AzureCosmosDBNoSQLCompositeKey
| +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| +| Supported index types |
  • Flat
  • QuantizedFlat
  • DiskAnn
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | ## Limitations @@ -247,9 +248,158 @@ var record = await collection.GetAsync(new AzureCosmosDBNoSQLCompositeKey("hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Overview -More info coming soon. +The Azure CosmosDB NoSQL Vector Store connector can be used to access and manage data in Azure CosmosDB NoSQL. The connector has the following characteristics. + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Azure Cosmos DB NoSQL Container | +| Supported key property types |
  • string
  • AzureCosmosDBNoSQLCompositeKey
| +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • DateTimeOffset
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • ndarray
| +| Supported index types |
  • Flat
  • QuantizedFlat
  • DiskAnn
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| is_filterable supported? | Yes | +| is_full_text_searchable supported? | Yes | +| HybridSearch supported? | No | + +## Getting started + +Add the Azure extra package to your project. + +```bash +pip install semantic-kernel[azure] +``` + +Next you can create a Azure CosmosDB NoSQL Vector Store instance directly. This reads certain environment variables to configure the connection to Azure CosmosDB NoSQL: + +- AZURE_COSMOS_DB_NO_SQL_URL +- AZURE_COSMOS_DB_NO_SQL_DATABASE_NAME + +And optionally: + +- AZURE_COSMOS_DB_NO_SQL_KEY + +When this is not set, a `AsyncDefaultAzureCredential` is used to authenticate. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLStore + +vector_store = AzureCosmosDBNoSQLStore() +``` + +You can also supply these values in the constructor: + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLStore + +vector_store = AzureCosmosDBNoSQLStore( + url="https://.documents.azure.com:443/", + key="", + database_name="" +) +``` + +And you can pass in a CosmosClient instance, just make sure it is a async client. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLStore +from azure.cosmos.aio import CosmosClient + +client = CosmosClient( + url="https://.documents.azure.com:443/", + credential="" or AsyncDefaultAzureCredential() +) +vector_store = AzureCosmosDBNoSQLStore( + client=client, + database_name="" +) +``` + +The next step needs a data model, a variable called Hotels is used in the example below. + +With a store, you can get a collection: + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLStore + +vector_store = AzureCosmosDBNoSQLStore() +collection = vector_store.get_collection(collection_name="skhotels", data_model=Hotel) +``` + +It is possible to construct a direct reference to a named collection, this uses the same environment variables as above. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLCollection + +collection = AzureCosmosDBNoSQLCollection( + collection_name="skhotels", + data_model_type=Hotel, +) +``` + +## Using partition key + +In the Azure Cosmos DB for NoSQL connector, the partition key property defaults to the key property - `id`. You can also supply a value for the partition key in the constructor. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLCollection + +collection = AzureCosmosDBNoSQLCollection( + collection_name="skhotels", + data_model_type=Hotel, + partition_key="hotel_name" +) +``` + +This can be a more complex key, when using the `PartitionKey` object: + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLCollection +from azure.cosmos import PartitionKey + +partition_key = PartitionKey(path="/hotel_name") +collection = AzureCosmosDBNoSQLCollection( + collection_name="skhotels", + data_model_type=Hotel, + partition_key=partition_key +) +``` + +The `AzureCosmosDBNoSQLVectorStoreRecordCollection` class supports two key types: `string` and `AzureCosmosDBNoSQLCompositeKey`. The `AzureCosmosDBNoSQLCompositeKey` consists of `key` and `partition_key`. + +If the partition key property is not set (and the default key property is used), `string` keys can be used for operations with database records. However, if a partition key property is specified, it is recommended to use `AzureCosmosDBNoSQLCompositeKey` to provide both the key and partition key values to the `get` and `delete` methods. + +```python +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLCollection +from semantic_kernel.connectors.memory.azure_cosmos_db import AzureCosmosDBNoSQLCompositeKey +from semantic_kernel.data import VectorStoreRecordDataField + +@vectorstoremodel +class data_model_type: + id: Annotated[str, VectorStoreRecordKeyField] + product_type: Annotated[str, VectorStoreRecordDataField()] + ... + +collection = store.get_collection( + collection_name=collection_name, + data_model=data_model_type, + partition_key=PartitionKey(path="/product_type"), +) + +# when there is data in the collection +composite_key = AzureCosmosDBNoSQLCompositeKey( + key='key value', partition_key='partition key value' +) +# get a record, with the partition key +record = await collection.get(composite_key) + +# or delete +await collection.delete(composite_key) +``` ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/chroma-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/chroma-connector.md new file mode 100644 index 00000000..9cfc7ee8 --- /dev/null +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/chroma-connector.md @@ -0,0 +1,104 @@ +--- +title: Using the Semantic Kernel Chroma Vector Store connector (Preview) +description: Contains information on how to use a Semantic Kernel Vector store connector to access and manipulate data in ChromaDB. +zone_pivot_groups: programming-languages +author: eavanvalkenburg +ms.topic: conceptual +ms.author: edvan +ms.date: 02/26/2025 +ms.service: semantic-kernel +--- + +# Using the Chroma connector (Preview) + +> [!WARNING] +> The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. + +::: zone pivot="programming-language-csharp" + +## Not supported + +Not supported. + +::: zone-end +::: zone pivot="programming-language-python" + +## Overview + +The Chroma Vector Store connector can be used to access and manage data in Chroma. The connector has the +following characteristics. + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------ | +| Collection maps to | Chroma collection | +| Supported key property types | string | +| Supported data property types | All types | +| Supported vector property types |
  • list[float]
  • list[int]
  • ndarray
| +| Supported index types |
  • HNSW
| +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | No | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | + +## Limitations + +Notable Chroma connector functionality limitations. + +| Feature Area | Workaround | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------- | +| Client-server mode | Use the client.HttpClient and pass the result to the `client` parameter, we do not support a AsyncHttpClient at this time | +| Chroma Cloud | Unclear at this time, as Chroma Cloud is still in private preview | + +## Getting Started + +Add the Chroma Vector Store connector dependencies to your project. + +```bash +pip install semantic-kernel[chroma] +``` + +You can then create the vector store. + +```python +from semantic_kernel.connectors.memory.chroma import ChromaStore + +store = ChromaStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +from chromadb import Client +from semantic_kernel.connectors.memory.chroma import ChromaStore + +client = Client(...) +store = ChromaStore(client=client) +``` + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.chroma import ChromaCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = ChromaCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +The Chroma client returns both `get` and `search` results in tabular form, this means that there are between 3 and 5 lists being returned in a dict, the lists are 'keys', 'documents', 'embeddings', and optionally 'metadatas' and 'distances'. The Semantic Kernel Chroma connector will automatically convert this into a list of `dict` objects, which are then parsed back to your data model. + +It could be very interesting performance wise to do straight serialization from this format into a dataframe-like structure as that saves a lot of rebuilding of the data structure. This is not done for you, even when using container mode, you would have to specify this yourself, for more details on this concept see the [serialization documentation](./../serialization.md). + +::: zone-end +::: zone pivot="programming-language-java" + +## Not supported + +Not supported. + +::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/couchbase-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/couchbase-connector.md index 63ba111f..e27ad81a 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/couchbase-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/couchbase-connector.md @@ -34,6 +34,7 @@ following characteristics. | IsFilterable supported? | No | | IsFullTextSearchable supported? | No | | StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | No | ## Getting Started diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/elasticsearch-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/elasticsearch-connector.md index 632ea9f8..6f41b9e3 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/elasticsearch-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/elasticsearch-connector.md @@ -13,6 +13,8 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. +::: zone pivot="programming-language-csharp" + ## Overview The Elasticsearch Vector Store connector can be used to access and manage data in Elasticsearch. The connector has the following characteristics. @@ -30,8 +32,7 @@ The Elasticsearch Vector Store connector can be used to access and manage data i | IsFilterable supported? | Yes | | IsFullTextSearchable supported? | Yes | | StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | - -::: zone pivot="programming-language-csharp" +| HybridSearch supported? | No | ## Getting started diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/faiss-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/faiss-connector.md new file mode 100644 index 00000000..7fc2c4a2 --- /dev/null +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/faiss-connector.md @@ -0,0 +1,121 @@ +--- +title: Using the Semantic Kernel Faiss Vector Store connector (Preview) +description: Contains information on how to use a Semantic Kernel Vector store connector to access and manipulate data in an in-memory Faiss vector store. +zone_pivot_groups: programming-languages +author: eavanvalkenburg +ms.topic: conceptual +ms.author: edvan +ms.date: 03/13/2025 +ms.service: semantic-kernel +--- +# Using the Faiss connector (Preview) + +> [!WARNING] +> The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. + +::: zone pivot="programming-language-csharp" + +## Not supported at this time + +::: zone-end +::: zone pivot="programming-language-python" + +## Overview + +The [Faiss](https://github.com/facebookresearch/faiss) Vector Store connector is a Vector Store implementation provided by Semantic Kernel that uses no external database and stores data in memory and vectors in a Faiss Index. It uses the [`InMemoryVectorCollection`](./inmemory-connector.md) for the other parts of the records, while using the Faiss indexes for search. +This Vector Store is useful for prototyping scenarios or where high-speed in-memory operations are required. + +The connector has the following characteristics. + +| Feature Area | Support | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | In-memory and Faiss indexes dictionary | +| Supported key property types | Any that is allowed to be a dict key, see the python documentation for details [here](https://docs.python.org/3/library/stdtypes.html#typesmapping) | +| Supported data property types | Any type | +| Supported vector property types |
  • list[float]
  • list[int]
  • numpy array
| +| Supported index types | Flat (see [custom indexes](#custom-indexes)) | +| Supported distance functions |
  • Dot Product Similarity
  • Euclidean Squared Distance
| +| Supports multiple vectors in a record | Yes | +| is_filterable supported? | Yes | +| is_full_text_searchable supported? | Yes | + +## Getting started + +Add the Semantic Kernel package to your project. + +```cmd +pip install semantic-kernel[faiss] +``` + +In the snippets below, it is assumed that you have a data model class defined named 'DataModel'. + +```python +from semantic_kernel.connectors.memory.faiss import FaissStore + +vector_store = FaissStore() +vector_collection = vector_store.get_collection("collection_name", DataModel) +``` + +It is possible to construct a direct reference to a named collection. + +```python +from semantic_kernel.connectors.memory.faiss import FaissCollection + +vector_collection = FaissCollection("collection_name", DataModel) +``` + +## Custom indexes + +The Faiss connector is limited to the Flat index type. + +Given the complexity of Faiss indexes, you are free to create your own index(es), including building the faiss-gpu package and using indexes from that. When doing this, any metrics defined on a vector field is ignored. If you have multiple vectors in your datamodel, you can pass in custom indexes only for the ones you want and let the built-in indexes be created, with a flat index and the metric defined in the model. + +Important to note, if the index requires training, then make sure to do that as well, whenever we use the index, a check is done on the `is_trained` attribute of the index. + +The index is always available (custom or built-in) in the `indexes` property of the collection. You can use this to get the index and do any operations you want on it, so you can do training afterwards, just make sure to do that before you want to use any CRUD against it. + +To pass in your custom index, use either: + +```python + +import faiss + +from semantic_kernel.connectors.memory.faiss import FaissCollection + +index = faiss.IndexHNSW(d=768, M=16, efConstruction=200) # or some other index +vector_collection = FaissCollection( + collection_name="collection_name", + data_model_type=DataModel, + indexes={"vector_field_name": index} +) +``` + +or: + +```python + +import faiss + +from semantic_kernel.connectors.memory.faiss import FaissCollection + +index = faiss.IndexHNSW(d=768, M=16, efConstruction=200) # or some other index +vector_collection = FaissCollection( + collection_name="collection_name", + data_model_type=DataModel, +) +await vector_collection.create_collection( + indexes={"vector_field_name": index} +) +# or when you have only one vector field: +await vector_collection.create_collection( + index=index +) + +``` + +::: zone-end +::: zone pivot="programming-language-java" + +## Not supported at this time + +::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md index 15b30d1d..8d6dca7d 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/index.md @@ -22,60 +22,59 @@ Semantic Kernel provides a number of out-of-the-box Vector Store integrations ma ::: zone pivot="programming-language-csharp" -| Vector Store Connectors | C# | Uses officially supported SDK | Maintainer / Vendor | -|------------------------------------------------------------|:---------------:|:---------------------------------:|:----------------------------------:| -| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Couchbase](./couchbase-connector.md) | ✅ | ✅ | Couchbase | -| [Elasticsearch](./elasticsearch-connector.md) | ✅ | ✅ | Elastic | -| Chroma | Planned | | | -| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | -| Milvus | Planned | | | -| [MongoDB](./mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Pinecone](./pinecone-connector.md) | ✅ | ❌ | Microsoft Semantic Kernel Project | -| [Postgres](./postgres-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| Sql Server | Planned | | | -| [SQLite](./sqlite-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Volatile (In-Memory)](./volatile-connector.md) | Deprecated (use In-Memory) | N/A | Microsoft Semantic Kernel Project | -| [Weaviate](./weaviate-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| Vector Store Connectors | C# | Uses officially supported SDK | Maintainer / Vendor | +| ------------------------------------------------------------------ | :------------------------: | :---------------------------: | :-------------------------------: | +| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Couchbase](./couchbase-connector.md) | ✅ | ✅ | Couchbase | +| [Elasticsearch](./elasticsearch-connector.md) | ✅ | ✅ | Elastic | +| Chroma | Planned | | | +| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| Milvus | Planned | | | +| [MongoDB](./mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Pinecone](./pinecone-connector.md) | ✅ | ❌ | Microsoft Semantic Kernel Project | +| [Postgres](./postgres-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| Sql Server | Planned | | | +| [SQLite](./sqlite-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Volatile (In-Memory)](./volatile-connector.md) | Deprecated (use In-Memory) | N/A | Microsoft Semantic Kernel Project | +| [Weaviate](./weaviate-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | ::: zone-end ::: zone pivot="programming-language-python" -| Vector Store Connectors | Python | Uses officially supported SDK | Maintainer / Vendor | -|------------------------------------------------------------|:---------------:|:----------------------------------:|:----------------------------------:| -| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Elasticsearch](./elasticsearch-connector.md) | Planned | | | -| Chroma | Planned | | | -| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | -| Milvus | Planned | | | -| [MongoDB](./mongodb-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Pinecone](./pinecone-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Postgres](./postgres-connector.md) | ✅ | | Microsoft Semantic Kernel Project | -| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| Sql Server | Planned | | | -| [SQLite](./sqlite-connector.md) | In Development | ✅ | Microsoft Semantic Kernel Project | -| [Volatile (In-Memory)](./volatile-connector.md) | Deprecated (use In-Memory) | N/A | Microsoft Semantic Kernel Project | -| [Weaviate](./weaviate-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| Vector Store Connectors | Python | Uses officially supported SDK | Maintainer / Vendor | +| ------------------------------------------------------------------ | :-----: | :---------------------------: | :-------------------------------: | +| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB MongoDB (vCore)](./azure-cosmosdb-mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Cosmos DB No SQL](./azure-cosmosdb-nosql-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Chroma](./chroma-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Elasticsearch](./elasticsearch-connector.md) | Planned | | | +| [Faiss](./faiss-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [In-Memory](./inmemory-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| [MongoDB](./mongodb-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Pinecone](./pinecone-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Postgres](./postgres-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Qdrant](./qdrant-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| SQL Server | Planned | | Microsoft Semantic Kernel Project | +| SQLite | Planned | | Microsoft Semantic Kernel Project | +| [Weaviate](./weaviate-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | ::: zone-end ::: zone pivot="programming-language-java" -| Vector Store Connectors | Java | Uses officially supported SDK | Maintainer / Vendor | -|------------------------------------------------------------|:--------------:|:----------------------------------:|:----------------------------------:| -| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| HSQLDB | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | -| [JDBC](./jdbc-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| MySQL | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | -| Postgres | Use [JDBC](./jdbc-connector.md) | | Microsoft Semantic Kernel Project | -| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | -| SQLite | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | -| [Volatile (In-Memory)](./volatile-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | +| Vector Store Connectors | Java | Uses officially supported SDK | Maintainer / Vendor | +| ------------------------------------------------- | :-----------------------------: | :---------------------------: | :-------------------------------: | +| [Azure AI Search](./azure-ai-search-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| HSQLDB | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | +| [JDBC](./jdbc-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| MySQL | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | +| Postgres | Use [JDBC](./jdbc-connector.md) | | Microsoft Semantic Kernel Project | +| [Redis](./redis-connector.md) | ✅ | ✅ | Microsoft Semantic Kernel Project | +| SQLite | Use [JDBC](./jdbc-connector.md) | ✅ | Microsoft Semantic Kernel Project | +| [Volatile (In-Memory)](./volatile-connector.md) | ✅ | N/A | Microsoft Semantic Kernel Project | ::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/inmemory-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/inmemory-connector.md index 1a08b629..fd626301 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/inmemory-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/inmemory-connector.md @@ -35,6 +35,7 @@ The connector has the following characteristics. | IsFilterable supported? | Yes | | IsFullTextSearchable supported? | Yes | | StoragePropertyName supported? | No, since storage is in-memory and data reuse is therefore not possible, custom naming is not applicable. | +| HybridSearch supported? | No | ## Getting started diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md index 6c406610..7591b678 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/mongodb-connector.md @@ -13,25 +13,50 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. -::: zone pivot="programming-language-csharp" - ## Overview The MongoDB Vector Store connector can be used to access and manage data in MongoDB. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | MongoDB Collection + Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| -| Supported index types | N/A | -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| -| Supported filter clauses |
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | No | -| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | MongoDB Collection + Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| +| Supported index types | N/A | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +| StoragePropertyName supported? | No, use BsonElementAttribute instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | MongoDB Collection + Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • ndarray
| +| Supported index types |
  • Hnsw
  • IvfFlat
| +| Supported distance functions |
  • CosineDistance
  • DotProductSimilarity
  • EuclideanDistance
| +| Supported filter clauses |
  • EqualTo
  • AnyTagsEqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | + +::: zone-end +::: zone pivot="programming-language-java" + +More info coming soon. + +::: zone-end +::: zone pivot="programming-language-csharp" ## Getting started @@ -136,9 +161,52 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Getting started -More info coming soon. +Add the MongoDB Atlas Vector Store dependencies to your environment. It needs the `pymongo` package which is included in the mongo extra: , you need to install with these extras: + +```bash +pip install semantic-kernel[mongo] +``` + +You can then create the vector store. + +```python +from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasStore + +# If the right environment settings are set, namely MONGODB_ATLAS_CONNECTION_STRING and optionally MONGODB_ATLAS_DATABASE_NAME and MONGODB_ATLAS_INDEX_NAME, this is enough to create the Store: +store = MongoDBAtlasStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +from pymongo import AsyncMongoClient +from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasStore + +client = AsyncMongoClient(...) +store = MongoDBAtlasStore(mongo_client=client) +``` + +When a client is passed in, Semantic Kernel will not close the connection for you, so you need to ensure to close it, for instance with a `async with` statement. + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = MongoDBAtlasCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +Since the MongoDB Atlas connector needs a simple dict with the fields corresponding to the index as the input, the serialization is quite easy, it only uses a predetermined key `_id`, so we replace the key of the data model with that if it is not already `_id`. + +For more details on this concept see the [serialization documentation](./../serialization.md). ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/pinecone-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/pinecone-connector.md index 134a29bc..cbd94d3b 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/pinecone-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/pinecone-connector.md @@ -19,19 +19,21 @@ ms.service: semantic-kernel The Pinecone Vector Store connector can be used to access and manage data in Pinecone. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Pinecone serverless Index | -| Supported key property types | string | -| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • decimal
  • *enumerables of type* string
| -| Supported vector property types | ReadOnlyMemory\ | -| Supported index types | PGA (Pinecone Graph Algorithm) | -| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| -| Supported filter clauses |
  • EqualTo
| -| Supports multiple vectors in a record | No | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | No | -| StoragePropertyName supported? | Yes | +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| Collection maps to | Pinecone serverless Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • decimal
  • *enumerables of type* string
| +| Supported vector property types | ReadOnlyMemory\ | +| Supported index types | PGA (Pinecone Graph Algorithm) | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| +| Supported filter clauses |
  • EqualTo
| +| Supports multiple vectors in a record | No | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +| StoragePropertyName supported? | Yes | +| HybridSearch supported? | No | +| Integrated Embeddings supported? | No | ## Getting started @@ -174,7 +176,133 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -The Pinecone connector is not yet available in Python. +## Overview + +The Pinecone Vector Store connector can be used to access and manage data in Pinecone. The connector has the following characteristics. + +| Feature Area | Support | +| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Pinecone serverless Index | +| Supported key property types | string | +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • numpy array
| +| Supported index types | PGA (Pinecone Graph Algorithm) | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanSquaredDistance
| +| Supported filter clauses |
  • EqualTo
  • AnyTagEqualTo
| +| Supports multiple vectors in a record | No | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | No | +| Integrated Embeddings supported? | Yes, see [here](#integrated-embeddings) | +| GRPC Supported? | Yes, see [here](#grpc-support) | + +## Getting started + +Add the Pinecone Vector Store connector extra to your project. + +```bash +pip install semantic-kernel[pinecone] +``` + +You can then create a PineconeStore instance and use it to create a collection. +This will read the Pinecone API key from the environment variable `PINECONE_API_KEY`. + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeStore + +store = PineconeStore() +collection = store.get_collection(collection_name="collection_name", data_model=DataModel) +``` + +It is possible to construct a direct reference to a named collection. + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection(collection_name="collection_name", data_model=DataModel) +``` + +You can also create your own Pinecone client and pass it into the constructor. +The client needs to be either `PineconeAsyncio` or `PineconeGRPC` (see [GRPC Support](#grpc-support)). + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeStore, PineconeCollection +from pinecone import PineconeAsyncio + +client = PineconeAsyncio(api_key="your_api_key") +store = PineconeStore(client=client) +collection = store.get_collection(collection_name="collection_name", data_model=DataModel) +``` + +### GRPC support + +We also support two options on the collection constructor, the first is to enable GRPC support: + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection(collection_name="collection_name", data_model=DataModel, use_grpc=True) +``` + +Or with your own client: + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeStore +from pinecone.grpc import PineconeGRPC + +client = PineconeGRPC(api_key="your_api_key") +store = PineconeStore(client=client) +collection = store.get_collection(collection_name="collection_name", data_model=DataModel) +``` + +### Integrated Embeddings + +The second is to use the integrated embeddings of Pinecone, this will check for a environment variable called `PINECONE_EMBED_MODEL` with the model name, or you can pass in a `embed_settings` dict, which can contain just the model key, or the full settings for the embedding model. In the former case, the other settings will be derived from the data model definition. + +See [Pinecone docs](https://docs.pinecone.io/guides/indexes/create-an-index) and then the `Use integrated embeddings` sections. + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection(collection_name="collection_name", data_model=DataModel) +``` + +Alternatively, when not settings the environment variable, you can pass the embed settings into the constructor: + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection(collection_name="collection_name", data_model=DataModel, embed_settings={"model": "multilingual-e5-large"}) +``` + +This can include other details about the vector setup, like metric and field mapping. +You can also pass the embed settings into the `create_collection` method, this will override the default settings set during initialization. + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection(collection_name="collection_name", data_model=DataModel) +await collection.create_collection(embed_settings={"model": "multilingual-e5-large"}) +``` + +> Important: GRPC and Integrated embeddings cannot be used together. + +## Index Namespace + +The Vector Store abstraction does not support a multi tiered record grouping mechanism. Collections in the abstraction map to a Pinecone serverless index +and no second level exists in the abstraction. Pinecone does support a second level of grouping called namespaces. + +By default the Pinecone connector will pass `''` as the namespace for all operations. However it is possible to pass a single namespace to the +Pinecone collection when constructing it and use this instead for all operations. + +```python +from semantic_kernel.connectors.memory.pinecone import PineconeCollection + +collection = PineconeCollection( + collection_name="collection_name", + data_model=DataModel, + namespace="seasidehotels" +) +``` ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 8eb24e45..e253ab5e 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -32,6 +32,7 @@ The Postgres Vector Store connector can be used to access and manage data in Pos | IsFilterable supported? | No | | IsFullTextSearchable supported? | No | | StoragePropertyName supported? | Yes | +| HybridSearch supported? | No | ## Getting started diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/qdrant-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/qdrant-connector.md index 03ee4e4b..bd6adebc 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/qdrant-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/qdrant-connector.md @@ -17,6 +17,8 @@ ms.service: semantic-kernel The Qdrant Vector Store connector can be used to access and manage data in Qdrant. The connector has the following characteristics. +::: zone pivot="programming-language-csharp" + | Feature Area | Support | |-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| | Collection maps to | Qdrant collection with payload indices for filterable data fields | @@ -30,6 +32,33 @@ The Qdrant Vector Store connector can be used to access and manage data in Qdran | IsFilterable supported? | Yes | | IsFullTextSearchable supported? | Yes | | StoragePropertyName supported? | Yes | +| HybridSearch supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| +| Collection maps to | Qdrant collection with payload indices for filterable data fields | +| Supported key property types |
  • ulong
  • Guid
| +| Supported data property types |
  • string
  • int
  • long
  • double
  • float
  • bool
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
| +| Supported index types | Hnsw | +| Supported distance functions |
  • CosineSimilarity
  • DotProductSimilarity
  • EuclideanDistance
  • ManhattanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes (configurable) | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-java" + +## Not Supported + +Not currently supported. + +::: zone-end ::: zone pivot="programming-language-csharp" @@ -203,6 +232,8 @@ For more details on this concept see the [serialization documentation](./../seri ::: zone pivot="programming-language-java" ::: zone-end +::: zone pivot="programming-language-csharp" + ### Qdrant vector modes Qdrant supports two modes for vector storage and the Qdrant Connector with default mapper supports both modes. @@ -213,8 +244,6 @@ The default mode is *single unnamed vector*. With this option a collection may only contain a single vector and it will be unnamed in the storage model in Qdrant. Here is an example of how an object is represented in Qdrant when using *single unnamed vector* mode: -::: zone pivot="programming-language-csharp" - ```csharp new Hotel { @@ -236,6 +265,16 @@ new Hotel ::: zone-end ::: zone pivot="programming-language-python" +### Qdrant vector modes + +Qdrant supports two modes for vector storage and the Qdrant Connector with default mapper supports both modes. +The default mode is *single unnamed vector*. + +#### Single unnamed vector + +With this option a collection may only contain a single vector and it will be unnamed in the storage model in Qdrant. +Here is an example of how an object is represented in Qdrant when using *single unnamed vector* mode: + ```python Hotel( hotel_id = 1, @@ -254,18 +293,18 @@ PointStruct( vector=[0.9, 0.1, 0.1, 0.1], ) ``` + ::: zone-end ::: zone pivot="programming-language-java" ::: zone-end +::: zone pivot="programming-language-csharp" #### Named vectors If using the named vectors mode, it means that each point in a collection may contain more than one vector, and each will be named. Here is an example of how an object is represented in Qdrant when using *named vectors* mode: -::: zone pivot="programming-language-csharp" - ```csharp new Hotel { @@ -291,6 +330,11 @@ new Hotel ::: zone-end ::: zone pivot="programming-language-python" +#### Named vectors + +If using the named vectors mode, it means that each point in a collection may contain more than one vector, and each will be named. +Here is an example of how an object is represented in Qdrant when using *named vectors* mode: + ```python Hotel( hotel_id = 1, @@ -318,11 +362,11 @@ PointStruct( ::: zone pivot="programming-language-java" ::: zone-end +::: zone pivot="programming-language-csharp" + To enable named vectors mode, pass this as an option when constructing a Vector Store or collection. The same options can also be passed to any of the provided dependency injection container extension methods. -::: zone pivot="programming-language-csharp" - ```csharp using Microsoft.SemanticKernel.Connectors.Qdrant; using Qdrant.Client; @@ -340,6 +384,9 @@ var collection = new QdrantVectorStoreRecordCollection( ::: zone-end ::: zone pivot="programming-language-python" +To enable named vectors mode, pass this as an option when constructing a Vector Store or collection. +The same options can also be passed to any of the provided dependency injection container extension methods. + In python the default value for `named_vectors` is True, but you can also disable this as shown below. ```python @@ -351,6 +398,7 @@ collection = QdrantCollection( named_vectors=False, ) ``` + ::: zone-end ::: zone pivot="programming-language-java" ::: zone-end diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/redis-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/redis-connector.md index 72d8ef82..f1436bc9 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/redis-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/redis-connector.md @@ -32,6 +32,7 @@ The connector has the following characteristics. | IsFilterable supported? | Yes | | IsFullTextSearchable supported? | Yes | | StoragePropertyName supported? | **When using Hashes:** Yes
**When using JSON:** No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | No | ::: zone pivot="programming-language-csharp" diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sqlite-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sqlite-connector.md index 0e1d9c25..5f13fcf8 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sqlite-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/sqlite-connector.md @@ -32,6 +32,7 @@ The SQLite Vector Store connector can be used to access and manage data in SQLit | IsFilterable supported? | No | | IsFullTextSearchable supported? | No | | StoragePropertyName supported? | Yes | +| HybridSearch supported? | No | ## Limitations diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md index 52a0178a..7b30706d 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/weaviate-connector.md @@ -13,33 +13,55 @@ ms.service: semantic-kernel > [!WARNING] > The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release. -::: zone pivot="programming-language-csharp" - ## Overview The Weaviate Vector Store connector can be used to access and manage data in Weaviate. The connector has the following characteristics. -| Feature Area | Support | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------| -| Collection maps to | Weaviate Collection | -| Supported key property types | Guid | -| Supported data property types |
  • string
  • byte
  • short
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • DateTimeOffset
  • Guid
  • *and enumerables of each of these types*
| -| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| -| Supported index types |
  • Hnsw
  • Flat
  • Dynamic
| -| Supported distance functions |
  • CosineDistance
  • NegativeDotProductSimilarity
  • EuclideanSquaredDistance
  • Hamming
  • ManhattanDistance
| -| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| -| Supports multiple vectors in a record | Yes | -| IsFilterable supported? | Yes | -| IsFullTextSearchable supported? | Yes | -| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +::: zone pivot="programming-language-csharp" + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Weaviate Collection | +| Supported key property types | Guid | +| Supported data property types |
  • string
  • byte
  • short
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • DateTime
  • DateTimeOffset
  • Guid
  • *and enumerables of each of these types*
| +| Supported vector property types |
  • ReadOnlyMemory\
  • ReadOnlyMemory\
| +| Supported index types |
  • Hnsw
  • Flat
  • Dynamic
| +| Supported distance functions |
  • CosineDistance
  • NegativeDotProductSimilarity
  • EuclideanSquaredDistance
  • Hamming
  • ManhattanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | +| StoragePropertyName supported? | No, use `JsonSerializerOptions` and `JsonPropertyNameAttribute` instead. [See here for more info.](#data-mapping) | +| HybridSearch supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-python" + +| Feature Area | Support | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Collection maps to | Weaviate Collection | +| Supported key property types | Guid | +| Supported data property types |
  • string
  • byte
  • short
  • int
  • long
  • double
  • float
  • decimal
  • bool
  • *and iterables of each of these types*
| +| Supported vector property types |
  • list[float]
  • list[int]
  • ndarray
| +| Supported index types |
  • Hnsw
  • Flat
  • Dynamic
| +| Supported distance functions |
  • CosineDistance
  • NegativeDotProductSimilarity
  • EuclideanSquaredDistance
  • Hamming
  • ManhattanDistance
| +| Supported filter clauses |
  • AnyTagEqualTo
  • EqualTo
| +| Supports multiple vectors in a record | Yes | +| IsFilterable supported? | Yes | +| IsFullTextSearchable supported? | Yes | + +::: zone-end +::: zone pivot="programming-language-java" +Coming soon. +::: zone-end ## Limitations Notable Weaviate connector functionality limitations. -| Feature Area | Workaround | -|------------------------------------------------------------------------| -----------------------------------------------------------------------------------------------| -| Using the 'vector' property for single vector objects is not supported | Use of the 'vectors' property is supported instead. | +| Feature Area | Workaround | +| ---------------------------------------------------------------------- | --------------------------------------------------- | +| Using the 'vector' property for single vector objects is not supported | Use of the 'vectors' property is supported instead. | > [!WARNING] > Weaviate requires collection names to start with an upper case letter. If you do not provide a collection name with an upper case letter, Weaviate will return an error when you try and create your collection. The error that you will see is `Cannot query field "mycollection" on type "GetObjectsObj". Did you mean "Mycollection"?` where `mycollection` is your collection name. In this example, if you change your collection name to `Mycollection` instead, this will fix the error. @@ -181,9 +203,64 @@ public class Hotel ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Getting Started -More info coming soon. +Add the Weaviate Vector Store connector dependencies to your project. + +```bash +pip install semantic-kernel[weaviate] +``` + +You can then create the vector store, it uses environment settings to connect: + +For using Weaviate Cloud: + +- url: WEAVIATE_URL +- api_key: WEAVIATE_API_KEY + +For using Weaviate Local (i.e. Weaviate in a Docker container): + +- local_host: WEAVIATE_LOCAL_HOST +- local_port: WEAVIATE_LOCAL_PORT +- local_grpc_port: WEAVIATE_LOCAL_GRPC_PORT + +If you want to use embedded: + +- use_embed: WEAVIATE_USE_EMBED + +These should be set exclusively, so only one set of the above is present, otherwise it will raise an exception. + +```python +from semantic_kernel.connectors.memory.weaviate import WeaviateStore + +store = WeaviateStore() +``` + +Alternatively, you can also pass in your own mongodb client if you want to have more control over the client construction: + +```python +import weaviate +from semantic_kernel.connectors.memory.weaviate import WeaviateStore + +client = weaviate.WeaviateAsyncClient(...) +store = WeaviateStore(async_client=client) +``` + +You can also create a collection directly, without the store. + +```python +from semantic_kernel.connectors.memory.weaviate import WeaviateCollection + +# `hotel` is a class created with the @vectorstoremodel decorator +collection = WeaviateCollection( + collection_name="my_collection", + data_model_type=hotel +) +``` + +## Serialization + +The Weaviate client returns it's own objects which are parsed and turned into dicts in the regular flow, for more details on this concept see the [serialization documentation](./../serialization.md). ::: zone-end ::: zone pivot="programming-language-java" diff --git a/semantic-kernel/concepts/vector-store-connectors/vector-search.md b/semantic-kernel/concepts/vector-store-connectors/vector-search.md index 27d71481..8d6855ac 100644 --- a/semantic-kernel/concepts/vector-store-connectors/vector-search.md +++ b/semantic-kernel/concepts/vector-store-connectors/vector-search.md @@ -19,7 +19,7 @@ Semantic Kernel provides vector search capabilities as part of its Vector Store ## Vector Search -The `VectorizedSearchAsync` method allows searching using data that has already been vectorized. This method takes a vector and an optional `VectorSearchOptions` class as input. +The `VectorizedSearchAsync` method allows searching using data that has already been vectorized. This method takes a vector and an optional `VectorSearchOptions` class as input. This method is available on the following interfaces: 1. `IVectorizedSearch` @@ -64,26 +64,23 @@ await foreach (var record in searchResult.Results) ## Supported Vector Types `VectorizedSearchAsync` takes a generic type as the vector parameter. -The types of vectors supported y each data store vary. +The types of vectors supported by each data store vary. See [the documentation for each connector](./out-of-the-box-connectors/index.md) for the list of supported vector types. It is also important for the search vector type to match the target vector that is being searched, e.g. if you have two vectors on the same record with different vector types, make sure that the search vector you supply matches the type of the specific vector you are targeting. -See [VectorPropertyName](#vectorpropertyname) for how to pick a target vector if you have more than one per record. +See [VectorProperty](#vectorproperty) for how to pick a target vector if you have more than one per record. ## Vector Search Options -The following options can be provided using the `VectorSearchOptions` class. - -### VectorPropertyName +The following options can be provided using the `VectorSearchOptions` class. -The `VectorPropertyName` option can be used to specify the name of the vector property to target during the search. -If none is provided, the first vector found on the data model or specified in the record definition will be used. +### VectorProperty -Note that when specifying the `VectorPropertyName`, use the name of the property as defined on the data model or in the record definition. -Use this property name even if the property may be stored under a different name in the vector store. The storage name may e.g. be different -because of custom serialization settings. +The `VectorProperty` option can be used to specify the vector property to target during the search. +If none is provided and the data model contains only one vector, that vector will be used. +If the data model contains no vector or multiple vectors and `VectorProperty` is not provided, the search method will throw. ```csharp using Microsoft.Extensions.VectorData; @@ -93,13 +90,13 @@ var vectorStore = new InMemoryVectorStore(); var collection = vectorStore.GetCollection("skproducts"); // Create the vector search options and indicate that we want to search the FeatureListEmbedding property. -var vectorSearchOptions = new VectorSearchOptions +var vectorSearchOptions = new VectorSearchOptions { - VectorPropertyName = nameof(Product.FeatureListEmbedding) + VectorProperty = r => r.FeatureListEmbedding }; // This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. -var searchResult = await collection.VectorizedSearchAsync(searchVector, vectorSearchOptions).Results.ToListAsync(); +var searchResult = await collection.VectorizedSearchAsync(searchVector, vectorSearchOptions); public sealed class Product { @@ -128,7 +125,7 @@ Top and Skip can be used to do paging if you wish to retrieve a large number of ```csharp // Create the vector search options and indicate that we want to skip the first 40 results and then get the next 20. -var vectorSearchOptions = new VectorSearchOptions +var vectorSearchOptions = new VectorSearchOptions { Top = 20, Skip = 40 @@ -157,7 +154,7 @@ The default value for `IncludeVectors` is `false`. ```csharp // Create the vector search options and indicate that we want to include vectors in the search results. -var vectorSearchOptions = new VectorSearchOptions +var vectorSearchOptions = new VectorSearchOptions { IncludeVectors = true }; @@ -172,9 +169,9 @@ await foreach (var result in searchResult.Results) } ``` -### VectorSearchFilter +### Filter -The `VectorSearchFilter` option can be used to provide a filter for filtering the records in the chosen collection +The vector search filter option can be used to provide a filter for filtering the records in the chosen collection before applying the vector search. This has multiple benefits: @@ -191,22 +188,16 @@ set the `IsFilterable` property to true when defining your data model or when cr > [!TIP] > For more information on how to set the `IsFilterable` property, refer to [VectorStoreRecordDataAttribute parameters](./defining-your-data-model.md#vectorstorerecorddataattribute-parameters) or [VectorStoreRecordDataProperty configuration settings](./schema-with-record-definition.md#vectorstorerecorddataproperty-configuration-settings). -To create a filter use the `VectorSearchFilter` class. You can combine multiple filter clauses together in one `VectorSearchFilter`. -All filter clauses are combined with `and`. -Note that when providing a property name when constructing the filter, use the name of the property as defined on the data model or in the record definition. -Use this property name even if the property may be stored under a different name in the vector store. The storage name may e.g. be different -because of custom serialization settings. +Filters are expressed using LINQ expressions based on the type of the data model. +The set of LINQ expressions supported will vary depending on the functionality supported +by each database, but all databases support a broad base of common expressions, e.g. equals, +not equals, and, or, etc. ```csharp -// Filter where Category == 'External Definitions' and Tags contain 'memory'. -var filter = new VectorSearchFilter() - .EqualTo(nameof(Glossary.Category), "External Definitions") - .AnyTagEqualTo(nameof(Glossary.Tags), "memory"); - // Create the vector search options and set the filter on the options. -var vectorSearchOptions = new VectorSearchOptions +var vectorSearchOptions = new VectorSearchOptions { - Filter = filter + Filter = r => r.Category == "External Definitions" && r.Tags.Contains("memory") }; // This snippet assumes searchVector is already provided, having been created using the embedding model of your choice. @@ -242,15 +233,6 @@ sealed class Glossary } ``` -#### EqualTo filter clause - -Use `EqualTo` for a direct comparison between property and value. - -#### AnyTagEqualTo filter clause - -Use `AnyTagEqualTo` to check if any of the strings, stored in a tag property in the vector store, contains a provided value. -For a property to be considered a tag property, it needs to be a List, array or other enumerable of string. - ::: zone-end ::: zone pivot="programming-language-python" diff --git a/semantic-kernel/index.yml b/semantic-kernel/index.yml index 3f94c798..b8a4c02f 100644 --- a/semantic-kernel/index.yml +++ b/semantic-kernel/index.yml @@ -51,7 +51,7 @@ productDirectory: links: - url: /semantic-kernel/concepts/enterprise-readiness/observability/index text: Observability - - url: /semantic-kernel/concepts/enterprise-readiness/SECURITY + - url: /semantic-kernel/support/SECURITY.md text: Security - url: /semantic-kernel/concepts/enterprise-readiness/filters text: Filters diff --git a/semantic-kernel/media/agentSKdocs.png b/semantic-kernel/media/agentSKdocs.png new file mode 100644 index 00000000..e90d9fa4 Binary files /dev/null and b/semantic-kernel/media/agentSKdocs.png differ diff --git a/semantic-kernel/media/agentSKdocs2.png b/semantic-kernel/media/agentSKdocs2.png new file mode 100644 index 00000000..1d897fc6 Binary files /dev/null and b/semantic-kernel/media/agentSKdocs2.png differ diff --git a/semantic-kernel/media/agentSKdocs3.png b/semantic-kernel/media/agentSKdocs3.png new file mode 100644 index 00000000..e7fee71f Binary files /dev/null and b/semantic-kernel/media/agentSKdocs3.png differ diff --git a/semantic-kernel/media/agentSKdocs4.png b/semantic-kernel/media/agentSKdocs4.png new file mode 100644 index 00000000..816ada71 Binary files /dev/null and b/semantic-kernel/media/agentSKdocs4.png differ diff --git a/semantic-kernel/support/migration/toc.yml b/semantic-kernel/support/migration/toc.yml index 1718b55c..aa17fe21 100644 --- a/semantic-kernel/support/migration/toc.yml +++ b/semantic-kernel/support/migration/toc.yml @@ -11,4 +11,6 @@ - name: Kernel Events and Filters Migration href: kernel-events-and-filters-migration.md - name: Agent Framework Release Candidate Migration Guide - href: agent-framework-rc-migration-guide.md \ No newline at end of file + href: agent-framework-rc-migration-guide.md +- name: Vector Store changes - March 2025 + href: vectorstore-march-2025.md \ No newline at end of file diff --git a/semantic-kernel/support/migration/vectorstore-march-2025.md b/semantic-kernel/support/migration/vectorstore-march-2025.md new file mode 100644 index 00000000..dececefe --- /dev/null +++ b/semantic-kernel/support/migration/vectorstore-march-2025.md @@ -0,0 +1,149 @@ +--- +title: Vector Store changes - March 2025 +description: Describes the changes included in the March 2025 Vector Store release and how to migrate +zone_pivot_groups: programming-languages +author: westey-m +ms.topic: conceptual +ms.author: westey +ms.date: 03/06/2025 +ms.service: semantic-kernel +--- +::: zone pivot="programming-language-csharp" + +# Vector Store changes - March 2025 + +## LINQ based filtering + +When doing vector searches it is possible to create a filter (in addition to the vector similarity) +that act on data properties to constrain the list of records matched. + +This filter is changing to support more filtering options. Previously the filter would +have been expressed using a custom `VectorSearchFilter` type, but with this update the filter +would be expressed using LINQ expressions. + +The old filter clause is still preserved in a property called OldFilter, and will be removed in future. + +```csharp +// Before +var searchResult = await collection.VectorizedSearchAsync( + searchVector, + new() { Filter = new VectorSearchFilter().EqualTo(nameof(Glossary.Category), "External Definitions") }); + +// After +var searchResult = await collection.VectorizedSearchAsync( + searchVector, + new() { Filter = g => g.Category == "External Definitions" }); + +// The old filter option is still available +var searchResult = await collection.VectorizedSearchAsync( + searchVector, + new() { OldFilter = new VectorSearchFilter().EqualTo(nameof(Glossary.Category), "External Definitions") }); +``` + +## Target Property Selection for Search + +When doing a vector search, it is possible to choose the vector property that the search should +be executed against. +Previously this was done via an option on the `VectorSearchOptions` class called `VectorPropertyName`. +`VectorPropertyName` was a string that could contain the name of the target property. + +`VectorPropertyName` is being obsoleted in favour of a new property called `VectorProperty`. +`VectorProperty` is an expression that references the required property directly. + +```csharp +// Before +var options = new VectorSearchOptions() { VectorPropertyName = "DescriptionEmbedding" }; + +// After +var options = new VectorSearchOptions() { VectorProperty = r => r.DescriptionEmbedding }; +``` + +Specifying `VectorProperty` will remain optional just like `VectorPropertyName` was optional. +The behavior when not specifying the property name is changing. +Previously if not specifying a target property, and more than one vector property existed on the +data model, the search would target the first available vector property in the schema. + +Since the property which is 'first' can change in many circumstances unrelated to the search code, using this +strategy is risky. We are therefore changing this behavior, so that if there are more than +one vector property, one must be chosen. + +## `VectorSearchOptions` change to generic type + +The `VectorSearchOptions` class is changing to `VectorSearchOptions`, to accomodate the +LINQ based filtering and new property selectors metioned above. + +If you are currently constructing the options class without providing the name of the options class +there will be no change. E.g. `VectorizedSearchAsync(embedding, new() { Top = 5 })`. + +On the other hand if you are using `new` with the type name, you will need to add the record type as a +generic parameter. + +```csharp +// Before +var options = new VectorSearchOptions() { Top = 5 }; + +// After +var options = new VectorSearchOptions() { Top = 5 }; +``` + +## Removal of collection factories in favour of inheritance/decorator pattern + +Each VectorStore implementation allows you to pass a custom factory to use for +constructing collections. This pattern is being removed and the recommended approach +is now to inherit from the VectorStore where you want custom construction and override +the GetCollection method. + +```csharp +// Before +var vectorStore = new QdrantVectorStore( + new QdrantClient("localhost"), + new() + { + VectorStoreCollectionFactory = new CustomQdrantCollectionFactory(productDefinition) + }); + +// After +public class QdrantCustomCollectionVectorStore(QdrantClient qdrantClient) : QdrantVectorStore(qdrantClient) +{ + public override IVectorStoreRecordCollection GetCollection(string name, VectorStoreRecordDefinition? vectorStoreRecordDefinition = null) + { + // custom construction logic... + } +} + +var vectorStore = new QdrantCustomCollectionVectorStore(new QdrantClient("localhost")); +``` + +## Removal of DeleteRecordOptions and UpsertRecordOptions + +The `DeleteRecordOptions` and `UpsertRecordOptions` parameters have been removed from the +`DeleteAsync`, `DeleteBatchAsync`, `UpsertAsync` and `UpsertBatchAsync` methods on the +`IVectorStoreRecordCollection` interface. + +These parameters were all optional and the options classes did not contain any options to set. + +If you were passing these options in the past, you will need to remove these with this update. + +```csharp +// Before +collection.DeleteAsync("mykey", new DeleteRecordOptions(), cancellationToken); + +// After +collection.DeleteAsync("mykey", cancellationToken); +``` + +::: zone-end +::: zone pivot="programming-language-python" + +## Not Applicable + +These changes are currently only applicable in C# + +::: zone-end +::: zone pivot="programming-language-java" + +## Coming soon + +These changes are currently only applicable in C# + +::: zone-end