Merge pull request #3001 from liyun95/v2.5.x

add index types: hnsw, flat, ivf, ivf-pq

site/en/menuStructure/en.json

@@ -627,9 +627,41 @@
         "children": [
           {
             "label": "Vector Indexes",
-            "id": "index-vector-fields.md",
+            "id": "vector-index",
             "order": 0,
-            "children": []
+            "isMenu": true,
+            "children": [
+              {
+                "label": "Index Vector Fields",
+                "id": "index-vector-fields.md",
+                "order": 0,
+                "children": []
+              },
+              {
+                "label": "FLAT",
+                "id": "flat.md",
+                "order": 1,
+                "children": []
+              },
+              {
+                "label": "IVF_FLAT",
+                "id": "ivf-flat.md",
+                "order": 2,
+                "children": []
+              },
+              {
+                "label": "IVF_PQ",
+                "id": "ivf-pq.md",
+                "order": 3,
+                "children": []
+              },
+              {
+                "label": "HNSW",
+                "id": "hnsw.md",
+                "order": 3,
+                "children": []
+              }
+            ]
           },
           {
             "label": "Scalar Indexes",

site/en/userGuide/manage-indexes/floating-vector/flat.md

@@ -0,0 +1,56 @@
+---
+id: flat.md
+order: 0
+summary: This article will introduce the FLAT index in Milvus.
+title: FLAT
+---
+
+# FLAT
+
+The **FLAT** index is one of the simplest and most straightforward methods for indexing and searching floating-point vectors. It relies on a brute-force approach, where each query vector is directly compared to every vector in the dataset, without any advanced preprocessing or data structuring. This approach guarantees accuracy, offering 100% recall, as every potential match is evaluated.
+
+However, this exhaustive search method comes with trade-offs. The FLAT index is the slowest indexing option compared to other index types in Milvus, as it performs a full scan of the dataset for every query. Consequently, it is not well-suited for environments with massive datasets, where performance is a concern. The primary advantage of the FLAT index is its simplicity and reliability, as it requires no training or complex parameter configurations.
+
+## Build index
+
+To build an `FLAT` index on a vector field in Milvus, use the `add_index()` method, specifying the `index_type` and `metric_type` parameters for the index.
+
+```python
+from pymilvus import MilvusClient
+
+# Prepare index building params
+index_params = MilvusClient.prepare_index_params()
+
+index_params.add_index(
+    field_name="your_vector_field_name", # Name of the vector field to be indexed
+    index_type="FLAT", # Type of the index to create
+    index_name="vector_index", # Name of the index to create
+    metric_type="L2", # Metric type used to measure similarity
+    params={} # No additional parameters required for FLAT
+)
+```
+
+In this configuration:
+
+- `index_type`: The type of index to be built. In this example, set the value to `FLAT`.
+- `metric_type`: The method used to calculate the distance between vectors. Supported values include `COSINE`, `L2`, and `IP`. For details, refer to [Metric Types](metric.md).
+- `params`: No extra parameters are needed for the FLAT index.
+
+Once the index parameters are configured, you can create the index by using the `create_index()` method directly or passing the index params in the `create_collection` method. For details, refer to [Create Collection](create-collection.md).
+
+## Search on index
+
+Once the index is built and entities are inserted, you can perform similarity searches on the index.
+
+```python
+res = MilvusClient.search(
+    collection_name="your_collection_name", # Collection name
+    data=[[0.1, 0.2, 0.3, 0.4, 0.5]],  # Query vector
+    limit=3,  # TopK results to return
+    search_params={"params": {}}  # No additional parameters required for FLAT
+)
+```
+
+## Index params
+
+For the FLAT index, no additional parameters are needed either during the index creation or the search process.

site/en/userGuide/manage-indexes/floating-vector/hnsw.md

@@ -0,0 +1,116 @@
+---
+id: hnsw.md
+order: 1
+summary: This article will introduce the HNSW index in Milvus.
+title: HNSW
+---
+
+# HNSW
+
+The **HNSW** index is a **graph-based** indexing algorithm that can improve performance when searching for high-dimensional floating vectors. It offers **excellent** search accuracy and **low** latency, while it requires **high** memory overhead to maintain its hierarchical graph structure.
+
+## Overview
+
+The Hierarchical Navigable Small World (HNSW) algorithm builds a multi-layered graph, kind of like a map with different zoom levels. The **bottom layer** contains all the data points, while the **upper layers** consist of a subset of data points sampled from the lower layer.
+
+In this hierarchy, each layer contains nodes representing data points, connected by edges that indicate their proximity. The higher layers provide long-distance jumps to quickly get close to the target, while the lower layers enable a fine-grained search for the most accurate results.
+
+Here's how it works:
+
+1. **Entry point**: The search starts at a fixed entry point at the top layer, which is a pre-determined node in the graph.
+2. **Greedy search**: The algorithm greedily moves to the closest neighbor at the current layer until it cannot get any closer to the query vector. The upper layers serve a navigational purpose, acting as a coarse filter to locate potential entry points for the finer search at the lower levels.
+3. **Layer descend**: Once a **local minimum** is reached at the current layer, the algorithm jumps down to the lower layer, using a pre-established connection, and repeats the greedy search.
+4. **Final** **refinement**: This process continues until the bottom layer is reached, where a final refinement step identifies the nearest neighbors.
+
+![HNSW](../../../../../assets/hnsw.png)
+
+The performance of HNSW depends on several key parameters that control both the structure of the graph and the search behavior. These include:
+
+- `M`: The maximum number of edges or connections each node can have in the graph at each level of the hierarchy. A higher `M` results in a denser graph and increases recall and accuracy as the search has more pathways to explore, which also consumes more memory and slows down insertion time due to additional connections. As shown in the image above, **M = 5** indicates that each node in the HNSW graph is directly connected to a maximum of 5 other nodes. This creates a moderately dense graph structure where nodes have multiple pathways to reach other nodes.
+- `efConstruction`: The number of candidates considered during index construction. A higher `efConstruction` generally results in a better quality graph but requires more time to build.
+- `ef`: The number of neighbors evaluated during a search. Increasing `ef` improves the likelihood of finding the nearest neighbors but slows down the search process.
+
+For details on how to adjust these settings to suit your needs, refer to [Index params](#index-params).
+
+## Build index
+
+To build an `HNSW` index on a vector field in Milvus, use the `add_index()` method, specifying the `index_type`, `metric_type`, and additional parameters for the index.
+
+```python
+from pymilvus import MilvusClient
+
+# Prepare index building params
+index_params = MilvusClient.prepare_index_params()
+
+index_params.add_index(
+    field_name="your_vector_field_name", # Name of the vector field to be indexed
+    index_type="HNSW", # Type of the index to create
+    index_name="vector_index", # Name of the index to create
+    metric_type="L2", # Metric type used to measure similarity
+    params={
+        "M": 64, # Maximum number of neighbors each node can connect to in the graph
+        "efConstruction": 100 # Number of candidate neighbors considered for connection during index construction
+    } # Index building params
+)
+```
+
+In this configuration:
+
+- `index_type`: The type of index to be built. In this example, set the value to `HNSW`.
+- `metric_type`: The method used to calculate the distance between vectors. Supported values include `COSINE`, `L2`, and `IP`. For details, refer to [Metric Types](metric.md).
+- `params`: Additional configuration options for building the index.
+    - `M`: Maximum number of neighbors each node can connect to.
+    - `efConstruction`: Number of candidate neighbors considered for connection during index construction.
+
+    To learn more building parameters available for the `HNSW` index, refer to [Index building params](#Index-building-params).
+
+
+Once the index parameters are configured, you can create the index by using the `create_index()` method directly or passing the index params in the `create_collection` method. For details, refer to [Create Collection](create-collection.md).
+
+## Search on index
+
+Once the index is built and entities are inserted, you can perform similarity searches on the index.
+
+```python
+search_params = {
+    "params": {
+        "ef": 10, # Number of neighbors to consider during the search
+    }
+}
+
+res = MilvusClient.search(
+    collection_name="your_collection_name", # Collection name
+    data=[[0.1, 0.2, 0.3, 0.4, 0.5]],  # Query vector
+    limit=10,  # TopK results to return
+    search_params=search_params
+)
+```
+
+In this configuration:
+
+- `params`: Additional configuration options for searching on the index.
+    - `ef`: Number of neighbors to consider during a search.
+
+    To learn more search parameters available for the `HNSW` index, refer to [Index-specific search params](#index-specific-search-params).
+
+
+## Index params
+
+This section provides an overview of the parameters used for building an index and performing searches on the index.
+
+### Index building params
+
+The following table lists the parameters that can be configured in `params` when [building an index](#Build-index).
+
+| **Parameter** | **Description** | **Value Range** | **Tuning Suggestion** |
+| --- | --- | --- | --- |
+| `M` | Maximum number of connections （or edges) each node can have in the graph, including both outgoing and incoming edges.<br>This parameter directly affects both index construction and search. | **Type**: Integer<br>**Range**: [2, 2048]<br>**Default value**: `30` (up to 30 outgoing and 30 incoming edges per node) | A larger `M` generally leads to **higher accuracy** but **increases memory overhead** and **slows down both index building and search**.<br>Consider increasing `M` for datasets with high dimensionality or when high recall is crucial.<br>Consider decreasing `M` when memory usage and search speed are primary concerns.<br>In most cases, we recommend you set a value within this range: [5, 100]. |
+| `efConstruction` | Number of candidate neighbors considered for connection during index construction.<br>A larger pool of candidates is evaluated for each new element, but the maximum number of connections actually established is still limited by `M`. | **Type**: Integer<br>**Range**: [1, *int_max*]<br>**Default value**: `360` | A higher `efConstruction` typically results in a **more accurate index**, as more potential connections are explored. However, this also leads to **longer indexing time and increased memory usage** during construction.<br>Consider increasing `efConstruction` for improved accuracy, especially in scenarios where indexing time is less critical.<br>Consider decreasing `efConstruction` to speed up index construction when resource constraints are a concern.<br>In most cases, we recommend you set a value within this range: [50, 500]. |
+
+### Index-specific search params
+
+The following table lists the parameters that can be configured in `search_params.params` when [searching on the index](#Search-on-index).
+
+| **Parameter** | **Description** | **Value Range** | **Tuning Suggestion** |
+| --- | --- | --- | --- |
+| `ef` | **Controls the breadth of search during nearest neighbor retrieval.** It determines how many nodes are visited and evaluated as potential nearest neighbors. This parameter affects only the search process and applies exclusively to the bottom layer of the graph. | **Type**: Integer<br>**Range**: [1, *int_max*]<br>**Default value**: *limit* (TopK nearest neighbors to return) | A larger `ef` generally leads to **higher search accuracy** as more potential neighbors are considered. However, this also **increases search time**.<br>Consider increasing `ef` when achieving high recall is critical and search speed is less of a concern.<br>Consider decreasing `ef` to prioritize faster searches, especially in scenarios where a slight reduction in accuracy is acceptable.<br>In most cases, we recommend you set a value within this range: [K, 10K]. |

site/en/userGuide/manage-indexes/floating-vector/ivf-flat.md

@@ -0,0 +1,118 @@
+---
+id: ivf-flat.md
+order: 1
+summary: This article will introduce the IVF_FLAT index in Milvus.
+title: IVF_FLAT
+---
+
+# IVF_FLAT
+
+The **IVF_FLAT** index is an indexing algorithm that can improve search performance for floating-point vectors.
+
+This index type is ideal for large-scale datasets that require fast query responses and high accuracy, especially when clustering your dataset can reduce the search space and sufficient memory is available to store cluster data.
+
+## Overview
+
+The term **IVF_FLAT** stands for **Inverted File Flat**, which encapsulates its dual-layered approach to indexing and searching for floating-point vectors:
+
+- **Inverted File (IVF):** Refers to clustering the vector space into manageable regions using [k-means clustering](https://en.wikipedia.org/wiki/K-means_clustering). Each cluster is represented by a **centroid**, serving as a reference point for the vectors within.
+- **Flat:** Indicates that within each cluster, vectors are stored in their original form (flat structure), without any compression or quantization, for precise distance computations.
+
+The following figure shows how it works:
+
+![ivf-flat-1.png](../../../../../assets/ivf-1.png)
+
+This indexing method speeds up the search process, but it comes with a potential drawback: the candidate found as the nearest to the query embedding may not be the exact nearest one. This can happen if the nearest embedding to the query embedding resides in a cluster different from the one selected based on the nearest centroid (see visualization below).
+
+To address this issue, **IVF_FLAT** provides two hyperparameters that we can tune:
+
+- `nlist`: Specifies the number of partitions to create using the k-means algorithm.
+- `nprobe`: Specifies the number of partitions to consider during the search for candidates.
+
+Now if we set `nprobe` to 3 instead of 1, we get the following result:
+
+![ivf-flat-2.png](../../../../../assets/ivf-2.png)
+
+By increasing the `nprobe` value, you can include more partitions in the search, which can help ensure that the nearest embedding to the query is not missed, even if it resides in a different partition. However, this comes at the cost of increased search time, as more candidates need to be evaluated. For more information on index parameter tuning, refer to [Index params](#index-params).
+
+## Build index
+
+To build an `IVF_FLAT` index on a vector field in Milvus, use the `add_index()` method, specifying the `index_type`, `metric_type`, and additional parameters for the index.
+
+```python
+from pymilvus import MilvusClient
+
+# Prepare index building params
+index_params = MilvusClient.prepare_index_params()
+
+index_params.add_index(
+    field_name="your_vector_field_name", # Name of the vector field to be indexed
+    index_type="IVF_FLAT", # Type of the index to create
+    index_name="vector_index", # Name of the index to create
+    metric_type="L2", # Metric type used to measure similarity
+    params={
+        "nlist": 64, # Number of clusters for the index
+    } # Index building params
+)
+
+```
+
+In this configuration:
+
+- `index_type`: The type of index to be built. In this example, set the value to `IVF_FLAT`.
+- `metric_type`: The method used to calculate the distance between vectors. Supported values include `COSINE`, `L2`, and `IP`. For details, refer to [Metric Types](metric.md).
+- `params`: Additional configuration options for building the index.
+    - `nlist`: Number of clusters to divide the dataset.
+
+    To learn more building parameters available for the `IVF_FLAT` index, refer to [Index building params](#Index-building-params).
+
+
+Once the index parameters are configured, you can create the index by using the `create_index()` method directly or passing the index params in the `create_collection` method. For details, refer to [Create Collection](create-collection.md).
+
+## Search on index
+
+Once the index is built and entities are inserted, you can perform similarity searches on the index.
+
+```python
+search_params = {
+    "params": {
+        "nprobe": 10, # Number of clusters to search
+    }
+}
+
+res = MilvusClient.search(
+    collection_name="your_collection_name", # Collection name
+    data=[[0.1, 0.2, 0.3, 0.4, 0.5]],  # Query vector
+    limit=3,  # TopK results to return
+    search_params=search_params
+)
+
+```
+
+In this configuration:
+
+- `params`: Additional configuration options for searching on the index.
+    - `nprobe`: Number of clusters to search for.
+
+    To learn more search parameters available for the `IVF_FLAT` index, refer to [Index-specific search params](#index-specific-search-params).
+
+
+## Index params
+
+This section provides an overview of the parameters used for building an index and performing searches on the index.
+
+### Index building params
+
+The following table lists the parameters that can be configured in `params` when [building an index](#Build-index).
+
+| **Parameter** | **Description** | **Value Range** | **Tuning Suggestion** |
+| --- | --- | --- | --- |
+| `nlist` | The number of clusters to create using the k-means algorithm during index building.Each cluster, represented by a centroid, stores a list of vectors. Increasing this parameter reduces the number of vectors in each cluster, creating smaller, more focused partitions. | **Type**: Integer<br>**Range**: [1, 65536]<br>**Default value**: `128` | Larger `nlist` values improve recall by creating more refined clusters but increase index building time. Optimize based on dataset size and available resources.In most cases, we recommend you set a value within this range: [32, 4096]. |
+
+### Index-specific search params
+
+The following table lists the parameters that can be configured in `search_params.params` when [searching on the index](#Search-on-index).
+
+| **Parameter** | **Description**                                                                                                                                                                              | **Value Range**                                                | **Tuning Suggestion**                                                                                                                                                                                              |
+|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `nprobe`      | The number of clusters to search for candidates.Higher values allow more clusters to be searched, improving recall by expanding the search scope but at the cost of increased query latency. | **Type**: Integer<br>**Range**: [1, *nlist*]<br>**Default value**: `8` | Increasing this value improves recall but may slow down the search.Set `nprobe` proportionally to `nlist` to balance speed and accuracy.<br>In most cases, we recommend you set a value within this range: [1, nlist]. |