-
+## Deploy Milvus
-## Using Azure Blob Storage
+Now the Kubernetes cluster is ready. Let's deploy Milvus right now.
-Azure Blob Storage is Azure's version of AWS Simple Storage Service (S3).
+```bash
+helm repo add milvus https://zilliztech.github.io/milvus-helm/
+helm repo update
+helm install -f values.yaml my-release milvus/milvus
+```
-[MinIO Azure Gateway](https://blog.min.io/deprecation-of-the-minio-gateway/) allows accessing Azure. Essentially, MinIO Azure Gateway translates and forwards all connections to Azure by using APIs. You can use MinIO Azure Gateway instead of a MinIO server.
+In the preceding commands, we add the repo of Milvus Helm charts locally and update the repo to fetch the latest charts. Then we install a Milvus instance and name it **my-release**.
-### Set variables
+Notice the config `service.type` value, which indicates that we would like to expose the Milvus instance through a Layer-4 load balancer.
-Set variables before you use MinIO Azure Gateway. Modify the default values as needed.
-#### Metadata
+## Verify the deployment
-The following table lists the metadata that you can configure.
+Once all pods are running, run the following command to get the external IP address.
-|Option|Description|Default|
-|:---|:---|:---|
-|`minio.azuregateway.enabled`|Set the value to ```true``` to enable MinIO Azure Gateway.|`false`|
-|`minio.accessKey`|The MinIO access key.|`""`|
-|`minio.secretKey`|The MinIO secret key.|`""`|
-|`externalAzure.bucketName`|The name of the Azure bucket to use. Unlike an S3/MinIO bucket, an Azure bucket must be globally unique.|`""`|
+```bash
+kubectl get services|grep my-release-milvus|grep LoadBalancer|awk '{print $4}'
+```
-The following table lists the metadata that you might want to leave as default.
-|Option|Description|Default|
-|:---|:---|:---|
-|`minio.azuregateway.replicas`|The number of replica nodes to use for the gateway. We recommend that you use one because MinIO does not support well for more than one replica.|`1`|
+## Hello Milvus
-Continue to use all predefined MinIO metadata variables.
+Please refer to [Hello Milvus](https://milvus.io/docs/example_code.md), change the host value to external IP address, then run the code.
-The following example installs a chart named `my-release`.
-```shell
-helm install my-release ./milvus --set service.type=LoadBalancer --set minio.persistence.enabled=false --set externalAzure.bucketName=milvusbuckettwo --set minio.azuregateway.enabled=true --set minio.azuregateway.replicas=1 --set minio.accessKey=milvusstorage --set minio.secretKey=your-azure-key
-```
## What's next
If you want to learn how to deploy Milvus on other clouds:
diff --git a/preview/site/en/adminGuide/clouds/azure/ingress.md b/preview/site/en/adminGuide/clouds/azure/ingress.md
new file mode 100644
index 000000000..3be4488f8
--- /dev/null
+++ b/preview/site/en/adminGuide/clouds/azure/ingress.md
@@ -0,0 +1,143 @@
+---
+id: ingress.md
+title: Configure ingress nginx with Milvus
+related_key: ingress nginx
+summary: Learn how to configure ingress nginx with Milvus.
+---
+
+# Configure ingress nginx with Milvus
+This topic introduces how to configure ingress nginx with Milvus.
+For more details, refer to [ingress-nginx](https://learn.microsoft.com/en-us/azure/aks/ingress-tls?tabs=azure-cli).
+
+
+## Configure ingress nginx
+
+- Set env.
+```bash
+export DNS_LABEL="milvustest" # Your DNS label must be unique within its Azure location.
+export NAMESPACE="ingress-basic"
+```
+
+- Install ingress nginx
+```bash
+helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
+helm repo update
+helm install ingress-nginx ingress-nginx/ingress-nginx \
+ --create-namespace \
+ --namespace $NAMESPACE \
+ --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"=$DNS_LABEL \
+ --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz
+```
+
+- Get External IP address.
+```bash
+kubectl --namespace $NAMESPACE get services -o wide -w ingress-nginx-controller
+```
+
+- Configure an FQDN for your ingress controller.
+```bash
+# Public IP address of your ingress controller
+IP="MY_EXTERNAL_IP"
+
+# Get the resource-id of the public IP
+PUBLICIPID=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$IP')].[id]" --output tsv)
+
+# Update public IP address with DNS name
+az network public-ip update --ids $PUBLICIPID --dns-name $DNS_LABEL
+
+# Display the FQDN
+az network public-ip show --ids $PUBLICIPID --query "[dnsSettings.fqdn]" --output tsv
+# sample output: milvustest.eastus2.cloudapp.azure.com
+```
+
+
+## Install cert-manager
+
+```bash
+helm repo add jetstack https://charts.jetstack.io
+helm repo update
+helm install cert-manager jetstack/cert-manager \
+ --namespace $NAMESPACE \
+ --set installCRDs=true
+```
+
+## Create a CA cluster issuer
+
+- Create a cluster issuer, such as cluster-issuer.yaml, using the following example manifest. Replace MY_EMAIL_ADDRESS with a valid address from your organization.
+```yaml
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+ name: letsencrypt
+spec:
+ acme:
+ server: https://acme-v02.api.letsencrypt.org/directory
+ email: MY_EMAIL_ADDRESS
+ privateKeySecretRef:
+ name: letsencrypt
+ solvers:
+ - http01:
+ ingress:
+ class: nginx
+```
+
+- Apply the issuer using the kubectl apply command.
+```bash
+kubectl apply -f cluster-issuer.yaml
+```
+
+
+## Deploy Milvus
+refer to [Azure](https://milvus.io/docs/azure.md), notice the config `service.type` value, you need change to `ClusterIP`.
+
+
+## Create Milvus ingress route
+```bash
+kubectl apply -f ingress.yaml
+```
+
+the ingress.yaml contents:
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: my-release-milvus
+ annotations:
+ cert-manager.io/cluster-issuer: letsencrypt
+ nginx.ingress.kubernetes.io/backend-protocol: GRPC
+ nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+ nginx.ingress.kubernetes.io/proxy-body-size: 2048m
+spec:
+ ingressClassName: nginx
+ tls:
+ - hosts:
+ - milvustest.eastus2.cloudapp.azure.com # the FQDN
+ secretName: tls-secret
+ rules:
+ - host: milvustest.eastus2.cloudapp.azure.com
+ http:
+ paths:
+ - path: /
+ pathType: Prefix
+ backend:
+ service:
+ name: my-release-milvus
+ port:
+ number: 19530
+```
+
+## Verify
+```bash
+kubectl get certificate
+NAME READY SECRET AGE
+tls-secret True tls-secret 8m7s
+kubectl get ingress
+NAME CLASS HOSTS ADDRESS PORTS AGE
+my-release-milvus nginx milvustest.eastus2.cloudapp.azure.com EXTERNAL-IP 80, 443 8m15s
+```
+
+## Hello Milvus
+Please refer to [Hello Milvus](https://milvus.io/docs/example_code.md), change uri args, then run the code.
+```python
+connections.connect("default",uri="https://milvustest.eastus2.cloudapp.azure.com:443")
+```
diff --git a/preview/site/en/adminGuide/clouds/deploy_on_clouds.md b/preview/site/en/adminGuide/clouds/deploy_on_clouds.md
deleted file mode 100644
index 93f209f34..000000000
--- a/preview/site/en/adminGuide/clouds/deploy_on_clouds.md
+++ /dev/null
@@ -1,14 +0,0 @@
----
-id: deploy_on_clouds.md
-title: Deploy on Clouds
----
-
-# Deploy on Clouds
-
-Milvus can be deployed on various cloud platforms.
-
-- AWS
- - [EC2](aws.md)
- - [EKS](eks.md)
-- [GCP](gcp.md)
-- [Azure](azure.md)
diff --git a/preview/site/en/adminGuide/clouds/gcp/gcp.md b/preview/site/en/adminGuide/clouds/gcp/gcp.md
index 670e9c5c1..8d03be1a1 100644
--- a/preview/site/en/adminGuide/clouds/gcp/gcp.md
+++ b/preview/site/en/adminGuide/clouds/gcp/gcp.md
@@ -5,13 +5,13 @@ related_key: cluster
summary: Learn how to deploy a Milvus cluster on GCP.
---
-## Deploy a Milvus Cluster on GCP
+# Deploy a Milvus Cluster on GCP
Milvus is a cloud-native vector database and can be deployed on various cloud environments. This guide walks you through every detail about setting up Milvus on Google Cloud Platform (GCP).
-### Before you start
+## Before you start
To deploy Milvus on GCP, ensure that
@@ -23,7 +23,7 @@ To deploy Milvus on GCP, ensure that
- You have [initialized the gcloud CLI](https://cloud.google.com/sdk/docs/install-sdk#initializing_the) with your GCP account credentials.
-### Set up the network
+## Set up the network
To ensure Milvus security, you need to create a logically isolated virtual network in your GCP project. The following command creates a VPC.
@@ -90,7 +90,7 @@ gcloud compute firewall-rules create allow-milvus-in \
--source-ranges=0.0.0.0/0
```
-### Provision a Kubernetes cluster
+## Provision a Kubernetes cluster
In this guide, we will use the Google Kubernetes Engine (GKE) service to provision a Kubernetes cluster with two nodes in the **us-west1-a** zone. Each node is an **e2-standard-4** Compute Engine virtual machine running the **COS_CONTAINERD** image.
@@ -101,19 +101,19 @@ You are advised to use the types of machines that offer a minimum memory of 16 G
```bash
-gcloud beta container clusters create "milvus-cluster-1" \
+gcloud container clusters create "milvus-cluster-1" \
--project "milvus-testing-nonprod" \
--zone "us-west1-a" \
+ --workload-pool "milvus-testing-nonprod.svc.id.goog" \
--no-enable-basic-auth \
- --cluster-version "1.20.8-gke.900" \
+ --cluster-version "1.27.3-gke.100" \
--release-channel "regular" \
- --machine-type "e2-standard-4" \
+ --machine-type "c2-standard-4" \
--image-type "COS_CONTAINERD" \
--disk-type "pd-standard" \
--disk-size "100" \
--max-pods-per-node "110" \
- --num-nodes "2" \
- --enable-stackdriver-kubernetes \
+ --num-nodes "3" \
--enable-ip-alias \
--network "projects/milvus-testing-nonprod/global/networks/milvus-network" \
--subnetwork "projects/milvus-testing-nonprod/regions/us-west1/subnetworks/milvus-network"
@@ -122,33 +122,67 @@ gcloud beta container clusters create "milvus-cluster-1" \
It would take a couple of minutes for the Kubernetes cluster to go up. Once the cluster is ready, use the following command to fetch its credentials so that you can run `kubectl` commands in your terminal to communicate with the cluster remotely.
```bash
-gcloud container clusters get-credentials milvus-cluster-1
+gcloud container clusters get-credentials milvus-cluster-1 --zone "us-west1-a"
```
-### Deploy Milvus
+## Use Google Cloud Storage (GCS) as external object storage
+
+- Create bucket.
+```bash
+gcloud storage buckets create gs://milvus-testing-nonprod --project=milvus-testing-nonprod --default-storage-class=STANDARD --location=us-west1 --uniform-bucket-level-access
+```
+- Generate User Access Key and Secret Key, you should go to your project’s storage page. In the left sidebar of the dashboard, click Google Cloud Storage and then Settings. Select the INTEROPERABILITY tab. If you haven't enabled it already, click on Interoperable Access. Then click CREATE A KEY button to create.
+
+
+
+- Add values.yaml
+```yaml
+cluster:
+ enabled: true
+
+service:
+ type: LoadBalancer
+
+minio:
+ enabled: false
+
+externalS3:
+ enabled: true
+ host: storage.googleapis.com
+ port: 443
+ rootPath: milvus/my-release
+ bucketName: milvus-testing-nonprod
+ cloudProvider: gcp
+ useSSL: true
+ accessKey: "20.81.111.155 in the the EXTERNAL-IP column is the IP address of the load balancer. The default Milvus port is 19530.
-/volumes folder according to the default con
Having finished modifying the configuration file and installation file, you can then start Milvus.
```
-$ sudo docker-compose up -d
+$ sudo docker compose up -d
```
## What's next
diff --git a/preview/site/en/adminGuide/configure-helm.md b/preview/site/en/adminGuide/configure-helm.md
index 71240f886..6477eaf19 100644
--- a/preview/site/en/adminGuide/configure-helm.md
+++ b/preview/site/en/adminGuide/configure-helm.md
@@ -3,6 +3,7 @@ id: configure-helm.md
label: Helm
related_key: configure
summary: Configure Milvus with Helm Charts.
+title: Configure Milvus with Helm Charts
---
# Configure Milvus with Helm Charts
@@ -79,7 +80,6 @@ Sorted by:
In addition, when configuring proxy, there is an extra field called `serviceType`. This field defines the type of service Milvus provides in the K8s cluster.
@@ -137,7 +138,6 @@ To configure resources for a specific component, add the component name in the f
Proxy
Query coord
Query node
- Index coord
Index node
Data coord
Data node
@@ -208,6 +208,7 @@ To configure resources for a specific component, add the component name in the f
Quota and Limits
@@ -165,9 +170,10 @@ Under this section, you can configure MinIO or S3 address, relevant access keys,
Root prefix of the key to where Milvus stores data in MinIO or S3.
- Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
It is recommended to change this parameter before starting Milvus for the first time.
+ To share an MinIO instance among multiple Milvus instances, consider changing this to a different value for each Milvus instance before you start them. For details, see Operation FAQs.
Set an easy-to-identify root key prefix for Milvus if etcd service already exists.
+ Changing this for an already running Milvus instance may result in failures to read legacy data.
files
diff --git a/preview/site/en/reference/sys_config/configure_nats.md b/preview/site/en/reference/sys_config/configure_nats.md
new file mode 100644
index 000000000..915339077
--- /dev/null
+++ b/preview/site/en/reference/sys_config/configure_nats.md
@@ -0,0 +1,266 @@
+---
+id: configure_nats.md
+related_key: configure
+group: system_configuration.md
+summary: Learn how to configure NATs for Milvus standalone.
+title: NATS-related Configurations
+---
+
+# NATS-related Configurations
+
+This topic introduces the NATs-related configurations of Milvus.
+
+NATS is a message-oriented middleware that allows data exchange between applications and services, segmented in the form of messages. Milvus uses NATS as a underlying engine for reliable storage and pub/sub of message streams. You can use it as an alternative to RocksMQ.
+
+Under this section, you can configure message size, retention time and size, etc.
+
+## `natsmq.server.port`
+
+
+
+
+
+## `natsmq.server.storeDir`
+
+
+
+
+
+## `natsmq.server.maxFileStore`
+
+
+
+
+
+## `natsmq.server.maxPayload`
+
+
+
+
+
+## `natsmq.server.maxPending`
+
+
+
+
+
+## `natsmq.server.initializeTimeout`
+
+
+
+
+
+## `natsmq.monitor.debug`
+
+
+
+
+
+## `natsmq.monitor.logTime`
+
+
+
+
+
+## `natsmq.monitor.logFile`
+
+
+
+
+
+## `natsmq.monitor.logSizeLimit`
+
+
+
+
+
+## `natsmq.rentention.maxAge`
+
+
+
+
+
+## `natsmq.rentention.maxBytes`
+
+
+
+
+
+## `natsmq.rentention.maxMsgs`
+
+
+
+
diff --git a/preview/site/en/reference/sys_config/configure_proxy.md b/preview/site/en/reference/sys_config/configure_proxy.md
index 2e3bafbd3..a5278b7b3 100644
--- a/preview/site/en/reference/sys_config/configure_proxy.md
+++ b/preview/site/en/reference/sys_config/configure_proxy.md
@@ -3,6 +3,7 @@ id: configure_proxy.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure proxy of Milvus.
+title: Proxy-related Configurations
---
# Proxy-related Configurations
@@ -238,3 +239,230 @@ Under this section, you can configure proxy port, system limits, etc.
+
+## `proxy.maxVectorFieldNum`
+
+
+
+
+
+## `proxy.accessLog.enable`
+
+
+
+
+
+## `proxy.accessLog.filename`
+
+
+
+
+
+## `proxy.accessLog.localPath`
+
+
+
+
+
+## `proxy.accessLog.maxSize`
+
+
+
+
+
+## `proxy.accessLog.rotatedTime`
+
+
+
+
+
+## `proxy.accessLog.maxBackups`
+
+
+
+
+
+## `proxy.accessLog.minioEnable`
+
+
+
+
+
+## `proxy.accessLog.remotePath`
+
+
+
+
+
+## `proxy.accessLog.remoteMaxTime`
+
+
+
+
+
+## `proxy.accessLog.base.format`
+
+
+
+
+
+## `proxy.accessLog..format`
+
+
+
+
+
+## `proxy.accessLog..methods`
+
+
+
+
diff --git a/preview/site/en/reference/sys_config/configure_pulsar.md b/preview/site/en/reference/sys_config/configure_pulsar.md
index af0a5f1dd..594e25cf6 100644
--- a/preview/site/en/reference/sys_config/configure_pulsar.md
+++ b/preview/site/en/reference/sys_config/configure_pulsar.md
@@ -3,6 +3,7 @@ id: configure_pulsar.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure Pulsar for Milvus cluster.
+title: Pulsar-related Configurations
---
# Pulsar-related Configurations
@@ -13,6 +14,12 @@ Pulsar is the underlying engine supporting Milvus cluster's reliable storage and
Under this section, you can configure Pulsar address, the message size, etc.
+
+
+
+
## `pulsar.maxMessageSize`
@@ -82,3 +109,42 @@ Under this section, you can configure Pulsar address, the message size, etc.
+## `pulsar.tenant`
+
+
+
+
+
+## `pulsar.namespace`
+
+
+
+
diff --git a/preview/site/en/reference/sys_config/configure_querycoord.md b/preview/site/en/reference/sys_config/configure_querycoord.md
index cd49e9972..d24965fc9 100644
--- a/preview/site/en/reference/sys_config/configure_querycoord.md
+++ b/preview/site/en/reference/sys_config/configure_querycoord.md
@@ -3,6 +3,7 @@ id: configure_querycoord.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure query coordinator of Milvus.
+title: Query Coordinator-related Configurations
---
# Query Coordinator-related Configurations
@@ -266,4 +267,4 @@ Under this section, you can configure query coord address, auto handoff, auto lo
104857600
-
+
\ No newline at end of file
diff --git a/preview/site/en/reference/sys_config/configure_querynode.md b/preview/site/en/reference/sys_config/configure_querynode.md
index 36c906cac..ca5f3b47a 100644
--- a/preview/site/en/reference/sys_config/configure_querynode.md
+++ b/preview/site/en/reference/sys_config/configure_querynode.md
@@ -3,6 +3,7 @@ id: configure_querynode.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure query node of Milvus.
+title: Query Node-related Configurations
---
# Query Node-related Configurations
@@ -135,6 +136,25 @@ Under this section, you can configure query node port, graceful time, etc.
+## `queryNode.replicas`
+
+
+
+
+
## `queryNode.stats.publishInterval`
@@ -190,7 +210,32 @@ Under this section, you can configure query node port, graceful time, etc.
+
+## `queryNode.segcore.InterimIndex`
+
+
+
+
\ No newline at end of file
+
diff --git a/preview/site/en/reference/sys_config/configure_quota_limits.md b/preview/site/en/reference/sys_config/configure_quota_limits.md
index 0658c7f6a..cb8f1430e 100644
--- a/preview/site/en/reference/sys_config/configure_quota_limits.md
+++ b/preview/site/en/reference/sys_config/configure_quota_limits.md
@@ -3,6 +3,7 @@ id: configure_quota_limits.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure quotas and limitations.
+title: Quota- and Limit-related configurations
---
# Quota- and Limit-related configurations
@@ -13,6 +14,23 @@ Some of these configuration items are used to set thresholds for Milvus to proac
Some of them are used to set backpressure signals that force Milvus to lower the rate of DDL/DML/DQL requests.
+## `quotaAndLimits.limits.maxCollectionNumPerDB`
+
+
+
+
+
## `quotaAndLimits.ddl.enabled`
@@ -224,6 +242,27 @@ Some of them are used to set backpressure signals that force Milvus to lower the
+## `quotaAndLimits.dml.insertRate.collection.max`
+
+
+
+
+
## `quotaAndLimits.dml.deleteRate.max`
@@ -236,8 +275,29 @@ Some of them are used to set backpressure signals that force Milvus to lower the
+
+## `quotaAndLimits.dml.deleteRate.collection.max`
+
+
+
+
+## `quotaAndLimits.dql.searchRate.collection.max`
+
+
+
+
+
## `quotaAndLimits.dql.queryRate.max`
@@ -304,6 +385,27 @@ Some of them are used to set backpressure signals that force Milvus to lower the
+## `quotaAndLimits.dql.queryRate.collection.max`
+
+
+
+
+
## `quotaAndLimits.limitWriting.ttProtection.enabled`
@@ -334,7 +436,7 @@ Some of them are used to set backpressure signals that force Milvus to lower the
+## `quotaAndLimits.limitWriting.diskProtection.diskQuotaPerCollection`
+
+
+
+
+
## `quotaAndLimits.limitWriting.forceDeny`
@@ -610,4 +733,4 @@ Some of them are used to set backpressure signals that force Milvus to lower the
\ No newline at end of file
+
diff --git a/preview/site/en/reference/sys_config/configure_rocksmq.md b/preview/site/en/reference/sys_config/configure_rocksmq.md
index bd201fdf0..e80ef6f46 100644
--- a/preview/site/en/reference/sys_config/configure_rocksmq.md
+++ b/preview/site/en/reference/sys_config/configure_rocksmq.md
@@ -3,6 +3,7 @@ id: configure_rocksmq.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure RocksMQ for Milvus standalone.
+title: RocksMQ-related Configurations
---
# RocksMQ-related Configurations
@@ -98,3 +99,42 @@ Under this section, you can configure message size, retention time and size, etc
+
+## `rocksmq.compactionInterval`
+
+
+
+
+
+## `rocksmq.lrucacheratio`
+
+
+
+
diff --git a/preview/site/en/reference/sys_config/configure_rootcoord.md b/preview/site/en/reference/sys_config/configure_rootcoord.md
index 287818545..beee0c14d 100644
--- a/preview/site/en/reference/sys_config/configure_rootcoord.md
+++ b/preview/site/en/reference/sys_config/configure_rootcoord.md
@@ -3,6 +3,7 @@ id: configure_rootcoord.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure root coordinator of Milvus.
+title: Root Coordinator-related Configurations
---
# Root Coordinator-related Configurations
@@ -276,4 +277,4 @@ Under this section, you can configure root coord address, index building thresho
86400
-
+
\ No newline at end of file
diff --git a/preview/site/en/reference/sys_config/system_configuration.md b/preview/site/en/reference/sys_config/system_configuration.md
index 35f5c0eba..8597db584 100644
--- a/preview/site/en/reference/sys_config/system_configuration.md
+++ b/preview/site/en/reference/sys_config/system_configuration.md
@@ -3,6 +3,7 @@ id: system_configuration.md
related_key: configure
group: system_configuration.md
summary: Learn about the system configuration of Milvus.
+title: Milvus System Configurations Checklist
---
# Milvus System Configurations Checklist
@@ -51,6 +52,19 @@ Under this section, you can configure message size, retention time and size, etc
See [RocksMQ-related Configurations](configure_rocksmq.md) for detailed description for each parameter under this section.
+### `nats`
+
+NATS is a message-oriented middleware that allows data exchange between applications and services, segmented in the form of messages. Milvus uses NATS as a underlying engine for reliable storage and pub/sub of message streams. You can use it as an alternative to RocksMQ.
+
+Under this section, you can configure NATS server, monitoring properties, and rention time and size, etc.
+
+See [NATS-related Configurations](configure_nats.md) for detailed description for each parameter under this section.
+
+### `kafka`
+
+Apache Kafka is an open-source distributed event streaming platform for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. It serves as an alternative to RocksMQ and Pulsar for reliable storage and publication/subscription of message streams.
+
+See [Kafka-related Configurations](configure_kafka.md) for detailed description for each parameter under this section.
### `rootCoord`
@@ -84,14 +98,6 @@ Under this section, you can configure query node port, graceful time, etc.
See [Query Node-related Configurations](configure_querynode.md) for detailed description for each parameter under this section.
-### `indexCoord`
-
-Index coordinator (index coord) manages topology of the index nodes, and maintains index metadata.
-
-Under this section, you can configure index coord address, etc.
-
-See [Index Coordinator-related Configurations](configure_indexcoord.md) for detailed description for each parameter under this section.
-
### `indexNode`
Index node builds indexes for vectors.
@@ -146,8 +152,6 @@ See [Common Configurations](configure_common.md) for detailed description for ea
### `knowhere`
-[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus.
-
Under this section, you can configure the default SIMD instruction set type of the system.
See [Knowhere-related Configurations](configure_knowhere.md) for detailed description for each parameter under this section.
@@ -237,5 +241,4 @@ The following parameters control the log output and object storage access.
- Learn how to [configure Milvus](configure-docker.md) before installation.
- Learn more about the installation of Milvus:
- - [Install Milvus Standalone](install_standalone-docker.md)
- - [Install Milvus Cluster](install_cluster-docker.md)
\ No newline at end of file
+ - [Install Milvus Standalone](install_standalone-docker.md)
\ No newline at end of file
diff --git a/preview/site/en/reference/timestamp.md b/preview/site/en/reference/timestamp.md
index 78ae33ca8..89ae0e9b6 100644
--- a/preview/site/en/reference/timestamp.md
+++ b/preview/site/en/reference/timestamp.md
@@ -30,7 +30,7 @@ Several timestamp-related parameters are involved when you conduct a vector simi
If the `Guarantee_timestamp` is not configured, Milvus automatically takes the point in time when the search request is made. Therefore, the search is conducted on a data view with all data updates by DML operations before the search.
-To save you the trouble of understanding the [TSO](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/milvus_hybrid_ts_en.md?from=from_parent_mindnote) inside Milvus, as a user, you do not have to directly configure the `Guarantee_timestamp` parameter. You only need to choose the [consistency level](https://milvus.io/docs/v2.1.x/consistency.md), and Milvus automatically handles the `Guarantee_timestamp` parameter for you. Each consistency level corresponds to a certain `Guarantee_timestamp` value.
+To save you the trouble of understanding the [TSO](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/20211214-milvus_hybrid_ts.md) inside Milvus, as a user, you do not have to directly configure the `Guarantee_timestamp` parameter. You only need to choose the [consistency level](https://milvus.io/docs/v2.1.x/consistency.md), and Milvus automatically handles the `Guarantee_timestamp` parameter for you. Each consistency level corresponds to a certain `Guarantee_timestamp` value.
.
@@ -78,21 +78,7 @@ As shown in the figure 1, the value of `Guarantee_timestamp` is set as `2021-08-
As shown in the figure 2 , the value of `Guarantee_timestamp` is set as `2021-08-26T18:15:01`, and `Graceful_time` as `2s`. The current value of `Service_timestamp` is only `2021-08-26T18:14:54`. This means that the expected DML operations are not completed yet and even given the 2 second of graceful time, data invisibility is still intolerable. Therefore, the query node needs to put off the search or query request until certain DML requests are completed (i.e. when `Service_timestamp` + `Graceful_time` >= `Guarantee_timestamp`).
-### `Travel_timestamp`
-
-`Travel_timestamp` is a type of configurable timestamp used to indicate that the search or query needs to be conducted on a data view before a certain point in time. `Travel_timestamp` can be seen as a data snapshot. And [search with Time Travel](timetravel.md) means conducting a search or query on the data snapshot indicated by the value of `Travel_timestamp`. All DML operations after `Travel_timestamp` are not involved in the search or query.
-
-When there is an incoming search or query request, suppose `Service_timestamp` > `Guarantee_timestamp`.
-
-.
-
-The value of `Guarantee_timestamp` has nothing to do with the Time Travel feature.
-
-As shown in the figure above, the value of `Guarantee_timestamp` is set as `2021-08-26T18:14:55`. And the `Service_timestamp` is grown to `2021-08-26T18:15:01` meaning that more DML operations are executed and completed after this time point. However, no matter the value of `Travel_timestamp` is `2021-08-26T18:14:56` or `2021-08-26T18:14:54`, only data before `Travel_timestamp` are searched or queried.
-
## What's next
- Learn how [guarantee timestamp enables tunable consistency in Milvus](consistency.md)
-- Learn how to [search with Time Travel](timetravel.md)
-
diff --git a/preview/site/en/reference/timetravel_ref.md b/preview/site/en/reference/timetravel_ref.md
deleted file mode 100644
index 05a6dcbdc..000000000
--- a/preview/site/en/reference/timetravel_ref.md
+++ /dev/null
@@ -1,74 +0,0 @@
----
-id: timetravel_ref.md
-related_key: Time Travel
-summary: Learn the design and implementation details of Time Travel in Milvus.
----
-
-# Time Travel
-
-This topic introduces the Time Travel feature in detail. See [Search with Time Travel](timetravel.md) for more information about how to use this feature.
-
-## Overview
-
-Time Travel is a feature that allows you to access historical data at any point within a specified time period, making it possible to query, restore, and back up data in the past. With Time Travel, you can:
-
-- Search or query data that has been deleted.
-
-- Restore data that has been deleted or updated.
-
-- Back up data before a specific point of time.
-
-Unlike traditional databases that use snapshots or retain data to support the Time Travel feature, the Milvus vector database maintains a timeline for all data insert or delete operations and adopts a timestamp mechanism. This means you can specify the timestamp in a search or query to retrieve data at a specific point of time in the past to significantly reduce maintenance costs.
-
-## Timestamp in Milvus
-
-In the Milvus vector database, each entity has its own timestamp attribute. All data manipulation language (DML) operations including data insertion and deletion, mark entities with a timestamp. For instance, if you inserted several entities all at one go, this batch of data will be marked with timestamps and share the same timestamp value.
-
-### DML operations
-
-When the proxy receives a data insert or delete request, it also gets a timestamp from the [root coord](https://milvus.io/docs/v2.1.x/four_layers.md#Root-coordinator-root-coord). Then, the proxy adds the timestamp as an additional field to the inserted or deleted data. Timestamp is a data field just like primary key (`pk`). The timestamp field is stored together with other data fields of a collection.
-
-When you load a collection to memory, all data in the collection, including their corresponding timestamps, are loaded into memory.
-
-### Search or query with Time Travel
-
-`travel_timestamp` is a timestamp specified by you to indicate that you need to conduct a query or search on the data view before this point in time. In parallel to a traditional database, you can consider `travel_timestamp` as a data snapshot, and the query or search needs to be conducted on the data in this snapshot.
-
-During a search, if the search request received by the proxy contains the parameter, `travel_timestamp`, the value of this parameter will be passed to [segcore](https://github.com/milvus-io/milvus/tree/master/docs/design_docs/segcore), the execution engine which supports concurrent insertion, deletion, query, index loading, monitoring and statistics of a segment data in memory. The segcore filters the search results by timestamp. In other words, you can deem the Time Travel feature as data filtering with the condition limited by the value of `travel_timestamp`. This means that data whose timestamp value is greater than `travel_timestamp` are filtered out and will not be involved in the search or query process. The expression for filtering is `data.timestamp <= travel_timestamp`.
-
-## Bitset for timestamp
-
-To go into details, searches and queries with filtering in [knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) are facilitated by bitset. And the underlying mechanism behind Time Travel is enabled by bitset.
-
-When a search is conducted, the segcore obtains a bitset indicating if the timestamp meets the condition. Then Milvus judges the range of data to query or search based on this bitset.
-
-### Sealed segment
-
-When loading a sealed segment, Milvus loads all the timestamps to memory and the segcore builds an index, `TimestampIndex` , on the timestamp field. The index contains information about the smallest and the largest timestamp value of this sealed segment, the offset and the row number of timestamps in this sealed segment.
-
-When you search with Time Travel, Milvus first filters the sealed segment according to the smallest and largest timestamp in the `TimestampIndex`:
-
-- If the value you set for `travel_timestamp` is greater than the largest timestamp of the segment, this means all the data in this segment meets the requirement. Therefore, the bitset of the data in this segment is marked as 1.
-- If the value you set for `travel_timestamp` is smaller than the smallest timestamp of the segment, this means the data in this segment does not meet the requirement. Therefore, the bitset of the data in this segment is marked as 0.
-- If the value you set for `travel_timestamp` is between the largest and the smallest timestamp of the segment, Milvus compares the timestamps in the segment one by one, and generates a bitset accordingly. In the bitset, if the data meet the requirement, they are marked with 1, and 0 if they do not.
-
-
-
-### Growing segment
-
-For growing segments, you do not need to load the collection to memory. All inserted data exists in memory, with the timestamp field attached. Data in growing segments are sorted according to the order of timestamp. When new data are inserted, they are added to the segment in the order of their timestamp. Segment data are organized in segcore memory in the same way.
-
-When you search with Time Travel, Milvus uses binary search to find the first offset, or the row number data, with their timestamp value greater than the value you set for the `travel_timestamp` parameter. Then subsequent operations including filtering and vector similarity search are conducted within this range of offsets.
-
-## What's next
-After learning how Time Travel works in Milvus, you might also want to:
-
-- Learn how to [search with Time Travel](timetravel.md)
-- Learn the [architecture](architecture_overview.md) of Milvus.
-- Understand [how data are processed](data_processing.md) in Milvus.
-
-
-
-
-
-
diff --git a/preview/site/en/reference/users_and_roles.md b/preview/site/en/reference/users_and_roles.md
index 2cf6589ce..ceb5b2823 100644
--- a/preview/site/en/reference/users_and_roles.md
+++ b/preview/site/en/reference/users_and_roles.md
@@ -2,6 +2,7 @@
id: users_and_roles.md
related_key: users, roles
summary: Learn about the definition of users, roles, objects, and privileges in role-based access control (RBAC).
+title: Users and Roles
---
# Users and Roles
@@ -22,45 +23,67 @@ The following figure illustrates the relationship between objects, privileges, r
The relationship between object, privilege, role and user.
-Milvus creates a `root` user by default. The `root` user is granted the `admin` privileges, which means that this `root` user can have access to all resources and perform all actions.
+Milvus creates a `root` user by default with a default password `Milvus`. The `root` user is granted the `admin` privileges, which means that this `root` user can have access to all resources and perform all actions.
If a user is bind with a `public` role, this user is entitled to the privileges of `DescribeCollection`, `ShowCollections`, and `IndexDetail`.
The following table lists the values you can choose when [enabling RBAC](rbac.md).
-| Object name | Privilege name | Relevant API description on the client side |
-|-------------|--------------------|---------------------------------------------------|
-| Collection | CreateIndex | CreateIndex |
-| Collection | DropIndex | DropIndex |
-| Collection | IndexDetail | DescribeIndex/GetIndexState/GetIndexBuildProgress |
-| Collection | Load | LoadCollection |
-| Collection | Release | ReleaseCollection |
-| Collection | Insert | Insert |
-| Collection | Delete | Delete |
-| Collection | Search | Search |
-| Collection | Flush | Flush |
-| Collection | Query | Query |
-| Collection | GetStatistics | GetCollectionStatistics |
-| Collection | Compaction | Compact |
-| Collection | Alias | CreateAlias/DropAlias/AlterAlias |
-| Collection | Import | BulkInsert/Import |
-| Collection | LoadBalance | LoadBalance |
-| Global | *(All) | All API operation permissions in this table |
-| Global | CreateCollection | CreateCollection |
-| Global | DropCollection | DropCollection |
-| Global | DescribeCollection | DescribeCollection |
-| Global | ShowCollections | ShowCollections |
-| Global | CreateOwnership | CreateUser CreateRole |
-| Global | DropOwnership | DeleteCredential DropRole |
-| Global | SelectOwnership | SelectRole/SelectGrant |
-| Global | ManageOwnership | OperateUserRole OperatePrivilege |
-| User | UpdateUser | UpdateCredential |
-| User | SelectUser | SelectUser |
-
+| Object name | Privilege name | Relevant API description on the client side |
+| ----------- | --------------------- | ------------------------------------------------- |
+| Collection | CreateIndex | CreateIndex |
+| Collection | DropIndex | DropIndex |
+| Collection | IndexDetail | DescribeIndex/GetIndexState/GetIndexBuildProgress |
+| Collection | Load | LoadCollection/GetLoadingProgress/GetLoadState |
+| Collection | GetLoadingProgress | GetLoadingProgress |
+| Collection | GetLoadState | GetLoadState |
+| Collection | Release | ReleaseCollection |
+| Collection | Insert | Insert |
+| Collection | Delete | Delete |
+| Collection | Upsert | Upsert |
+| Collection | Search | Search |
+| Collection | Flush | Flush/GetFlushState |
+| Collection | GetFlushState | GetFlushState |
+| Collection | Query | Query |
+| Collection | GetStatistics | GetCollectionStatistics |
+| Collection | Compaction | Compact |
+| Collection | Import | BulkInsert/Import |
+| Collection | LoadBalance | LoadBalance |
+| Collection | CreatePartition | CreatePartition |
+| Collection | DropPartition | DropPartition |
+| Collection | ShowPartitions | ShowPartitions |
+| Collection | HasPartition | HasPartition |
+| Global | All | All API operation permissions in this table |
+| Global | CreateCollection | CreateCollection |
+| Global | DropCollection | DropCollection |
+| Global | DescribeCollection | DescribeCollection |
+| Global | ShowCollections | ShowCollections |
+| Global | RenameCollection | RenameCollection |
+| Global | FlushAll | FlushAll |
+| Global | CreateOwnership | CreateUser CreateRole |
+| Global | DropOwnership | DeleteCredential DropRole |
+| Global | SelectOwnership | SelectRole/SelectGrant |
+| Global | ManageOwnership | OperateUserRole OperatePrivilege |
+| Global | CreateResourceGroup | CreateResourceGroup |
+| Global | DropResourceGroup | DropResourceGroup |
+| Global | DescribeResourceGroup | DescribeResourceGroup |
+| Global | ListResourceGroups | ListResourceGroups |
+| Global | TransferNode | TransferNode |
+| Global | TransferReplica | TransferReplica |
+| Global | CreateDatabase | CreateDatabase |
+| Global | DropDatabase | DropDatabase |
+| Global | ListDatabases | ListDatabases |
+| Global | CreateAlias | CreateAlias |
+| Global | DropAlias | DropAlias |
+| Global | DescribeAlias | DescribeAlias |
+| Global | ListAliases | ListAliases |
+| User | UpdateUser | UpdateCredential |
+| User | SelectUser | SelectUser |
Object and privilege names are case-sensitive.
-To grant all privileges to a global object, use "*" for privilege name.
+To grant all privileges to a kind of object, like Collection, Global, User, use "*" for privilege name.
+The "*" privilege name for the Global object doesn't include the All privilege, because the All privilege includes all permissions, including any collection and user object.
## What's next
diff --git a/preview/site/en/release_notes.md b/preview/site/en/release_notes.md
index 76740cc2d..bdb6f5c45 100644
--- a/preview/site/en/release_notes.md
+++ b/preview/site/en/release_notes.md
@@ -1,288 +1,453 @@
---
id: release_notes.md
summary: Milvus Release Notes
+title: Release Notes
---
# Release Notes
-Find out what’s new in Milvus! This page summarizes information about new features, improvements, known issues, and bug fixes in each release. You can find the release notes for each released version after v2.2.0 in this section. We suggest that you regularly visit this page to learn about updates.
+Find out what’s new in Milvus! This page summarizes new features, improvements, known issues, and bug fixes in each release. You can find the release notes for each released version after v2.4.0 in this section. We suggest that you regularly visit this page to learn about updates.
+
+## v2.4.5
+
+Release date: June 18, 2024
+
+| Milvus version | Python SDK version | Java SDK version | Node.js SDK version |
+|----------------|--------------------| --------------------| --------------------|
+| 2.4.5 | 2.4.4 | 2.4.1 | 2.4.3 |
+
+The release of Milvus 2.4.5 introduces several improvements and bug fixes to enhance performance, stability, and functionality. Milvus 2.4.5 simplifies sparse, float16, and bfloat16 vector search with auto-indexing, speeds up searches, deletions, and compactions with Bloom filter optimizations, and tackles data management through faster loading times and import L0 segment support. It also introduces the sparse HNSW index for efficient high-dimensional sparse data search, enhances the RESTful API with sparse float vector support, and fixes critical bugs for better stability.
+
+### New Features
+
+- Added rbac support to describe/alter database api ([#33804](https://github.com/milvus-io/milvus/pull/33804))
+- Supported building the HNSW index for sparse vectors ([#33653](https://github.com/milvus-io/milvus/pull/33653), [#33662](https://github.com/milvus-io/milvus/pull/33662))
+- Supported building the Disk index for binary vector ([#33575](https://github.com/milvus-io/milvus/pull/33575))
+- Supported sparse vector type on RESTful v2 ([#33555](https://github.com/milvus-io/milvus/pull/33555))
+- Add /management/stop RESTful api to stop a component ([#33799](https://github.com/milvus-io/milvus/pull/33799))
+
+### Improvements
+
+- Set maxPartitionNum default value to 1024 ([#33950](https://github.com/milvus-io/milvus/pull/33950))
+- Enabled to Force reset connection for unavailable error ([#33910](https://github.com/milvus-io/milvus/pull/33910))
+- Enabled flush rate limiter of collection level ([#33864](https://github.com/milvus-io/milvus/pull/33864))
+- Executed bloom filter apply in parallel to speed up segment predict ([#33793](https://github.com/milvus-io/milvus/pull/33793))
+- Used fastjson lib for unmarshal delete log to speed up json.Unmarshal([#33802](https://github.com/milvus-io/milvus/pull/33802))
+- Used BatchPkExist to reduce bloom filter func call cost ([#33752](https://github.com/milvus-io/milvus/pull/33752))
+- Speeded up the loading of small collections ([#33746](https://github.com/milvus-io/milvus/pull/33746))
+- Supported import delete data to L0 segment ([#33712](https://github.com/milvus-io/milvus/pull/33712))
+- Skipped mark compaction tasks to be timeouted to aviod executing the same task over and over again ([#33833](https://github.com/milvus-io/milvus/pull/33833))
+- Handled float16 and bfloat16 vectors as same as BinaryVector in numpy bulk insert ([#33788](https://github.com/milvus-io/milvus/pull/33788))
+- Added the includeCurrentMsg flag for the seek method ([#33743](https://github.com/milvus-io/milvus/pull/33743))
+- Added mergeInterval, targetBufSize, maxTolerantLagof msgdispatcher to configs ([#33680](https://github.com/milvus-io/milvus/pull/33680))
+- Improved GetVectorByID of sparse vector ([#33652](https://github.com/milvus-io/milvus/pull/33652))
+- Removed StringPrimarykey to reduce unnecessary copy and function call cost ([#33649](https://github.com/milvus-io/milvus/pull/33649))
+- Added autoindex mapping for binary/sparse data type ([#33625](https://github.com/milvus-io/milvus/pull/33625))
+- Optimized some cache to reduce memory usage ([#33560](https://github.com/milvus-io/milvus/pull/33560))
+- Abstracted execute interface for import/preimport task ([#33607](https://github.com/milvus-io/milvus/pull/33607))
+- Used map pk to timestamp in buffer insert to reduce bf causes ([#33582](https://github.com/milvus-io/milvus/pull/33582))
+- Avoided redundant meta operations of import ([#33519](https://github.com/milvus-io/milvus/pull/33519))
+- Improve logs by logging better disk quota info, adding UseDefaultConsistency flag, removing unnecessary logs ([#33597](https://github.com/milvus-io/milvus/pull/33597), [#33644](https://github.com/milvus-io/milvus/pull/33644), [#33670](https://github.com/milvus-io/milvus/pull/33670))
+
+### Bug fixes
+
+- Fixed a bug that queryHook unable to recognize vector type ([#33911](https://github.com/milvus-io/milvus/pull/33911))
+- Prevented use captured iteration variable partitionID ([#33970](https://github.com/milvus-io/milvus/pull/33970))
+- Fixed a bug that may cause Milvus to be unable to create AutoIndex on binary and sparse vectors ([#33867](https://github.com/milvus-io/milvus/pull/33867))
+- Fixed a bug that may cause indexnode to retry creating index on invalid index params of all vectors（[#33878](https://github.com/milvus-io/milvus/pull/33878))
+- Fixed the bug that when loads and releases happen concurrently may crash the Server([#33699](https://github.com/milvus-io/milvus/pull/33699))
+- Improved cache consistency for configuration values ([#33797](https://github.com/milvus-io/milvus/pull/33797))
+- Prevented possible data loss during deletion ([#33821](https://github.com/milvus-io/milvus/pull/33821))
+- Ensured the DroppedAt field (likely deletion timestamp) is set after dropping collections([#33767](https://github.com/milvus-io/milvus/pull/33767))
+- Fixed an issue that might have caused Milvus to handle binary vector data sizes incorrectly([#33751](https://github.com/milvus-io/milvus/pull/33751))
+- Prevented sensitive Kafka credentials from being logged in plain text.([#33694](https://github.com/milvus-io/milvus/pull/33694), [#33747](https://github.com/milvus-io/milvus/pull/33747))
+- Ensured Milvus can correctly import data with multiple vector fields.([#33724](https://github.com/milvus-io/milvus/pull/33724))
+- Enhanced import reliability by checking if an import job exists before starting. ([#33673](https://github.com/milvus-io/milvus/pull/33673))
+- Improved handling of the sparse HNSW index (internal functionality) ([#33714](https://github.com/milvus-io/milvus/pull/33714))
+- Cleaned vector memory to avoid memory leak([#33708](https://github.com/milvus-io/milvus/pull/33708))
+- Ensured smoother asynchronous warmup by fixing a state lock issue.([#33687](https://github.com/milvus-io/milvus/pull/33687))
+- Addressed a bug that might have caused missing results in query iterators. ([#33506](https://github.com/milvus-io/milvus/pull/33506))
+- Fixed a bug that might cause import segment size is uneven ([#33634](https://github.com/milvus-io/milvus/pull/33634))
+- Fixed incorrect data size handling for bf16, fp16, and binary vector types ([#33488](https://github.com/milvus-io/milvus/pull/33488))
+- Improved stability by addressing potential issues with the L0 compactor([#33564](https://github.com/milvus-io/milvus/pull/33564))
+- Ensured dynamic configuration updates are reflected correctly in the cache. ([#33590](https://github.com/milvus-io/milvus/pull/33590))
+- Improved the accuracy of the RootCoordQuotaStates metric ([#33601](https://github.com/milvus-io/milvus/pull/33601))
+- Ensured accurate reporting of the number of loaded entities in metric([#33522](https://github.com/milvus-io/milvus/pull/33522))
+- Provided more complete information in exception logs. ([#33396](https://github.com/milvus-io/milvus/pull/33396))
+- Optimized query pipeline by removing unnecessary group checker ([#33485](https://github.com/milvus-io/milvus/pull/33485))
+- Used the local storage path for a more accurate disk capacity check on the index node. ([#33505](https://github.com/milvus-io/milvus/pull/33505))
+- Corrected hasMoreResult may return false when hit number larger than limit ([#33642](https://github.com/milvus-io/milvus/pull/33642))
+- Delayed load bf in delegator to prevent bfs been loaded over and over again when worker has no more memory ([#33650](https://github.com/milvus-io/milvus/pull/33650))- Fixed a bug that queryHook unable to recognize vector type ([#33911](https://github.com/milvus-io/milvus/pull/33911))
+- Prevented use captured iteration variable partitionID ([#33970](https://github.com/milvus-io/milvus/pull/33970))
+- Fixed a bug that may cause Milvus to be unable to create AutoIndex on binary and sparse vectors ([#33867](https://github.com/milvus-io/milvus/pull/33867))
+- Fixed a bug that may cause indexnode to retry creating index on invalid index params of all vectors（[#33878](https://github.com/milvus-io/milvus/pull/33878))
+- Fixed the bug that when loads and releases happen concurrently may crash the Server([#33699](https://github.com/milvus-io/milvus/pull/33699))
+- Improved cache consistency for configuration values ([#33797](https://github.com/milvus-io/milvus/pull/33797))
+- Prevented possible data loss during deletion ([#33821](https://github.com/milvus-io/milvus/pull/33821))
+- Ensured the DroppedAt field (likely deletion timestamp) is set after dropping collections([#33767](https://github.com/milvus-io/milvus/pull/33767))
+- Fixed an issue that might have caused Milvus to handle binary vector data sizes incorrectly([#33751](https://github.com/milvus-io/milvus/pull/33751))
+- Prevented sensitive Kafka credentials from being logged in plain text.([#33694](https://github.com/milvus-io/milvus/pull/33694), [#33747](https://github.com/milvus-io/milvus/pull/33747))
+- Ensured Milvus can correctly import data with multiple vector fields.([#33724](https://github.com/milvus-io/milvus/pull/33724))
+- Enhanced import reliability by checking if an import job exists before starting. ([#33673](https://github.com/milvus-io/milvus/pull/33673))
+- Improved handling of the sparse HNSW index (internal functionality) ([#33714](https://github.com/milvus-io/milvus/pull/33714))
+- Cleaned vector memory to avoid memory leak([#33708](https://github.com/milvus-io/milvus/pull/33708))
+- Ensured smoother asynchronous warmup by fixing a state lock issue.([#33687](https://github.com/milvus-io/milvus/pull/33687))
+- Addressed a bug that might have caused missing results in query iterators. ([#33506](https://github.com/milvus-io/milvus/pull/33506))
+- Fixed a bug that might cause import segment size is uneven ([#33634](https://github.com/milvus-io/milvus/pull/33634))
+- Fixed incorrect data size handling for bf16, fp16, and binary vector types ([#33488](https://github.com/milvus-io/milvus/pull/33488))
+- Improved stability by addressing potential issues with the L0 compactor([#33564](https://github.com/milvus-io/milvus/pull/33564))
+- Ensured dynamic configuration updates are reflected correctly in the cache. ([#33590](https://github.com/milvus-io/milvus/pull/33590))
+- Improved the accuracy of the RootCoordQuotaStates metric ([#33601](https://github.com/milvus-io/milvus/pull/33601))
+- Ensured accurate reporting of the number of loaded entities in metric([#33522](https://github.com/milvus-io/milvus/pull/33522))
+- Provided more complete information in exception logs. ([#33396](https://github.com/milvus-io/milvus/pull/33396))
+- Optimized query pipeline by removing unnecessary group checker ([#33485](https://github.com/milvus-io/milvus/pull/33485))
+- Used the local storage path for a more accurate disk capacity check on the index node. ([#33505](https://github.com/milvus-io/milvus/pull/33505))
+- Corrected hasMoreResult may return false when hit number larger than limit ([#33642](https://github.com/milvus-io/milvus/pull/33642))
+- Delayed load bf in delegator to prevent bfs been loaded over and over again when worker has no more memory ([#33650](https://github.com/milvus-io/milvus/pull/33650))
+
+## v2.4.4
+
+Release date: May 31, 2024
+
+| Milvus version | Python SDK version | Java SDK version | Node.js SDK version |
+|----------------|--------------------| --------------------| --------------------|
+| 2.4.4 | 2.4.3 | 2.4.1 | 2.4.2 |
+
+Milvus v2.4.4 includes several critical bug fixes and improvements aimed at enhancing performance and stability. Notably, we've **resolved a critical issue where bulk insert statistics logs were incorrectly garbage collected**, potentially affecting data integrity. **We strongly recommend all v2.4 users upgrade to this version to benefit from these fixes.**
+
+**If you are using bulk insert, upgrade to v2.4.4 at the earliest opportunity for data integrity.**
+
+### Critical bug fixes
+
+- Filled stats log ID and validated its correctness ([#33478](https://github.com/milvus-io/milvus/pull/33478))
+
+### Improvements
+
+- Upgraded bitset for ARM SVE ([#33440](https://github.com/milvus-io/milvus/pull/33440))
+- Enabled Milvus compilation with GCC-13 ([#33441](https://github.com/milvus-io/milvus/pull/33441))
+
+### Bug fixes
+
+- Displayed empty collections when all privilege is granted ([#33454](https://github.com/milvus-io/milvus/pull/33454))
+- Ensured CMake downloads and installs for the current platform, not just x86_64 ([#33439](https://github.com/milvus-io/milvus/pull/33439))
+
+## v2.4.3
+
+Release date: May 29, 2024
+
+| Milvus version | Python SDK version | Java SDK version | Node.js SDK version |
+|----------------|--------------------| --------------------| --------------------|
+| 2.4.3 | 2.4.3 | 2.4.1 | 2.4.2 |
+
+Milvus version 2.4.3 introduced a host of features, improvements, and bug fixes to elevate performance and reliability. Notable enhancements included support for sparse float vector bulk insert and optimized bloom filter acceleration. Improvements covered various areas, from dynamic configuration updates to memory usage optimization. Bug fixes addressed critical issues like panic scenarios and ensured smoother system operations. This release underscored Milvus's ongoing commitment to enhancing functionality, optimizing performance, and delivering a robust user experience.
-## 2.2.3
-Release date: 10 Feburary, 2023
-
-| Milvus version | Python SDK version | Java SDK version | Go SDK version | Node.js SDK version |
-| -------------- | ------------------ | ---------------- | -------------- | ------------------- |
-| 2.2.3 | 2.2.2 | Coming soon | Coming soon | Coming soon |
-
-Milvus 2.2.3 introduces the rolling upgrade capability to Milvus clusters and brings high availability settings to RootCoords. The former greatly reduces the impacts brought by the upgrade and restart of the Milvus cluster in production to the minimum, while the latter enables coordinators to work in active-standby mode and ensures a short failure recovery time of no more than 30 seconds.
+### Features
-In this release, Milvus also ships with a lot of improvements and enhancements in performance, including a fast bulk-insert experience with reduced memory usage and less loading time.
+- Supported sparse float vector bulk insert for binlog/json/parquet ([#32649](https://github.com/milvus-io/milvus/pull/32649))
+
+### Improvements
+
+- Implemented Datacoord/node watch channel based on RPC ([#32036](https://github.com/milvus-io/milvus/pull/32036))
+- Optimized bloom filter to accelerate delete filtering ([#32642](https://github.com/milvus-io/milvus/pull/32642), [#33329](https://github.com/milvus-io/milvus/pull/33329), [#33284](https://github.com/milvus-io/milvus/pull/33284))
+- Loaded raw data via mmap if scalar index did not have raw data ([#33317](https://github.com/milvus-io/milvus/pull/33317))
+- Synced milvus config to milvus.yaml ([#33322](https://github.com/milvus-io/milvus/pull/33322), [#32920](https://github.com/milvus-io/milvus/pull/32920), [#32857](https://github.com/milvus-io/milvus/pull/32857), [#32946](https://github.com/milvus-io/milvus/pull/32946))
+- Updated knowhere version ([#33310](https://github.com/milvus-io/milvus/pull/33310), [#32931](https://github.com/milvus-io/milvus/pull/32931), [#33043](https://github.com/milvus-io/milvus/pull/33043))
+- Enabled dynamic updating of balancer policy in QueryCoord ([#33272](https://github.com/milvus-io/milvus/pull/33272))
+- Used a pre-built logger in the write buffer to minimize logger allocation ([#33304](https://github.com/milvus-io/milvus/pull/33304))
+- Improved parameter checking ([#32777](https://github.com/milvus-io/milvus/pull/32777), [#33271](https://github.com/milvus-io/milvus/pull/33271), [#33218](https://github.com/milvus-io/milvus/pull/33218))
+- Added a parameter to ignore incorrect message IDs in the checkpoint ([#33249](https://github.com/milvus-io/milvus/pull/33249))
+- Added config to control initialization failure handling for plugins ([#32680](https://github.com/milvus-io/milvus/pull/32680))
+- Added score compute consistency config for knowhere ([#32997](https://github.com/milvus-io/milvus/pull/32997))
+- Introduced a configuration option to control the initialization of public role permissions ([#33174](https://github.com/milvus-io/milvus/pull/33174))
+- Optimized memory usage when reading fields ([#33196](https://github.com/milvus-io/milvus/pull/33196))
+- Refined implementation of Channel Manager v2 ([#33172](https://github.com/milvus-io/milvus/pull/33172), [#33121](https://github.com/milvus-io/milvus/pull/33121), [#33014](https://github.com/milvus-io/milvus/pull/33014))
+- Added feature to track the size of data in memory for binlog ([#33025](https://github.com/milvus-io/milvus/pull/33025))
+- Added metrics for segment index files size ([#32979](https://github.com/milvus-io/milvus/pull/32979), [#33305](https://github.com/milvus-io/milvus/pull/33305))
+- Replaced Delete with DeletePartialMatch to remove metrics ([#33029](https://github.com/milvus-io/milvus/pull/33029))
+- Got related data size according to segment type ([#33017](https://github.com/milvus-io/milvus/pull/33017))
+- Cleaned channel node info in meta store ([#32988](https://github.com/milvus-io/milvus/pull/32988))
+- Removed rootcoord from datanode broker ([#32818](https://github.com/milvus-io/milvus/pull/32818))
+- Enabled batch uploading ([#32788](https://github.com/milvus-io/milvus/pull/32788))
+- Changed default partition number to 16 when using partition key ([#32950](https://github.com/milvus-io/milvus/pull/32950))
+- Improved reduce performance on very large top-k queries ([#32871](https://github.com/milvus-io/milvus/pull/32871))
+- Utilized TestLocations ability to accelerate write & compaction ([#32948](https://github.com/milvus-io/milvus/pull/32948))
+- Optimized plan parser pool to avoid unnecessary recycling ([#32869](https://github.com/milvus-io/milvus/pull/32869))
+- Improved load speed ([#32898](https://github.com/milvus-io/milvus/pull/32898))
+- Used collection default consistency level for restv2 ([#32956](https://github.com/milvus-io/milvus/pull/32956))
+- Added cost response for the rest API ([#32620](https://github.com/milvus-io/milvus/pull/32620))
+- Enabled channel exclusive balance policy ([#32911](https://github.com/milvus-io/milvus/pull/32911))
+- Exposed describedatabase API in proxy ([#32732](https://github.com/milvus-io/milvus/pull/32732))
+- Utilized coll2replica mapping when getting RG by collection ([#32892](https://github.com/milvus-io/milvus/pull/32892))
+- Added more tracing for search & query ([#32734](https://github.com/milvus-io/milvus/pull/32734))
+- Supported dynamic config for opentelemetry trace ([#32169](https://github.com/milvus-io/milvus/pull/32169))
+- Avoided iteration over channel results when updating leaderview ([#32887](https://github.com/milvus-io/milvus/pull/32887))
+- Optimized vector offsets handling for parquet ([#32822](https://github.com/milvus-io/milvus/pull/32822))
+- Improved datacoord segment filtering with collection ([#32831](https://github.com/milvus-io/milvus/pull/32831))
+- Adjusted log level and frequency ([#33042](https://github.com/milvus-io/milvus/pull/33042), [#32838](https://github.com/milvus-io/milvus/pull/32838), [#33337](https://github.com/milvus-io/milvus/pull/33337))
+- Enabled stopping balance after balance had been suspended ([#32812](https://github.com/milvus-io/milvus/pull/32812))
+- Updated shard leader cache when leader location changed ([#32470](https://github.com/milvus-io/milvus/pull/32470))
+- Removed deprecated API and field ([#32808](https://github.com/milvus-io/milvus/pull/32808), [#32704](https://github.com/milvus-io/milvus/pull/32704))
+- Added metautil.channel to convert string compare to int ([#32749](https://github.com/milvus-io/milvus/pull/32749))
+- Added type info for payload writer error message and log when querynode found new collection ([#32522](https://github.com/milvus-io/milvus/pull/32522))
+- Checked partition number when creating collection with partition key ([#32670](https://github.com/milvus-io/milvus/pull/32670))
+- Removed legacy l0 segment if watch failed ([#32725](https://github.com/milvus-io/milvus/pull/32725))
+- Improved printing type of request ([#33319](https://github.com/milvus-io/milvus/pull/33319))
+- Checked array field data was nil before getting the type ([#33311](https://github.com/milvus-io/milvus/pull/33311))
+- Returned error when startup Delete/AddNode node operation failed ([#33258](https://github.com/milvus-io/milvus/pull/33258))
+- Allowed datanode's server ID to be updated ([#31597](https://github.com/milvus-io/milvus/pull/31597))
+- Unified querynode metrics cleanup in collection release ([#32805](https://github.com/milvus-io/milvus/pull/32805))
+- Fixed scalar auto index config incorrect version ([#32795](https://github.com/milvus-io/milvus/pull/32795))
+- Refined index param check for create/alter index ([#32712](https://github.com/milvus-io/milvus/pull/32712))
+- Removed redundant replica recovery ([#32985](https://github.com/milvus-io/milvus/pull/32985))
+- Enabled channel meta table to write more than 200k segments ([#33300](https://github.com/milvus-io/milvus/pull/33300))
+
+### Bug fixes
+
+- Fixed panic when the database didn't exist in the rate limit interceptor ([#33308](https://github.com/milvus-io/milvus/pull/33308))
+- Fixed quotacenter metrics collection failure due to incorrect parameters ([#33399](https://github.com/milvus-io/milvus/pull/33399))
+- Fixed panic if processactivestandby returned an error ([#33372](https://github.com/milvus-io/milvus/pull/33372))
+- Fixed search result truncation in restful v2 when nq > 1 ([#33363](https://github.com/milvus-io/milvus/pull/33363))
+- Added database name field for role operations in restful v2 ([#33291](https://github.com/milvus-io/milvus/pull/33291))
+- Fixed global rate limit not working ([#33336](https://github.com/milvus-io/milvus/pull/33336))
+- Fixed panic caused by failure of building index ([#33314](https://github.com/milvus-io/milvus/pull/33314))
+- Added validation for sparse vector in segcore to ensure legality ([#33312](https://github.com/milvus-io/milvus/pull/33312))
+- Removed task from syncmgr after task completion ([#33303](https://github.com/milvus-io/milvus/pull/33303))
+- Fixed partition key filtering failure during data import ([#33277](https://github.com/milvus-io/milvus/pull/33277))
+- Fixed inability to generate traceID when using noop exporter ([#33208](https://github.com/milvus-io/milvus/pull/33208))
+- Improved query results retrieval ([#33179](https://github.com/milvus-io/milvus/pull/33179))
+- Marked channel checkpoint dropped to prevent checkpoint lag metrics leakage ([#33201](https://github.com/milvus-io/milvus/pull/33201))
+- Fixed query node getting stuck during stopping progress ([#33154](https://github.com/milvus-io/milvus/pull/33154))
+- Fixed missing segments in flush response ([#33061](https://github.com/milvus-io/milvus/pull/33061))
+- Made submit operation idempotent ([#33053](https://github.com/milvus-io/milvus/pull/33053))
+- Allocated new slice for each batch in streaming reader ([#33360](https://github.com/milvus-io/milvus/pull/33360))
+- Cleaned offline node from resource group after QueryCoord restart ([#33233](https://github.com/milvus-io/milvus/pull/33233))
+- Removed l0 compactor in completedCompactor ([#33216](https://github.com/milvus-io/milvus/pull/33216))
+- Reset quota value when initializing the limiter ([#33152](https://github.com/milvus-io/milvus/pull/33152))
+- Fixed issue where etcd limit was exceeded ([#33041](https://github.com/milvus-io/milvus/pull/33041))
+- Resolved etcd transaction limit exceedance due to too many fields ([#33040](https://github.com/milvus-io/milvus/pull/33040))
+- Removed RLock re-entry in GetNumRowsOfPartition ([#33045](https://github.com/milvus-io/milvus/pull/33045))
+- Started LeaderCacheObserver before SyncAll ([#33035](https://github.com/milvus-io/milvus/pull/33035))
+- Enabled balancing of released standby channel ([#32986](https://github.com/milvus-io/milvus/pull/32986))
+- Initialized access logger before server initialization ([#32976](https://github.com/milvus-io/milvus/pull/32976))
+- Made compactor capable of clearing empty segments ([#32821](https://github.com/milvus-io/milvus/pull/32821))
+- Filled deltalog entry number and time range in l0 compactions ([#33004](https://github.com/milvus-io/milvus/pull/33004))
+- Fixed proxy crash due to shard leader cache data race ([#32971](https://github.com/milvus-io/milvus/pull/32971))
+- Corrected time unit for load index metric ([#32935](https://github.com/milvus-io/milvus/pull/32935))
+- Fixed issue where segment on stopping query node couldn't be released successfully ([#32929](https://github.com/milvus-io/milvus/pull/32929))
+- Fixed index resource estimation ([#32842](https://github.com/milvus-io/milvus/pull/32842))
+- Set channel checkpoint to delta position ([#32878](https://github.com/milvus-io/milvus/pull/32878))
+- Made syncmgr lock key before returning future ([#32865](https://github.com/milvus-io/milvus/pull/32865))
+- Ensured inverted index had only one segment ([#32858](https://github.com/milvus-io/milvus/pull/32858))
+- Fixed compaction trigger choosing two identical segments ([#32800](https://github.com/milvus-io/milvus/pull/32800))
+- Fixed issue where partition name could not be specified in binlog import ([#32730](https://github.com/milvus-io/milvus/pull/32730), [#33027](https://github.com/milvus-io/milvus/pull/33027))
+- Made dynamic column optional in parquet import ([#32738](https://github.com/milvus-io/milvus/pull/32738))
+- Skipped checking auto ID when inserting data ([#32775](https://github.com/milvus-io/milvus/pull/32775))
+- Validated number of rows for insert field data with schema ([#32770](https://github.com/milvus-io/milvus/pull/32770))
+- Added Wrapper and Keepalive for CTraceContext IDs ([#32746](https://github.com/milvus-io/milvus/pull/32746))
+- Fixed issue where database name was not found in the datacoord meta object ([#33412](https://github.com/milvus-io/milvus/pull/33412))
+- Synchronized dropped segment for dropped partition ([#33332](https://github.com/milvus-io/milvus/pull/33332))
+- Fixed quotaCenter metrics collection failure due to incorrect parameters ([#33399](https://github.com/milvus-io/milvus/pull/33399))
+
+## v2.4.1
+
+Release date: May 6, 2024
+
+| Milvus version | Python SDK version | Java SDK version | Node.js SDK version |
+|----------------|--------------------| --------------------| --------------------|
+| 2.4.1 | 2.4.1 | 2.4.0 | 2.4.2 |
+
+Milvus version 2.4.1 brings numerous improvements and bug fixes that aim to enhance the software's performance, observability, and stability. These improvements include a declarative resource group API, enhanced bulk insert functionality that supports Float16/BFloat16 vector data types, a refined garbage collection (GC) mechanism that reduces list operations for object storage, and other changes related to performance optimizations. Additionally, bug fixes address issues such as compilation errors, failed fuzzy matches on newline characters, incorrect parameter datatypes for RESTful interfaces, and BulkInsert raising errors on numpy files when dynamic fields are enabled.
### Breaking changes
-In 2.2.3, the maximum number of fields in a collection is reduced from 256 to 64. ([#22030](https://github.com/milvus-io/milvus/pull/22030))
+- Discontinued support for delete with an empty filter expression. ([#32472](https://github.com/milvus-io/milvus/pull/32472))
### Features
-- Rolling upgrade
-
- The rolling upgrade feature allows Milvus to respond to incoming requests during the upgrade, which is not possible in previous releases. In such releases, upgrading a Milvus instance requires it to be stopped first and then restarted after the upgrade is complete, leaving all incoming requests unanswered.
-
- Related issues:
-
- - Graceful stop of index nodes implemented ([#21556](https://github.com/milvus-io/milvus/pull/21556))
- - Graceful stop of query nodes implemented ([#21528](https://github.com/milvus-io/milvus/pull/21528))
- - Auto-sync of segments on closing implemented ([#21576](https://github.com/milvus-io/milvus/pull/21576))
- - Graceful stop APIs and error messages improved ([#21580](https://github.com/milvus-io/milvus/pull/21580))
- - Issues identified and fixed in the code of QueryNode and QueryCoord ([#21565](https://github.com/milvus-io/milvus/pull/21565))
-
-- Coordinator HA
-
- Coordinator HA allows Milvus coordinators to work in active-standby mode to avoid single-point of failures.
-
- Related issues:
-
- - HA-related issues identified and fixed in QueryCoordV2 ([#21501](https://github.com/milvus-io/milvus/pull/21501))
- - Auto-registration on the startup of Milvus was implemented to prevent both coordinators from working as the active coordinators. ([#21641](https://github.com/milvus-io/milvus/pull/21641))
- - HA-related issues identified and fixed in RootCoords ([#21700](https://github.com/milvus-io/milvus/pull/21700))
- - Issues identified and fixed in active-standby switchover ([#21747](https://github.com/milvus-io/milvus/pull/21747))
-
-### Enhancements
-
-- Bulk-insert performance enhanced
-
- - Bulk-insert enhancement implemented ([#20986](https://github.com/milvus-io/milvus/pull/20986) [#21532](https://github.com/milvus-io/milvus/pull/21532))
- - JSON parser optimized for data import ([#21332](https://github.com/milvus-io/milvus/pull/21332))
- - Stream-reading NumPy data implemented ([#21540](https://github.com/milvus-io/milvus/pull/21540))
- - Bulk-insert progress report implemented ([#21612](https://github.com/milvus-io/milvus/pull/21612))
- - Issues identified and fixed so that Milvus checks indexes before flushes segments before bulk-insert is complete ([#21604](https://github.com/milvus-io/milvus/pull/21604))
- - Issues related to bulk-insert progress identified and fixed ([#21668](https://github.com/milvus-io/milvus/pull/21668))
- - Issues related to bulk-insert report identified and fixed ([#21758](https://github.com/milvus-io/milvus/pull/21758))
- - Issues identified and fixed so that Milvus does not seal failed segments while performing bulk-insert operations. ([#21779](https://github.com/milvus-io/milvus/pull/21779))
- - Issues identified and fixed so that bulk-insert operations do not cause a slow flush ([#21918](https://github.com/milvus-io/milvus/pull/21918))
- - Issues identified and fixed so that bulk-insert operations do not crash the DataNodes ([#22040](https://github.com/milvus-io/milvus/pull/22040))
- - Refresh option added to LoadCollection and LoadPartition APIs ([#21811](https://github.com/milvus-io/milvus/pull/21811))
- - Segment ID update on data import implemented ([#21583](https://github.com/milvus-io/milvus/pull/21583))
-
-- Memory usage reduced
-
- - Issues identified and fixed so that loading failures do not return insufficient memory ([#21592](https://github.com/milvus-io/milvus/pull/21592))
- - Arrow usage removed from FieldData ([#21523](https://github.com/milvus-io/milvus/pull/21523))
- - Memory usage reduced in indexing scalar fields ([#21970](https://github.com/milvus-io/milvus/pull/21970)) ([#21978](https://github.com/milvus-io/milvus/pull/21978))
-
-- Monitoring metrics optimized
-
- - Issues related to unregistered metrics identified and fixed ([#22098](https://github.com/milvus-io/milvus/pull/22098))
- - A new segment metric that counts the number of binlog files added ([#22085](https://github.com/milvus-io/milvus/pull/22085))
- - Many new metrics added ([#21975](https://github.com/milvus-io/milvus/pull/21975))
- - Minor fix on segment metric ([#21977](https://github.com/milvus-io/milvus/pull/21977))
-
-- Meta storage performance improved
-
- - Improved ListSegments performance for Datacoord catalog. ([#21600](https://github.com/milvus-io/milvus/pull/21600))
- - Improved LoadWithPrefix performance for SuffixSnapshot. ([#21601](https://github.com/milvus-io/milvus/pull/21601))
- - Removed redundant LoadPrefix requests for Catalog ListCollections. ([#21551](https://github.com/milvus-io/milvus/pull/21551)) ([#21594](https://github.com/milvus-io/milvus/pull/21594))
- - Added A WalkWithPrefix API for MetaKv interface. ([#21585](https://github.com/milvus-io/milvus/pull/21585))
- - Added GC for snapshot KV based on time-travel. ([#21417](https://github.com/milvus-io/milvus/pull/21417)) ([#21763](https://github.com/milvus-io/milvus/pull/21763))
-
-- Performance improved
-
- - Upgraded Knowhere to 1.3.7. ([#21735](https://github.com/milvus-io/milvus/pull/21735))
- - Upgraded Knowhere to 1.3.8. ([#22024](https://github.com/milvus-io/milvus/pull/22024))
- - Skipped search GRPC call for standalone. ([#21630](https://github.com/milvus-io/milvus/pull/21630))
- - Optimized some low-efficient code. ([#20529](https://github.com/milvus-io/milvus/pull/20529)) ([#21683](https://github.com/milvus-io/milvus/pull/21683))
- - Fixed fill the string field twice when string index exists. ([#21852](https://github.com/milvus-io/milvus/pull/21852)) ([#21865](https://github.com/milvus-io/milvus/pull/21865))
- - Used all() API for bitset check. ([#20462](https://github.com/milvus-io/milvus/pull/20462)) ([#21682](https://github.com/milvus-io/milvus/pull/21682))
-
-- Others
-
- - Implemented the GetLoadState API. ([#21533](https://github.com/milvus-io/milvus/pull/21533))
- - Added a task to unsubscribe dmchannel. ([#21513](https://github.com/milvus-io/milvus/pull/21513)) ([#21794](https://github.com/milvus-io/milvus/pull/21794))
- - Explicitly list the triggering reasons when Milvus denies reading/writing. ([#21553](https://github.com/milvus-io/milvus/pull/21553))
- - Verified and adjusted the number of rows in a segment before saving and passing SegmentInfo. ([#21200](https://github.com/milvus-io/milvus/pull/21200))
- - Added a segment seal policy by the number of binlog files. ([#21941](https://github.com/milvus-io/milvus/pull/21941))
- - Upgraded etcd to 3.5.5. ([#22007](https://github.com/milvus-io/milvus/pull/22007)）
+- Added support for Float16/BFloat16 vector data types in bulk insert ([#32157](https://github.com/milvus-io/milvus/pull/32157))
+- Enhanced sparse float vector to support brute force iterator search and range search ([#32635](https://github.com/milvus-io/milvus/pull/32635))
+
+### Improvements
+
+- Added declarative resource group api ([#31930](https://github.com/milvus-io/milvus/pull/31930), [#32297](https://github.com/milvus-io/milvus/pull/32297), [#32536](https://github.com/milvus-io/milvus/pull/32536), [#32666](https://github.com/milvus-io/milvus/pull/32666))
+- Rewrote the collection observer in QueryCoord to make it task-driven ([#32441](https://github.com/milvus-io/milvus/pull/32441))
+- Refactored the data structure used in the SyncManager of DataNode to reduce memory usage and prevent errors ([#32673](https://github.com/milvus-io/milvus/pull/32673))
+- Revised the implementation of garbage collection to minimize list operations associated with object storage ([#31740](https://github.com/milvus-io/milvus/pull/31740))
+- Reduced the cpu usage when collection number is high ([#32245](https://github.com/milvus-io/milvus/pull/32245))
+- Enhanced the management of milvus.yaml by automatically generating relevant configuration items in the milvus.yaml file through code ([#31832](https://github.com/milvus-io/milvus/pull/31832), [#32357](https://github.com/milvus-io/milvus/pull/32357))
+- Enhanced the performance of the Query by retrieving the data after performing local reduction ([#32346](https://github.com/milvus-io/milvus/pull/32346))
+- Added WithBlock option for etcd client creation ([#32641](https://github.com/milvus-io/milvus/pull/32641))
+- Used client_request_id specified by the client as the TraceID if client provided ([#32264](https://github.com/milvus-io/milvus/pull/32264))
+- Added db label to the metrics for the delete and bulk insert operations ([#32611](https://github.com/milvus-io/milvus/pull/32611))
+- Added logic to skip the verification through configuration for AutoID and PartitionKey columns ([#32592](https://github.com/milvus-io/milvus/pull/32592))
+- Refined errors related to authentication ([#32253](https://github.com/milvus-io/milvus/pull/32253))
+- Refined error logs for AllocSegmentID in DataCoord ([#32351](https://github.com/milvus-io/milvus/pull/32351), [#32335](https://github.com/milvus-io/milvus/pull/32335))
+- Removed duplicate metrics ([#32380](https://github.com/milvus-io/milvus/pull/32380), [#32308](https://github.com/milvus-io/milvus/pull/32308)) and cleaned up unused metrics ([#32404](https://github.com/milvus-io/milvus/pull/32404), [#32515](https://github.com/milvus-io/milvus/pull/32515))
+- Added configuration option to control whether to enforce the activation of the partitionKey feature ([#32433](https://github.com/milvus-io/milvus/pull/32433))
+- Added configuration option to control the maximum amount of data that can be inserted in a single request ([#32433](https://github.com/milvus-io/milvus/pull/32433))
+- Parallelize the applyDelete operation at the segment level to accelerate the processing of Delete messages by the Delegator ([#32291](https://github.com/milvus-io/milvus/pull/32291))
+- Used index ([#32232](https://github.com/milvus-io/milvus/pull/32232), [#32505](https://github.com/milvus-io/milvus/pull/32505), [#32533](https://github.com/milvus-io/milvus/pull/32533), [#32595](https://github.com/milvus-io/milvus/pull/32595)) and add cache ([#32580](https://github.com/milvus-io/milvus/pull/32580)) to accelerate frequent filtering operations in QueryCoord.
+- Rewrote the data structure ([#32273](https://github.com/milvus-io/milvus/pull/32273)) and refactored code ([#32389](https://github.com/milvus-io/milvus/pull/32389)) to accelerate common operations in DataCoord.
+- Removed openblas from conan ([#32002](https://github.com/milvus-io/milvus/pull/32002))
+
+### Bug fixes
+
+- Fixed build milvus in rockylinux8 ([#32619](https://github.com/milvus-io/milvus/pull/32619))
+- Fixed compilation errors for SVE on ARM ([#32463](https://github.com/milvus-io/milvus/pull/32463), [#32270](https://github.com/milvus-io/milvus/pull/32270))
+- Fixed the crash issue on ARM-based GPU images ([#31980](https://github.com/milvus-io/milvus/pull/31980))
+- Fixed regex query can't handle text with newline ([#32569](https://github.com/milvus-io/milvus/pull/32569))
+- Fixed search get empty result caused by GetShardLeaders return empty node list ([#32685](https://github.com/milvus-io/milvus/pull/32685))
+- Fixed BulkInsert raised error when encountering dynamic fields in numpy files ([#32596](https://github.com/milvus-io/milvus/pull/32596))
+- Fixed bugs related to the RESTFulV2 interface, including an important fix that allows numeric parameters in requests to accept numerical input instead of string type ([#32485](https://github.com/milvus-io/milvus/pull/32485), [#32355](https://github.com/milvus-io/milvus/pull/32355))
+- Fixed memory leak in proxy by removing watching config event in rate limiter ([#32313](https://github.com/milvus-io/milvus/pull/32313))
+- Fixed the issue where the rate limiter incorrectly reports that the partition cannot be found when partitionName is not specified ([#32647](https://github.com/milvus-io/milvus/pull/32647))
+- Added detection between the cases of Collection being in the recovery state and not being loaded in the error type. ([#32447](https://github.com/milvus-io/milvus/pull/32447))
+- Corrected the negative queryable num entities metric ([#32361](https://github.com/milvus-io/milvus/pull/32361))
+
+
+## v2.4.0
+
+Release date: April 17, 2024
+
+| Milvus version | Python SDK version | Node.js SDK version |
+|----------------|--------------------| --------------------|
+| 2.4.0 | 2.4.0 | 2.4.0 |
+
+We are excited to announce the official launch of Milvus 2.4.- Building upon the solid foundation of the 2.4.0-rc.1 release, we have focused on addressing critical bugs reported by our users, while preserving the existing functionality. In addition, Milvus 2.4.0 introduces a range of optimizations aimed at enhancing system performance, improving observability through the incorporation of various metrics, and streamlining the codebase for increased simplicity.
+
+### Improvements
+
+- Support for MinIO TLS connections ([#31396](https://github.com/milvus-io/milvus/pull/31396), [#31618](https://github.com/milvus-io/milvus/pull/31618))
+- AutoIndex support for scalar fields ([#31593](https://github.com/milvus-io/milvus/pull/31593))
+- Hybrid search refactoring for consistent execution paths with regular search ([#31742](https://github.com/milvus-io/milvus/pull/31742), [#32178](https://github.com/milvus-io/milvus/pull/32178))
+- Accelerated filtering through bitset and bitset_view refactoring ([#31592](https://github.com/milvus-io/milvus/pull/31592), [#31754](https://github.com/milvus-io/milvus/pull/31754), [#32139](https://github.com/milvus-io/milvus/pull/32139))
+- Import tasks now support waiting for data index completion ([#31733](https://github.com/milvus-io/milvus/pull/31733))
+- Enhanced Import compatibility ([#32121](https://github.com/milvus-io/milvus/pull/32121)), task scheduling ([#31475](https://github.com/milvus-io/milvus/pull/31475)), and limits on imported file size and number ([#31542](https://github.com/milvus-io/milvus/pull/31542))
+- Code simplification efforts including interface standardization for type checking ([#31945](https://github.com/milvus-io/milvus/pull/31945), [#31857](https://github.com/milvus-io/milvus/pull/31857)), removal of deprecated code and metrics ([#32079](https://github.com/milvus-io/milvus/pull/32079), [#32134](https://github.com/milvus-io/milvus/pull/32134), [#31535](https://github.com/milvus-io/milvus/pull/31535), [#32211](https://github.com/milvus-io/milvus/pull/32211), [#31935](https://github.com/milvus-io/milvus/pull/31935)), and normalization of constant names ([#31515](https://github.com/milvus-io/milvus/pull/31515))
+- New metrics for QueryCoord current target channel check point lag latency ([#31420](https://github.com/milvus-io/milvus/pull/31420))
+- New db label for common metrics ([#32024](https://github.com/milvus-io/milvus/pull/32024))
+- New metrics regarding the count of deleted, indexed, and loaded entities, with the inclusion of labels such as collectionName and dbName ([#31861](https://github.com/milvus-io/milvus/pull/31861))
+- Error handling improvements for mismatched vector types ([#31766](https://github.com/milvus-io/milvus/pull/31766))
+- Support for throwing errors instead of crashing when index cannot be built ([#31845](https://github.com/milvus-io/milvus/pull/31845))
+- Support for invalidating the database meta cache when dropping databases ([#32092](https://github.com/milvus-io/milvus/pull/32092))
+- Interface refactoring for channel distribution ([#31814](https://github.com/milvus-io/milvus/pull/31814)) and leader view management ([#32127](https://github.com/milvus-io/milvus/pull/32127))
+- Refactor channel dist manager interface ([#31814](https://github.com/milvus-io/milvus/pull/31814)) and Refactor leader view manager interface ([#32127](https://github.com/milvus-io/milvus/pull/32127))
+- Batch processing ([#31632](https://github.com/milvus-io/milvus/pull/31632)), adding mapping information ([#32234](https://github.com/milvus-io/milvus/pull/32234), [#32249](https://github.com/milvus-io/milvus/pull/32249)), and avoiding usage of lock ([#31787](https://github.com/milvus-io/milvus/pull/31787)) to accelerate frequently invoked operations
+
+### Breaking Changes
+
+- Discontinued grouping search on binary vectors ([#31735](https://github.com/milvus-io/milvus/pull/31735))
+- Discontinued grouping search with hybrid search ([#31812](https://github.com/milvus-io/milvus/pull/31812))
+- Discontinued HNSW index on binary vectors ([#31883](https://github.com/milvus-io/milvus/pull/31883))
### Bug Fixes
-- QueryCoord segment replacement fixed
-
- - Fixed the mismatch of sealed segments IDs after enabling load-balancing in 2.2. ([#21322](https://github.com/milvus-io/milvus/pull/21322))
- - Fixed the sync logic of the leader observer. ([#20478](https://github.com/milvus-io/milvus/pull/20478)) ([#21315](https://github.com/milvus-io/milvus/pull/21315))
- - Fixed the issues that observers may update the current target to an unfinished next target. ([#21107](https://github.com/milvus-io/milvus/pull/21107)) ([#21280](https://github.com/milvus-io/milvus/pull/21280))
- - Fixed the load timeout after the next target updates. ([#21759](https://github.com/milvus-io/milvus/pull/21759)) ([#21770](https://github.com/milvus-io/milvus/pull/21770))
- - Fixed the issue that the current target may be updated to an invalid target. ([#21742](https://github.com/milvus-io/milvus/pull/21742)) ([#21762](https://github.com/milvus-io/milvus/pull/21762))
- - Fixed the issue that a failed node may update the current target to an unavailable target. ([#21743](https://github.com/milvus-io/milvus/pull/21743))
-
-- Improperly invalidated proxy cache fixed
-
- - Fixed the issue that the proxy does not update the shard leaders cache for some types of error ([#21185](https://github.com/milvus-io/milvus/pull/21185)) ([#21303](https://github.com/milvus-io/milvus/pull/21303))
- - Fixed the issue that Milvus invalidates the proxy cache first when the shard leader list contains error ([#21451](https://github.com/milvus-io/milvus/pull/21451)) ([#21464](https://github.com/milvus-io/milvus/pull/21464))
-
-- CheckPoint and GC Related issues fixed
-
- - Fixed the issue that the checkpoint will not update after data delete and compact ([#21495](https://github.com/milvus-io/milvus/pull/21495))
- - Fixed issues related to channel checkpoint and GC ([#22027](https://github.com/milvus-io/milvus/pull/22027))
- - Added restraints on segment GC of DML position before channel copy ([#21773](https://github.com/milvus-io/milvus/pull/21773))
- - Removed collection meta after GC is complete ([#21595](https://github.com/milvus-io/milvus/pull/21595)) ([#21671](https://github.com/milvus-io/milvus/pull/21671))
-
-- Issues related to not being able to use embedded etcd with Milvus fixed
-
- - Added setup config files for embedded etcd ([#22076](https://github.com/milvus-io/milvus/pull/22076))
+- Enhanced data type and value checks for queries and insertions to prevent crashes ([#31478](https://github.com/milvus-io/milvus/pull/31478), [#31653](https://github.com/milvus-io/milvus/pull/31653), [#31698](https://github.com/milvus-io/milvus/pull/31698), [#31842](https://github.com/milvus-io/milvus/pull/31842), [#32042](https://github.com/milvus-io/milvus/pull/32042), [#32251](https://github.com/milvus-io/milvus/pull/32251), [#32204](https://github.com/milvus-io/milvus/pull/32204))
+- RESTful API bug fixes ([#32160](https://github.com/milvus-io/milvus/pull/32160))
+- Improved prediction of inverted index resource usage ([#31641](https://github.com/milvus-io/milvus/pull/31641))
+- Resolution of connection issues with etcd when authorization is enabled ([#31668](https://github.com/milvus-io/milvus/pull/31668))
+- Security update for nats server ([#32023](https://github.com/milvus-io/milvus/pull/32023))
+- Stored inverted index files into a local storage path of QueryNode instead of /tmp ([#32210](https://github.com/milvus-io/milvus/pull/32210))
+- Addressed datacoord memory leaks for collectionInfo ([#32243](https://github.com/milvus-io/milvus/pull/32243))
+- Fixes for fp16/bf16 related bugs potentially causing system panic ([#31677](https://github.com/milvus-io/milvus/pull/31677), [#31841](https://github.com/milvus-io/milvus/pull/31841), [#32196](https://github.com/milvus-io/milvus/pull/32196))
+- Resolved issues with grouping search returning insufficient results ([#32151](https://github.com/milvus-io/milvus/pull/32151))
+- Adjustment of search with iterators to handle offsets in the Reduce step more effectively and ensure adequate results with "reduceStopForBest" enabled ([#32088](https://github.com/milvus-io/milvus/pull/32088))
-- Others
+## v2.4.0-rc.1
+Release date: March 20, 2024
- - Fixed the offset panic in queries ([#21292](https://github.com/milvus-io/milvus/pull/21292)) ([#21751](https://github.com/milvus-io/milvus/pull/21751))
- - Fixed the issue that small candidate compaction should only happen with more than one segment ([#21250](https://github.com/milvus-io/milvus/pull/21250))
- - Fixed the issue of memory usage calculation ([#21798](https://github.com/milvus-io/milvus/pull/21798))
- - Fixed the issue that a timestamp allocation failure blocks compaction queue forever ([#22039](https://github.com/milvus-io/milvus/pull/22039)) ([#22046](https://github.com/milvus-io/milvus/pull/22046))
- - Fixed the issue that QueryNode may panic when stopped ([#21406](https://github.com/milvus-io/milvus/pull/21406)) ([#21419](https://github.com/milvus-io/milvus/pull/21419))
- - Modified lastSyncTime in advance to prevent multiple flush binlogs ([#22048](https://github.com/milvus-io/milvus/pull/22048))
- - Fixed the issue that a collection does not exist when users try to recover it ([#21471](https://github.com/milvus-io/milvus/pull/21471)) ([#21628](https://github.com/milvus-io/milvus/pull/21628))
- - Use tt msg stream for consume delete msg ([#21478](https://github.com/milvus-io/milvus/pull/21478))
- - Prevent users from deleting entities by any non-primary-key field ([#21459](https://github.com/milvus-io/milvus/pull/21459)) ([#21472](https://github.com/milvus-io/milvus/pull/21472))
- - Fixed potential nil access on segments ([#22104](https://github.com/milvus-io/milvus/pull/22104))
+| Milvus version | Python SDK version |
+|----------------|--------------------|
+| 2.4.0-rc.1 | 2.4.0 |
-## 2.2.2
-Release date: 22 December, 2022
+This release introduces several scenario-based features:
-| Milvus version | Python SDK version | Java SDK version | Go SDK version | Node.js SDK version |
-| -------------- | ------------------ | ---------------- | -------------- | ------------------- |
-| 2.2.2 | 2.2.1 | 2.2.1 | 2.2.0 | 2.2.1 |
+- **New GPU Index - CAGRA**: Thanks to NVIDIA's contribution, this new GPU index offers a 10x performance boost, especially for batch searches. For details, refer to [GPU Index](gpu_index.md).
-Milvus 2.2.2 is a minor fix of Milvus 2.2.1. It fixed a few loading failure issues as of the upgrade to 2.2.1 and the issue that the proxy cache is not cleaned upon some types of errors.
+- **Multi-vector** and **Hybrid Search**: This feature enables storing vector embeddings from multiple models and conducting hybrid searches. For details, refer to [Hybrid Search](multi-vector-search.md).
+- **Sparse Vectors**: Ideal for keyword interpretation and analysis, sparse vectors are now supported for processing in your collection. For details, refer to [Sparse Vectors](sparse_vector.md).
-
+
+In this tutorial, we will show you how to build a RAG(Retrieval-Augmented Generation) pipeline with Milvus.
+
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+
+
+## Preparation
+### Dependencies and Environment
+
+
+```python
+$ pip install --upgrade pymilvus openai requests tqdm
+```
+
+ tags to provide an answer to the question enclosed in tags.
+
+{context}
+
+
+{question}
+
+"""
+```
+
+Use OpenAI ChatGPT to generate a response based on the prompts.
+
+
+```python
+response = openai_client.chat.completions.create(
+ model="gpt-3.5-turbo",
+ messages=[
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": USER_PROMPT},
+ ],
+)
+print(response.choices[0].message.content)
+```
+
+ The hardware requirements specification for building Milvus and running it from the source code are as follows:
+
+ - 8GB of RAM
+ - 50GB of free disk space
+
diff --git a/preview/site/en/tutorials/dna_sequence_classification.md b/preview/site/en/tutorials/dna_sequence_classification.md
index ae8499f32..e84371264 100644
--- a/preview/site/en/tutorials/dna_sequence_classification.md
+++ b/preview/site/en/tutorials/dna_sequence_classification.md
@@ -1,13 +1,12 @@
---
id: dna_sequence_classification.md
-summary: Build a DNA sequence classification system with Milvus.
+summary: Build a DNA sequence classification system with Milvus.
+title: DNA Sequence Classification
---
# DNA Sequence Classification
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a DNA sequence classification model.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/medical/dna_sequence_classification/dna_sequence_classification.ipynb)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/medical/dna_sequence_classification/quick_deploy)
The ML model and third-party software used include:
- CountVectorizer
diff --git a/preview/site/en/tutorials/example_applications.md b/preview/site/en/tutorials/example_applications.md
deleted file mode 100644
index c0fd677b8..000000000
--- a/preview/site/en/tutorials/example_applications.md
+++ /dev/null
@@ -1,24 +0,0 @@
----
-id: example_applications.md
-title: Example applications
----
-
-# Example applications
-
-Milvus makes it easy to supercharge your AI application with vector similarity search. Check out the following example applications of Milvus.
-
-- [Image Similarity Search](image_similarity_search.md)
-
-- [Question Answering System](question_answering_system.md)
-
-- [Recommender System](recommendation_system.md)
-
-- [Video Similarity Search](video_similarity_search.md)
-
-- [Audio Similarity Search](audio_similarity_search.md)
-
-- [Molecular Similarity Search](molecular_similarity_search.md)
-
-- [DNA Sequence Classification](dna_sequence_classification.md)
-
-- [Text Search Engine](text_search_engine.md)
diff --git a/preview/site/en/tutorials/image_deduplication_system.md b/preview/site/en/tutorials/image_deduplication_system.md
index e77fca81a..6116a84da 100644
--- a/preview/site/en/tutorials/image_deduplication_system.md
+++ b/preview/site/en/tutorials/image_deduplication_system.md
@@ -1,6 +1,7 @@
---
id: image_deduplication_system.md
-summary: Build an image deduplication system with Milvus.
+summary: Build an image deduplication system with Milvus.
+title: Image Deduplication
---
# Image Deduplication
diff --git a/preview/site/en/tutorials/image_similarity_search.md b/preview/site/en/tutorials/image_similarity_search.md
index 9c7cdca32..f4263b001 100644
--- a/preview/site/en/tutorials/image_similarity_search.md
+++ b/preview/site/en/tutorials/image_similarity_search.md
@@ -1,31 +1,201 @@
---
id: image_similarity_search.md
-summary: Build an image similarity search system with Milvus.
+summary: image search with Milvus
+title: Image Search with Milvus
---
-# Image Similarity Search
+# Image Search with Milvus
-This tutorial demonstrates how to use Milvus, the open-source vector database, to build a reverse image search system.
-- [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/image/reverse_image_search)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/image/reverse_image_search/quick_deploy)
-- [Try online demo](https://milvus.io/milvus-demos/)
+
-The ML models and third-party software used include:
-- YOLOv3
-- ResNet-50
-- MySQL
-- [Towhee](https://towhee.io/)
+In this notebook, we will show you how to use Milvus to search for similar images in a dataset. We will use a subset of the [ImageNet](https://www.image-net.org/) dataset, then search for an image of an Afghan hound to demonstrate this.
-
-Major search engines like Google already give users the option to search by image. Additionally, e-commerce platforms have realized the benefits this functionality offers online shoppers, with Amazon incorporating image search into its smartphone applications.
+## Dataset Preparation
+First, we need to load the dataset and unextract it for further processing.
-
-In this tutorial, you will learn how to build a reverse image search system that can detect image patterns and return similar images to the one you upload. To build such an image similarity search system, download PASCAL VOC image set which contains 17125 images of 20 categories. Alternatively, you can prepare your own image datasets. Use YOLOv3 for object detection and ResNet-50 for image feature extraction. After going through the two ML models, images are converted into 256-dimensional vectors. Then store the vectors in Milvus and a unique ID for each vector is automatically generated by Milvus. MySQL is then used to map the vector IDs to the images in the dataset. Whenever you upload a new image to the image search system, it will be converted into a new vector and compared against the vectors previously stored in Milvus. Milvus then returns the IDs of the most similar vectors and you can query the corresponding images in MySQL.
+```python
+!wget https://github.com/milvus-io/pymilvus-assets/releases/download/imagedata/reverse_image_search.zip
+!unzip -q -o reverse_image_search.zip
+```
-
+## Prequisites
-
+To run this notebook, you need to have the following dependencies installed:
+- pymilvus>=2.4.2
+- timm
+- torch
+- numpy
+- sklearn
+- pillow
-
+To run Colab, we provide the handy commands to install the necessary dependencies.
+
+
+```python
+$ pip install pymilvus --upgrade
+$ pip install timm
+```
+
+
+
+
+
+ 'results'
+
+
+
+
+
+
+
+
+We can see that most of the images are from the same category as the search image, which is the Afghan hound. This means that we found similar images to the search image.
diff --git a/preview/site/en/tutorials/molecular_similarity_search.md b/preview/site/en/tutorials/molecular_similarity_search.md
deleted file mode 100644
index a82294dc3..000000000
--- a/preview/site/en/tutorials/molecular_similarity_search.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-id: molecular_similarity_search.md
-summary: Build a molecular similarity search system with Milvus.
----
-
-# Molecular Similarity Search
-
-This tutorial demonstrates how to use Milvus, the open-source vector database, to build a molecular similarity search system.
-- [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/medical/molecular_search)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/tree/master/solutions/medical/molecular_similarity_search/quick_deploy)
-- [Try demo](https://milvus.io/milvus-demos/)
-
-The third-party software used include:
-- RDKit
-- MySQL
-- [Towhee](https://towhee.io/)
-
-
- -Drug discovery is an important part of new medicine research and development. The process of drug discovery includes target selection and confirmation. When fragments or lead compounds are discovered, researchers usually search for similar compounds in internal or commercial compound libraries in order to discover structure-activity relationship (SAR), compound availability. Ultimately, they will evaluate the potential of the lead compounds to be optimized to candidate compounds. In order to discover available compounds from billion-scale compound libraries, chemical fingerprint is usually retrieved for substructure search and molecule similarity search. - -
- -In this tutorial, you will learn how to build a molecular similarity search system that can retrieve the substructure, superstructure, and similar structure of a particular molecule. RDKit is an open-source cheminformatics software that can convert molecule structures into vectors. Then, the vectors are stored in Milvus and Milvus can perform similarity search on vectors. Milvus also automatically generates a unique ID for each vector. The mapping of vector IDs and structure of molecules are stored in MySQL. - -
- - - - diff --git a/preview/site/en/tutorials/question_answering_system.md b/preview/site/en/tutorials/question_answering_system.md index 6d8c62522..dec6aafce 100644 --- a/preview/site/en/tutorials/question_answering_system.md +++ b/preview/site/en/tutorials/question_answering_system.md @@ -1,13 +1,13 @@ --- id: question_answering_system.md -summary: Build a question answering system with Milvus. +summary: Build a question answering system with Milvus. +title: Question Answering System --- # Question Answering System This tutorial demonstrates how to use Milvus, the open-source vector database, to build a question answering (QA) system. - [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/nlp/question_answering) -- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/question_answering_system/quick_deploy) - [Try online demo](https://milvus.io/milvus-demos/) The ML model and third-party software used include: diff --git a/preview/site/en/tutorials/recommendation_system.md b/preview/site/en/tutorials/recommendation_system.md index 7fcb612d6..cee4bf2db 100644 --- a/preview/site/en/tutorials/recommendation_system.md +++ b/preview/site/en/tutorials/recommendation_system.md @@ -1,13 +1,12 @@ --- id: recommendation_system.md -summary: Build a personalized recommender system with Milvus. +summary: Build a personalized recommender system with Milvus. +title: Recommender System --- # Recommender System This tutorial demonstrates how to use Milvus, the open-source vector database, to build a recommendation system. -- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/recommender_system.ipynb) -- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/quick_deploy) The ML model and third-party software used include: - PaddlePaddle diff --git a/preview/site/en/tutorials/text_image_search.md b/preview/site/en/tutorials/text_image_search.md index ea4cb53f2..bc38dda35 100644 --- a/preview/site/en/tutorials/text_image_search.md +++ b/preview/site/en/tutorials/text_image_search.md @@ -1,6 +1,7 @@ --- id: text_image_search.md -summary: Build a text to image search engine with Milvus. +summary: Build a text to image search engine with Milvus. +title: Text to Image Search Engine --- # Text to Image Search Engine diff --git a/preview/site/en/tutorials/text_search_engine.md b/preview/site/en/tutorials/text_search_engine.md index f99bb749a..f223d1579 100644 --- a/preview/site/en/tutorials/text_search_engine.md +++ b/preview/site/en/tutorials/text_search_engine.md @@ -1,6 +1,7 @@ --- id: text_search_engine.md -summary: Build a text search engine with Milvus. +summary: Build a text search engine with Milvus. +title: Text Search Engine --- # Text Search Engine diff --git a/preview/site/en/tutorials/video_similarity_search.md b/preview/site/en/tutorials/video_similarity_search.md index 4292487d3..7071bc136 100644 --- a/preview/site/en/tutorials/video_similarity_search.md +++ b/preview/site/en/tutorials/video_similarity_search.md @@ -1,13 +1,13 @@ --- id: video_similarity_search.md -summary: Build a video similarity search system with Milvus. +summary: Build a video similarity search system with Milvus. +title: Video Similarity Search --- # Video Similarity Search This tutorial demonstrates how to use Milvus, the open-source vector database, to build a video similarity search system. - [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/video/reverse_video_search) -- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/video/video_similarity_search/quick_deploy) The ML models and third-party software used include: - OpenCV diff --git a/preview/site/en/userGuide/build_index.md b/preview/site/en/userGuide/build_index.md deleted file mode 100644 index d9ddd48cc..000000000 --- a/preview/site/en/userGuide/build_index.md +++ /dev/null @@ -1,536 +0,0 @@ ---- -id: build_index.md -related_key: create index -summary: Learn how to build an index for vectors in Milvus. ---- - -# Build an Index on Vectors - -This guide describes how to build an index on vectors in Milvus. - -Vector indexes are an organizational unit of metadata used to accelerate [vector similarity search](search.md). Without the index built on vectors, Milvus will perform a brute-force search by default. - -See [Vector Index](index.md) for more information about the mechanism and varieties of vector indexes. - -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Build index
-
-Build the index by specifying the vector field name and index parameters.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.create_index(
- field_name="book_intro",
- index_params=index_params
-)
-```
-
-```python
-Status(code=0, message='')
-```
-
-```javascript
-await milvusClient.indexManager.createIndex({
- collection_name: "book",
- field_name: "book_intro",
- extra_params: index_params,
-});
-```
-
-```go
-err = milvusClient.CreateIndex(
- context.Background(), // ctx
- "book", // CollectionName
- "book_intro", // fieldName
- idx, // entity.Index
- false, // async
-)
-if err != nil {
- log.Fatal("fail to create index:", err.Error())
-}
-```
-
-```java
-milvusClient.createIndex(
- CreateIndexParam.newBuilder()
- .withCollectionName("book")
- .withFieldName("book_intro")
- .withIndexType(INDEX_TYPE)
- .withMetricType(MetricType.L2)
- .withExtraParam(INDEX_PARAM)
- .withSyncMode(Boolean.FALSE)
- .build()
-);
-```
-
-```shell
-# Follow the previous step.
-```
-
-```curl
-# Follow the previous step.
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
- - [Search with Time Travel](timetravel.md)
diff --git a/preview/site/en/userGuide/build_scalar_index.md b/preview/site/en/userGuide/build_scalar_index.md
deleted file mode 100644
index 132175d3f..000000000
--- a/preview/site/en/userGuide/build_scalar_index.md
+++ /dev/null
@@ -1,56 +0,0 @@
----
-id: build_scalar_index.md
-related_key: create scalar index
-summary: Learn how to build an index for scalar fields in Milvus.
----
-
-# Build an Index on Scalars
-
-This guide describes how to build an index on scalar fields.
-
-## Overview
-
-Unlike vectors, which have both magnitude and direction, scalars have only magnitude. Milvus regards single numbers and strings as scalars. [Here](schema.md#Supported-data-type) is a list of the available data types for scalar fields in Milvus.
-
-To speed up [attribute filtering](boolean.md) in [hybrid searches](hybridsearch.md), you can build indexes on scalar fields since Milvus v2.1.0. You can read more about scalar field indexing [here](scalar_index.md).
-
-## Build index
-
-To build an index on scalar fields, you do not need to set any index parameters. The default value of a scalar field index name is **_default_idx_**. You can set it to another value that seems fit.
-
-The following code snippet assumes that a collection named `book` already exists and an index is to be created on the string field `book_name`.
-
-```python
-from pymilvus import Collection
-
-collection = Collection("book")
-collection.create_index(
- field_name="book_name",
- index_name="scalar_index",
-)
-collection.load()
-```
-
-Once the index has been created, you can include a boolean expression on this string field in a vector similarity search as follows:
-
-```python
-search_param = {
- "data": [[0.1, 0.2]],
- "anns_field": "book_intro",
- "param": {"metric_type": "L2", "params": {"nprobe": 10}},
- "limit": 2,
- "expr": "book_name like \"Hello%\"",
-}
-res = collection.search(**search_param)
-```
-
-## What's next
-
-- To learn more about scalar field indexing, read [Scalar Index](scalar_index.md).
-- To learn more about the related terms and rules mentioned above, read
-
- - [Bitset](bitset.md)
- - [Hybrid search](hybridsearch.md)
- - [Boolean expression rules](boolean.md)
- - [Supported data types](schema.md#Supported-data-type)
-
diff --git a/preview/site/en/userGuide/bulk_insert.md b/preview/site/en/userGuide/bulk_insert.md
deleted file mode 100644
index 61ee310b4..000000000
--- a/preview/site/en/userGuide/bulk_insert.md
+++ /dev/null
@@ -1,514 +0,0 @@
----
-id: bulk_insert.md
-related_key: bulk load
-summary: Learn how to insert multiple entities in a batch from a JSON file.
----
-
-# Insert Entities from Files
-
-Milvus 2.2 now supports inserting a batch of entities from a file. Compared to the `insert()` method, this feature reduces network transmission across the Milvus client, proxy, Pulsar, and data nodes. You can now import a batch of entities in one file or multiple files into a collection with just a few lines of code.
-
-## Prepare the data file
-
-Organize the data to be inserted into a Milvus collection in a row-based JSON file or multiple NumPy files.
-
-### Row-based JSON file
-
-You can name the file whatever makes sense, but the root key must be **root**. In the file, each entity is organized in a dictionary. The keys in the dictionary are field names, and the values are field values in the corresponding entity.
-
-The following is an example of a row-based JSON file.
-
-```
-{
- "rows":[
- {"book_id": 101, "word_count": 13, "book_intro": [1.1, 1.2]},
- {"book_id": 102, "word_count": 25, "book_intro": [2.1, 2.2]},
- {"book_id": 103, "word_count": 7, "book_intro": [3.1, 3.2]},
- {"book_id": 104, "word_count": 12, "book_intro": [4.1, 4.2]},
- {"book_id": 105, "word_count": 34, "book_intro": [5.1, 5.2]}
- ]
-}
-```
-
- response = milvusClient.bulkInsert(param);
- BulkInsertResponseWrapper wrapper = new BulkInsertResponseWrapper(response.getData());
- task_id = wrapper.getTaskID();
- ```
-
-- For a set of column-based NumPy files, parameter **files** should be a multi-member list containing the paths to the NumPy files.
-
-
-
- ```python
- from pymilvus import utility
- task_id = utility.do_bulk_insert(
- collection_name="book",
- partition_name="2022",
- files=["book_id.npy", "word_count.npy", "book_intro.npy"]
- )
- ```
-
- ```java
- import io.milvus.param.bulkinsert.BulkInsertParam;
- import io.milvus.response.BulkInsertResponseWrapper;
- import io.milvus.grpc.ImportResponse;
- import io.milvus.param.R;
-
- BulkInsertParam param = BulkInsertParam.newBuilder()
- .withCollectionName("book")
- .withPartitionName("2022")
- .addFile("book_id.npy")
- .addFile("word_count.npy")
- .addFile("book_intro.npy")
- .build()
- R response = milvusClient.bulkInsert(param);
- BulkInsertResponseWrapper wrapper = new BulkInsertResponseWrapper(response.getData());
- task_id = wrapper.getTaskID();
- ```
-
- Each call to the bulk-insert API returns immediately. The return value is the ID of a data-import task running in the background. Milvus maintains a queue of such tasks to be dispatched in parallel to idle data nodes.
-
- response = milvusClient.getBulkInsertState(param);
-GetBulkInsertStateWrapper wrapper = new GetBulkInsertStateWrapper(response.getData());
-ImportState state = wrapper.getState();
-long row_count = wrapper.getImportedCount();
-String create_ts = wrapper.getCreateTimeStr();
-String failed_reason = wrapper.getFailedReason();
-String files = wrapper.getFiles();
-int progress = wrapper.getProgress();
-```
-
-The following table lists the state of a data-import task returned.
-
-| State | Code | Description |
-| ------------------ | -------- | ------------------------------------------------------------ |
-| Pending | 0 | The task is pending. |
-| Failed | 1 | The task fails. Use response = milvusClient.listBulkInsertTasks(param);
-List tasks = response.getTasksList();
-for (GetImportStateResponse task : tasks) {
- GetBulkInsertStateWrapper wrapper = new GetBulkInsertStateWrapper(task);
- ImportState state = wrapper.getState();
- long row_count = wrapper.getImportedCount();
- String create_ts = wrapper.getCreateTimeStr();
- String failed_reason = wrapper.getFailedReason();
- String files = wrapper.getFiles();
-}
-```
-
-| Parameter | Description |
-| -------------------------- | ------------------------------------------------------------ |
-| collection_name (optional) | Specify the target collection name to list all tasks on this collection. Leave the value empty if you want to list all tasks recorded by Milvus root coords. |
-| limit (optional) | Specify this parameter to limit the number of returned tasks. |
-
-
-See [System Configurations](configure_rootcoord.md) for more information about import task configurations.
-
-## Limits
-
-| Feature | Maximum limit |
-| ------------------------------ | ------------- |
-| Max. size of task pending list | 65536 |
-| Max. size of a data file | 16 GB |
-
-
-## Reference
-
-### Configure Milvus for data import
-
-To have Milvus remove failed or old data-import tasks automatically, you can specify a timeout duration and retention period for data-import tasks in the Milvus configuration file.
-
-```yaml
-rootCoord:
- # (in seconds) Duration after which an import task will expire (be killed). Default 900 seconds (15 minutes).
- # Note: If default value is to be changed, change also the default in: internal/util/paramtable/component_param.go
- importTaskExpiration: 900
- # (in seconds) Milvus will keep the record of import tasks for at least `importTaskRetention` seconds. Default 86400
- # seconds (24 hours).
- # Note: If default value is to be changed, change also the default in: internal/util/paramtable/component_param.go
- importTaskRetention: 86400
-```
-
-### Create NumPy files
-
-The following examples demonstrate how to create NumPy files for columns of data types that Milvus supports.
-
-- Create a Numpy file from a boolean array
-
- ```
- import numpy as np
- data = [True, False, True, False]
- dt = np.dtype('bool', (len(data)))
- arr = np.array(data, dtype=dt)
- np.save(file_path, arr)
- ```
-
-- Create a NumPy file from an int8 array
-
- ```
- import numpy as np
- data = [1, 2, 3, 4]
- dt = np.dtype('int8', (len(data)))
- arr = np.array(data, dtype=dt)
- np.save(file_path, arr)
- ```
-
-- Create a NumPy file from an int16 array
-
- ```
- import numpy as np
- data = [1, 2, 3, 4]
- dt = np.dtype('int16', (len(data)))
- arr = np.array(data, dtype=dt)
- np.save(file_path, arr)
- ```
-
-- Create a NumPy file from an int32 array
-
- ```
- import numpy as np
- data = [1, 2, 3, 4]
- dt = np.dtype('int32', (len(data)))
- arr = np.array(data, dtype=dt)
- np.save(file_path, arr)
- ```
-
-- Create a NumPy file from an int64 array
-
- ```
- import numpy as np
- data = [1, 2, 3, 4]
- dt = np.dtype('int64', (len(data)))
- arr = np.array(data, dtype=dt)
- np.save(file_path, arr)
- ```
-
-- Create a NumPy file from a float array
-
- ```
- import numpy as np
- data = [0.1, 0.2, 0.3, 0.4]
- dt = np.dtype('float32', (len(data)))
- arr = np.array(data, dtype=dt)
- np.save(file_path, arr)
- ```
-
-- Create a NumPy file from a double float array
-
- ```
- import numpy as np
- data = [0.1, 0.2, 0.3, 0.4]
- dt = np.dtype('float64', (len(data)))
- arr = np.array(data, dtype=dt)
- np.save(file_path, arr)
- ```
-
-- Create a NumPy file from a VARCHAR array
-
- ```
- data = ["a", "b", "c", "d"]
- arr = np.array(data)
- np.save(file_path, arr)
- ```
-
-- Create a NumPy file from a binary vector array
-
- For binary vectors, use **uint8** as the NumPy data type. Each uint8 value represents 8 dimensions. For a 32-dimensional binary vector, use four uint8 values.
-
- ```
- data = [
- [43, 35, 124, 90],
- [65, 212, 12, 57],
- [6, 126, 232, 78],
- [87, 189, 38, 22],
- ]
- dt = np.dtype('uint8', (len(data), 4))
- arr = np.array(data)
- np.save(file_path, arr)
- ```
-
-- Create a NumPy file from a float vector array
-
- In Milvus, you can use either float32 or float64 values to form a float vector.
-
- The following snippet creates a NumPy file from an 8-dimensional vector array formed using float32 values.
-
- ```
- data = [
- [1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8],
- [2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8],
- [3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8],
- [4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8],
- ]
- dt = np.dtype('float32', (len(data), 8))
- arr = np.array(data)
- np.save(file_path, arr)
- ```
-
-### Import multiple NumPy files in parallel
-
-You can upload NumPy files into different subdirectories, create multiple import tasks, and execute them in parallel.
-
-Assume the data structure is as follows:
-
- response = milvusClient.describeIndex(
- DescribeIndexParam.newBuilder()
- .withCollectionName(collection_name)
- .withIndexName(index_name)
- .build());
- IndexDescription desc = response.getData().getIndexDescriptions(0);
- if (desc.getIndexedRows() == desc.getTotalRows()) {
- break;
- }
- }
- ```
-
-2. Load new segments into query nodes
-
-Newly indexed segments need to be loaded manually as follows:
-
-
-
-```python
-collection.load(_refresh = True)
-```
-
-```java
-R response = milvusClient.loadCollection(
- LoadCollectionParam.newBuilder()
- .withCollectionName(collection_name)
- .withRefresh(Boolean.TRUE)
- .build());
-```
-
-
-
-
-
-
-
-
-
-## Prepare calculation parameters
-
-Specify the parameters used for the calculation.
-
-
-
-
-```python
-params = {
- "metric": "IP",
- "dim": 2
-}
-```
-
-```javascript
-// Node User Guide will be ready soon.
-```
-
-```go
-// GO User Guide will be ready soon.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-// CLI User Guide will be ready soon.
-```
-
-```curl
-params='[
- {"key": "metric", "value": "IP"}
-]'
-```
-
-
-
-
-
-
-
-
-
-## (Optional) Load collection
-
-If you calculate with the vectors in a collection in Milvus, you must load the collection to memory first.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.load()
-```
-
-```javascript
-await milvusClient.collectionManager.loadCollection({
- collection_name: "book",
-});
-```
-
-```go
-err := milvusClient.LoadCollection(
- context.Background(), // ctx
- "book", // CollectionName
- false // async
-)
-if err != nil {
- log.Fatal("failed to load collection:", err.Error())
-}
-```
-
-```java
-milvusClient.loadCollection(
- LoadCollectionParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-```
-
-```shell
-load -c book
-```
-
-``` curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/collection/load' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book"
- }'
-```
-
-## Calculate vector distance
-
-Calculate the distance between vectors based on the vectors and parameters provided.
-
-```python
-from pymilvus import utility
-results = utility.calc_distance(
- vectors_left=vectors_left,
- vectors_right=vectors_right,
- params=params
-)
-print(results)
-```
-
-```javascript
-// Node User Guide will be ready soon.
-```
-
-```go
-// GO User Guide will be ready soon.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-// CLI User Guide will be ready soon.
-```
-
-```curl
-curl -X 'GET' \
- 'http://localhost:9091/api/v1/distance' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d "{
- \"op_left\": $vectors_left,
- \"op_right\": $vectors_right,
- \"params\": $params
- }"
-```
-
- respHasCollection = milvusClient.hasCollection(
- HasCollectionParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-if (respHasCollection.getData() == Boolean.TRUE) {
- System.out.println("Collection exists.");
-}
-```
-
-```shell
-describe collection -c book
-```
-
-```curl
-curl -X 'GET' \
- 'http://localhost:9091/api/v1/collection/existence' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book"
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Check collection details
-
-Check the details of a collection.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-
-collection.schema # Return the schema.CollectionSchema of the collection.
-collection.description # Return the description of the collection.
-collection.name # Return the name of the collection.
-collection.is_empty # Return the boolean value that indicates if the collection is empty.
-collection.num_entities # Return the number of entities in the collection.
-collection.primary_field # Return the schema.FieldSchema of the primary key field.
-collection.partitions # Return the list[Partition] object.
-collection.indexes # Return the list[Index] object.
-collection.properties # Return the expiration time of data in the collection.
-```
-
-```javascript
-await milvusClient.collectionManager.describeCollection({ // Return the name and schema of the collection.
- collection_name: "book",
-});
-
-await milvusClient.collectionManager.getCollectionStatistics({ // Return the statistics information of the collection.
- collection_name: "book",
-});
-```
-
-```go
-collDesc, err := milvusClient.DescribeCollection( // Return the name and schema of the collection.
- context.Background(), // ctx
- "book", // CollectionName
-)
-if err != nil {
- log.Fatal("failed to check collection schema:", err.Error())
-}
-log.Printf("%v\n", collDesc)
-
-collStat, err := milvusClient.GetCollectionStatistics( // Return the statistics information of the collection.
- context.Background(), // ctx
- "book", // CollectionName
-)
-if err != nil {
- log.Fatal("failed to check collection statistics:", err.Error())
-}
-```
-
-```java
-R respDescribeCollection = milvusClient.describeCollection(
- // Return the name and schema of the collection.
- DescribeCollectionParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-DescCollResponseWrapper wrapperDescribeCollection = new DescCollResponseWrapper(respDescribeCollection.getData());
-System.out.println(wrapperDescribeCollection);
-
-R respCollectionStatistics = milvusClient.getCollectionStatistics(
- // Return the statistics information of the collection.
- GetCollectionStatisticsParam.newBuilder()
- .withCollectionName("book")
- .build()
- );
-GetCollStatResponseWrapper wrapperCollectionStatistics = new GetCollStatResponseWrapper(respCollectionStatistics.getData());
-System.out.println("Collection row count: " + wrapperCollectionStatistics.getRowCount());
-```
-
-```shell
-describe collection -c book
-```
-
-```curl
-curl -X 'GET' \
- 'http://localhost:9091/api/v1/collection' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book"
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## List all collections
-
-List all collections in this Milvus Instance.
-
-
-
-
-```python
-from pymilvus import utility
-utility.list_collections()
-```
-
-```javascript
-await milvusClient.collectionManager.showCollections();
-```
-
-```go
-listColl, err := milvusClient.ListCollections(
- context.Background(), // ctx
-)
-if err != nil {
- log.Fatal("failed to list all collections:", err.Error())
-}
-log.Println(listColl)
-```
-
-```java
-R respShowCollections = milvusClient.showCollections(
- ShowCollectionsParam.newBuilder().build()
- );
-System.out.println(respShowCollections);
-```
-
-```shell
-list collections
-```
-
-```curl
-curl -X 'GET' \
- 'http://localhost:9091/api/v1/collections' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json'
-```
-
-
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Create a partition](create_partition.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
diff --git a/preview/site/en/userGuide/check_partition.md b/preview/site/en/userGuide/check_partition.md
deleted file mode 100644
index 915af85f6..000000000
--- a/preview/site/en/userGuide/check_partition.md
+++ /dev/null
@@ -1,354 +0,0 @@
----
-id: check_partition.md
-related_key: Partition
-summary: Learn how to check partition information in Milvus.
----
-
-# Check Partition Information
-
-This topic describes how to check the information of partitions in Milvus.
-
-## Verify if a partition exists
-
-Verify if a partition exists in the specified collection.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.has_partition("novel")
-```
-
-```javascript
-await milvusClient.partitionManager.hasPartition({
- collection_name: "book",
- partition_name: "novel",
-});
-```
-
-```go
-hasPar, err := milvusClient.HasPartition(
- context.Background(), // ctx
- "book", // CollectionName
- "novel", // partitionName
-)
-if err != nil {
- log.Fatal("failed to check the partition:", err.Error())
-}
-log.Println(hasPar)
-```
-
-```java
-R respHasPartition = milvusClient.hasPartition(
- HasPartitionParam.newBuilder()
- .withCollectionName("book")
- .withPartitionName("novel")
- .build()
-);
-if (respHasPartition.getData() == Boolean.TRUE) {
- System.out.println("Partition exists.");
-}
-```
-
-```shell
-describe partition -c book -p novel
-```
-
-``` curl
-curl -X 'GET' \
- 'http://localhost:9091/api/v1/partition/existence' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "partition_name": "novel"
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## List all partitions
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.partitions
-```
-
-```javascript
-await milvusClient.partitionManager.showPartitions({
- collection_name: "book",
-});
-```
-
-```go
-listPar, err := milvusClient.ShowPartitions(
- context.Background(), // ctx
- "book", // CollectionName
-)
-if err != nil {
- log.Fatal("failed to list partitions:", err.Error())
-}
-log.Println(listPar)
-```
-
-```java
-R respShowPartitions = milvusClient.showPartitions(
- ShowPartitionsParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-System.out.println(respShowPartitions);
-```
-
-```shell
-list partitions -c book
-```
-
-``` curl
-curl -X 'GET' \
- 'http://localhost:9091/api/v1/partitions' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book"
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
-
diff --git a/preview/site/en/userGuide/collection_alias.md b/preview/site/en/userGuide/collection_alias.md
deleted file mode 100644
index d7a0a016c..000000000
--- a/preview/site/en/userGuide/collection_alias.md
+++ /dev/null
@@ -1,489 +0,0 @@
----
-id: collection_alias.md
-related_key: collection alias
-summary: Learn how to manage collection alias in Milvus.
----
-
-# Collection Alias
-
-This topic describes how to manage collection aliases. Milvus supports specifying a unique alias for a collection.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Drop a collection alias
-
-Drop a specified alias.
-
-
-
-
-```python
-from pymilvus import utility
-utility.drop_alias(alias = "publication")
-```
-
-```javascript
-await milvusClient.collectionManager.dropAlias({
- alias: "publication",
-});
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-milvusClient.dropAlias(
- DropAliasParam.newBuilder()
- .withAlias("publication")
- .build()
-);
-```
-
-```shell
-delete alias -a publication
-```
-
-```curl
-curl -X 'DELETE' \
- 'http://localhost:9091/api/v1/alias' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "alias":"publication"
- }'
-```
-```curl
-# Output:
-{}
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Alter a collection alias
-
-Alter an existing alias to another collection. The following example is based on the situation that the alias `publication` was originally created for another collection.
-
-
-
-
-```python
-from pymilvus import utility
-utility.alter_alias(
- collection_name = "book",
- alias = "publication"
-)
-```
-
-```javascript
-await milvusClient.collectionManager.alterAlias({
- collection_name: "book",
- alias: "publication",
-});
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-milvusClient.alterAlias(
- AlterAliasParam.newBuilder()
- .withCollectionName("book")
- .withAlias("publication")
- .build()
-);
-```
-
-```shell
-create alias -c book -A -a publication
-```
-
-```curl
-curl -X 'PATCH' \
- 'http://localhost:9091/api/v1/alias' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "alias":"publication"
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Limits
-
-|Feature|Maximum limit|
-|---|---|
-|Length of an alias|255 characters|
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Create a partition](create_partition.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
-
diff --git a/preview/site/en/userGuide/compact_data.md b/preview/site/en/userGuide/compact_data.md
deleted file mode 100644
index b30b1e429..000000000
--- a/preview/site/en/userGuide/compact_data.md
+++ /dev/null
@@ -1,192 +0,0 @@
----
-id: compact_data.md
-related_key: compact data
-summary: Learn how to compact data in Milvus.
----
-
-# Compact Data
-
-This topic describes how to compact data in Milvus.
-
-Milvus supports automatic data compaction by default. You can [configure](configure-docker.md) your Milvus to enable or disable [compaction](configure_datacoord.md#dataCoordenableCompaction) and [automatic compaction](configure_datacoord.md#dataCoordcompactionenableAutoCompaction).
-
-If automatic compaction is disabled, you can still compact data manually.
-
- response = milvusClient.manualCompaction(
- ManualCompactionParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-long compactionID = response.getData().getCompactionID();
-```
-
-```shell
-compact -c book
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/compaction' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collectionID": 434262071120432449
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Check compaction status
-
-You can check the compaction status with the compaction ID returned when the manual compaction is triggered.
-
-
-
-
-```python
-collection.get_compaction_state()
-```
-
-```javascript
-const state = await milvusClient.collectionManager.getCompactionState({
- compactionID
-});
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-milvusClient.getCompactionState(GetCompactionStateParam.newBuilder()
- .withCompactionID(compactionID)
- .build()
-);
-```
-
-```shell
-show compaction_state -c book
-```
-
-```curl
-curl -X 'GET' \
- 'http://localhost:9091/api/v1/compaction/state' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "compactionID": 434262132129005569
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Create a collection with the schema
-
-Then, create a collection with the schema you specified above.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection(
- name=collection_name,
- schema=schema,
- using='default',
- shards_num=2
- )
-```
-
-```javascript
-await milvusClient.collectionManager.createCollection(params);
-```
-
-```go
-err = milvusClient.CreateCollection(
- context.Background(), // ctx
- schema,
- 2, // shardNum
-)
-if err != nil {
- log.Fatal("failed to create collection:", err.Error())
-}
-```
-
-```java
-milvusClient.createCollection(createCollectionReq);
-```
-
-```shell
-# Follow the previous step.
-```
-
-```curl
-# Follow the previous step.
-```
-
-
-
-
-
-
-
-
-
-
-
-## Limits
-|Feature|Maximum limit|
-|---|---|
-|Length of a collection name|255 characters|
-|Number of partitions in a collection|4,096|
-|Number of fields in a collection|64|
-|Number of shards in a collection|256|
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Create a partition](create_partition.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
-
diff --git a/preview/site/en/userGuide/create_partition.md b/preview/site/en/userGuide/create_partition.md
deleted file mode 100644
index 62a1ba84a..000000000
--- a/preview/site/en/userGuide/create_partition.md
+++ /dev/null
@@ -1,211 +0,0 @@
----
-id: create_partition.md
-related_key: Partition
-summary: Learn how to create a partition in Milvus.
----
-
-# Create a Partition
-
-This topic describes how to create a partition in Milvus.
-
-Milvus allows you to divide the bulk of vector data into a small number of partitions. Search and other operations can then be limited to one partition to improve the performance.
-
-A collection consists of one or more partitions. While creating a new collection, Milvus creates a default partition `_default`. See [Glossary - Partition](glossary.md#Partition) for more information.
-
-The following example builds a partition `novel` in the collection `book`.
-
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.create_partition("novel")
-```
-
-```javascript
-await milvusClient.partitionManager.createPartition({
- collection_name: "book",
- partition_name: "novel",
-});
-```
-
-```go
-err := milvusClient.CreatePartition(
- context.Background(), // ctx
- "book", // CollectionName
- "novel" // partitionName
-)
-if err != nil {
- log.Fatal("failed to create partition:", err.Error())
-}
-```
-
-```java
-milvusClient.createPartition(
- CreatePartitionParam.newBuilder()
- .withCollectionName("book")
- .withPartitionName("novel")
- .build()
-);
-```
-
-```shell
-create partition -c book -p novel
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/partition' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "partition_name": "novel"
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Limits
-|Feature |Maximum limit|
-|---|---|
-|Number of partitions in a collection|4,096|
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
-
diff --git a/preview/site/en/userGuide/data-import/import-data.md b/preview/site/en/userGuide/data-import/import-data.md
new file mode 100644
index 000000000..2f43829ef
--- /dev/null
+++ b/preview/site/en/userGuide/data-import/import-data.md
@@ -0,0 +1,256 @@
+---
+id: import-data.md
+order: 1
+title: Import Data
+summary: This page demonstrates the procedure to import the prepared data.
+---
+
+# Import data
+
+This page demonstrates the procedure to import the prepared data.
+
+## Before you start
+
+- You have already prepared your data and placed it into the Milvus bucket.
+
+ If not, you should use **RemoteBulkWriter** to prepare your data first, and ensure that the prepared data has already been transferred to the Milvus bucket on the MinIO instance started along with your Milvus instance. For details, refer to [Prepare Source Data](prepare-source-data.md).
+
+- You have already created a collection with the schema you use to prepare your data. If not, refer to [Manage Collections](manage-collections.md).
+
+
-
-
-
-
-## Delete entities
-
-Delete the entities with the boolean expression you created. Milvus returns the ID list of the deleted entities.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.delete(expr)
-```
-
-```javascript
-await milvusClient.dataManager.deleteEntities({
- collection_name: "book",
- expr: expr,
-});
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-milvusClient.delete(
- DeleteParam.newBuilder()
- .withCollectionName("book")
- .withExpr(DELETE_EXPR)
- .build()
-);
-```
-
-```shell
-You are trying to delete the entities of collection. This action cannot be undone!
-Do you want to continue? [y/N]: y
-```
-
-```curl
-curl -X 'DELETE' \
- 'http://localhost:9091/api/v1/entities' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "expr": "book_id in [0,1]"
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
diff --git a/preview/site/en/userGuide/drop_collection.md b/preview/site/en/userGuide/drop_collection.md
deleted file mode 100644
index ee45f4cbe..000000000
--- a/preview/site/en/userGuide/drop_collection.md
+++ /dev/null
@@ -1,177 +0,0 @@
----
-id: drop_collection.md
-related_key: drop collection
-summary: Learn how to drop a collection in Milvus.
----
-
-# Drop a collection
-
-This topic describes how to drop a collection and the data within.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Create a partition](create_partition.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
diff --git a/preview/site/en/userGuide/drop_index.md b/preview/site/en/userGuide/drop_index.md
deleted file mode 100644
index 00c2a2d42..000000000
--- a/preview/site/en/userGuide/drop_index.md
+++ /dev/null
@@ -1,169 +0,0 @@
----
-id: drop_index.md
-related_key: drop index
-summary: Learn how to drop an index in Milvus.
----
-
-# Drop an Index
-
-This topic describes how to drop an index in Milvus.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
- - [Search with Time Travel](timetravel.md)
-
diff --git a/preview/site/en/userGuide/drop_partition.md b/preview/site/en/userGuide/drop_partition.md
deleted file mode 100644
index ebd2563c7..000000000
--- a/preview/site/en/userGuide/drop_partition.md
+++ /dev/null
@@ -1,196 +0,0 @@
----
-id: drop_partition.md
-related_key: Partition
-summary: Learn how to drop a partition in Milvus.
----
-
-# Drop Partitions
-
-This topic describes how to drop a partition in a specified collection.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
-
diff --git a/preview/site/en/userGuide/enable-dynamic-field.md b/preview/site/en/userGuide/enable-dynamic-field.md
new file mode 100644
index 000000000..fb07c6de9
--- /dev/null
+++ b/preview/site/en/userGuide/enable-dynamic-field.md
@@ -0,0 +1,570 @@
+---
+id: enable-dynamic-field.md
+title: Enable Dynamic Field
+---
+
+# Enable Dynamic Field
+
+This page explains how to use the dynamic field in a collection for flexible data insertion and retrieval.
+
+## Overview
+
+Milvus allows you to define the schema of a collection by setting the name and the data type of each specific field so that you can create indexes in these fields for improved search performance.
+
+Once a field is defined, you need to include this field when you insert data. What if some fields are not always present in all your data entries? This is where the dynamic field comes in.
+
+The dynamic field in a collection is a reserved JSON field named $meta. It can hold non-schema-defined fields and their values as key-value pairs. Using the dynamic field, you can search and query both schema-defined fields and any non-schema-defined fields they may have.
+
+## Enable dynamic field
+
+When defining a schema for a collection, you can set `enable_dynamic_field` to `True` to enable the reserved dynamic field, indicating that any non-schema-defined fields and their values inserted later on will be saved as key-value pairs in the reserved dynamic field.
+
+The following snippet creates a collection with two schema-defined fields, namely id and vector, and enables the dynamic field.
+
+"
+# }
+```
+
+```java
+import io.milvus.v2.client.ConnectConfig;
+import io.milvus.v2.client.MilvusClientV2;
+import io.milvus.v2.common.DataType;
+import io.milvus.v2.common.IndexParam;
+import io.milvus.v2.service.collection.request.AddFieldReq;
+import io.milvus.v2.service.collection.request.CreateCollectionReq;
+import io.milvus.v2.service.collection.request.GetLoadStateReq;
+
+String CLUSTER_ENDPOINT = "http://localhost:19530";
+
+// 1. Connect to Milvus server
+ConnectConfig connectConfig = ConnectConfig.builder()
+ .uri(CLUSTER_ENDPOINT)
+ .build();
+
+MilvusClientV2 client = new MilvusClientV2(connectConfig);
+
+// 2. Create a collection in customized setup mode
+
+// 2.1 Create schema
+CreateCollectionReq.CollectionSchema schema = client.createSchema();
+
+// 2.2 Add fields to schema
+schema.addField(AddFieldReq.builder().fieldName("id").dataType(DataType.Int64).isPrimaryKey(true).autoID(false).build());
+schema.addField(AddFieldReq.builder().fieldName("vector").dataType(DataType.FloatVector).dimension(5).build());
+
+// 2.3 Prepare index parameters
+IndexParam indexParamForIdField = IndexParam.builder()
+ .fieldName("id")
+ .indexType(IndexParam.IndexType.STL_SORT)
+ .build();
+
+IndexParam indexParamForVectorField = IndexParam.builder()
+ .fieldName("vector")
+ .indexType(IndexParam.IndexType.IVF_FLAT)
+ .metricType(IndexParam.MetricType.IP)
+ .extraParams(Map.of("nlist", 1024))
+ .build();
+
+List indexParams = new ArrayList<>();
+indexParams.add(indexParamForIdField);
+indexParams.add(indexParamForVectorField);
+
+// 2.4 Create a collection with schema and index parameters
+CreateCollectionReq customizedSetupReq = CreateCollectionReq.builder()
+ .collectionName("customized_setup")
+ .collectionSchema(schema)
+ .indexParams(indexParams)
+ // highlight-next-line
+ .enableDynamicField(true)
+ .build();
+
+client.createCollection(customizedSetupReq);
+
+Thread.sleep(5000);
+
+// 2.5 Get load state of the collection
+GetLoadStateReq customSetupLoadStateReq1 = GetLoadStateReq.builder()
+ .collectionName("customized_setup")
+ .build();
+
+boolean res = client.getLoadState(customSetupLoadStateReq1);
+
+System.out.println(res);
+
+// Output:
+// true
+```
+
+```javascript
+const { MilvusClient, DataType, sleep } = require("@zilliz/milvus2-sdk-node")
+
+const address = "http://localhost:19530"
+
+async function main() {
+// 1. Set up a Milvus Client
+client = new MilvusClient({address});
+
+// 2. Create a collection
+// 2.1 Define fields
+const fields = [
+ {
+ name: "id",
+ data_type: DataType.Int64,
+ is_primary_key: true,
+ auto_id: false
+ },
+ {
+ name: "vector",
+ data_type: DataType.FloatVector,
+ dim: 5
+ },
+]
+
+// 2.2 Prepare index parameters
+const index_params = [{
+ field_name: "id",
+ index_type: "STL_SORT"
+},{
+ field_name: "vector",
+ index_type: "IVF_FLAT",
+ metric_type: "IP",
+ params: { nlist: 1024}
+}]
+
+// 2.3 Create a collection with fields and index parameters
+res = await client.createCollection({
+ collection_name: "test_collection",
+ fields: fields,
+ index_params: index_params,
+ // highlight-next-line
+ enable_dynamic_field: true
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+
+res = await client.getLoadState({
+ collection_name: "test_collection",
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateLoaded
+//
+```
+
+## Insert dynamic data
+
+Once the collection is created, you can start inserting data, including the dynamic data into the collection.
+
+### Prepare data
+
+In this section, you need to prepare some randomly generated data for the insertion later on.
+
+
+
+```python
+colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
+data = []
+
+for i in range(1000):
+ current_color = random.choice(colors)
+ current_tag = random.randint(1000, 9999)
+ data.append({
+ "id": i,
+ "vector": [ random.uniform(-1, 1) for _ in range(5) ],
+ "color": current_color,
+ "tag": current_tag,
+ "color_tag": f"{current_color}_{str(current_tag)}"
+ })
+
+print(data[0])
+```
+
+```java
+List colors = Arrays.asList("green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey");
+List data = new ArrayList<>();
+
+for (int i=0; i<1000; i++) {
+ Random rand = new Random();
+ String current_color = colors.get(rand.nextInt(colors.size()-1));
+ int current_tag = rand.nextInt(8999) + 1000;
+ JSONObject row = new JSONObject();
+ row.put("id", Long.valueOf(i));
+ row.put("vector", Arrays.asList(rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat()));
+ row.put("color", current_color);
+ row.put("tag", current_tag);
+ row.put("color_tag", current_color + "_" + String.valueOf(rand.nextInt(8999) + 1000));
+ data.add(row);
+}
+
+System.out.println(JSONObject.toJSON(data.get(0)));
+```
+
+```javascript
+const colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
+var data = []
+
+for (let i = 0; i < 1000; i++) {
+ const current_color = colors[Math.floor(Math.random() * colors.length)]
+ const current_tag = Math.floor(Math.random() * 8999 + 1000)
+ data.push({
+ id: i,
+ vector: [Math.random(), Math.random(), Math.random(), Math.random(), Math.random()],
+ color: current_color,
+ tag: current_tag,
+ color_tag: `${current_color}_${current_tag}`
+ })
+}
+
+console.log(data[0])
+```
+
+You can view the structure of the generated data by checking its first entry.
+
+```
+{
+ id: 0,
+ vector: [
+ 0.1275656405044483,
+ 0.47417858592773277,
+ 0.13858264437643286,
+ 0.2390904907020377,
+ 0.8447862593689635
+ ],
+ color: 'blue',
+ tag: 2064,
+ color_tag: 'blue_2064'
+}
+```
+
+### Insert data
+
+Then you can safely insert the data into the collection.
+
+> queryVectors = Arrays.asList(Arrays.asList(0.3580376395471989f, -0.6023495712049978f, 0.18414012509913835f, -0.26286205330961354f, 0.9029438446296592f));
+
+SearchReq searchReq = SearchReq.builder()
+ .collectionName("customized_setup")
+ .data(queryVectors)
+ .filter("$meta[\"color\"] in [\"red\", \"green\"]")
+ .outputFields(List.of("id", "color_tag"))
+ .topK(3)
+ .build();
+
+SearchResp searchResp = client.search(searchReq);
+
+System.out.println(JSONObject.toJSON(searchResp));
+
+// Output:
+// {"searchResults": [[
+// {
+// "distance": 1.3159835,
+// "id": 979,
+// "entity": {
+// "color_tag": "red_7155",
+// "id": 979
+// }
+// },
+// {
+// "distance": 1.0744804,
+// "id": 44,
+// "entity": {
+// "color_tag": "green_8006",
+// "id": 44
+// }
+// },
+// {
+// "distance": 1.0060014,
+// "id": 617,
+// "entity": {
+// "color_tag": "red_4056",
+// "id": 617
+// }
+// }
+// ]]}
+```
+
+```javascript
+// 4. Search with non-schema-defined fields
+const query_vectors = [[0.1, 0.2, 0.3, 0.4, 0.5]]
+
+res = await client.search({
+ collection_name: "test_collection",
+ data: query_vectors,
+ filter: "color in [\"red\", \"green\"]",
+ output_fields: ["color_tag"],
+ limit: 3
+})
+
+console.log(res.results)
+
+// Output
+//
+// [
+// { score: 1.2284551858901978, id: '301', color_tag: 'red_1270' },
+// { score: 1.2195171117782593, id: '205', color_tag: 'red_2780' },
+// { score: 1.2055039405822754, id: '487', color_tag: 'red_6653' }
+// ]
+//
+```
+
+## Recaps
+
+It is worth noting that __color__, __tag__, and __color_tag__ are not present when you define the collection schema, but you can use them as schema-defined fields when you conduct searches and queries.
+
+If the name of a non-schema-defined field contains characters other than digits, letters, and underscores, such as plus signs (+), asterisks (*), or dollar signs ($), you have to include the key within __$meta[]__ as shown in the following code snippet when using it in a boolean expression or including it in the output fields.
+
+```python
+...
+filter='$meta["$key"] in ["a", "b", "c"]',
+output_fields='$meta["$key"]'
+...
+```
diff --git a/preview/site/en/userGuide/import_data.md b/preview/site/en/userGuide/import_data.md
deleted file mode 100644
index ad1e0c8f6..000000000
--- a/preview/site/en/userGuide/import_data.md
+++ /dev/null
@@ -1,191 +0,0 @@
----
-id: import_data.md
-related_key: bulk load
-summary: Learn how to bulk load data in Milvus.
----
-
-# Import Data
-
-This topic describes how to import data in Milvus via bulk load.
-
-Regular method to insert a large batch of entities to Milvus usually leads to a massive network transmission across client, proxy, Pulsar and data nodes. To avoid such situation, Milvus 2.1 supports loading data from files via bulk load. You can import large amounts of data into a collection by just a few lines of code, and endow atomicity to a whole batch of entities.
-
-You can also migrate data to Milvus with [MilvusDM](migrate_overview.md), an open-source tool designed specifically for importing and exporting data with Milvus.
-
-## Prepare data file
-
-You can prepare the data file on row base or column base.
-
-- Row-based data file
-
-A row-based data file is a JSON file containing multiple rows. The root key must be "rows". The file name can be specified arbitrarily.
-
-```json
-{
- "rows":[
- {"book_id": 101, "word_count": 13, "book_intro": [1.1, 1.2]},
- {"book_id": 102, "word_count": 25, "book_intro": [2.1, 2.2]},
- {"book_id": 103, "word_count": 7, "book_intro": [3.1, 3.2]},
- {"book_id": 104, "word_count": 12, "book_intro": [4.1, 4.2]},
- {"book_id": 105, "word_count": 34, "book_intro": [5.1, 5.2]},
- ]
-}
-```
-
-- Column-based data file
-
-A column-based data file can be a JSON file containing multiple columns, several Numpy files, each contains a single column, or a JSON file contains multiple columns and some Numpy files.
-
- - JSON file containing multiple columns
- ```json
- {
- "book_id": [101, 102, 103, 104, 105],
- "word_count": [13, 25, 7, 12, 34],
- "book_intro": [
- [1.1, 1.2],
- [2.1, 2.2],
- [3.1, 3.2],
- [4.1, 4.2],
- [5.1, 5.2]
- ]
- }
- ```
-
- - Numpy files
-
- ```python
- import numpy
- numpy.save('book_id.npy', numpy.array([101, 102, 103, 104, 105]))
- numpy.save('word_count.npy', numpy.array([13, 25, 7, 12, 34]))
- arr = numpy.array([[1.1, 1.2],
- [2.1, 2.2],
- [3.1, 3.2],
- [4.1, 4.2],
- [5.1, 5.2]])
- numpy.save('book_intro.npy', arr)
- ```
-
- - A JSON file contains multiple columns and some Numpy files.
-
- ```json
- {
- "book_id": [101, 102, 103, 104, 105],
- "word_count": [13, 25, 7, 12, 34]
- }
- ```
-
- ```python
- {
- "book_id": [101, 102, 103, 104, 105],
- "word_count": [13, 25, 7, 12, 34]
- }
- ```
-
-## Upload data file
-
-Upload data files to object storage.
-
-You can upload data file to MinIO or local storage (available only in Milvus Standalone).
-
-- Upload to MinIO
-
-upload the data files to the bucket which is defined by [`minio.bucketName`](configure_minio.md#miniobucketName) in the configuration file `milvus.yml`.
-
-- Upload to local storage
-
-copy the data files into the directory which is defined by [`localStorage.path`](configure_localstorage.md#localStoragepath) in the configuration file `milvus.yml`.
-
-
-## Insert data to Milvus
-
-Import the data to the collection.
-
-- For row-based files
-
-```python
-from pymilvus import utility
-tasks = utility.bulk_load(
- collection_name="book",
- is_row_based=True,
- files=["row_based_1.json", "row_based_2.json"]
-)
-```
-
-- For column-based files
-
-```python
-from pymilvus import utility
-tasks = utility.bulk_load(
- collection_name="book",
- is_row_based=False,
- files=["columns.json", "book_intro.npy"]
-)
-```
-
-
-
-
-
-## Check the import task state
-
-Check the state of the import task.
-
-```python
-state = utility.get_bulk_load_state(tasks[0])
-print(state.state_name())
-print(state.ids())
-print(state.infos())
-```
-The state codes and their corresponding descriptions.
-
-| State code | State | Description |
-| ---------- | ----------------------- | -------------------------------------------------------------- |
-| 0 | BulkLoadPending | Task is in pending list |
-| 1 | BulkLoadFailed | Task failed, get the failed reason with `state.infos["failed_reason"]` |
-| 2 | BulkLoadStarted | Task is dispatched to data node, gonna to be executed |
-| 3 | BulkLoadDownloaded | Data file has been downloaded from MinIO to local |
-| 4 | BulkLoadParsed | Data file has been validated and parsed |
-| 5 | BulkLoadPersisted | New segments have been generated and persisted |
-| 6 | BulkLoadCompleted | Task completed |
-
-
-## Limits
-
-|Feature|Maximum limit|
-|---|---|
-|Max size of task pending list|32|
-|Max size of a data file|4GB|
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
-- Explore API references for Milvus SDKs:
- - [PyMilvus API reference](/api-reference/pymilvus/v2.2.2/tutorial.html)
- - [Node.js API reference](/api-reference/node/v2.2.x/tutorial.html)
-
diff --git a/preview/site/en/userGuide/insert-update-delete.md b/preview/site/en/userGuide/insert-update-delete.md
new file mode 100644
index 000000000..10f7b225b
--- /dev/null
+++ b/preview/site/en/userGuide/insert-update-delete.md
@@ -0,0 +1,751 @@
+---
+id: insert-update-delete.md
+summary: This guide walks you through the data manipulation operations within a collection, including insertion, upsertion, and deletion.
+title: Insert, Upsert & Delete
+---
+
+# Insert, Upsert & Delete
+
+This guide walks you through the data manipulation operations within a collection, including insertion, upsertion, and deletion.
+
+## Before you start
+
+- You have installed the SDK of your choice. To install an SDK, refer to [Install SDKs](https://milvus.io/docs/install-pymilvus.md).
+
+- You have created a collection. To create a collection, refer to [Manage Collections](manage-collections.md).
+
+- To insert a large volume of data, you are advised to use [Data Import](https://milvus.io/api-reference/pymilvus/v2.4.x/DataImport/LocalBulkWriter/LocalBulkWriter.md).
+
+## Overview
+
+An entity, within the context of Milvus collections, is a singular, identifiable instance within a collection. It represents a distinct member of a particular class, be it a book in a library, a gene in a genome, or any other identifiable entity.
+
+Entities within a collection share a common set of attributes, termed schema, outlining the structure that each entity must adhere to, including field names, data types, and any other constraints.
+
+Successful insertion of entities into a collection requires that the provided data should contain all the schema-defined fields of the target collection. Additionally, you can also include non-schema-defined fields only if you have enabled the dynamic field. For details, refer to [Enable Dynamic Field](enable-dynamic-field.md).
+
+## Preparations
+
+The code snippet below repurposes the existing code to establish a connection to a Milvus cluster and quickly set up a collection.
+
+ data = Arrays.asList(
+ new JSONObject(Map.of("id", 0L, "vector", Arrays.asList(0.3580376395471989f, -0.6023495712049978f, 0.18414012509913835f, -0.26286205330961354f, 0.9029438446296592f), "color", "pink_8682")),
+ new JSONObject(Map.of("id", 1L, "vector", Arrays.asList(0.19886812562848388f, 0.06023560599112088f, 0.6976963061752597f, 0.2614474506242501f, 0.838729485096104f), "color", "red_7025")),
+ new JSONObject(Map.of("id", 2L, "vector", Arrays.asList(0.43742130801983836f, -0.5597502546264526f, 0.6457887650909682f, 0.7894058910881185f, 0.20785793220625592f), "color", "orange_6781")),
+ new JSONObject(Map.of("id", 3L, "vector", Arrays.asList(0.3172005263489739f, 0.9719044792798428f, -0.36981146090600725f, -0.4860894583077995f, 0.95791889146345f), "color", "pink_9298")),
+ new JSONObject(Map.of("id", 4L, "vector", Arrays.asList(0.4452349528804562f, -0.8757026943054742f, 0.8220779437047674f, 0.46406290649483184f, 0.30337481143159106f), "color", "red_4794")),
+ new JSONObject(Map.of("id", 5L, "vector", Arrays.asList(0.985825131989184f, -0.8144651566660419f, 0.6299267002202009f, 0.1206906911183383f, -0.1446277761879955f), "color", "yellow_4222")),
+ new JSONObject(Map.of("id", 6L, "vector", Arrays.asList(0.8371977790571115f, -0.015764369584852833f, -0.31062937026679327f, -0.562666951622192f, -0.8984947637863987f), "color", "red_9392")),
+ new JSONObject(Map.of("id", 7L, "vector", Arrays.asList(-0.33445148015177995f, -0.2567135004164067f, 0.8987539745369246f, 0.9402995886420709f, 0.5378064918413052f), "color", "grey_8510")),
+ new JSONObject(Map.of("id", 8L, "vector", Arrays.asList(0.39524717779832685f, 0.4000257286739164f, -0.5890507376891594f, -0.8650502298996872f, -0.6140360785406336f), "color", "white_9381")),
+ new JSONObject(Map.of("id", 9L, "vector", Arrays.asList(0.5718280481994695f, 0.24070317428066512f, -0.3737913482606834f, -0.06726932177492717f, -0.6980531615588608f), "color", "purple_4976"))
+);
+
+InsertReq insertReq = InsertReq.builder()
+ .collectionName("quick_setup")
+ .data(data)
+ .build();
+
+InsertResp insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 10}
+```
+
+```javascript
+// 3. Insert some data
+
+var data = [
+ {id: 0, vector: [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592], color: "pink_8682"},
+ {id: 1, vector: [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, 0.2614474506242501, 0.838729485096104], color: "red_7025"},
+ {id: 2, vector: [0.43742130801983836, -0.5597502546264526, 0.6457887650909682, 0.7894058910881185, 0.20785793220625592], color: "orange_6781"},
+ {id: 3, vector: [0.3172005263489739, 0.9719044792798428, -0.36981146090600725, -0.4860894583077995, 0.95791889146345], color: "pink_9298"},
+ {id: 4, vector: [0.4452349528804562, -0.8757026943054742, 0.8220779437047674, 0.46406290649483184, 0.30337481143159106], color: "red_4794"},
+ {id: 5, vector: [0.985825131989184, -0.8144651566660419, 0.6299267002202009, 0.1206906911183383, -0.1446277761879955], color: "yellow_4222"},
+ {id: 6, vector: [0.8371977790571115, -0.015764369584852833, -0.31062937026679327, -0.562666951622192, -0.8984947637863987], color: "red_9392"},
+ {id: 7, vector: [-0.33445148015177995, -0.2567135004164067, 0.8987539745369246, 0.9402995886420709, 0.5378064918413052], color: "grey_8510"},
+ {id: 8, vector: [0.39524717779832685, 0.4000257286739164, -0.5890507376891594, -0.8650502298996872, -0.6140360785406336], color: "white_9381"},
+ {id: 9, vector: [0.5718280481994695, 0.24070317428066512, -0.3737913482606834, -0.06726932177492717, -0.6980531615588608], color: "purple_4976"}
+]
+
+var res = await client.insert({
+ collection_name: "quick_setup",
+ data: data,
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 10
+//
+```
+
+### Insert into partitions
+
+To insert data into a specific partition, you can specify the name of the partition in the insert request as follows:
+
+
+
+```python
+# 4. Insert some more data into a specific partition
+data=[
+ {"id": 10, "vector": [-0.5570353903748935, -0.8997887893201304, -0.7123782431855732, -0.6298990746450119, 0.6699215060604258], "color": "red_1202"},
+ {"id": 11, "vector": [0.6319019033373907, 0.6821488267878275, 0.8552303045704168, 0.36929791364943054, -0.14152860714878068], "color": "blue_4150"},
+ {"id": 12, "vector": [0.9483947484855766, -0.32294203351925344, 0.9759290319978025, 0.8262982148666174, -0.8351194181285713], "color": "orange_4590"},
+ {"id": 13, "vector": [-0.5449109892498731, 0.043511240563786524, -0.25105249484790804, -0.012030655265886425, -0.0010987671273892108], "color": "pink_9619"},
+ {"id": 14, "vector": [0.6603339372951424, -0.10866551787442225, -0.9435597754324891, 0.8230244263466688, -0.7986720938400362], "color": "orange_4863"},
+ {"id": 15, "vector": [-0.8825129181091456, -0.9204557711667729, -0.935350065513425, 0.5484069690287079, 0.24448151140671204], "color": "orange_7984"},
+ {"id": 16, "vector": [0.6285586391568163, 0.5389064528263487, -0.3163366239905099, 0.22036279378888013, 0.15077052220816167], "color": "blue_9010"},
+ {"id": 17, "vector": [-0.20151825016059233, -0.905239387635804, 0.6749305353372479, -0.7324272081377843, -0.33007998971889263], "color": "blue_4521"},
+ {"id": 18, "vector": [0.2432286610792349, 0.01785636564206139, -0.651356982731391, -0.35848148851027895, -0.7387383128324057], "color": "orange_2529"},
+ {"id": 19, "vector": [0.055512329053363674, 0.7100266349039421, 0.4956956543575197, 0.24541352586717702, 0.4209030729923515], "color": "red_9437"}
+]
+
+client.create_partition(
+ collection_name="quick_setup",
+ partition_name="partitionA"
+)
+
+res = client.insert(
+ collection_name="quick_setup",
+ data=data,
+ partition_name="partitionA"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 10,
+# "ids": [
+# 10,
+# 11,
+# 12,
+# 13,
+# 14,
+# 15,
+# 16,
+# 17,
+# 18,
+# 19
+# ]
+# }
+```
+
+```java
+// 4. Insert some more data into a specific partition
+data = Arrays.asList(
+ new JSONObject(Map.of("id", 10L, "vector", Arrays.asList(-0.5570353903748935f, -0.8997887893201304f, -0.7123782431855732f, -0.6298990746450119f, 0.6699215060604258f), "color", "red_1202")),
+ new JSONObject(Map.of("id", 11L, "vector", Arrays.asList(0.6319019033373907f, 0.6821488267878275f, 0.8552303045704168f, 0.36929791364943054f, -0.14152860714878068f), "color", "blue_4150")),
+ new JSONObject(Map.of("id", 12L, "vector", Arrays.asList(0.9483947484855766f, -0.32294203351925344f, 0.9759290319978025f, 0.8262982148666174f, -0.8351194181285713f), "color", "orange_4590")),
+ new JSONObject(Map.of("id", 13L, "vector", Arrays.asList(-0.5449109892498731f, 0.043511240563786524f, -0.25105249484790804f, -0.012030655265886425f, -0.0010987671273892108f), "color", "pink_9619")),
+ new JSONObject(Map.of("id", 14L, "vector", Arrays.asList(0.6603339372951424f, -0.10866551787442225f, -0.9435597754324891f, 0.8230244263466688f, -0.7986720938400362f), "color", "orange_4863")),
+ new JSONObject(Map.of("id", 15L, "vector", Arrays.asList(-0.8825129181091456f, -0.9204557711667729f, -0.935350065513425f, 0.5484069690287079f, 0.24448151140671204f), "color", "orange_7984")),
+ new JSONObject(Map.of("id", 16L, "vector", Arrays.asList(0.6285586391568163f, 0.5389064528263487f, -0.3163366239905099f, 0.22036279378888013f, 0.15077052220816167f), "color", "blue_9010")),
+ new JSONObject(Map.of("id", 17L, "vector", Arrays.asList(-0.20151825016059233f, -0.905239387635804f, 0.6749305353372479f, -0.7324272081377843f, -0.33007998971889263f), "color", "blue_4521")),
+ new JSONObject(Map.of("id", 18L, "vector", Arrays.asList(0.2432286610792349f, 0.01785636564206139f, -0.651356982731391f, -0.35848148851027895f, -0.7387383128324057f), "color", "orange_2529")),
+ new JSONObject(Map.of("id", 19L, "vector", Arrays.asList(0.055512329053363674f, 0.7100266349039421f, 0.4956956543575197f, 0.24541352586717702f, 0.4209030729923515f), "color", "red_9437"))
+);
+
+CreatePartitionReq createPartitionReq = CreatePartitionReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionA")
+ .build();
+
+client.createPartition(createPartitionReq);
+
+insertReq = InsertReq.builder()
+ .collectionName("quick_setup")
+ .data(data)
+ .partitionName("partitionA")
+ .build();
+
+insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 10}
+```
+
+```javascript
+// 4. Insert some more data into a specific partition
+data = [
+ {id: 10, vector: [-0.5570353903748935, -0.8997887893201304, -0.7123782431855732, -0.6298990746450119, 0.6699215060604258], color: "red_1202"},
+ {id: 11, vector: [0.6319019033373907, 0.6821488267878275, 0.8552303045704168, 0.36929791364943054, -0.14152860714878068], color: "blue_4150"},
+ {id: 12, vector: [0.9483947484855766, -0.32294203351925344, 0.9759290319978025, 0.8262982148666174, -0.8351194181285713], color: "orange_4590"},
+ {id: 13, vector: [-0.5449109892498731, 0.043511240563786524, -0.25105249484790804, -0.012030655265886425, -0.0010987671273892108], color: "pink_9619"},
+ {id: 14, vector: [0.6603339372951424, -0.10866551787442225, -0.9435597754324891, 0.8230244263466688, -0.7986720938400362], color: "orange_4863"},
+ {id: 15, vector: [-0.8825129181091456, -0.9204557711667729, -0.935350065513425, 0.5484069690287079, 0.24448151140671204], color: "orange_7984"},
+ {id: 16, vector: [0.6285586391568163, 0.5389064528263487, -0.3163366239905099, 0.22036279378888013, 0.15077052220816167], color: "blue_9010"},
+ {id: 17, vector: [-0.20151825016059233, -0.905239387635804, 0.6749305353372479, -0.7324272081377843, -0.33007998971889263], color: "blue_4521"},
+ {id: 18, vector: [0.2432286610792349, 0.01785636564206139, -0.651356982731391, -0.35848148851027895, -0.7387383128324057], color: "orange_2529"},
+ {id: 19, vector: [0.055512329053363674, 0.7100266349039421, 0.4956956543575197, 0.24541352586717702, 0.4209030729923515], color: "red_9437"}
+]
+
+await client.createPartition({
+ collection_name: "quick_setup",
+ partition_name: "partitionA"
+})
+
+res = await client.insert({
+ collection_name: "quick_setup",
+ data: data,
+ partition_name: "partitionA"
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 10
+//
+```
+
+The output is a dictionary containing the statistics on the affected entities. For details on partition operations, refer to [Manage Partitions](manage-partitions.md).
+
+## Upsert entities
+
+Upserting data is a combination of update and insert operations. In Milvus, an upsert operation performs a data-level action to either insert or update an entity based on whether its primary key already exists in a collection. Specifically:
+
+- If the primary key of the entity already exists in the collection, the existing entity will be overwritten.
+
+- If the primary key does not exist in the collection, a new entity will be inserted.
+
+ book_id_array = new ArrayList<>();
-List word_count_array = new ArrayList<>();
-List> book_intro_array = new ArrayList<>();
-for (long i = 0L; i < 2000; ++i) {
- book_id_array.add(i);
- word_count_array.add(i + 10000);
- List vector = new ArrayList<>();
- for (int k = 0; k < 2; ++k) {
- vector.add(ran.nextFloat());
- }
- book_intro_array.add(vector);
-}
-```
-
-```shell
-# Prepare your data in a CSV file. Milvus CLI only supports importing data from local or remote files.
-```
-
-```curl
-# See the following step.
-```
-
-## Insert data to Milvus
-
-Insert the data to the collection.
-
-By specifying `partition_name`, you can optionally decide to which partition to insert the data.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-mr = collection.insert(data)
-```
-
-```javascript
-const mr = await milvusClient.dataManager.insert({
- collection_name: "book",
- fields_data: data,
-});
-```
-
-```go
-_, err = milvusClient.Insert(
- context.Background(), // ctx
- "book", // CollectionName
- "", // partitionName
- idColumn, // columnarData
- wordColumn, // columnarData
- introColumn, // columnarData
-)
-if err != nil {
- log.Fatal("failed to insert data:", err.Error())
-}
-```
-
-```java
-List fields = new ArrayList<>();
-fields.add(new InsertParam.Field("book_id", DataType.Int64, book_id_array));
-fields.add(new InsertParam.Field("word_count", DataType.Int64, word_count_array));
-fields.add(new InsertParam.Field("book_intro", DataType.FloatVector, book_intro_array));
-
-InsertParam insertParam = InsertParam.newBuilder()
- .withCollectionName("book")
- .withPartitionName("novel")
- .withFields(fields)
- .build();
-milvusClient.insert(insertParam);
-```
-
-```shell
-import -c book 'https://raw.githubusercontent.com/milvus-io/milvus_cli/main/examples/user_guide/search.csv'
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/entities' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "fields_data": [
- {
- "field_name": "book_id",
- "type": 5,
- "field": [
- 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100
- ]
- },
- {
- "field_name": "word_count",
- "type": 5,
- "field": [
- 1000,2000,3000,4000,5000,6000,7000,8000,9000,10000,11000,12000,13000,14000,15000,16000,17000,18000,19000,20000,21000,22000,23000,24000,25000,26000,27000,28000,29000,30000,31000,32000,33000,34000,35000,36000,37000,38000,39000,40000,41000,42000,43000,44000,45000,46000,47000,48000,49000,50000,51000,52000,53000,54000,55000,56000,57000,58000,59000,60000,61000,62000,63000,64000,65000,66000,67000,68000,69000,70000,71000,72000,73000,74000,75000,76000,77000,78000,79000,80000,81000,82000,83000,84000,85000,86000,87000,88000,89000,90000,91000,92000,93000,94000,95000,96000,97000,98000,99000,100000
- ]
- },
- {
- "field_name": "book_intro",
- "type": 101,
- "field": [
- [1,1],[2,1],[3,1],[4,1],[5,1],[6,1],[7,1],[8,1],[9,1],[10,1],[11,1],[12,1],[13,1],[14,1],[15,1],[16,1],[17,1],[18,1],[19,1],[20,1],[21,1],[22,1],[23,1],[24,1],[25,1],[26,1],[27,1],[28,1],[29,1],[30,1],[31,1],[32,1],[33,1],[34,1],[35,1],[36,1],[37,1],[38,1],[39,1],[40,1],[41,1],[42,1],[43,1],[44,1],[45,1],[46,1],[47,1],[48,1],[49,1],[50,1],[51,1],[52,1],[53,1],[54,1],[55,1],[56,1],[57,1],[58,1],[59,1],[60,1],[61,1],[62,1],[63,1],[64,1],[65,1],[66,1],[67,1],[68,1],[69,1],[70,1],[71,1],[72,1],[73,1],[74,1],[75,1],[76,1],[77,1],[78,1],[79,1],[80,1],[81,1],[82,1],[83,1],[84,1],[85,1],[86,1],[87,1],[88,1],[89,1],[90,1],[91,1],[92,1],[93,1],[94,1],[95,1],[96,1],[97,1],[98,1],[99,1],[100,1]
- ]
- }
- ],
- "num_rows": 100
-}'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Limits
-
-|Feature|Maximum limit|
-|---|---|
-|Dimensions of a vector|32,768|
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
-
diff --git a/preview/site/en/userGuide/load_balance.md b/preview/site/en/userGuide/load_balance.md
index a7ff98a43..c1011e3b7 100644
--- a/preview/site/en/userGuide/load_balance.md
+++ b/preview/site/en/userGuide/load_balance.md
@@ -2,6 +2,8 @@
id: load_balance.md
related_key: Load balance
summary: Learn how to balance query load in Milvus.
+title: Balance Query Load
+deprecate: true
---
# Balance Query Load
@@ -44,7 +46,7 @@ milvusClient.getQuerySegmentInfo(
```
```javascript
-await dataManager.getQuerySegmentInfo({
+await getQuerySegmentInfo({
collectionName: "book",
});
```
@@ -151,7 +153,7 @@ milvusClient.loadBalance(
```
```javascript
-await dataManager.loadBalance({
+await loadBalance({
src_nodeID: 3,
dst_nodeIDs: [4],
sealed_segmentIDs: [431067441441538050]
@@ -257,9 +259,10 @@ load_balance -s 3 -d 4 -ss 431067441441538050
## What's next
- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Create a partition](create_partition.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Index Vector Fields](index-vector-fields.md)
+ - [Index Scalar Fields](index-scalar-fields.md)
+ - [Single-vector search](single-vector-search.md)
+ - [Hybrid search](multi-vector-search.md)
diff --git a/preview/site/en/userGuide/load_collection.md b/preview/site/en/userGuide/load_collection.md
deleted file mode 100644
index 36c3e7bc5..000000000
--- a/preview/site/en/userGuide/load_collection.md
+++ /dev/null
@@ -1,226 +0,0 @@
----
-id: load_collection.md
-related_key: load collection
-summary: Learn how to load a collection into memory for CRUD operations in Milvus.
----
-
-# Load a Collection
-
-This topic describes how to load the collection to memory before a search or a query. All search and query operations within Milvus are executed in memory.
-
-Milvus 2.1 allows users to load a collection as multiple replicas to utilize the CPU and memory resources of extra query nodes. This feature boost the overall QPS and throughput without extra hardware. It is supported on PyMilvus in current release.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Get replica information
-
-You can check the information of the loaded replicas.
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.load(replica_number=2) # Load collection as 2 replicas
-result = collection.get_replicas()
-print(result)
-```
-
-Below is an example of the output.
-
-```
-Replica groups:
-- Group: , , , , , Shard: , , ]>
-- Group: , , , , , Shard: , , ]>
-```
-
-## Constraints
-
-- Error will be returned at the attempt to load partition(s) when the parent collection is already loaded. Future releases will support releasing partitions from a loaded collection, and (if needed) then loading some other partition(s).
-- "Load successfully" will be returned at the attempt to load the collection that is already loaded.
-- Error will be returned at the attempt to load the collection when the child partition(s) is/are already loaded. Future releases will support loading the collection when some of its partitions are already loaded.
-- Loading different partitions in a same collection via separate RPCs is not allowed.
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Create a partition](create_partition.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
-
diff --git a/preview/site/en/userGuide/load_partition.md b/preview/site/en/userGuide/load_partition.md
deleted file mode 100644
index 1814d2585..000000000
--- a/preview/site/en/userGuide/load_partition.md
+++ /dev/null
@@ -1,239 +0,0 @@
----
-id: load_partition.md
-related_key: load partition
-summary: Learn how to load a partition into memory for search or query in Milvus.
----
-
-# Load a Partition
-
-This topic describes how to load a partition to memory. Loading partitions instead of the whole collection to memory can significantly reduce the memory usage. All search and query operations within Milvus are executed in memory.
-
-Milvus 2.1 allows users to load a partition as multiple replicas to utilize the CPU and memory resources of extra query nodes. This feature boost the overall QPS and throughput with extra hardware. It is supported on PyMilvus in current release.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Get replica information
-
-You can check the information of the loaded replicas.
-
-```python
-from pymilvus import Partition
-partition = Partition("novel") # Get an existing partition.
-partition.load(replica_number=2) # Load partition as 2 replicas
-result = partition.get_replicas()
-print(result)
-```
-
-
-## Constraints
-
-- Error will be returned at the attempt to load partition(s) when the parent collection is already loaded. Future releases will support releasing partitions from a loaded collection, and (if needed) then loading some other partition(s).
-- "Load successfully" will be returned at the attempt to load the collection that is already loaded.
-- Error will be returned at the attempt to load the collection when the child partition(s) is/are already loaded. Future releases will support loading the collection when some of its partitions are already loaded.
-- Loading different partitions in a same collection via separate RPCs is not allowed.
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
-
diff --git a/preview/site/en/userGuide/manage-collections.md b/preview/site/en/userGuide/manage-collections.md
new file mode 100644
index 000000000..2d0d68f35
--- /dev/null
+++ b/preview/site/en/userGuide/manage-collections.md
@@ -0,0 +1,2812 @@
+---
+id: manage-collections.md
+title: Manage Collections
+---
+
+# Manage Collections
+
+This guide walks you through creating and managing collections using the SDK of your choice.
+
+## Before you start
+
+- You have installed [Milvus standalone](https://milvus.io/docs/install_standalone-docker.md) or [Milvus cluster](https://milvus.io/docs/install_cluster-milvusoperator.md).
+
+- You have installed preferred SDKs. You can choose among various languages, including [Python](https://milvus.io/docs/install-pymilvus.md), [Java](https://milvus.io/docs/install-java.md), [Go](https://milvus.io/docs/install-go.md), and [Node.js](https://milvus.io/docs/install-node.md).
+
+## Overview
+
+In Milvus, you store your vector embeddings in collections. All vector embeddings within a collection share the same dimensionality and distance metric for measuring similarity.
+
+Milvus collections support dynamic fields (i.e., fields not pre-defined in the schema) and automatic incrementation of primary keys.
+
+To accommodate different preferences, Milvus offers two methods for creating a collection. One provides a quick setup, while the other allows for detailed customization of the collection schema and index parameters.
+
+Additionally, you can view, load, release, and drop a collection when necessary.
+
+## Create Collection
+
+You can create a collection in either of the following manners:
+
+- __Quick setup__
+
+ In this manner, you can create a collection by simply giving it a name and specifying the number of dimensions of the vector embeddings to be stored in this collection. For details, refer to [Quick setup](manage-collections.md).
+
+- __Customized setup__
+
+ Instead of letting In Milvus decide almost everything for your collection, you can determine the __schema__ and __index parameters__ of the collection on your own. For details, refer to [Customized setup](manage-collections.md).
+
+### Quick setup
+
+Against the backdrop of the great leap in the AI industry, most developers just need a simple yet dynamic collection to start with. Milvus allows a quick setup of such a collection with just three arguments:
+
+- Name of the collection to create,
+
+- Dimension of the vector embeddings to insert, and
+
+- Metric type used to measure similarities between vector embeddings.
+
+"
+# }
+```
+
+```java
+import io.milvus.v2.client.ConnectConfig;
+import io.milvus.v2.client.MilvusClientV2;
+import io.milvus.v2.service.collection.request.GetLoadStateReq;
+import io.milvus.v2.service.collection.request.CreateCollectionReq;
+
+String CLUSTER_ENDPOINT = "http://localhost:19530";
+
+// 1. Connect to Milvus server
+ConnectConfig connectConfig = ConnectConfig.builder()
+ .uri(CLUSTER_ENDPOINT)
+ .build();
+
+MilvusClientV2 client = new MilvusClientV2(connectConfig);
+
+// 2. Create a collection in quick setup mode
+CreateCollectionReq quickSetupReq = CreateCollectionReq.builder()
+ .collectionName("quick_setup")
+ .dimension(5)
+ .build();
+
+client.createCollection(quickSetupReq);
+
+// Thread.sleep(5000);
+
+GetLoadStateReq quickSetupLoadStateReq = GetLoadStateReq.builder()
+ .collectionName("quick_setup")
+ .build();
+
+Boolean res = client.getLoadState(quickSetupLoadStateReq);
+
+System.out.println(res);
+
+// Output:
+// true
+```
+
+```javascript
+address = "http://localhost:19530"
+
+// 1. Set up a Milvus Client
+client = new MilvusClient({address});
+
+// 2. Create a collection in quick setup mode
+let res = await client.createCollection({
+ collection_name: "quick_setup",
+ dimension: 5,
+});
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+
+res = await client.getLoadState({
+ collection_name: "quick_setup"
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateLoaded
+//
+```
+
+```shell
+$ export MILVUS_URI="localhost:19530"
+
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/create" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "quick_setup",
+ "dimension": 5
+}'
+
+# Output
+#
+# {
+# "code": 0,
+# "data": {},
+# }
+
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/get_load_state" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "quick_setup"
+}'
+
+# {
+# "code": 0,
+# "data": {
+# "loadProgress": 100,
+# "loadState": "LoadStateLoaded"
+# }
+# }
+```
+
+The collection generated in the above code contains only two fields: `id` (as the primary key) and `vector` (as the vector field), with `auto_id` and `enable_dynamic_field` settings enabled by default.
+
+- `auto_id`
+
+ Enabling this setting ensures that the primary key increments automatically. There's no need for manual provision of primary keys during data insertion.
+
+- `enable_dynamic_field`
+
+ When enabled, all fields, excluding `id` and `vector` in the data to be inserted, are treated as dynamic fields. These additional fields are saved as key-value pairs within a special field named `$meta`. This feature allows the inclusion of extra fields during data insertion.
+
+The automatically indexed and loaded collection from the provided code is ready for immediate data insertions.
+
+### Customized setup
+
+Instead of letting Milvus decide almost everything for your collection, you can determine the __schema__ and __index parameters__ of the collection on your own.
+
+#### Step 1: Set up schema
+
+A schema defines the structure of a collection. Within the schema, you have the option to enable or disable `enable_dynamic_field`, add pre-defined fields, and set attributes for each field. For a detailed explanation of the concept and available data types, refer to [Schema Explained](schema.md).
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+#### Step 2: Set up index parameters
+
+Index parameters dictate how Milvus organizes your data within a collection. You can tailor the indexing process for specific fields by adjusting their `metric_type` and `index_type`. For the vector field, you have the flexibility to select `COSINE`, `L2`, `IP`, `HAMMING`, or `JACCARD` as the `metric_type`, depending on the type of vectors you are working with. For more information, refer to [Similarity Metrics](metric.md).
+
+ indexParams = new ArrayList<>();
+indexParams.add(indexParamForIdField);
+indexParams.add(indexParamForVectorField);
+```
+
+```javascript
+// 3.2 Prepare index parameters
+const index_params = [{
+ field_name: "my_id",
+ index_type: "STL_SORT"
+},{
+ field_name: "my_vector",
+ index_type: "IVF_FLAT",
+ metric_type: "IP",
+ params: { nlist: 1024}
+}]
+```
+
+```shell
+export indexParams='[{ \
+ "fieldName": "my_id", \
+ "indexName": "my_id", \
+ "params": { \
+ "index_type": "SLT_SORT" \
+ } \
+}, { \
+ "fieldName": "my_vector", \
+ "metricType": "COSINE", \
+ "indexName": "my_vector", \
+ "params": { \
+ "index_type": "IVF_FLAT", \
+ "nlist": 1024 \
+ } \
+}]'
+```
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+The code snippet above demonstrates how to set up index parameters for the vector field and a scalar field, respectively. For the vector field, set both the metric type and the index type. For a scalar field, set only the index type. It is recommended to create an index for the vector field and any scalar fields that are frequently used for filtering.
+
+#### Step 3: Create the collection
+
+You have the option to create a collection and an index file separately or to create a collection with the index loaded simultaneously upon creation.
+
+"
+ # }
+ ```
+
+ ```java
+ import io.milvus.v2.service.collection.request.CreateCollectionReq;
+ import io.milvus.v2.service.collection.request.GetLoadStateReq;
+
+ // 3.4 Create a collection with schema and index parameters
+ CreateCollectionReq customizedSetupReq1 = CreateCollectionReq.builder()
+ .collectionName("customized_setup_1")
+ .collectionSchema(schema)
+ .indexParams(indexParams)
+ .build();
+
+ client.createCollection(customizedSetupReq1);
+
+ // Thread.sleep(5000);
+
+ // 3.5 Get load state of the collection
+ GetLoadStateReq customSetupLoadStateReq1 = GetLoadStateReq.builder()
+ .collectionName("customized_setup_1")
+ .build();
+
+ res = client.getLoadState(customSetupLoadStateReq1);
+
+ System.out.println(res);
+
+ // Output:
+ // true
+ ```
+
+ ```javascript
+ // 3.3 Create a collection with fields and index parameters
+ res = await client.createCollection({
+ collection_name: "customized_setup_1",
+ fields: fields,
+ index_params: index_params,
+ })
+
+ console.log(res.error_code)
+
+ // Output
+ //
+ // Success
+ //
+
+ res = await client.getLoadState({
+ collection_name: "customized_setup_1"
+ })
+
+ console.log(res.state)
+
+ // Output
+ //
+ // LoadStateLoaded
+ //
+ ```
+
+ ```shell
+ $ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/create" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "collectionName": "customized_setup_1",
+ "schema": {
+ "autoId": false,
+ "enabledDynamicField": false,
+ "fields": [
+ {
+ "fieldName": "my_id",
+ "dataType": "Int64",
+ "isPrimary": true
+ },
+ {
+ "fieldName": "my_vector",
+ "dataType": "FloatVector",
+ "elementTypeParams": {
+ "dim": "5"
+ }
+ }
+ ]
+ },
+ "indexParams": [
+ {
+ "fieldName": "my_vector",
+ "metricType": "COSINE",
+ "indexName": "my_vector",
+ "params": {
+ "index_type": "IVF_FLAT",
+ "nlist": "1024"
+ }
+ },
+ {
+ "fieldName": "my_id",
+ "indexName": "my_id",
+ "params": {
+ "index_type": "STL_SORT"
+ }
+ }
+ ]
+ }'
+
+ # Output
+ #
+ # {
+ # "code": 0,
+ # "data": {},
+ # }
+
+ $ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/get_load_state" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "collectionName": "customized_setup_1"
+ }'
+
+ # {
+ # "code": 0,
+ # "data": {
+ # "loadProgress": 100,
+ # "loadState": "LoadStateLoaded"
+ # }
+ # }
+ ```
+
+ The collection created above is loaded automatically. To learn more about loading and releasing a collection, refer to [Load & Release Collection](manage-collections.md#Load--Release-Collection).
+
+- __Create a collection and an index file separately.__
+
+
+
+ ```python
+ # 3.6. Create a collection and index it separately
+ client.create_collection(
+ collection_name="customized_setup_2",
+ schema=schema,
+ )
+
+ res = client.get_load_state(
+ collection_name="customized_setup_2"
+ )
+
+ print(res)
+
+ # Output
+ #
+ # {
+ # "state": ""
+ # }
+ ```
+
+ ```java
+ // 3.6 Create a collection and index it separately
+ CreateCollectionReq customizedSetupReq2 = CreateCollectionReq.builder()
+ .collectionName("customized_setup_2")
+ .collectionSchema(schema)
+ .build();
+
+ client.createCollection(customizedSetupReq2);
+ ```
+
+ ```javascript
+ // 3.4 Create a collection and index it seperately
+ res = await client.createCollection({
+ collection_name: "customized_setup_2",
+ fields: fields,
+ })
+
+ console.log(res.error_code)
+
+ // Output
+ //
+ // Success
+ //
+
+ res = await client.getLoadState({
+ collection_name: "customized_setup_2"
+ })
+
+ console.log(res.state)
+
+ // Output
+ //
+ // LoadStateNotLoad
+ //
+ ```
+
+ ```shell
+ $ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/create" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "collectionName": "customized_setup_2",
+ "schema": {
+ "autoId": false,
+ "enabledDynamicField": false,
+ "fields": [
+ {
+ "fieldName": "my_id",
+ "dataType": "Int64",
+ "isPrimary": true
+ },
+ {
+ "fieldName": "my_vector",
+ "dataType": "FloatVector",
+ "elementTypeParams": {
+ "dim": "5"
+ }
+ }
+ ]
+
+ }
+ }'
+
+ # Output
+ #
+ # {
+ # "code": 0,
+ # "data": {},
+ # }
+
+ $ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/get_load_state" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "collectionName": "customized_setup_2"
+ }'
+
+ # {
+ # "code": 0,
+ # "data": {
+ # "loadState": "LoadStateNotLoaded"
+ # }
+ # }
+ ```
+
+ The collection created above is not loaded automatically. You can create an index for the collection as follows. Creating an index for the collection in a separate manner does not automatically load the collection. For details, refer to [Load & Release Collection](manage-collections.md#Load--Release-Collection).
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ The collection created above is not loaded automatically. You can create an index for the collection as follows. Creating an index for the collection in a separate manner does not automatically load the collection. For details, refer to [Load & Release Collection](manage-collections.md).
+
+
+
+ ```python
+ # 3.6 Create index
+ client.create_index(
+ collection_name="customized_setup_2",
+ index_params=index_params
+ )
+
+ res = client.get_load_state(
+ collection_name="customized_setup_2"
+ )
+
+ print(res)
+
+ # Output
+ #
+ # {
+ # "state": ""
+ # }
+ ```
+
+ ```java
+ CreateIndexReq createIndexReq = CreateIndexReq.builder()
+ .collectionName("customized_setup_2")
+ .indexParams(indexParams)
+ .build();
+
+ client.createIndex(createIndexReq);
+
+ // Thread.sleep(1000);
+
+ // 3.7 Get load state of the collection
+ GetLoadStateReq customSetupLoadStateReq2 = GetLoadStateReq.builder()
+ .collectionName("customized_setup_2")
+ .build();
+
+ res = client.getLoadState(customSetupLoadStateReq2);
+
+ System.out.println(res);
+
+ // Output:
+ // false
+ ```
+
+ ```javascript
+ // 3.5 Create index
+ res = await client.createIndex({
+ collection_name: "customized_setup_2",
+ field_name: "my_vector",
+ index_type: "IVF_FLAT",
+ metric_type: "IP",
+ params: { nlist: 1024}
+ })
+
+ res = await client.getLoadState({
+ collection_name: "customized_setup_2"
+ })
+
+ console.log(res.state)
+
+ // Output
+ //
+ // LoadStateNotLoad
+ //
+ ```
+
+ ```shell
+ $ curl -X POST "http://${MILVUS_URI}/v2/vectordb/indexes/create" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "collectionName": "customized_setup_2",
+ "indexParams": [
+ {
+ "metricType": "L2",
+ "fieldName": "my_vector",
+ "indexName": "my_vector",
+ "indexConfig": {
+ "index_type": "IVF_FLAT",
+ "nlist": "1024"
+ }
+ }
+ ]
+ }'
+
+ # Output
+ #
+ # {
+ # "code": 0,
+ # "data": {},
+ # }
+
+ $ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/get_load_state" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "collectionName": "customized_setup_2"
+ }'
+
+ # {
+ # "code": 0,
+ # "data": {
+ # "loadState": "LoadStateNotLoaded"
+ # }
+ # }
+ ```
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+## View Collections
+
+"
+# }
+```
+
+```java
+import io.milvus.v2.service.collection.request.LoadCollectionReq;
+
+// 6. Load the collection
+LoadCollectionReq loadCollectionReq = LoadCollectionReq.builder()
+ .collectionName("customized_setup_2")
+ .build();
+
+client.loadCollection(loadCollectionReq);
+
+// Thread.sleep(5000);
+
+// 7. Get load state of the collection
+GetLoadStateReq loadStateReq = GetLoadStateReq.builder()
+ .collectionName("customized_setup_2")
+ .build();
+
+res = client.getLoadState(loadStateReq);
+
+System.out.println(res);
+
+// Output:
+// true
+```
+
+```javascript
+// 7. Load the collection
+res = await client.loadCollection({
+ collection_name: "customized_setup_2"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+
+await sleep(3000)
+
+res = await client.getLoadState({
+ collection_name: "customized_setup_2"
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateLoaded
+//
+```
+
+```shell
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/load" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_2"
+}'
+
+# Output
+#
+# {
+# "code": 0,
+# "data": {},
+# }
+
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/get_load_state" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_2"
+}'
+
+# {
+# "code": 0,
+# "data": {
+# "loadProgress": 100,
+# "loadState": "LoadStateLoaded"
+# }
+# }
+```
+
+### Release a collection
+
+"
+# }
+```
+
+```java
+import io.milvus.v2.service.collection.request.ReleaseCollectionReq;
+
+// 8. Release the collection
+ReleaseCollectionReq releaseCollectionReq = ReleaseCollectionReq.builder()
+ .collectionName("customized_setup_2")
+ .build();
+
+client.releaseCollection(releaseCollectionReq);
+
+// Thread.sleep(1000);
+
+res = client.getLoadState(loadStateReq);
+
+System.out.println(res);
+
+// Output:
+// false
+```
+
+```javascript
+// 8. Release the collection
+res = await client.releaseCollection({
+ collection_name: "customized_setup_2"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+
+res = await client.getLoadState({
+ collection_name: "customized_setup_2"
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateNotLoad
+//
+```
+
+```shell
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/release" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_2"
+}'
+
+# Output
+#
+# {
+# "code": 0,
+# "data": {},
+# }
+
+
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/get_load_state" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_2"
+}'
+
+
+# {
+# "code": 0,
+# "data": {
+# "loadState": "LoadStateNotLoad"
+# }
+# }
+```
+
+## Set up aliases
+
+You can assign aliases for collections to make them more meaningful in a specific context. You can assign multiple aliases for a collection, but multiple collections cannot share an alias.
+
+### Create aliases
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+### List aliases
+
+ indexParams = new ArrayList<>();
+indexParams.add(indexParamForVectorField);
+
+CreateIndexReq createIndexReq = CreateIndexReq.builder()
+ .collectionName("test_scalar_index") // Specify the collection name
+ .indexParams(indexParams)
+ .build();
+
+client.createIndex(createIndexReq);
+```
+
+```javascript
+await client.createIndex({
+ collection_name: "test_scalar_index", // Specify the collection name
+ field_name: "scalar_1", // Name of the scalar field to be indexed
+ index_name: "default_index", // Name of the index to be created
+ index_type: "" // Type of index to be created. For auto indexing, leave it empty or omit this parameter.
+})
+```
+
+## Custom indexing
+
+ indexParams = new ArrayList<>();
+indexParams.add(indexParamForVectorField);
+
+CreateIndexReq createIndexReq = CreateIndexReq.builder()
+ .collectionName("test_scalar_index") // Specify the collection name
+ .indexParams(indexParams)
+ .build();
+
+client.createIndex(createIndexReq);
+```
+
+```javascript
+await client.createIndex({
+ collection_name: "test_scalar_index", // Specify the collection name
+ field_name: "scalar_1", // Name of the scalar field to be indexed
+ index_name: "inverted_index", // Name of the index to be created
+ index_type: "INVERTED" // Type of index to be created
+})
+```
+
+ indexNames = client.listIndexes(listIndexesReq);
+
+System.out.println(indexNames);
+
+// Output:
+// [
+// "default_index",
+// "inverted_index"
+// ]
+```
+
+```javascript
+res = await client.listIndexes({
+ collection_name: 'test_scalar_index'
+})
+
+console.log(res.indexes)
+
+// Output:
+// [
+// "default_index",
+// "inverted_index"
+// ]
+```
+
+## Limits
+
+- Currently, scalar indexing supports INT8, INT16, INT32, INT64, FLOAT, DOUBLE, BOOL, and VARCHAR data types, but not JSON and ARRAY types.
+
diff --git a/preview/site/en/userGuide/manage-indexes/index-vector-fields.md b/preview/site/en/userGuide/manage-indexes/index-vector-fields.md
new file mode 100644
index 000000000..df4b4782b
--- /dev/null
+++ b/preview/site/en/userGuide/manage-indexes/index-vector-fields.md
@@ -0,0 +1,616 @@
+---
+id: index-vector-fields.md
+order: 1
+summary: This guide walks you through the basic operations on creating and managing indexes on vector fields in a collection.
+title: Index Vector Fields
+---
+
+# Index Vector Fields
+
+This guide walks you through the basic operations on creating and managing indexes on vector fields in a collection.
+
+## Overview
+
+Leveraging the metadata stored in an index file, Milvus organizes your data in a specialized structure, facilitating rapid retrieval of requested information during searches or queries.
+
+Milvus provides several index types and metrics to sort field values for efficient similarity searches. The following table lists the supported index types and metrics for different vector field types. For details, refer to [In-memory Index](index.md) and [Similarity Metrics](metric.md).
+
+
+
+ indexParams = new ArrayList<>();
+indexParams.add(indexParamForVectorField);
+
+// 4.3 Crate an index file
+CreateIndexReq createIndexReq = CreateIndexReq.builder()
+ .collectionName("customized_setup")
+ .indexParams(indexParams)
+ .build();
+
+client.createIndex(createIndexReq);
+```
+
+```javascript
+// 4. Set up index for the collection
+// 4.1. Set up the index parameters
+res = await client.createIndex({
+ collection_name: "customized_setup",
+ field_name: "vector",
+ index_type: "AUTOINDEX",
+ metric_type: "COSINE",
+ index_name: "vector_index",
+ params: { "nlist": 128 }
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+```
+
+
+
+
+
+
+
+
+
+
+
+
+
+ indexNames = client.listIndexes(listIndexesReq);
+
+System.out.println(indexNames);
+
+// Output:
+// [
+// "vector_index"
+// ]
+
+// 5.2 Describe an index
+DescribeIndexReq describeIndexReq = DescribeIndexReq.builder()
+ .collectionName("customized_setup")
+ .indexName("vector_index")
+ .build();
+
+DescribeIndexResp describeIndexResp = client.describeIndex(describeIndexReq);
+
+System.out.println(JSONObject.toJSON(describeIndexResp));
+
+// Output:
+// {
+// "metricType": "COSINE",
+// "indexType": "AUTOINDEX",
+// "fieldName": "vector",
+// "indexName": "vector_index"
+// }
+```
+
+```javascript
+// 5. Describe the index
+res = await client.describeIndex({
+ collection_name: "customized_setup",
+ index_name: "vector_index"
+})
+
+console.log(JSON.stringify(res.index_descriptions, null, 2))
+
+// Output
+//
+// [
+// {
+// "params": [
+// {
+// "key": "index_type",
+// "value": "AUTOINDEX"
+// },
+// {
+// "key": "metric_type",
+// "value": "COSINE"
+// }
+// ],
+// "index_name": "vector_index",
+// "indexID": "449007919953063141",
+// "field_name": "vector",
+// "indexed_rows": "0",
+// "total_rows": "0",
+// "state": "Finished",
+// "index_state_fail_reason": "",
+// "pending_index_rows": "0"
+// }
+// ]
+//
+```
+
+You can check the index file created on a specific field, and collect the statistics on the number of rows indexed using this index file.
+
+## Drop an Index
+
+You can simply drop an index if it is no longer needed.
+
+ partitionNames = client.listPartitions(listPartitionsReq);
+
+System.out.println(partitionNames);
+
+// Output:
+// ["_default"]
+```
+
+```javascript
+// 3. List partitions
+res = await client.listPartitions({
+ collection_name: "quick_setup"
+})
+
+console.log(res.partition_names)
+
+// Output
+//
+// [ '_default' ]
+//
+```
+
+The output of the above code snippet includes the names of the partitions within the specified collection.
+
+"
+# }
+
+res = client.get_load_state(
+ collection_name="quick_setup",
+ partition_name="partitionA"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "state": ""
+# }
+
+res = client.get_load_state(
+ collection_name="quick_setup",
+ partition_name="partitionB"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "state": ""
+# }
+
+```
+
+```java
+import io.milvus.v2.service.collection.request.GetLoadStateReq;
+import io.milvus.v2.service.collection.request.ReleaseCollectionReq;
+import io.milvus.v2.service.partition.request.LoadPartitionsReq;
+import io.milvus.v2.service.partition.request.ReleasePartitionsReq;
+
+// 6. Load a partition independantly
+// 6.1 Release the collection
+ReleaseCollectionReq releaseCollectionReq = ReleaseCollectionReq.builder()
+ .collectionName("quick_setup")
+ .build();
+
+client.releaseCollection(releaseCollectionReq);
+
+// 6.2 Load partitionA
+LoadPartitionsReq loadPartitionsReq = LoadPartitionsReq.builder()
+ .collectionName("quick_setup")
+ .partitionNames(List.of("partitionA"))
+ .build();
+
+client.loadPartitions(loadPartitionsReq);
+
+Thread.sleep(3000);
+
+// 6.3 Check the load status of the collection and its partitions
+GetLoadStateReq getLoadStateReq = GetLoadStateReq.builder()
+ .collectionName("quick_setup")
+ .build();
+
+boolean state = client.getLoadState(getLoadStateReq);
+
+System.out.println(state);
+
+// Output:
+// true
+
+getLoadStateReq = GetLoadStateReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionA")
+ .build();
+
+state = client.getLoadState(getLoadStateReq);
+
+System.out.println(state);
+
+// Output:
+// true
+
+getLoadStateReq = GetLoadStateReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionB")
+ .build();
+
+state = client.getLoadState(getLoadStateReq);
+
+System.out.println(state);
+
+// Output:
+// false
+```
+
+```javascript
+// 6. Load a partition indenpendantly
+await client.releaseCollection({
+ collection_name: "quick_setup"
+})
+
+res = await client.getLoadState({
+ collection_name: "quick_setup"
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateNotLoad
+//
+
+await client.loadPartitions({
+ collection_name: "quick_setup",
+ partition_names: ["partitionA"]
+})
+
+await sleep(3000)
+
+res = await client.getLoadState({
+ collection_name: "quick_setup"
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateLoaded
+//
+
+res = await client.getLoadState({
+ collection_name: "quick_setup",
+ partition_name: "partitionA"
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateLoaded
+//
+
+res = await client.getLoadState({
+ collection_name: "quick_setup",
+ partition_name: "partitionB"
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateLoaded
+//
+```
+
+Possible load status may be either of the following
+
+- __Loaded__
+
+ A collection is marked as `Loaded` if at least one of its partitions has been loaded.
+
+- __NotLoad__
+
+ A collection is marked as `NotLoad` if none of its partitions has been loaded.
+
+- __Loading__
+
+ A collection is marked as Loading if at least one of its partitions is in the loading process.
+
+
+### Load Partitions
+
+"
+# }
+```
+
+```java
+LoadPartitionsReq loadPartitionsReq = LoadPartitionsReq.builder()
+ .collectionName("quick_setup")
+ .partitionNames(List.of("partitionA"))
+ .build();
+
+client.loadPartitions(loadPartitionsReq);
+
+getLoadStateReq = GetLoadStateReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionA")
+ .build();
+
+state = client.getLoadState(getLoadStateReq);
+
+System.out.println(state);
+
+// Output:
+// true
+```
+
+```javascript
+await client.loadPartitions({
+ collection_name: "quick_setup",
+ partition_names: ["partitionA"]
+})
+
+res = await client.getLoadState({
+ collection_name: "quick_setup",
+ partition_name: "partitionA"
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateLoaded
+//
+```
+
+To load multiple partitions at a time, do as follows:
+
+
+
+```python
+client.load_partitions(
+ collection_name="quick_setup",
+ partition_names=["partitionA", "partitionB"]
+)
+
+res = client.get_load_status(
+ collection_name="quick_setup",
+ partition_name="partitionA"
+)
+
+# Output
+#
+# {
+# "state": ""
+# }
+
+res = client.get_load_status(
+ collection_name="quick_setup",
+ partition_name="partitionB"
+)
+
+# Output
+#
+# {
+# "state": ""
+# }
+```
+
+```java
+LoadPartitionsReq loadPartitionsReq = LoadPartitionsReq.builder()
+ .collectionName("quick_setup")
+ .partitionNames(List.of("partitionA", "partitionB"))
+ .build();
+
+client.loadPartitions(loadPartitionsReq);
+
+getLoadStateReq = GetLoadStateReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionA")
+ .build();
+
+state = client.getLoadState(getLoadStateReq);
+
+System.out.println(state);
+
+// Output:
+// true
+
+getLoadStateReq = GetLoadStateReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionB")
+ .build();
+
+state = client.getLoadState(getLoadStateReq);
+
+System.out.println(state);
+
+// Output:
+// true
+```
+
+```javascript
+await client.loadPartitions({
+ collection_name: "quick_setup",
+ partition_names: ["partitionA", "partitionB"]
+})
+
+res = await client.getLoadState({
+ collection_name: "quick_setup",
+ partition_name: "partitionA"
+})
+
+console.log(res)
+
+// Output
+//
+// LoadStateLoaded
+//
+
+res = await client.getLoadState({
+ collection_name: "quick_setup",
+ partition_name: "partitionB"
+})
+
+console.log(res)
+
+// Output
+//
+// LoadStateLoaded
+//
+```
+
+### Release Partitions
+
+"
+# }
+
+```
+
+```java
+import io.milvus.v2.service.partition.request.ReleasePartitionsReq;
+
+// 7. Release a partition
+ReleasePartitionsReq releasePartitionsReq = ReleasePartitionsReq.builder()
+ .collectionName("quick_setup")
+ .partitionNames(List.of("partitionA"))
+ .build();
+
+client.releasePartitions(releasePartitionsReq);
+
+getLoadStateReq = GetLoadStateReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionA")
+ .build();
+
+state = client.getLoadState(getLoadStateReq);
+
+System.out.println(state);
+
+// Output:
+// false
+```
+
+```javascript
+// 7. Release a partition
+await client.releasePartitions({
+ collection_name: "quick_setup",
+ partition_names: ["partitionA"]
+})
+
+res = await client.getLoadState({
+ collection_name: "quick_setup"
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateNotLoad
+//
+```
+
+To release multiple partitions at a time, do as follows:
+
+```python
+client.release_partitions(
+ collection_name="quick_setup",
+ partition_names=["_default", "partitionA", "partitionB"]
+)
+
+res = client.get_load_status(
+ collection_name="quick_setup",
+)
+
+# Output
+#
+# {
+# "state": ""
+# }
+```
+
+## Drop Partitions
+
+Once you release a partition, you can drop it if it is no longer needed.
+
+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Limits
-
-The maximum number of connections is 65,536.
-
-## What's next
-
-Having connected to a Milvus server, you can:
-
-- [Create a collection](create_collection.md)
-- [Manage data](insert_data.md)
-- [Build a vector index](build_index.md)
-- [Conduct a vector search](search.md)
-- [Conduct a hybrid search](hybridsearch.md)
-
diff --git a/preview/site/en/userGuide/manage_data.md b/preview/site/en/userGuide/manage_data.md
deleted file mode 100644
index b5b91a5f0..000000000
--- a/preview/site/en/userGuide/manage_data.md
+++ /dev/null
@@ -1,14 +0,0 @@
----
-id: manage_data.md
-title: Manage Data
----
-
-# Manage Data
-
-Data in Milvus are called entities. An entity consists of a group of fields and is used to represent unstructured data like images, magazine articles, short videos, and more.
-
-- [Insert Data](insert_data.md): Prepare and insert entities into Milvus. Milvus 2.1 now supports VARCHAR data type on scalar field.
-
-- [Delete Data](delete_data.md): Milvus supports deleting entities by primary key filtered with boolean expression.
-
-- [Compact Data](compact_data.md): Milvus supports automatic data compaction by default. You can still compact data manually and check the compaction status.
diff --git a/preview/site/en/userGuide/manage_databases.md b/preview/site/en/userGuide/manage_databases.md
new file mode 100644
index 000000000..1360685ff
--- /dev/null
+++ b/preview/site/en/userGuide/manage_databases.md
@@ -0,0 +1,638 @@
+---
+id: manage_databases.md
+title: Manage Databases
+---
+
+# Manage Databases
+
+Similar to traditional database engines, you can also create databases in Milvus and allocate privileges to certain users to manage them. Then such users have the right to manage the collections in the databases. A Milvus cluster supports a maximum of 64 databases.
+
+ response = client.createDatabase(createDatabaseParam);
+```
+
+```javascript
+const address = "http://localhost:19530"
+
+// 1. Set up a Milvus Client
+client = new MilvusClient({address});
+
+// 3. Create a database
+res = await client.createDatabase({
+ db_name: "my_db"
+})
+
+console.log(res)
+
+// {
+// error_code: 'Success',
+// reason: '',
+// code: 0,
+// retriable: false,
+// detail: ''
+// }
+```
+
+The above code snippets connects to the default database and creates a new database named `my_database`.
+
+## Use a database
+
+A Milvus cluster ships with a default database, named 'default'. Collections are created in the default database unless otherwise specified.
+
+To change the default database, do as follows:
+
+
+
+```python
+db.using_database("book")
+```
+
+```java
+// No equivalent method is available.
+```
+
+```javascript
+// 4. Activate another database
+res = await client.useDatabase({
+ db_name: "my_db"
+})
+
+console.log(res)
+```
+
+You can also set a database to use upon connecting to your Milvus cluster as follows:
+
+
+
+```python
+conn = connections.connect(
+ host="127.0.0.1",
+ port="19530",
+ db_name="my_database"
+)
+```
+
+```java
+ConnectParam connectParam = ConnectParam.newBuilder()
+ .withDatabaseName("my_database")
+ .withUri(CLUSTER_ENDPOINT)
+ .withToken(TOKEN)
+ .build();
+
+MilvusServiceClient client = new MilvusServiceClient(connectParam);
+```
+
+```javascript
+const address = "http://localhost:19530";
+const db_name = "my_database";
+
+// 1. Set up a Milvus Client
+client = new MilvusClient({address, db_name});
+```
+
+## List databases
+
+ listDatabasesResponse = client.listDatabases();
+System.out.println(listDatabasesResponse.getData());
+
+// status {
+// }
+// db_names: "default"
+// db_names: "my_database"
+// created_timestamp: 1716794498117757990
+// created_timestamp: 1716797196479639477
+```
+
+```javascript
+res = await client.listDatabases()
+
+console.log(res.db_names)
+
+// [ 'default', 'my_db' ]
+```
+
+## Drop database
+
+To drop a database, you have to drop all its collections first. Otherwise, the drop fails.
+
+ response = client.createRole(CreateRoleParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .build());
+
+ if (response.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(response.getMessage());
+ }
+
+ response = client.grantRolePrivilege(GrantRolePriviledgeParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withObject("Collection")
+ .withObjectName("*")
+ .withPrivilege(PRIVILEGE_INSERT)
+ .build());
+
+ if (response.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(response.getMessage());
+ }
+
+ R grants = client.selectGrantForRole(SelectGrantForRoleParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .build());
+
+ if (grants.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(grants.getMessage());
+ }
+
+ System.out.println(grants.getData());
+
+ grants = client.selectGrantForRoleAndObject(SelectGrantForRoleAndObjectParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withObject("Collection")
+ .withObjectName("*")
+ .build());
+
+ if (grants.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(grants.getMessage());
+ }
+
+ System.out.println(grants.getData());
+
+ response = client.revokeRolePrivilege(RevokeRolePrivilegeParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withObject("Global")
+ .withObjectName("*")
+ .withPrivilege(PRIVILEGE_INSERT)
+ .build());
+
+ if (response.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(response.getMessage());
+ }
+
+ response = client.revokeRolePrivilege(RevokeRolePrivilegeParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withObject("Global")
+ .withObjectName("*")
+ .withPrivilege(PRIVILEGE_INSERT)
+ .build());
+
+ if (response.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(response.getMessage());
+ }
+ ```
+
+ ```javascript
+ const ROLE_NAME = "test_role";
+ const PRIVILEGE_INSERT = "Insert";
+
+ const client = connectToMilvus();
+
+ async function demo() {
+
+ }
+ await client.createRole({
+ roleName: ROLE_NAME
+ })
+
+ const grants = await client.listGrants({
+ roleName: ROLE_NAME
+ })
+
+ console.log(grants.grants);
+
+ await client.revokePrivilege({
+ roleName: ROLE_NAME,
+ object: "Global",
+ objectName: "*",
+ privilege: PRIVILEGE_INSERT
+ })
+ ```
+
+- If a Milvus connection specifies a `db_name`, but a Permission API call afterward does not, **database** refers to the database whose name was specified in the Milvus connection.
+
+
+
+ ```python
+ # NOTE: please make sure the 'foo' db has been created
+ connect_to_milvus(db_name="foo")
+
+ # This role will have the insert permission of all collections under foo db,
+ # excluding the insert permissions of collections under other dbs
+ role.grant("Collection", "*", _PRIVILEGE_INSERT)
+ print(role.list_grants())
+ print(role.list_grant("Collection", "*"))
+ role.revoke("Global", "*", _PRIVILEGE_INSERT)
+ ```
+
+ ```java
+ // NOTE: please make sure the 'foo' db has been created
+ MilvusServiceClient client = new ConnectToMilvus().withDbName("foo").build();
+
+ // This role will have the insert permission of all collections under foo db,
+ // excluding the insert permissions of collections under other dbs
+ R response = client.grantRolePrivilege(GrantRolePriviledgeParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withObject("Collection")
+ .withObjectName("*")
+ .withPrivilege(PRIVILEGE_INSERT)
+ .build());
+
+ if (response.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(response.getMessage());
+ }
+
+ R grants = client.selectGrantForRole(SelectGrantForRoleParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .build());
+
+ if (grants.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(grants.getMessage());
+ }
+
+ System.out.println(grants.getData());
+
+ grants = client.selectGrantForRoleAndObject(SelectGrantForRoleAndObjectParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withObject("Collection")
+ .withObjectName("*")
+ .build());
+
+ if (grants.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(grants.getMessage());
+ }
+
+ System.out.println(grants.getData());
+
+ response = client.revokeRolePrivilege(RevokeRolePrivilegeParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withObject("Global")
+ .withObjectName("*")
+ .withPrivilege(PRIVILEGE_INSERT)
+ .build());
+
+ if (response.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(response.getMessage());
+ }
+ ```
+
+ ```javascript
+ const client = connectToMilvus("foo");
+
+ async function demo() {
+
+ }
+ await client.createRole({
+ roleName: ROLE_NAME
+ })
+
+ const grants = await client.listGrants({
+ roleName: ROLE_NAME
+ })
+
+ console.log(grants.grants);
+
+ await client.revokePrivilege({
+ roleName: ROLE_NAME,
+ object: "Global",
+ objectName: "*",
+ privilege: PRIVILEGE_INSERT
+ })
+ ```
+
+- If a Permission API call is made upon a Milvus connection, with or without `db_name` specified, **database** refers to the database whose name was specified in the Permission API call.
+
+
+
+ ```python
+ # NOTE: please make sure the 'foo' db has been created
+
+ db_name = "foo"
+ connect_to_milvus()
+ role.grant("Collection", "*", _PRIVILEGE_INSERT, db_name=db_name)
+ print(role.list_grants(db_name=db_name))
+ print(role.list_grant("Collection", "*", db_name=db_name))
+ role.revoke("Global", "*", _PRIVILEGE_INSERT, db_name=db_name)
+ ```
+
+ ```java
+ // NOTE: please make sure the 'foo' db has been created
+
+ String dbName = "foo";
+ MilvusServiceClient client = new ConnectToMilvus().build();
+
+ R response = client.grantRolePrivilege(GrantRolePriviledgeParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withObject("Collection")
+ .withObjectName("*")
+ .withPrivilege(PRIVILEGE_INSERT)
+ .withDatabaseName(dbName)
+ .build());
+
+ if (response.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(response.getMessage());
+ }
+
+ R grants = client.selectGrantForRole(SelectGrantForRoleParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withDatabaseName(dbName)
+ .build());
+
+ if (grants.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(grants.getMessage());
+ }
+
+ System.out.println(grants.getData());
+
+ grants = client.selectGrantForRoleAndObject(SelectGrantForRoleAndObjectParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withObject("Collection")
+ .withObjectName("*")
+ .withDatabaseName(dbName)
+ .build());
+
+ if (grants.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(grants.getMessage());
+ }
+
+ System.out.println(grants.getData());
+
+ response = client.revokeRolePrivilege(RevokeRolePrivilegeParam.newBuilder()
+ .withRoleName(ROLE_NAME)
+ .withObject("Global")
+ .withObjectName("*")
+ .withPrivilege(PRIVILEGE_INSERT)
+ .withDatabaseName(dbName)
+ .build());
+
+ if (response.getStatus() != R.Status.Success.getCode()) {
+ throw new RuntimeException(response.getMessage());
+ }
+ ```
+
+ ```javascript
+ // The Node.js SDK currently cannot support this case.
+ ```
+
+## What's next
+
+- [Enable RBAC](rbac.md)
+
+- [Multi-tenancy](multi_tenancy.md)
diff --git a/preview/site/en/userGuide/manage_indexes.md b/preview/site/en/userGuide/manage_indexes.md
deleted file mode 100644
index ce535926d..000000000
--- a/preview/site/en/userGuide/manage_indexes.md
+++ /dev/null
@@ -1,12 +0,0 @@
----
-id: manage_indexes.md
-title: Manage Indexes
----
-
-# Manage Indexes
-
-Indexing is the process of efficiently organizing data, and it plays a major role in making similarity search useful by dramatically accelerating time-consuming queries on large datasets.
-
-- [Build an Index](build_index.md): Build an index to accelerate your vector similarity search. Milvus 2.1 now supports indexes on the scalar field.
-
-- [Drop an Index](drop_index.md): Dropping an index irreversibly removes all corresponding index files.
diff --git a/preview/site/en/userGuide/manage_partitions.md b/preview/site/en/userGuide/manage_partitions.md
deleted file mode 100644
index a0326d746..000000000
--- a/preview/site/en/userGuide/manage_partitions.md
+++ /dev/null
@@ -1,18 +0,0 @@
----
-id: manage_partitions.md
-title: Manage Partitions
----
-
-# Manage Partitions
-
-A partition is a division of a collection. Milvus supports dividing collection data into multiple parts on physical storage so that search and other operations can be limited to one partition to improve performance. Learn about the partition-level operations.
-
-- [Create a Partition](create_partition.md): Create a partition with the example of creating a novel partition within a book collection.
-
-- [Check Partition Information](check_partition.md): Check the basic information of a partition in Milvus including its name.
-
-- [Drop a Partition](drop_partition.md): Caution is needed as the delete operation irreversibly drops the partition and all data within it.
-
-- [Load a Partition](load_partition.md): Load the partition to memory before a search or a query instead of loading the whole collection can significantly reduce memory usage. Milvus 2.1 now supports loading a partition as multiple replicas.
-
-- [Release a Partition](release_partition.md): Release a partition from memory after a search or a query to reduce memory usage.
diff --git a/preview/site/en/userGuide/modify_collection.md b/preview/site/en/userGuide/modify_collection.md
deleted file mode 100644
index e970c8c23..000000000
--- a/preview/site/en/userGuide/modify_collection.md
+++ /dev/null
@@ -1,31 +0,0 @@
----
-id: modify_collection.md
-related_key: modify collection
-summary: Learn how to modify the properties of a collection in Milvus.
----
-
-# Modify a collection
-
-This topic describes how to modify the properties, especially the time to live (TTL), of a collection.
-
-Currently, the TTL feature is only available in Python.
-
-```
-collection.set_properties(properties={"collection.ttl.seconds": 1800})
-```
-
-The example above changes the collection TTL to 1800 seconds.
-
-| Parameter | Description | Option |
-| ---------------------------------- | ------------------------------------------------------------ | ----------------------------------- |
-| Properties: collection.ttl.seconds | Collection time to live (TTL) is the expiration time of data in a collection. Expired data in the collection will be cleaned up and will not be involved in searches or queries. Specify TTL in the unit of seconds. | The value should be 0 or greater. The default value is 0, which means TTL is disabled. |
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Create a partition](create_partition.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
diff --git a/preview/site/en/userGuide/release_collection.md b/preview/site/en/userGuide/release_collection.md
deleted file mode 100644
index 1bbc1631f..000000000
--- a/preview/site/en/userGuide/release_collection.md
+++ /dev/null
@@ -1,188 +0,0 @@
----
-id: release_collection.md
-related_key: release collection
-summary: Learn how to release a collection from memory in Milvus.
----
-
-# Release a Collection
-
-This topic describes how to release a collection from memory after a search or a query to reduce memory usage.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.release()
-```
-
-```javascript
-await milvusClient.collectionManager.releaseCollection({
- collection_name: "book",
-});
-```
-
-```go
-err := milvusClient.ReleaseCollection(
- context.Background(), // ctx
- "book", // CollectionName
-)
-if err != nil {
- log.Fatal("failed to release collection:", err.Error())
-}
-```
-
-```java
-milvusClient.releaseCollection(
- ReleaseCollectionParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-```
-
-```shell
-release -c book
-```
-
-``` curl
-curl -X 'DELETE' \
- 'http://localhost:9091/api/v1/collection/load' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book"
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Constraints
-
-- Releasing the collection that is successfully loaded is allowed.
-- Releasing the collection is allowed when its partition(s) are loaded.
-- Error will be returned at the attempt to release partition(s) when the parent collection is already loaded. Future releases will support releasing partitions from a loaded collection, and loading the collection when its partition(s) are released.
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Create a partition](create_partition.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
-
diff --git a/preview/site/en/userGuide/release_partition.md b/preview/site/en/userGuide/release_partition.md
deleted file mode 100644
index 83e0fbebc..000000000
--- a/preview/site/en/userGuide/release_partition.md
+++ /dev/null
@@ -1,199 +0,0 @@
----
-id: release_partition.md
-related_key: release partition
-summary: Learn how to release a partition into memory for search or query in Milvus.
----
-
-# Release a Partition
-
-This topic describes how to release a partition from memory after a search or a query to reduce memory usage.
-
-
-
-
-
-```python
-from pymilvus import Partition
-partition = Partition("novel") # Get an existing partition.
-partition.release()
-```
-
-```javascript
-await milvusClient.partitionManager.releasePartitions({
- collection_name: "book",
- partition_names: ["novel"],
- });
-```
-
-```go
-err := milvusClient.ReleasePartitions(
- context.Background(), // ctx
- "book", // CollectionName
- []string{"novel"} // partitionNames
-)
-if err != nil {
- log.Fatal("failed to release partitions:", err.Error())
-}
-```
-
-```java
-milvusClient.releasePartitions(
- ReleasePartitionsParam.newBuilder()
- .withCollectionName("book")
- .withPartitionNames(["novel"])
- .build()
-);
-```
-
-```shell
-release -c book -p novel
-```
-
-``` curl
-curl -X 'DELETE' \
- 'http://localhost:9091/api/v1/partitions/load' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "partition_names": ["novel"],
- "replica_number": 1
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Constraints
-
-- Error will be returned at the attempt to load partition(s) when the parent collection is already loaded. Future releases will support releasing partitions from a loaded collection, and (if needed) then loading some other partition(s).
-- "Load successfully" will be returned at the attempt to load the collection that is already loaded.
-- Error will be returned at the attempt to load the collection when the child partition(s) is/are already loaded. Future releases will support loading the collection when some of its partitions are already loaded.
-- Loading different partitions in a same collection via separate RPCs is not allowed.
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
diff --git a/preview/site/en/userGuide/search-query-get/get-and-scalar-query.md b/preview/site/en/userGuide/search-query-get/get-and-scalar-query.md
new file mode 100644
index 000000000..8a69ca489
--- /dev/null
+++ b/preview/site/en/userGuide/search-query-get/get-and-scalar-query.md
@@ -0,0 +1,1607 @@
+---
+id: get-and-scalar-query.md
+order: 3
+summary: This guide demonstrates how to get entities by ID and conduct scalar filtering.
+title: Get & Scalar Query
+---
+
+# Get & Scalar Query
+
+This guide demonstrates how to get entities by ID and conduct scalar filtering. A scalar filtering retrieves entities that match the specified filtering conditions.
+
+## Overview
+
+A scalar query filters entities in a collection based on a defined condition using boolean expressions. The query result is a set of entities that match the defined condition. Unlike a vector search, which identifies the closest vector to a given vector in a collection, queries filter entities based on specific criteria.
+
+In Milvus, __a filter is always a string compising field names joined by operators__. In this guide, you will find various filter examples. To learn more about the operator details, go to the [Reference](https://milvus.io/docs/get-and-scalar-query.md#Reference-on-scalar-filters) section.
+
+## Preparations
+
+The following steps repurpose the code to connect to Milvus, quickly set up a collection, and insert over 1,000 randomly generated entities into the collection.
+
+### Step 1: Create a collection
+
+ colors = Arrays.asList("green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey");
+List data = new ArrayList<>();
+
+for (int i=0; i<1000; i++) {
+ Random rand = new Random();
+ String current_color = colors.get(rand.nextInt(colors.size()-1));
+ int current_tag = rand.nextInt(8999) + 1000;
+ JSONObject row = new JSONObject();
+ row.put("id", Long.valueOf(i));
+ row.put("vector", Arrays.asList(rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat()));
+ row.put("color", current_color);
+ row.put("tag", current_tag);
+ row.put("color_tag", current_color + '_' + String.valueOf(rand.nextInt(8999) + 1000));
+ data.add(row);
+}
+
+InsertReq insertReq = InsertReq.builder()
+ .collectionName("quick_setup")
+ .data(data)
+ .build();
+
+InsertResp insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 1000}
+```
+
+```javascript
+// 3. Insert randomly generated vectors
+const colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
+var data = []
+
+for (let i = 0; i < 1000; i++) {
+ current_color = colors[Math.floor(Math.random() * colors.length)]
+ current_tag = Math.floor(Math.random() * 8999 + 1000)
+ data.push({
+ "id": i,
+ "vector": [Math.random(), Math.random(), Math.random(), Math.random(), Math.random()],
+ "color": current_color,
+ "tag": current_tag,
+ "color_tag": `${current_color}_${current_tag}`
+ })
+}
+
+console.log(data[0])
+
+// Output
+//
+// {
+// id: 0,
+// vector: [
+// 0.16022394821966035,
+// 0.6514875214491056,
+// 0.18294484964044666,
+// 0.30227694168725394,
+// 0.47553087493572255
+// ],
+// color: 'blue',
+// tag: 8907,
+// color_tag: 'blue_8907'
+// }
+//
+
+res = await client.insert({
+ collection_name: "quick_setup",
+ data: data
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 1000
+//
+```
+
+### Step 3: Create partitions and insert more entities
+
+
For example, like "prefix%" matches strings that begin with "prefix". | +| __in__ | Tests if an expression matches any value in a list of values. | + +### Advanced operators + +- `count(*)` + + Counts the exact number of entities in the collection. Use this as an output field to get the exact number of entities in a collection or partition. + + colors = Arrays.asList("green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey");
+List data = new ArrayList<>();
+
+for (int i=0; i<1000; i++) {
+ Random rand = new Random();
+ String current_color = colors.get(rand.nextInt(colors.size()-1));
+ JSONObject row = new JSONObject();
+ row.put("id", Long.valueOf(i));
+ row.put("vector", Arrays.asList(rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat()));
+ row.put("color_tag", current_color + "_" + String.valueOf(rand.nextInt(8999) + 1000));
+ data.add(row);
+}
+
+InsertReq insertReq = InsertReq.builder()
+ .collectionName("quick_setup")
+ .data(data)
+ .build();
+
+InsertResp insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 1000}
+
+// 6.1. Create a partition
+CreatePartitionReq partitionReq = CreatePartitionReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("red")
+ .build();
+
+client.createPartition(partitionReq);
+
+partitionReq = CreatePartitionReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("blue")
+ .build();
+
+client.createPartition(partitionReq);
+
+// 6.2 Insert data into the partition
+data = new ArrayList<>();
+
+for (int i=1000; i<1500; i++) {
+ Random rand = new Random();
+ String current_color = "red";
+ JSONObject row = new JSONObject();
+ row.put("id", Long.valueOf(i));
+ row.put("vector", Arrays.asList(rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat()));
+ row.put("color", current_color);
+ row.put("color_tag", current_color + "_" + String.valueOf(rand.nextInt(8999) + 1000));
+ data.add(row);
+}
+
+insertReq = InsertReq.builder()
+ .collectionName("quick_setup")
+ .data(data)
+ .partitionName("red")
+ .build();
+
+insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 500}
+
+data = new ArrayList<>();
+
+for (int i=1500; i<2000; i++) {
+ Random rand = new Random();
+ String current_color = "blue";
+ JSONObject row = new JSONObject();
+ row.put("id", Long.valueOf(i));
+ row.put("vector", Arrays.asList(rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat()));
+ row.put("color", current_color);
+ row.put("color_tag", current_color + "_" + String.valueOf(rand.nextInt(8999) + 1000));
+ data.add(row);
+}
+
+insertReq = InsertReq.builder()
+ .collectionName("quick_setup")
+ .data(data)
+ .partitionName("blue")
+ .build();
+
+insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 500}
+```
+
+```javascript
+const { MilvusClient, DataType, sleep } = require("@zilliz/milvus2-sdk-node")
+
+const address = "http://localhost:19530"
+
+// 1. Set up a Milvus Client
+client = new MilvusClient({address});
+
+// 2. Create a collection in quick setup mode
+await client.createCollection({
+ collection_name: "quick_setup",
+ dimension: 5,
+ metric_type: "IP"
+});
+
+// 3. Insert randomly generated vectors
+const colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
+data = []
+
+for (let i = 0; i < 1000; i++) {
+ current_color = colors[Math.floor(Math.random() * colors.length)]
+ data.push({
+ id: i,
+ vector: [Math.random(), Math.random(), Math.random(), Math.random(), Math.random()],
+ color: current_color,
+ color_tag: `${current_color}_${Math.floor(Math.random() * 8999) + 1000}`
+ })
+}
+
+var res = await client.insert({
+ collection_name: "quick_setup",
+ data: data
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 1000
+//
+
+await client.createPartition({
+ collection_name: "quick_setup",
+ partition_name: "red"
+})
+
+await client.createPartition({
+ collection_name: "quick_setup",
+ partition_name: "blue"
+})
+
+// 6.1 Insert data into partitions
+var red_data = []
+var blue_data = []
+
+for (let i = 1000; i < 1500; i++) {
+ red_data.push({
+ id: i,
+ vector: [Math.random(), Math.random(), Math.random(), Math.random(), Math.random()],
+ color: "red",
+ color_tag: `red_${Math.floor(Math.random() * 8999) + 1000}`
+ })
+}
+
+for (let i = 1500; i < 2000; i++) {
+ blue_data.push({
+ id: i,
+ vector: [Math.random(), Math.random(), Math.random(), Math.random(), Math.random()],
+ color: "blue",
+ color_tag: `blue_${Math.floor(Math.random() * 8999) + 1000}`
+ })
+}
+
+res = await client.insert({
+ collection_name: "quick_setup",
+ data: red_data,
+ partition_name: "red"
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 500
+//
+
+res = await client.insert({
+ collection_name: "quick_setup",
+ data: blue_data,
+ partition_name: "blue"
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 500
+//
+```
+
+## Basic search
+
+When sending a `search` request, you can provide one or more vector values representing your query embeddings and a `limit` value indicating the number of results to return.
+
+Depending on your data and your query vector, you may get fewer than `limit` results. This happens when `limit` is larger than the number of possible matching vectors for your query.
+
+### Single-vector search
+
+Single-vector search is the simplest form of `search` operations in Milvus, designed to find the most similar vectors to a given query vector.
+
+To perform a single-vector search, specify the target collection name, the query vector, and the desired number of results (`limit`). This operation returns a result set comprising the most similar vectors, their IDs, and distances from the query vector.
+
+Here is an example of searching for the top 5 entities that are most similar to the query vector:
+
+
+
+```python
+# Single vector search
+res = client.search(
+ collection_name="test_collection", # Replace with the actual name of your collection
+ # Replace with your query vector
+ data=[[0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]],
+ limit=5, # Max. number of search results to return
+ search_params={"metric_type": "IP", "params": {}} # Search parameters
+)
+
+# Convert the output to a formatted JSON string
+result = json.dumps(res, indent=4)
+print(result)
+```
+
+```java
+// 4. Single vector search
+List> query_vectors = Arrays.asList(Arrays.asList(0.3580376395471989f, -0.6023495712049978f, 0.18414012509913835f, -0.26286205330961354f, 0.9029438446296592f));
+
+SearchReq searchReq = SearchReq.builder()
+ .collectionName("quick_setup")
+ .data(query_vectors)
+ .topK(3) // The number of results to return
+ .build();
+
+SearchResp searchResp = client.search(searchReq);
+
+System.out.println(JSONObject.toJSON(searchResp));
+```
+
+```javascript
+// 4. Single vector search
+var query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592],
+
+res = await client.search({
+ collection_name: "quick_setup",
+ data: [query_vector],
+ limit: 3, // The number of results to return
+})
+
+console.log(res.results)
+```
+
+
+
+
+
+
+
+
+
+
+
+
+
+The output is similar to the following:
+
+
+
+```python
+[
+ [
+ {
+ "id": 0,
+ "distance": 1.4093276262283325,
+ "entity": {}
+ },
+ {
+ "id": 4,
+ "distance": 0.9902134537696838,
+ "entity": {}
+ },
+ {
+ "id": 1,
+ "distance": 0.8519943356513977,
+ "entity": {}
+ },
+ {
+ "id": 5,
+ "distance": 0.7972343564033508,
+ "entity": {}
+ },
+ {
+ "id": 2,
+ "distance": 0.5928734540939331,
+ "entity": {}
+ }
+ ]
+]
+```
+
+```java
+{"searchResults": [[
+ {
+ "score": 1.263043,
+ "fields": {
+ "vector": [
+ 0.9533119,
+ 0.02538395,
+ 0.76714665,
+ 0.35481733,
+ 0.9845762
+ ],
+ "id": 740
+ }
+ },
+ {
+ "score": 1.2377806,
+ "fields": {
+ "vector": [
+ 0.7411156,
+ 0.08687937,
+ 0.8254139,
+ 0.08370924,
+ 0.99095553
+ ],
+ "id": 640
+ }
+ },
+ {
+ "score": 1.1869997,
+ "fields": {
+ "vector": [
+ 0.87928146,
+ 0.05324632,
+ 0.6312755,
+ 0.28005534,
+ 0.9542448
+ ],
+ "id": 455
+ }
+ }
+]]}
+```
+
+```javascript
+[
+ { score: 1.7463608980178833, id: '854' },
+ { score: 1.744946002960205, id: '425' },
+ { score: 1.7258622646331787, id: '718' }
+]
+```
+
+The output showcases the top 5 neighbors nearest to your query vector, including their unique IDs and the calculated distances.
+
+### Bulk-vector search
+
+A bulk-vector search extends the [single-vector search](https://milvus.io/docs/single-vector-search.md#Single-Vector-Search) concept by allowing multiple query vectors to be searched in a single request. This type of search is ideal for scenarios where you need to find similar vectors for a set of query vectors, significantly reducing the time and computational resources required.
+
+In a bulk-vector search, you can include several query vectors in the `data` field. The system processes these vectors in parallel, returning a separate result set for each query vector, each set containing the closest matches found within the collection.
+
+Here is an example of searching for two distinct sets of the most similar entities from two query vectors:
+
+
+
+```python
+# Bulk-vector search
+res = client.search(
+ collection_name="test_collection", # Replace with the actual name of your collection
+ data=[
+ [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, 0.2614474506242501, 0.838729485096104],
+ [0.3172005263489739, 0.9719044792798428, -0.36981146090600725, -0.4860894583077995, 0.95791889146345]
+ ], # Replace with your query vectors
+ limit=2, # Max. number of search results to return
+ search_params={"metric_type": "IP", "params": {}} # Search parameters
+)
+
+result = json.dumps(res, indent=4)
+print(result)
+```
+
+```java
+// 5. Batch vector search
+query_vectors = Arrays.asList(
+ Arrays.asList(0.3580376395471989f, -0.6023495712049978f, 0.18414012509913835f, -0.26286205330961354f, 0.9029438446296592f),
+ Arrays.asList(0.19886812562848388f, 0.06023560599112088f, 0.6976963061752597f, 0.2614474506242501f, 0.838729485096104f)
+);
+
+searchReq = SearchReq.builder()
+ .collectionName("quick_setup")
+ .data(query_vectors)
+ .topK(2)
+ .build();
+
+searchResp = client.search(searchReq);
+
+System.out.println(JSONObject.toJSON(searchResp));
+```
+
+```javascript
+// 5. Batch vector search
+var query_vectors = [
+ [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592],
+ [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, 0.2614474506242501, 0.838729485096104]
+]
+
+res = await client.search({
+ collection_name: "quick_setup",
+ data: query_vectors,
+ limit: 2,
+})
+
+console.log(res.results)
+```
+
+The output is similar to the following:
+
+
+
+```python
+[
+ [
+ {
+ "id": 1,
+ "distance": 1.3017789125442505,
+ "entity": {}
+ },
+ {
+ "id": 7,
+ "distance": 1.2419954538345337,
+ "entity": {}
+ }
+ ], # Result set 1
+ [
+ {
+ "id": 3,
+ "distance": 2.3358664512634277,
+ "entity": {}
+ },
+ {
+ "id": 8,
+ "distance": 0.5642921924591064,
+ "entity": {}
+ }
+ ] # Result set 2
+]
+```
+
+```java
+// Two sets of vectors are returned as expected
+
+{"searchResults": [
+ [
+ {
+ "score": 1.263043,
+ "fields": {
+ "vector": [
+ 0.9533119,
+ 0.02538395,
+ 0.76714665,
+ 0.35481733,
+ 0.9845762
+ ],
+ "id": 740
+ }
+ },
+ {
+ "score": 1.2377806,
+ "fields": {
+ "vector": [
+ 0.7411156,
+ 0.08687937,
+ 0.8254139,
+ 0.08370924,
+ 0.99095553
+ ],
+ "id": 640
+ }
+ }
+ ],
+ [
+ {
+ "score": 1.8654699,
+ "fields": {
+ "vector": [
+ 0.4671427,
+ 0.8378432,
+ 0.98844475,
+ 0.82763994,
+ 0.9729997
+ ],
+ "id": 638
+ }
+ },
+ {
+ "score": 1.8581753,
+ "fields": {
+ "vector": [
+ 0.735541,
+ 0.60140246,
+ 0.86730254,
+ 0.93152493,
+ 0.98603314
+ ],
+ "id": 855
+ }
+ }
+ ]
+]}
+```
+
+```javascript
+[
+ [
+ { score: 2.3590476512908936, id: '854' },
+ { score: 2.2896690368652344, id: '59' }
+ [
+ { score: 2.664059638977051, id: '59' },
+ { score: 2.59483003616333, id: '854' }
+ ]
+]
+```
+
+The results include two sets of nearest neighbors, one for each query vector, showcasing the efficiency of bulk-vector searches in handling multiple query vectors at once.
+
+### Partition search
+
+Partition search narrows the scope of your search to a specific subset or partition of your collection. This is particularly useful for organized datasets where data is segmented into logical or categorical divisions, allowing for faster search operations by reducing the volume of data to scan.
+
+To conduct a partition search, simply include the name of the target partition in `partition_names` of your search request. This specifies that the `search` operation only considers vectors within the specified partition.
+
+Here is an example of searching for entities in `red`:
+
+
+
+```python
+# 6.2 Search within a partition
+query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]
+
+res = client.search(
+ collection_name="quick_setup",
+ data=[query_vector],
+ limit=5,
+ search_params={"metric_type": "IP", "params": {"level": 1}},
+ partition_names=["red"]
+)
+
+print(res)
+```
+
+```java
+// 6.3 Search within partitions
+query_vectors = Arrays.asList(Arrays.asList(0.3580376395471989f, -0.6023495712049978f, 0.18414012509913835f, -0.26286205330961354f, 0.9029438446296592f));
+
+searchReq = SearchReq.builder()
+ .collectionName("quick_setup")
+ .data(query_vectors)
+ .partitionNames(Arrays.asList("red"))
+ .topK(5)
+ .build();
+
+searchResp = client.search(searchReq);
+
+System.out.println(JSONObject.toJSON(searchResp));
+```
+
+```javascript
+// 6.2 Search within partitions
+query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]
+
+res = await client.search({
+ collection_name: "quick_setup",
+ data: [query_vector],
+ partition_names: ["red"],
+ limit: 5,
+})
+
+console.log(res.results)
+```
+
+The output is similar to the following:
+
+
+
+```python
+[
+ [
+ {
+ "id": 16,
+ "distance": 0.9200337529182434,
+ "entity": {}
+ },
+ {
+ "id": 14,
+ "distance": 0.4505271911621094,
+ "entity": {}
+ },
+ {
+ "id": 15,
+ "distance": 0.19924677908420563,
+ "entity": {}
+ },
+ {
+ "id": 17,
+ "distance": 0.0075093843042850494,
+ "entity": {}
+ },
+ {
+ "id": 13,
+ "distance": -0.14609718322753906,
+ "entity": {}
+ }
+ ]
+]
+```
+
+```java
+{"searchResults": [
+ [
+ {
+ "score": 1.1677284,
+ "fields": {
+ "vector": [
+ 0.9986977,
+ 0.17964739,
+ 0.49086612,
+ 0.23155272,
+ 0.98438674
+ ],
+ "id": 1435
+ }
+ },
+ {
+ "score": 1.1476475,
+ "fields": {
+ "vector": [
+ 0.6952647,
+ 0.13417172,
+ 0.91045254,
+ 0.119336545,
+ 0.9338931
+ ],
+ "id": 1291
+ }
+ },
+ {
+ "score": 1.0969629,
+ "fields": {
+ "vector": [
+ 0.3363194,
+ 0.028906643,
+ 0.6675426,
+ 0.030419827,
+ 0.9735209
+ ],
+ "id": 1168
+ }
+ },
+ {
+ "score": 1.0741848,
+ "fields": {
+ "vector": [
+ 0.9980543,
+ 0.36063594,
+ 0.66427994,
+ 0.17359233,
+ 0.94954175
+ ],
+ "id": 1164
+ }
+ },
+ {
+ "score": 1.0584627,
+ "fields": {
+ "vector": [
+ 0.7187005,
+ 0.12674773,
+ 0.987718,
+ 0.3110777,
+ 0.86093885
+ ],
+ "id": 1085
+ }
+ }
+ ],
+ [
+ {
+ "score": 1.8030131,
+ "fields": {
+ "vector": [
+ 0.59726167,
+ 0.7054632,
+ 0.9573117,
+ 0.94529945,
+ 0.8664103
+ ],
+ "id": 1203
+ }
+ },
+ {
+ "score": 1.7728865,
+ "fields": {
+ "vector": [
+ 0.6672442,
+ 0.60448086,
+ 0.9325822,
+ 0.80272985,
+ 0.8861626
+ ],
+ "id": 1448
+ }
+ },
+ {
+ "score": 1.7536311,
+ "fields": {
+ "vector": [
+ 0.59663296,
+ 0.77831805,
+ 0.8578314,
+ 0.88818026,
+ 0.9030075
+ ],
+ "id": 1010
+ }
+ },
+ {
+ "score": 1.7520742,
+ "fields": {
+ "vector": [
+ 0.854198,
+ 0.72294194,
+ 0.9245805,
+ 0.86126596,
+ 0.7969224
+ ],
+ "id": 1219
+ }
+ },
+ {
+ "score": 1.7452049,
+ "fields": {
+ "vector": [
+ 0.96419,
+ 0.943535,
+ 0.87611496,
+ 0.8268136,
+ 0.79786557
+ ],
+ "id": 1149
+ }
+ }
+ ]
+]}
+```
+
+```javascript
+[
+ { score: 3.0258803367614746, id: '1201' },
+ { score: 3.004319190979004, id: '1458' },
+ { score: 2.880324363708496, id: '1187' },
+ { score: 2.8246407508850098, id: '1347' },
+ { score: 2.797295093536377, id: '1406' }
+]
+```
+
+Then, search for entities in `blue`:
+
+
+
+```python
+res = client.search(
+ collection_name="quick_setup",
+ data=[query_vector],
+ limit=5,
+ search_params={"metric_type": "IP", "params": {"level": 1}},
+ partition_names=["blue"]
+)
+
+print(res)
+```
+
+```java
+searchReq = SearchReq.builder()
+ .collectionName("quick_setup")
+ .data(query_vectors)
+ .partitionNames(Arrays.asList("blue"))
+ .topK(5)
+ .build();
+
+searchResp = client.search(searchReq);
+
+System.out.println(JSONObject.toJSON(searchResp));
+```
+
+```javascript
+res = await client.search({
+ collection_name: "quick_setup",
+ data: [query_vector],
+ partition_names: ["blue"],
+ limit: 5,
+})
+
+console.log(res.results)
+```
+
+The output is similar to the following:
+
+
+
+```python
+[
+ [
+ {
+ "id": 20,
+ "distance": 2.363696813583374,
+ "entity": {}
+ },
+ {
+ "id": 26,
+ "distance": 1.0665391683578491,
+ "entity": {}
+ },
+ {
+ "id": 23,
+ "distance": 1.066049575805664,
+ "entity": {}
+ },
+ {
+ "id": 29,
+ "distance": 0.8353596925735474,
+ "entity": {}
+ },
+ {
+ "id": 28,
+ "distance": 0.7484277486801147,
+ "entity": {}
+ }
+ ]
+]
+```
+
+```java
+{"searchResults": [
+ [
+ {
+ "score": 1.1628494,
+ "fields": {
+ "vector": [
+ 0.7442872,
+ 0.046407282,
+ 0.71031404,
+ 0.3544345,
+ 0.9819991
+ ],
+ "id": 1992
+ }
+ },
+ {
+ "score": 1.1470042,
+ "fields": {
+ "vector": [
+ 0.5505825,
+ 0.04367262,
+ 0.9985836,
+ 0.18922359,
+ 0.93255126
+ ],
+ "id": 1977
+ }
+ },
+ {
+ "score": 1.1450152,
+ "fields": {
+ "vector": [
+ 0.89994013,
+ 0.052991092,
+ 0.8645576,
+ 0.6406729,
+ 0.95679337
+ ],
+ "id": 1573
+ }
+ },
+ {
+ "score": 1.1439825,
+ "fields": {
+ "vector": [
+ 0.9253267,
+ 0.15890503,
+ 0.7999555,
+ 0.19126713,
+ 0.898583
+ ],
+ "id": 1552
+ }
+ },
+ {
+ "score": 1.1029172,
+ "fields": {
+ "vector": [
+ 0.95661926,
+ 0.18777144,
+ 0.38115507,
+ 0.14323527,
+ 0.93137646
+ ],
+ "id": 1823
+ }
+ }
+ ],
+ [
+ {
+ "score": 1.8005109,
+ "fields": {
+ "vector": [
+ 0.5953582,
+ 0.7794224,
+ 0.9388869,
+ 0.79825854,
+ 0.9197286
+ ],
+ "id": 1888
+ }
+ },
+ {
+ "score": 1.7714822,
+ "fields": {
+ "vector": [
+ 0.56805456,
+ 0.89422905,
+ 0.88187534,
+ 0.914824,
+ 0.8944365
+ ],
+ "id": 1648
+ }
+ },
+ {
+ "score": 1.7561421,
+ "fields": {
+ "vector": [
+ 0.83421993,
+ 0.39865613,
+ 0.92319834,
+ 0.42695504,
+ 0.96633124
+ ],
+ "id": 1688
+ }
+ },
+ {
+ "score": 1.7553532,
+ "fields": {
+ "vector": [
+ 0.89994013,
+ 0.052991092,
+ 0.8645576,
+ 0.6406729,
+ 0.95679337
+ ],
+ "id": 1573
+ }
+ },
+ {
+ "score": 1.7543385,
+ "fields": {
+ "vector": [
+ 0.16542226,
+ 0.38248396,
+ 0.9888778,
+ 0.80913955,
+ 0.9501492
+ ],
+ "id": 1544
+ }
+ }
+ ]
+]}
+```
+
+```javascript
+[
+ { score: 2.8421106338500977, id: '1745' },
+ { score: 2.838560104370117, id: '1782' },
+ { score: 2.8134000301361084, id: '1511' },
+ { score: 2.718268871307373, id: '1679' },
+ { score: 2.7014894485473633, id: '1597' }
+]
+```
+
+The data in `red` differs from that in `blue`. Therefore, the search results will be constrained to the specified partition, reflecting the unique characteristics and data distribution of that subset.
+
+### Search with output fields
+
+Search with output fields allows you to specify which attributes or fields of the matched vectors should be included in the search results.
+
+You can specify `output_fields` in a request to return results with specific fields.
+
+Here is an example of returning results with `color` attribute values:
+
+
+
+```python
+# Search with output fields
+res = client.search(
+ collection_name="test_collection", # Replace with the actual name of your collection
+ data=[[0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]],
+ limit=5, # Max. number of search results to return
+ search_params={"metric_type": "IP", "params": {}}, # Search parameters
+ output_fields=["color"] # Output fields to return
+)
+
+result = json.dumps(res, indent=4)
+print(result)
+```
+
+```java
+// 7. Search with output fields
+query_vectors = Arrays.asList(Arrays.asList(0.3580376395471989f, -0.6023495712049978f, 0.18414012509913835f, -0.26286205330961354f, 0.9029438446296592f));
+
+searchReq = SearchReq.builder()
+ .collectionName("quick_setup")
+ .data(query_vectors)
+ .outputFields(Arrays.asList("color"))
+ .topK(5)
+ .build();
+
+searchResp = client.search(searchReq);
+
+System.out.println(JSONObject.toJSON(searchResp));
+```
+
+```javascript
+// 7. Search with output fields
+query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]
+
+res = await client.search({
+ collection_name: "quick_setup",
+ data: [query_vector],
+ limit: 5,
+ output_fields: ["color"],
+})
+
+console.log(res.results)
+```
+
+The output is similar to the following:
+
+
+
+```python
+[
+ [
+ {
+ "id": 0,
+ "distance": 1.4093276262283325,
+ "entity": {
+ "color": "pink_8682"
+ }
+ },
+ {
+ "id": 16,
+ "distance": 1.0159327983856201,
+ "entity": {
+ "color": "yellow_1496"
+ }
+ },
+ {
+ "id": 4,
+ "distance": 0.9902134537696838,
+ "entity": {
+ "color": "red_4794"
+ }
+ },
+ {
+ "id": 14,
+ "distance": 0.9803846478462219,
+ "entity": {
+ "color": "green_2899"
+ }
+ },
+ {
+ "id": 1,
+ "distance": 0.8519943356513977,
+ "entity": {
+ "color": "red_7025"
+ }
+ }
+ ]
+]
+```
+
+```java
+{"searchResults": [
+ [
+ {
+ "score": 1.263043,
+ "fields": {}
+ },
+ {
+ "score": 1.2377806,
+ "fields": {}
+ },
+ {
+ "score": 1.1869997,
+ "fields": {}
+ },
+ {
+ "score": 1.1748955,
+ "fields": {}
+ },
+ {
+ "score": 1.1720343,
+ "fields": {}
+ }
+ ]
+]}
+```
+
+```javascript
+
+[
+ { score: 3.036271572113037, id: '59', color: 'orange' },
+ { score: 3.0267879962921143, id: '1745', color: 'blue' },
+ { score: 3.0069446563720703, id: '854', color: 'black' },
+ { score: 2.984386682510376, id: '718', color: 'black' },
+ { score: 2.916019916534424, id: '425', color: 'purple' }
+]
+```
+
+Alongside the nearest neighbors, the search results will include the specified field `color`, providing a richer set of information for each matching vector.
+
+## Filtered search
+
+Filtered search applies scalar filters to vector searches, allowing you to refine the search results based on specific criteria. You can find more about filter expressions in [Boolean Expression Rules](https://milvus.io/docs/boolean.md) and examples in [Get & Scalar Query](https://milvus.io/docs/get-and-scalar-query.md).
+
+### Use the `like` operator
+
+The `like` operator enhances string searches by evaluating patterns including prefixes, infixes, and suffixes:
+
+- __Prefix matching__: To find values starting with a specific prefix, use the syntax `'like "prefix%"'`.
+- __Infix matching__: To find values containing a specific sequence of characters anywhere within the string, use the syntax `'like "%infix%"'`.
+- __Suffix matching__: To find values ending with a specific suffix, use the syntax `'like "%suffix"'`.
+
+For single-character matching, underscore (`_`) acts as a wildcard for one character, e.g., `'like "y_llow"'`.
+
+### Special characters in search strings
+
+If you want to search for a string that contains special characters like underscores (`_`) or percent signs (`%`), which are normally used as wildcards in search patterns (`_` for any single character and `%` for any sequence of characters), you must escape these characters to treat them as literal characters. Use a backslash (`\`) to escape special characters, and remember to escape the backslash itself. For instance:
+
+- To search for a literal underscore, use `\\_`.
+- To search for a literal percent sign, use `\\%`.
+
+So, if you need to search for the text `"_version_"`, your query should be formatted as `'like "\\_version\\_"'` to ensure the underscores are treated as part of the search term and not as wildcards.
+
+Filter results whose __color__ is prefixed with __red__:
+
+
+
+```python
+# Search with filter
+res = client.search(
+ collection_name="test_collection", # Replace with the actual name of your collection
+ data=[[0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]],
+ limit=5, # Max. number of search results to return
+ search_params={"metric_type": "IP", "params": {}}, # Search parameters
+ output_fields=["color"], # Output fields to return
+ filter='color like "red%"'
+)
+
+result = json.dumps(res, indent=4)
+print(result)
+```
+
+```java
+// 8. Filtered search
+query_vectors = Arrays.asList(Arrays.asList(0.3580376395471989f, -0.6023495712049978f, 0.18414012509913835f, -0.26286205330961354f, 0.9029438446296592f));
+
+searchReq = SearchReq.builder()
+ .collectionName("quick_setup")
+ .data(query_vectors)
+ .outputFields(Arrays.asList("color_tag"))
+ .filter("color_tag like \"red%\"")
+ .topK(5)
+ .build();
+
+searchResp = client.search(searchReq);
+
+System.out.println(JSONObject.toJSON(searchResp));
+```
+
+```javascript
+// 8. Filtered search
+// 8.1 Filter with "like" operator and prefix wildcard
+query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]
+
+res = await client.search({
+ collection_name: "quick_setup",
+ data: [query_vector],
+ limit: 5,
+ filters: "color_tag like \"red%\"",
+ output_fields: ["color_tag"]
+})
+
+console.log(res.results)
+```
+
+The output is similar to the following:
+
+
+
+```python
+[
+ [
+ {
+ "id": 4,
+ "distance": 0.9902134537696838,
+ "entity": {
+ "color": "red_4794"
+ }
+ },
+ {
+ "id": 1,
+ "distance": 0.8519943356513977,
+ "entity": {
+ "color": "red_7025"
+ }
+ },
+ {
+ "id": 6,
+ "distance": -0.4113418459892273,
+ "entity": {
+ "color": "red_9392"
+ }
+ }
+ ]
+]
+```
+
+```java
+{"searchResults": [
+ [
+ {
+ "score": 1.1869997,
+ "fields": {"color_tag": "red_3026"}
+ },
+ {
+ "score": 1.1677284,
+ "fields": {"color_tag": "red_9030"}
+ },
+ {
+ "score": 1.1476475,
+ "fields": {"color_tag": "red_3744"}
+ },
+ {
+ "score": 1.0969629,
+ "fields": {"color_tag": "red_4168"}
+ },
+ {
+ "score": 1.0741848,
+ "fields": {"color_tag": "red_9678"}
+ }
+ ]
+]}
+```
+
+```javascript
+[
+ { score: 2.5080761909484863, id: '1201', color_tag: 'red_8904' },
+ { score: 2.491129159927368, id: '425', color_tag: 'purple_8212' },
+ { score: 2.4889798164367676, id: '1458', color_tag: 'red_6891' },
+ { score: 2.42964243888855, id: '724', color_tag: 'black_9885' },
+ { score: 2.4004223346710205, id: '854', color_tag: 'black_5990' }
+]
+```
+
+Filter results whose __color__ contains the letters __ll__ anywhere within the string:
+
+
+
+```python
+# Infix match on color field
+res = client.search(
+ collection_name="test_collection", # Replace with the actual name of your collection
+ data=[[0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]],
+ limit=5, # Max. number of search results to return
+ search_params={"metric_type": "IP", "params": {}}, # Search parameters
+ output_fields=["color"], # Output fields to return
+ filter='color like "%ll%"' # Filter on color field, infix match on "ll"
+)
+
+result = json.dumps(res, indent=4)
+print(result)
+```
+
+```java
+// 8. Filtered search
+query_vectors = Arrays.asList(Arrays.asList(0.3580376395471989f, -0.6023495712049978f, 0.18414012509913835f, -0.26286205330961354f, 0.9029438446296592f));
+
+searchReq = SearchReq.builder()
+ .collectionName("quick_setup")
+ .data(query_vectors)
+ .outputFields(Arrays.asList("color_tag"))
+ .filter("color like \"%ll%\"")
+ .topK(5)
+ .build();
+
+searchResp = client.search(searchReq);
+
+System.out.println(JSONObject.toJSON(searchResp));
+```
+
+```javascript
+// 8. Filtered search
+// 8.1 Filter with "like" operator and prefix wildcard
+query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]
+
+res = await client.search({
+ collection_name: "quick_setup",
+ data: [query_vector],
+ limit: 5,
+ filters: "color_tag like \"%ll%\"",
+ output_fields: ["color_tag"]
+})
+
+console.log(res.results)
+```
+
+The output is similar to the following:
+
+
+
+```python
+[
+ [
+ {
+ "id": 5,
+ "distance": 0.7972343564033508,
+ "entity": {
+ "color": "yellow_4222"
+ }
+ }
+ ]
+]
+```
+
+```java
+{"searchResults": [
+ [
+ {
+ "score": 1.1869997,
+ "fields": {"color_tag": "yellow_4222"}
+ }
+ ]
+]}
+```
+
+```javascript
+[
+ { score: 2.5080761909484863, id: '1201', color_tag: 'yellow_4222' }
+]
+```
+
+## Range search
+
+Range search allows you to find vectors that lie within a specified distance range from your query vector.
+
+By setting `radius` and optionally `range_filter`, you can adjust the breadth of your search to include vectors that are somewhat similar to the query vector, providing a more comprehensive view of potential matches.
+
+- `radius`: Defines the outer boundary of your search space. Only vectors that are within this distance from the query vector are considered potential matches.
+
+- `range_filter`: While `radius` sets the outer limit of the search, `range_filter` can be optionally used to define an inner boundary, creating a distance range within which vectors must fall to be considered matches.
+
+
+
+```python
+# Conduct a range search
+search_params = {
+ "metric_type": "IP",
+ "params": {
+ "radius": 0.8, # Radius of the search circle
+ "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
+ }
+}
+
+res = client.search(
+ collection_name="test_collection", # Replace with the actual name of your collection
+ data=[[0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]],
+ limit=3, # Max. number of search results to return
+ search_params=search_params, # Search parameters
+ output_fields=["color"], # Output fields to return
+)
+
+result = json.dumps(res, indent=4)
+print(result)
+```
+
+```java
+// 9. Range search
+query_vectors = Arrays.asList(Arrays.asList(0.3580376395471989f, -0.6023495712049978f, 0.18414012509913835f, -0.26286205330961354f, 0.9029438446296592f));
+
+searchReq = SearchReq.builder()
+ .collectionName("quick_setup")
+ .data(query_vectors)
+ .outputFields(Arrays.asList("color_tag"))
+ .searchParams(Map.of("radius", 0.1, "range", 1.0))
+ .topK(5)
+ .build();
+
+searchResp = client.search(searchReq);
+
+System.out.println(JSONObject.toJSON(searchResp));
+```
+
+```javascript
+// 9. Range search
+query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]
+
+res = await client.search({
+ collection_name: "quick_setup",
+ data: [query_vector],
+ limit: 5,
+ params: {
+ radius: 0.1,
+ range: 1.0
+ },
+ output_fields: ["color_tag"]
+})
+
+console.log(res.results)
+```
+
+The output is similar to the following:
+
+
+
+```python
+[
+ [
+ {
+ "id": 4,
+ "distance": 0.9902134537696838,
+ "entity": {
+ "color": "red_4794"
+ }
+ },
+ {
+ "id": 14,
+ "distance": 0.9803846478462219,
+ "entity": {
+ "color": "green_2899"
+ }
+ },
+ {
+ "id": 1,
+ "distance": 0.8519943356513977,
+ "entity": {
+ "color": "red_7025"
+ }
+ }
+ ]
+]
+```
+
+```java
+{"searchResults": [
+ [
+ {
+ "score": 1.263043,
+ "fields": {"color_tag": "green_2052"}
+ },
+ {
+ "score": 1.2377806,
+ "fields": {"color_tag": "purple_3709"}
+ },
+ {
+ "score": 1.1869997,
+ "fields": {"color_tag": "red_3026"}
+ },
+ {
+ "score": 1.1748955,
+ "fields": {"color_tag": "black_1646"}
+ },
+ {
+ "score": 1.1720343,
+ "fields": {"color_tag": "green_4853"}
+ }
+ ]
+]}
+```
+
+```javascript
+[
+ { score: 2.3387961387634277, id: '718', color_tag: 'black_7154' },
+ { score: 2.3352415561676025, id: '1745', color_tag: 'blue_8741' },
+ { score: 2.290485382080078, id: '1408', color_tag: 'red_2324' },
+ { score: 2.285870313644409, id: '854', color_tag: 'black_5990' },
+ { score: 2.2593345642089844, id: '1309', color_tag: 'red_8458' }
+]
+```
+
+You will observe that all the entities returned have a distance that falls within the range of 0.8 to 1.0 from the query vector.
+
+The parameter settings for `radius` and `range_filter` vary with the metric type in use.
+
+| __Metric Type__ | __Charactericstics__ | __Range Search Settings__ |
+| ---------------- | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| `L2` | Smaller L2 distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
`range_filter` <= distance < `radius` | +| `IP` | Larger IP distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
`radius` < distance <= `range_filter` | +| `COSINE` | Larger cosine value indicates higher similarity. | To exclude the closest vectors from results, ensure that:
`radius` < distance <= `range_filter` | +| `JACCARD` | Smaller Jaccard distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
`range_filter` <= distance < `radius` | +| `HAMMING` | Smaller Hamming distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
`range_filter` <= distance < `radius` | + +To learn more about distance metric types, refer to [Similarity Metrics](metric.md). + +## Grouping search + +In Milvus, grouping search by a specific field can avoid redundancy of the same field item in the results. You can get a varied set of results for the specific field. + +Consider a collection of documents, each document splits into various passages. Each passage is represented by one vector embedding and belongs to one document. To find relevant documents instead of similar passages, you can include the `group_by_field` argument in the `search()` opeartion to group results by the document ID. This helps return the most relevant and unique documents, rather than separate passages from the same document. + +Here is the example code to group search results by field: + +```python +# Connect to Milvus +client = MilvusClient(uri='http://localhost:19530') # Milvus server address + +# Load data into collection +client.load_collection("group_search") # Collection name + +# Group search results +res = client.search( + collection_name="group_search", # Collection name + data=[[0.14529211512077012, 0.9147257273453546, 0.7965055218724449, 0.7009258593102812, 0.5605206522382088]], # Query vector + search_params={ + "metric_type": "L2", + "params": {"nprobe": 10}, + }, # Search parameters + limit=10, # Max. number of search results to return + group_by_field="doc_id", # Group results by document ID + output_fields=["doc_id", "passage_id"] +) + +# Retrieve the values in the `doc_id` column +doc_ids = [result['entity']['doc_id'] for result in res[0]] + +print(doc_ids) +``` + +The output is similar to the following: + +```python +[5, 10, 1, 7, 9, 6, 3, 4, 8, 2] +``` + +In the given output, it can be observed that the returned entities do not contain any duplicate `doc_id` values. + +For comparison, let's comment out the `group_by_field` and conduct a regular search: + +```python +# Connect to Milvus +client = MilvusClient(uri='http://localhost:19530') # Milvus server address + +# Load data into collection +client.load_collection("group_search") # Collection name + +# Search without `group_by_field` +res = client.search( + collection_name="group_search", # Collection name + data=query_passage_vector, # Replace with your query vector + search_params={ + "metric_type": "L2", + "params": {"nprobe": 10}, + }, # Search parameters + limit=10, # Max. number of search results to return + # group_by_field="doc_id", # Group results by document ID + output_fields=["doc_id", "passage_id"] +) + +# Retrieve the values in the `doc_id` column +doc_ids = [result['entity']['doc_id'] for result in res[0]] + +print(doc_ids) +``` + +The output is similar to the following: + +```python +[1, 10, 3, 10, 1, 9, 4, 4, 8, 6] +``` + +In the given output, it can be observed that the returned entities contain duplicate `doc_id` values. + +__Limitations__ + +- __Indexing__: This grouping feature works only for collections that are indexed with the __HNSW__, __IVF_FLAT__, or __FLAT__ type. For more information, refer to [In-memory Index](https://milvus.io/docs/index.md#HNSW). + +- __Vector__: Currently, grouping search does not support a vector field of the __BINARY_VECTOR__ type. For more information on data types, refer to [Supported data types](https://milvus.io/docs/schema.md#Supported-data-types). + +- __Field__: Currently, grouping search allows only for a single column. You cannot specify multiple field names in the `group_by_field` config. Additionally, grouping search is incompatible with data types of JSON, FLOAT, DOUBLE, ARRAY, or vector fields. + +- __Performance Impact__: Be mindful that performance degrades with increasing query vector counts. Using a cluster with 2 CPU cores and 8 GB of memory as an example, the execution time for grouping search increases proportionally with the number of input query vectors. + +- __Functionality__: Currently, grouping search is not supported by [range search](https://milvus.io/docs/single-vector-search.md#Range-search), [search iterators](https://milvus.io/docs/with-iterators.md#Search-with-iterator), or [hybrid search](multi-vector-search.md). + +## Search parameters + +In the above searches except the range search, the default search parameters apply. In normal cases, you do not need to manually set search parameters. + +```python +# In normal cases, you do not need to set search parameters manually +# Except for range searches. +search_parameters = { + 'metric_type': 'L2', + 'params': { + 'nprobe': 10, + 'level': 1， + 'radius': 1.0 + 'range_filter': 0.8 + } +} +``` + +The following table lists all possible settings in the search parameters. + +| __Parameter Name__ | __Parameter Description__ | +| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `metric_type` | How to measure similarity between vector embeddings.
Possible values are `IP`, `L2`, and `COSINE`, and defaults to that of the loaded index file. | +| `params.nprobe` | Number of units to query during the search.
The value falls in the range [1, nlist[1]]. | +| `params.level` | Search precision level.
Possible values are `1`, `2`, and `3`, and defaults to `1`. Higher values yield more accurate results but slower performance. | +| `params.radius` | Minimum similarity between the query vector and candidate vectors.
The value falls in the range [1, nlist[1]]. | +| `params.range_filter` | A similarity range, optionally refining the search for vectors that fall in the range.
The value falls in the range [top-K[2], ∞]. | + + colors = Arrays.asList("green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey");
+List data = new ArrayList<>();
+
+for (int i=0; i<10000; i++) {
+ Random rand = new Random();
+ String current_color = colors.get(rand.nextInt(colors.size()-1));
+ JSONObject row = new JSONObject();
+ row.put("id", Long.valueOf(i));
+ row.put("vector", Arrays.asList(rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat()));
+ row.put("color_tag", current_color + "_" + String.valueOf(rand.nextInt(8999) + 1000));
+ data.add(row);
+}
+
+InsertParam insertParam = InsertParam.newBuilder()
+ .withCollectionName("quick_setup")
+ .withRows(data)
+ .build();
+
+R insertRes = client.insert(insertParam);
+
+if (insertRes.getStatus() != R.Status.Success.getCode()) {
+ System.err.println(insertRes.getMessage());
+}
+
+MutationResultWrapper wrapper = new MutationResultWrapper(insertRes.getData());
+System.out.println(wrapper.getInsertCount());
+```
+
+## Search with iterator
+
+Iterators make similarity searches more scalable.
+
+ searchIteratorRes = client.searchIterator(iteratorParam);
+
+if (searchIteratorRes.getStatus() != R.Status.Success.getCode()) {
+ System.err.println(searchIteratorRes.getMessage());
+}
+
+SearchIterator searchIterator = searchIteratorRes.getData();
+List results = new ArrayList<>();
+
+while (true) {
+ List batchResults = searchIterator.next();
+ if (batchResults.isEmpty()) {
+ searchIterator.close();
+ break;
+ }
+ for (QueryResultsWrapper.RowRecord rowRecord : batchResults) {
+ results.add(rowRecord);
+ }
+}
+
+System.out.println(results.size());
+```
+
+
+
+
+
+
+
+
+
+## Query with an iterator
+
+ queryIteratRes = client.queryIterator(queryIteratorParam);
+
+if (queryIteratRes.getStatus() != R.Status.Success.getCode()) {
+ System.err.println(queryIteratRes.getMessage());
+}
+
+QueryIterator queryIterator = queryIteratRes.getData();
+
+while (true) {
+ List batchResults = queryIterator.next();
+ if (batchResults.isEmpty()) {
+ queryIterator.close();
+ break;
+ }
+
+ String jsonString = "";
+ List jsonObject = new ArrayList<>();
+ try {
+ jsonString = Files.readString(Path.of("results.json"));
+ jsonObject = JSON.parseArray(jsonString).toJavaList(null);
+ } catch (IOException e) {
+ // TODO Auto-generated catch block
+ e.printStackTrace();
+ }
+
+ for (QueryResultsWrapper.RowRecord queryResult : batchResults) {
+ JSONObject row = new JSONObject();
+ row.put("id", queryResult.get("id"));
+ row.put("vector", queryResult.get("vector"));
+ row.put("color_tag", queryResult.get("color_tag"));
+ jsonObject.add(row);
+ }
+
+ try {
+ Files.write(Path.of("results.json"), JSON.toJSONString(jsonObject).getBytes(), StandardOpenOption.WRITE);
+ } catch (IOException e) {
+ // TODO Auto-generated catch block
+ e.printStackTrace();
+ }
+}
+```
+
+
+
+
+
+
+
+
diff --git a/preview/site/en/userGuide/search/hybridsearch.md b/preview/site/en/userGuide/search/hybridsearch.md
deleted file mode 100644
index 8189ae294..000000000
--- a/preview/site/en/userGuide/search/hybridsearch.md
+++ /dev/null
@@ -1,579 +0,0 @@
----
-id: hybridsearch.md
-related_key: filter
-summary: Conduct a Hybrid Search with Milvus.
----
-
-# Conduct a Hybrid Search
-
-This topic describes how to conduct a hybrid search.
-
-A hybrid search is essentially a vector search with attribute filtering. By specifying [boolean expressions](boolean.md) that filter the scalar fields or the primary key field, you can limit your search with certain conditions.
-
-The following example shows how to perform a hybrid search on the basis of a regular [vector search](search.md). Suppose you want to search for certain books based on their vectorized introductions, but you only want those within a specific range of word count. You can then specify the boolean expression to filter the `word_count` field in the search parameters. Milvus will search for similar vectors only among those entities that match the expression.
-
-
-## Load collection
-
-All search and query operations within Milvus are executed in memory. Load the collection to memory before conducting a vector search.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.load()
-```
-
-```javascript
-await milvusClient.collectionManager.loadCollection({
- collection_name: "book",
-});
-```
-
-```go
-err := milvusClient.LoadCollection(
- context.Background(), // ctx
- "book", // CollectionName
- false // async
-)
-if err != nil {
- log.Fatal("failed to load collection:", err.Error())
-}
-```
-
-```java
-milvusClient.loadCollection(
- LoadCollectionParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-```
-
-```shell
-load -c book
-```
-
-```curl
-# See the following step.
-```
-
-## Conduct a hybrid vector search
-
-By specifying the boolean expression, you can filter the scalar field of the entities during the vector search. The following example limits the scale of search to the vectors within a specified `word_count` value range.
-
-
-
-
-```python
-search_param = {
- "data": [[0.1, 0.2]],
- "anns_field": "book_intro",
- "param": {"metric_type": "L2", "params": {"nprobe": 10}},
- "offset": 0,
- "limit": 2,
- "expr": "word_count <= 11000",
-}
-res = collection.search(**search_param)
-```
-
-```javascript
-const results = await milvusClient.dataManager.search({
- collection_name: "book",
- expr: "word_count <= 11000",
- vectors: [[0.1, 0.2]],
- search_params: {
- anns_field: "book_intro",
- topk: "2",
- metric_type: "L2",
- params: JSON.stringify({ nprobe: 10 }),
- },
- vector_type: 101, // DataType.FloatVector,
-});
-```
-
-```go
-sp, _ := entity.NewIndexFlatSearchParam( // NewIndex*SearchParam func
- 10, // searchParam
-)
-searchResult, err := milvusClient.Search(
- context.Background(), // ctx
- "book", // CollectionName
- []string{}, // partitionNames
- "word_count <= 11000", // expr
- []string{"book_id"}, // outputFields
- []entity.Vector{entity.FloatVector([]float32{0.1, 0.2})}, // vectors
- "book_intro", // vectorField
- entity.L2, // metricType
- 2, // topK
- sp, // sp
-)
-if err != nil {
- log.Fatal("fail to search collection:", err.Error())
-}
-```
-
-```java
-final Integer SEARCH_K = 2;
-final String SEARCH_PARAM = "{\"nprobe\":10, \”offset\”:5}";
-List search_output_fields = Arrays.asList("book_id");
-List> search_vectors = Arrays.asList(Arrays.asList(0.1f, 0.2f));
-
-SearchParam searchParam = SearchParam.newBuilder()
- .withCollectionName("book")
- .withMetricType(MetricType.L2)
- .withOutFields(search_output_fields)
- .withTopK(SEARCH_K)
- .withVectors(search_vectors)
- .withVectorFieldName("book_intro")
- .withExpr("word_count <= 11000")
- .withParams(SEARCH_PARAM)
- .build();
-R respSearch = milvusClient.search(searchParam);
-```
-
-```shell
-search
-
-Collection name (book): book
-
-The vectors of search data(the length of data is number of query (nq), the dim of every vector in data must be equal to vector field’s of collection. You can also import a csv file without headers): [[0.1, 0.2]]
-
-The vector field used to search of collection (book_intro): book_intro
-
-Metric type: L2
-
-Search parameter nprobe's value: 10
-
-The max number of returned record, also known as topk: 2
-
-The boolean expression used to filter attribute []: word_count <= 11000
-
-The names of partitions to search (split by "," if multiple) ['_default'] []:
-
-timeout []:
-
-Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/search' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "output_fields": ["book_id"],
- "search_params": [
- {"key": "anns_field", "value": "book_intro"},
- {"key": "topk", "value": "2"},
- {"key": "params", "value": "{\"nprobe\": 10}"},
- {"key": "metric_type", "value": "L2"},
- {"key": "round_decimal", "value": "-1"}
- ],
- "vectors": [ [0.1,0.2] ],
- "dsl": "word_count >= 11000",
- "dsl_type": 1
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Check the returned results.
-
-
-
-
-```python
-assert len(res) == 1
-hits = res[0]
-assert len(hits) == 2
-print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
-print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")
-```
-
-```javascript
-console.log(results.results)
-```
-
-```go
-fmt.Printf("%#v\n", searchResult)
-for _, sr := range searchResult {
- fmt.Println(sr.IDs)
- fmt.Println(sr.Scores)
-}
-```
-
-```java
-SearchResultsWrapper wrapperSearch = new SearchResultsWrapper(respSearch.getData().getResults());
-System.out.println(wrapperSearch.getIDScore(0));
-System.out.println(wrapperSearch.getFieldData("book_id", 0));
-```
-
-```shell
-# Milvus CLI automatically returns the primary key values of the most similar vectors and their distances.
-```
-
-```curl
-# See the output of the previous step.
-```
-
-## What's next
-
-- Try [Search with Time Travel](timetravel.md)
-
-- Explore API references for Milvus SDKs:
-
- - [PyMilvus API reference](/api-reference/pymilvus/v2.2.2/About.md)
- - [Node.js API reference](/api-reference/node/v2.2.x/About.md)
- - [Go API reference](/api-reference/go/v2.2.0/About.md)
- - [Java API reference](/api-reference/java/v2.2.1/About.md)
-
-
diff --git a/preview/site/en/userGuide/search/query.md b/preview/site/en/userGuide/search/query.md
deleted file mode 100644
index 8fb8b8f9a..000000000
--- a/preview/site/en/userGuide/search/query.md
+++ /dev/null
@@ -1,409 +0,0 @@
----
-id: query.md
-related_key: query vectors
-summary: Learn how to query vectors in Milvus.
----
-
-# Conduct a Vector Query
-
-This topic describes how to conduct a vector query.
-
-Unlike a vector similarity search, a vector query retrieves vectors via scalar filtering based on [boolean expression](boolean.md). Milvus supports many data types in the scalar fields and a variety of boolean expressions. The boolean expression filters on scalar fields or the primary key field, and it retrieves all results that match the filters.
-
-The following example shows how to perform a vector query on a 2000-row dataset of book ID (primary key), word count (scalar field), and book introduction (vector field), simulating the situation where you query for certain books based on their IDs.
-
-
-## Load collection
-
-All search and query operations within Milvus are executed in memory. Load the collection to memory before conducting a vector query.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.load()
-```
-
-```javascript
-await milvusClient.collectionManager.loadCollection({
- collection_name: "book",
-});
-```
-
-```go
-err := milvusClient.LoadCollection(
- context.Background(), // ctx
- "book", // CollectionName
- false // async
-)
-if err != nil {
- log.Fatal("failed to load collection:", err.Error())
-}
-```
-
-```java
-milvusClient.loadCollection(
- LoadCollectionParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-```
-
-```shell
-load -c book
-```
-
-```curl
-# See the following step.
-```
-
-## Conduct a vector query
-
-The following example filters the vectors with certain `book_id` values, and returns the `book_id` field and `book_intro` of the results.
-
-Milvus supports setting consistency level specifically for a query. The example in this topic sets the consistency level as `Strong`. You can also set the consistency level as `Bounded`, `Session` or `Eventually`. See [Consistency](consistency.md) for more information about the four consistency levels in Milvus.
-
-
-
-
-```python
-res = collection.query(
- expr = "book_id in [2,4,6,8]",
- offset = 0,
- limit = 10,
- output_fields = ["book_id", "book_intro"],
- consistency_level="Strong"
-)
-```
-
-```javascript
-const results = await milvusClient.dataManager.query({
- collection_name: "book",
- expr: "book_id in [2,4,6,8]",
- output_fields: ["book_id", "book_intro"],
-});
-```
-
-```go
-queryResult, err := milvusClient.Query(
- context.Background(), // ctx
- "book", // CollectionName
- "", // PartitionName
- entity.NewColumnInt64("book_id", []int64{2,4,6,8}), // expr
- []string{"book_id", "book_intro"} // OutputFields
-)
-if err != nil {
- log.Fatal("fail to query collection:", err.Error())
-}
-```
-
-```java
-List query_output_fields = Arrays.asList("book_id", "word_count");
-QueryParam queryParam = QueryParam.newBuilder()
- .withCollectionName("book")
- .withConsistencyLevel(ConsistencyLevelEnum.STRONG)
- .withExpr("book_id in [2,4,6,8]")
- .withOutFields(query_output_fields)
- .withOffset(0L)
- .withLimit(10L)
- .build();
-R respQuery = milvusClient.query(queryParam);
-```
-
-```shell
-query
-
-collection_name: book
-
-The query expression: book_id in [2,4,6,8]
-
-Name of partitions that contain entities(split by "," if multiple) []:
-
-A list of fields to return(split by "," if multiple) []: book_id, book_intro
-
-timeout []:
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/query' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "output_fields": ["book_id", "book_intro"],
- "vectors": [ [0.1,0.2] ],
- "expr": "book_id in [2,4,6,8]"
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Check the returned results.
-
-
-
-
-```python
-sorted_res = sorted(res, key=lambda k: k['book_id'])
-sorted_res
-```
-
-```javascript
-console.log(results.data)
-```
-
-```go
-fmt.Printf("%#v\n", queryResult)
-for _, qr := range queryResult {
- fmt.Println(qr.IDs)
-}
-```
-
-```java
-QueryResultsWrapper wrapperQuery = new QueryResultsWrapper(respQuery.getData());
-System.out.println(wrapperQuery.getFieldWrapper("book_id").getFieldData());
-System.out.println(wrapperQuery.getFieldWrapper("word_count").getFieldData());
-```
-
-```shell
-# Milvus CLI automatically returns the entities with the pre-defined output fields.
-```
-
-```curl
-# See the output of the previous step.
-```
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
- - [Search with Time Travel](timetravel.md)
-
-- Explore API references for Milvus SDKs:
-
- - [PyMilvus API reference](/api-reference/pymilvus/v2.2.2/About.md)
- - [Node.js API reference](/api-reference/node/v2.2.x/About.md)
- - [Go API reference](/api-reference/go/v2.2.0/About.md)
- - [Java API reference](/api-reference/java/v2.2.1/About.md)
-
-
diff --git a/preview/site/en/userGuide/search/search.md b/preview/site/en/userGuide/search/search.md
deleted file mode 100644
index bb89ff8ed..000000000
--- a/preview/site/en/userGuide/search/search.md
+++ /dev/null
@@ -1,758 +0,0 @@
----
-id: search.md
-related_key: search
-summary: Conduct a vector similarity search with Milvus.
----
-
-# Conduct a Vector Similarity Search
-
-This topic describes how to search entities with Milvus.
-
-A vector similarity search in Milvus calculates the distance between query vector(s) and vectors in the collection with specified similarity metrics, and returns the most similar results. By specifying a [boolean expression](boolean.md) that filters the scalar field or the primary key field, you can perform a [hybrid search](hybridsearch.md) or even a search with [Time Travel](timetravel.md).
-
-The following example shows how to perform a vector similarity search on a 2000-row dataset of book ID (primary key), word count (scalar field), and book introduction (vector field), simulating the situation that you search for certain books based on their vectorized introductions. Milvus will return the most similar results according to the query vector and search parameters you have defined.
-
-
-## Load collection
-
-All search and query operations within Milvus are executed in memory. Load the collection to memory before conducting a vector similarity search.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.load()
-```
-
-```javascript
-await milvusClient.collectionManager.loadCollection({
- collection_name: "book",
-});
-```
-
-```go
-err := milvusClient.LoadCollection(
- context.Background(), // ctx
- "book", // CollectionName
- false // async
-)
-if err != nil {
- log.Fatal("failed to load collection:", err.Error())
-}
-```
-
-```java
-milvusClient.loadCollection(
- LoadCollectionParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-```
-
-```shell
-load -c book
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/collection/load' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book"
- }'
-```
-
-## Prepare search parameters
-
-Prepare the parameters that suit your search scenario. The following example defines that the search will calculate the distance with Euclidean distance, and retrieve vectors from ten closest clusters built by the IVF_FLAT index.
-
-
-
-
-```python
-search_params = {"metric_type": "L2", "params": {"nprobe": 10}, "offset": 5}
-```
-
-```javascript
-const searchParams = {
- anns_field: "book_intro",
- topk: "2",
- metric_type: "L2",
- params: JSON.stringify({ nprobe: 10 }),
-};
-```
-
-```go
-sp, _ := entity.NewIndexFlatSearchParam( // NewIndex*SearchParam func
- 10, // searchParam
-)
-```
-
-```java
-final Integer SEARCH_K = 2; // TopK
-final String SEARCH_PARAM = "{\"nprobe\":10, \”offset\”:5}"; // Params
-```
-
-```shell
-search
-
-Collection name (book): book
-
-The vectors of search data(the length of data is number of query (nq), the dim of every vector in data must be equal to vector field’s of collection. You can also import a csv file without headers): [[0.1, 0.2]]
-
-The vector field used to search of collection (book_intro): book_intro
-
-Metric type: L2
-
-Search parameter nprobe's value: 10
-
-The max number of returned record, also known as topk: 10
-
-The boolean expression used to filter attribute []:
-
-The names of partitions to search (split by "," if multiple) ['_default'] []:
-
-timeout []:
-
-Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/search' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "output_fields": ["book_id"],
- "search_params": [
- {"key": "anns_field", "value": "book_intro"},
- {"key": "topk", "value": "2"},
- {"key": "params", "value": "{\"nprobe\": 10}"},
- {"key": "metric_type", "value": "L2"},
- {"key": "round_decimal", "value": "-1"}
- ],
- "vectors": [ [0.1,0.2] ],
- "dsl_type": 1
- }'
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-## Conduct a vector search
-
-Search vectors with Milvus. To search in a specific [partition](glossary.md#Partition), specify the list of partition names.
-
-Milvus supports setting consistency level specifically for a search. The example in this topic sets the consistency level as `Strong`. You can also set the consistency level as `Bounded`, `Session` or `Eventually`. See [Consistency](consistency.md) for more information about the four consistency levels in Milvus.
-
-
-
-
-```python
-results = collection.search(
- data=[[0.1, 0.2]],
- anns_field="book_intro",
- param=search_params,
- limit=10,
- expr=None,
- consistency_level="Strong"
-)
-```
-
-```javascript
-const results = await milvusClient.dataManager.search({
- collection_name: "book",
- expr: "",
- vectors: [[0.1, 0.2]],
- search_params: searchParams,
- vector_type: 101, // DataType.FloatVector
-});
-```
-
-```go
-searchResult, err := milvusClient.Search(
- context.Background(), // ctx
- "book", // CollectionName
- []string{}, // partitionNames
- "", // expr
- []string{"book_id"}, // outputFields
- []entity.Vector{entity.FloatVector([]float32{0.1, 0.2})}, // vectors
- "book_intro", // vectorField
- entity.L2, // metricType
- 2, // topK
- sp, // sp
-)
-if err != nil {
- log.Fatal("fail to search collection:", err.Error())
-}
-```
-
-```java
-List search_output_fields = Arrays.asList("book_id");
-List> search_vectors = Arrays.asList(Arrays.asList(0.1f, 0.2f));
-
-SearchParam searchParam = SearchParam.newBuilder()
- .withCollectionName("book")
- .withConsistencyLevel(ConsistencyLevelEnum.STRONG)
- .withMetricType(MetricType.L2)
- .withOutFields(search_output_fields)
- .withTopK(SEARCH_K)
- .withVectors(search_vectors)
- .withVectorFieldName("book_intro")
- .withParams(SEARCH_PARAM)
- .build();
-R respSearch = milvusClient.search(searchParam);
-```
-
-```shell
-# Follow the previous step.
-```
-
-```curl
-# Follow the previous step.
-```
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Check the primary key values of the most similar vectors and their distances.
-
-
-
-
-```python
-results[0].ids
-results[0].distances
-```
-
-```javascript
-console.log(results.results)
-```
-
-```go
-fmt.Printf("%#v\n", searchResult)
-for _, sr := range searchResult {
- fmt.Println(sr.IDs)
- fmt.Println(sr.Scores)
-}
-```
-
-```java
-SearchResultsWrapper wrapperSearch = new SearchResultsWrapper(respSearch.getData().getResults());
-System.out.println(wrapperSearch.getIDScore(0));
-System.out.println(wrapperSearch.getFieldData("book_id", 0));
-```
-
-```shell
-# Milvus CLI automatically returns the primary key values of the most similar vectors and their distances.
-```
-
-Release the collection loaded in Milvus to reduce memory consumption when the search is completed.
-
-
-
-
-```python
-collection.release()
-```
-
-```javascript
-await milvusClient.collectionManager.releaseCollection({ collection_name: "book",});
-```
-
-```go
-err := milvusClient.ReleaseCollection(
- context.Background(), // ctx
- "book", // CollectionName
-)
-if err != nil {
- log.Fatal("failed to release collection:", err.Error())
-}
-```
-
-```java
-milvusClient.releaseCollection(
- ReleaseCollectionParam.newBuilder()
- .withCollectionName("book")
- .build());
-```
-
-```shell
-release -c book
-```
-
-``` curl
-curl -X 'DELETE' \
- 'http://localhost:9091/api/v1/collection/load' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book"
- }'
-```
-
-## Limits
-|Feature|Maximum limit|
-|---|---|
-|Length of a collection name|255 characters|
-|Number of partitions in a collection|4,096|
-|Number of fields in a collection|256|
-|Number of shards in a collection|256|
-|Dimensions of a vector|32,768|
-|Top K|16,384|
-|Target input vectors|16,384|
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Query vectors](query.md)
- - [Conduct a hybrid search](hybridsearch.md)
- - [Search with Time Travel](timetravel.md)
-
-- Explore API references for Milvus SDKs:
-
- - [PyMilvus API reference](/api-reference/pymilvus/v2.2.2/About.md)
- - [Node.js API reference](/api-reference/node/v2.2.x/About.md)
- - [Go API reference](/api-reference/go/v2.2.0/About.md)
- - [Java API reference](/api-reference/java/v2.2.1/About.md)
-
-
diff --git a/preview/site/en/userGuide/search/search_and_query.md b/preview/site/en/userGuide/search/search_and_query.md
deleted file mode 100644
index 6dfd051a6..000000000
--- a/preview/site/en/userGuide/search/search_and_query.md
+++ /dev/null
@@ -1,14 +0,0 @@
----
-id: search_and_query.md
-title: Search and Query
----
-
-# Search and Query
-
-Conduct a search, query, or hybrid search to find relevant results.
-
-- [Search](search.md): A vector similarity search in Milvus calculates the distance between query vector(s) and vectors in a collection and returns the most similar results.
-
-- [Hybrid Search](hybridsearch.md): A hybrid search in Milvus is essentially a vector search with attribute filtering.
-
-- [Query](query.md): A vector query in Milvus retrieves vectors via scalar filtering based on boolean expression.
diff --git a/preview/site/en/userGuide/search/timetravel.md b/preview/site/en/userGuide/search/timetravel.md
deleted file mode 100644
index 894d33533..000000000
--- a/preview/site/en/userGuide/search/timetravel.md
+++ /dev/null
@@ -1,1027 +0,0 @@
----
-id: timetravel.md
-related_key: Time Travel
-summary: Learn how to search with Time Travel in Milvus.
----
-
-# Search with Time Travel
-
-This topic describes how to use the Time Travel feature during vector search.
-
-Milvus maintains a timeline for all data insert and delete operations. It allows users to specify a timestamp in a search to retrieve a data view at a specified point in time, without spending tremendously on maintenance for data rollback.
-
-
-
-
diff --git a/preview/site/en/userGuide/tools/attu_overview.md b/preview/site/en/userGuide/tools/attu_overview.md
deleted file mode 100644
index 8fe91ec85..000000000
--- a/preview/site/en/userGuide/tools/attu_overview.md
+++ /dev/null
@@ -1,20 +0,0 @@
----
-id: attu_overview.md
-related_key: attu
-summary: Attu, an intuitive GUI for Milvus service management.
----
-
-# Attu Overview Page
-
-This topic describes the Home page of Attu.
-
-Attu consists of **Overview** page, **Collection** page, **Vector Search** page, and **System View** page, corresponding to the four icons on the left-side navigation pane respectively.
-
-The Overview page lists the following information:
-
-1. Loaded Collections: the number of loaded collections.
-2. All Collections: the total number of collections.
-3. Data: the total number of entities.
-4. **Loaded For Search** card: an interactive shortcut panel that allows you to perform a [vector search](attu_search.md) on or [release the collection](attu_collection.md).
-
-
diff --git a/preview/site/en/userGuide/tools/attu_partition.md b/preview/site/en/userGuide/tools/attu_partition.md
deleted file mode 100644
index d993bec46..000000000
--- a/preview/site/en/userGuide/tools/attu_partition.md
+++ /dev/null
@@ -1,35 +0,0 @@
----
-id: attu_partition.md
-related_key: attu
-summary: Learn how to manage partitions with Attu, an intuitive GUI tool for Milvus.
----
-
-# Manage Partitions with Attu
-
-This topic describes how to manage partitions with Attu.
-
-Milvus creates a partition automatically after a collection is created, which cannot be deleted.
-
-## Create a partition
-
-1. Click **Partitions** tab on the **Collection** page.
-2. Click **Create Partition** on the **Partitions** tab page, and the **Create Partition** dialog box appears as shown below.
-3. In the **Create Partition** dialog box, enter the new partition name in the **Name** field.
-4. Click **Create** to create a partition.
-
-
-
-If successful, the new partition appears on the **Partitions** tab page.
-
-
-
-Choose the Default partition or the newly created partition to store imported data as needed.
-
-## Delete a partition
-
-1. Tick the partition you want to delete.
-2. Click the **Trash** icon and the **Delete Partition** dialog box appears as shown below.
-3. Type `delete` to confirm the deletion.
-4. Click **Delete** to delete the partition.
-
-
diff --git a/preview/site/en/userGuide/tools/attu_query.md b/preview/site/en/userGuide/tools/attu_query.md
deleted file mode 100644
index f6935e1e8..000000000
--- a/preview/site/en/userGuide/tools/attu_query.md
+++ /dev/null
@@ -1,37 +0,0 @@
----
-id: attu_query.md
-related_key: attu
-summary: Learn how to query data with Attu, an intuitive GUI tool for Milvus.
----
-
-# Query Data with Attu
-
-This topic describes how to query data with Attu.
-
-## Query data with advanced filters
-
-1. Click the entry of the collection that you want to query data in, and the corresponding detail page appears.
-2. On the **Data Query** tab page, click the **Filter** icon and the **Advanced Filter** dialog box appears.
-3. Specify a complicated query condition such as **color > 10 && color < 20** by using the **Field Name** dropdown lists, the **Logic** dropdown lists, the **Value** fields, and the **AND** operator. Then click **Apply** Filter to apply the query condition.
-
-
-
-3. Click **Query** to retrieve all query results that match the query condition.
-
-
-
-## Query data with Time Travel
-
-TBD (Not supported yet)
-
-## Delete data
-
-1. Tick the entities you want to delete and click the **Trash** icon.
-2. Type `delete` to confirm the deletion in the **Delete entities** dialog box.
-3. Click **Delete** to delete the selected entities.
-
-
-
-You can perform a query to retrieve the deleted entities. No results will be returned if the deletion is successful.
-
-
diff --git a/preview/site/en/userGuide/tools/attu_search.md b/preview/site/en/userGuide/tools/attu_search.md
deleted file mode 100644
index 396549224..000000000
--- a/preview/site/en/userGuide/tools/attu_search.md
+++ /dev/null
@@ -1,47 +0,0 @@
----
-id: attu_search.md
-related_key: attu
-summary: Learn how to conduct a vector similarity search using Attu, an intuitive GUI tool for Milvus.
----
-
-# Search Data with Attu
-
-This topic describes how to search data with Attu.
-
-## Conduct a vector similarity search
-
-On the basis of the regular vector similarity search, you can perform hybrid search of search with Time Travel.
-
-### Load the collection to memory
-
-All CRUD operations within Milvus are executed in memory. Load the collection to memory before conducting a vector similarity search. See [Load a collection](attu_collection.md#Load-a-collection) for more instruction.
-
-
-
-### Set search parameters
-
-1. Select the collection and the vector field you wish to search in in the dropdown lists of the **Choose collection and field** section.
-2. In the **Enter vector value** field, enter a vector (or vectors) with the same dimensions of the selected field as the target vector(s) to search with.
-3. In the **Set search parameters** section, specify the specific parameter(s) to the index and other search-related parameters.
-
-
-
-### Hybrid search with advanced filters (optional)
-
-Click **Advanced Filter** and the **Advanced Filter** dialog box appears. You can use the **AND** or **OR** operators to combine multiple conditions into a compound condition. The filter expression updates automatically with any changes to the conditions. See [boolean expression rule](boolean.md) for more information.
-
-
-
-### Search with Time Travel (optional)
-
-Milvus maintains a timeline for all data insert and delete operations. It allows users to specify a timestamp in a search to retrieve a data view at a specified point in time.
-
-1. Click **Time Travel**, and select a time point in the dialog box that appears.
-
-
-
-2. Specify the number of search results to return in the **TopK** dropdown list.
-3. Click **Search** to retrieve the nearest search results, which indicate the most similar vectors.
-
-
-
diff --git a/preview/site/en/userGuide/tools/attu_system.md b/preview/site/en/userGuide/tools/attu_system.md
deleted file mode 100644
index 9d48a02d9..000000000
--- a/preview/site/en/userGuide/tools/attu_system.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-id: attu_system.md
-related_key: attu
-summary: Learn how to monitor your Milvus system with Attu, an intuitive GUI tool for Milvus.
----
-
-# Monitor System with Attu
-
-This topic describes how to monitor your Milvus system with Attu.
-
-## System view
-
-Click the **System View** icon on the left-side navigation pane to visit the System View page.
-
-
-
-The System View dashboard consists of the following tiles:
-
-- **Topology**: Shows the structure of your Milvus instance. Click Milvus node or a coordinator node and the corresponding information appears in the Info tile on the right.
-- **Info**: Shows the system and configuration information of the selected node.
-
-## Node list view
-
-All nodes managed by their parent coordinator node are listed in the table. You can sort the nodes by metrics including **CPU Core Count**, **CPU Core Usage**, **Disk Usage**, and **Memory Usage**.
-
-
-
-Right to the table is a mini topology showing the selected node and its coordinator node. Under the mini topology is a tile showing relevant information.
-
-Click the down arrow to collapse rows in the table.
diff --git a/preview/site/en/userGuide/tools/birdwatcher_install_guides.md b/preview/site/en/userGuide/tools/birdwatcher_install_guides.md
new file mode 100644
index 000000000..974572db8
--- /dev/null
+++ b/preview/site/en/userGuide/tools/birdwatcher_install_guides.md
@@ -0,0 +1,116 @@
+---
+id: birdwatcher_install_guides.md
+summary: Learn how to install Birdwatch to debug Milvus.
+title: Install Birdwatcher
+---
+
+# Install Birdwatcher
+
+This page demonstrates how to install Birdwatcher.
+
+## Local install
+
+If you have installed Milvus Standalone [using docker](install_standalone-docker.md), you'd better download and install the built binary, install Birdwatcher as a common Go module, or build Birdwatcher from the source.
+
+- Install it as a common Go module.
+
+ ```shell
+ git clone https://github.com/milvus-io/birdwatcher.git
+ cd birdwatcher
+ go install github.com/milvus-io/birdwatcher
+ ```
+
+ Then you can run Birdwatcher as follows:
+
+ ```shell
+ go run main.go
+ ```
+
+- Build it from the source.
+
+ ```shell
+ git clone https://github.com/milvus-io/birdwatcher.git
+ cd birdwatcher
+ go build -o birdwatcher main.go
+ ```
+
+ Then you can run Birdwatcher as follows:
+
+ ```shell
+ ./birdwatcher
+ ```
+
+- Download the already-built binary
+
+ First, open the [latest release page](https://github.com/milvus-io/birdwatcher/releases/latest), and find the prepared binaries.
+
+ ```shell
+ wget -O birdwatcher.tar.gz \
+ https://github.com/milvus-io/birdwatcher/releases/download/latest/birdwatcher__.tar.gz
+ ```
+
+ Then you can decompress the tarball and use Birdwatcher as follows:
+
+ ```shell
+ tar -xvzf birdwatcher.tar.gz
+ ./birdwatcher
+ ```
+
+## Install as a Kubernetes pod
+
+If you have installed either Milvus Standalone [using the Helm charts](install_standalone-helm.md) or [the Milvus Operator](install_standalone-operator.md) or Milvus Cluster [using the Helm charts](install_cluster-helm.md) or [the Milvus Operator](install_cluster-milvusoperator.md), you are advised to install Birdwatcher as a Kubernetes pod.
+
+### Prepare deployment.yml
+
+```yml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: birdwatcher
+spec:
+ selector:
+ matchLabels:
+ app: birdwatcher
+ template:
+ metadata:
+ labels:
+ app: birdwatcher
+ spec:
+ containers:
+ - name: birdwatcher
+ image: milvusdb/birdwatcher
+ resources:
+ limits:
+ memory: "128Mi"
+ cpu: "500m"
+```
+
+ }
+Field title, value: &{string_data: }
+Field title_vector, value: &{dim:768 float_vector: }
+Field article_meta, value: &{json_data: }
+```
+
+### Probe all collections with mock queries
+
+You can also have Birdwatcher probe all collections with mock queries.
+
+```shell
+Milvus(by-dev) > probe query
+probing collection 442682158191982314
+Found vector field vector(103) with dim[384], indexID: 442682158191990455
+failed to generated mock request probing index type IVF_FLAT not supported yet
+probing collection 442844725212086932
+Found vector field title_vector(102) with dim[768], indexID: 442844725212086936
+Shard my-release-rootcoord-dml_1_442844725212086932v0 leader[298] probe with search success.
+probing collection 442844725212299747
+Found vector field title_vector(102) with dim[768], indexID: 442844725212299751
+Shard my-release-rootcoord-dml_4_442844725212299747v0 leader[298] probe with search success.
+probing collection 443294505839900248
+Found vector field vector(101) with dim[256], indexID: 443294505839900251
+Shard my-release-rootcoord-dml_5_443294505839900248v0 leader[298] probe with search success.
+```
+
+
+
diff --git a/preview/site/en/userGuide/tools/cli_commands.md b/preview/site/en/userGuide/tools/cli_commands.md
index 33eb8beef..100664cad 100644
--- a/preview/site/en/userGuide/tools/cli_commands.md
+++ b/preview/site/en/userGuide/tools/cli_commands.md
@@ -1,6 +1,7 @@
---
id: cli_commands.md
summary: Interact with Milvus using commands.
+title: Milvus_CLI Command Reference
---
# Milvus_CLI Command Reference
@@ -9,164 +10,144 @@ Milvus Command-Line Interface (CLI) is a command-line tool that supports databas
This topic introduces all supported commands and the corresponding options. Some examples are also included for your reference.
-## calc distance
+## clear
-Calculates the distance between two vector arrays.
+Clears the screen.
-::` format. |
| -p | --schema-primary-field | The name of the primary key field. |
| -a | --schema-auto-id | (Optional) Flag to generate IDs automatically. |
-| -d | --schema-description | (Optional) The description of the collection. |
+| -desc | --schema-description | (Optional) The description of the collection. |
+| -level | --consistency-level | (Optional) Consistency level: Bounded,Session,Strong, Eventual . |
+| -d | --is-dynamic | (Optional) Collection schema supports dynamic fields or not. |
+| -s | --shards-num | (Optional) Shards number |
| --help | n/a | Displays help for using the command. |
:::(:if Varchar)
+
+milvus_cli > create collection -c car -f id:INT64:primary_field -f vector:FLOAT_VECTOR:128 -f color:INT64:color -f brand:ARRAY:64:VARCHAR:128 -p id -A -d 'car_collection'
```
## create partition
@@ -322,11 +307,11 @@ The name of the field to create an index for (vector): vector
Index name: vectorIndex
-Index type (FLAT, IVF_FLAT, IVF_SQ8, IVF_PQ, RNSG, HNSW, ANNOY): IVF_FLAT
+# Default is ''
+Index type FLAT, IVF_FLAT, IVF_SQ8, IVF_PQ, RNSG, HNSW, ANNOY, AUTOINDEX, DISKANN, GPU_IVF_FLAT, GPU_IVF_PQ, SPARSE_INVERTED_INDEX, SPARSE_WAND, SCANN, STL_SORT, Trie, INVERTED, ) []: IVF_FLAT
-Index metric type (L2, IP, HAMMING, TANIMOTO): L2
-
-Index params nlist: 2
+# Default is ''
+Index metric type (L2, IP, HAMMING, TANIMOTO, COSINE, ) []:
Timeout []:
```
@@ -338,20 +323,20 @@ Deletes a user
### Syntax
```shell
-delete alias -p (text)
+delete user -u (text)
```
### Options
| Option | Full name | Description |
| :----- | :--------- | :----------------------------------- |
-| -u | --usernmae | The username. |
+| -u | --username | The username. |
| --help | n/a | Displays help for using the command. |
### Example
```shell
-milvus_cli > delete user -p zilliz
+milvus_cli > delete user -u zilliz
```
## delete alias
@@ -361,7 +346,7 @@ Deletes an alias.
| Command | Description |
| :----------- | :------------------------------------------------------------------------------------------------------------------ |
-| calc | Calculates the distance between two vector arrays, mkts_from_hybridts, mkts_from_unixtime, or hybridts_to_unixtime. |
| clear | Clears the screen. |
| connect | Connects to Milvus. |
-| create | Creates a collection, partition, index, or alias. |
-| delete | Deletes a collection, partition, index, entity, or alias. |
-| describe | Describes a collection, partition, or index. |
+| create | Create collection, database, partition,user and index. |
+| delete | Delete collection, database, partition,alias,user or index. |
| exit | Closes the command line window. |
| help | Displays help for using a command. |
-| import | Imports data into a partition. |
-| list | Lists collections, partitions, or indexes. |
+| insert | Imports data into a partition. |
+| list | List collections,databases, partitions,users or indexes. |
| load | Loads a collection or partition. |
-| load_balance | Performs load balancing on a query node. |
| query | Shows query results that match all the criteria that you enter. |
| release | Releases a collection or partition. |
| search | Performs a vector similarity search or hybrid search. |
-| show | Shows the current collection, progress of entity loading, progress of entity indexing, or segment information. |
+| show | Show connection, database,collection, loading_progress or index_progress. |
+| rename | Rename collection |
+| use | Use database |
| version | Shows the version of Milvus_CLI. |
## import
@@ -601,7 +585,7 @@ Imports local or remote data into a partition.
+import -c (text)[-p (text)]
```
| :----- | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| -c | --collection-name | The name of the collection that the data are inserted into. |
| -p | --partition | (Optional) The name of the partition that the data are inserted into. Not passing this partition option indicates choosing the "\_default" partition. |
-| -t | --timeout | (Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs. |
| --help | n/a | Displays help for using the command. |
Syntax
@@ -65,7 +66,7 @@ This step is optional. If you skip this, ensure that you already have some data
Note that running Milvus Backup against a Milvus instance will not normally affect the running of the instance. Your Milvus instance is fully functional during backup or restore.
--
+
quotaAndLimits.limits.maxCollectionNumPerDBquotaAndLimits.ddl.enabledquotaAndLimits.ddl.collectionRatequotaAndLimits.ddl.partitionRate
@@ -219,10 +220,14 @@ To configure resources for a specific component, add the component name in the f
quotaAndLimits.compaction.maxquotaAndLimits.dml.enabledquotaAndLimits.dml.insertRate.max
+ quotaAndLimits.dml.insertRate.collection.maxquotaAndLimits.dml.deleteRate.max
+ quotaAndLimits.dml.deleteRate.collection.maxquotaAndLimits.dql.enabledquotaAndLimits.dql.searchRate.max
+ quotaAndLimits.dql.searchRate.collection.maxquotaAndLimits.dql.queryRate.max
+ quotaAndLimits.dql.queryRate.collection.maxquotaAndLimits.limitWriting.ttProtection.enabledquotaAndLimits.limitWriting.ttProtection.maxTimeTickDelayquotaAndLimits.limitWriting.memProtection.enabled
@@ -232,6 +237,7 @@ To configure resources for a specific component, add the component name in the f
quotaAndLimits.limitWriting.memProtection.queryNodeMemoryHighWaterLevelquotaAndLimits.limitWriting.diskProtection.enabledquotaAndLimits.limitWriting.diskProtection.diskQuota
+ quotaAndLimits.limitWriting.diskProtection.diskQuotaPerCollectionquotaAndLimits.limitWriting.forceDenyquotaAndLimits.limitReading.queueProtection.enabledquotaAndLimits.limitReading.queueProtection.nqInQueueThreshold
diff --git a/preview/site/en/adminGuide/deploy_etcd.md b/preview/site/en/adminGuide/deploy_etcd.md
index 12b4b6c2c..d78c9b3df 100644
--- a/preview/site/en/adminGuide/deploy_etcd.md
+++ b/preview/site/en/adminGuide/deploy_etcd.md
@@ -47,10 +47,10 @@ See [etcd-related Configurations](configure_etcd.md) for more information.
Run the following command to start Milvus that uses the etcd configurations.
```
-docker-compose up
+docker compose up
```
-- The release name should only contain letters, numbers and dashes. Dots are not allowed in the release name. -
- The default command line installs cluster version of Milvus while installing Milvus with Helm. Further setting is needed while installing Milvus standalone. -
- According to the deprecated API migration guide of Kuberenetes, the policy/v1beta1 API version of PodDisruptionBudget is not longer served as of v1.25. You are suggested to migrate manifests and API clients to use the policy/v1 API version instead.
As a workaround for users who still use the policy/v1beta1 API version of PodDisruptionBudget on Kuberenetes v1.25 and later, you can instead run the following command to install Milvus:
-helm install my-release milvus/milvus --set pulsar.bookkeeper.pdb.usePolicy=false,pulsar.broker.pdb.usePolicy=false,pulsar.proxy.pdb.usePolicy=false,pulsar.zookeeper.pdb.usePolicy=false
- - See Milvus Helm Chart and Helm for more information. -
- When you delete the Milvus cluster using the default configuration, dependencies like etcd, Pulsar, and MinIO are not deleted. Therefore, next time when you install the same Milvus cluster instance, these dependencies will be used again. -
- To delete the dependencies and private virtual clouds (PVCs) along with the Milvus cluster, see configuration file. - -
- Intel 2nd Gen Core CPU or higher
- Apple Silicon
- Standalone: 8 core or more
- Cluster: 16 core or more
- Intel 2nd Gen Core CPU or higher
- Apple Silicon
- Standalone: 4 core or more
- Cluster: 8 core or more
- SSE4.2
- AVX
- AVX2
- AVX-512
- SSE4.2
- AVX
- AVX2
- AVX-512
- Standalone: 16G
- Cluster: 64G
- Standalone: 32G
- Cluster: 128G
- Standalone: 8G
- Cluster: 32G
- Standalone: 16G
- Cluster: 128G
- The release name should only contain letters, numbers and dashes. Dots are not allowed in the release name. +
- The default command line installs cluster version of Milvus while installing Milvus with Helm. Further setting is needed while installing Milvus standalone. +
- According to the deprecated API migration guide of Kuberenetes, the policy/v1beta1 API version of PodDisruptionBudget is not longer served as of v1.25. You are suggested to migrate manifests and API clients to use the policy/v1 API version instead.
As a workaround for users who still use the policy/v1beta1 API version of PodDisruptionBudget on Kuberenetes v1.25 and later, you can instead run the following command to install Milvus:
+helm install my-release milvus/milvus --set pulsar.bookkeeper.pdb.usePolicy=false,pulsar.broker.pdb.usePolicy=false,pulsar.proxy.pdb.usePolicy=false,pulsar.zookeeper.pdb.usePolicy=false
+ - See Milvus Helm Chart and Helm for more information. +
- Kubernetes 1.16 or later
- kubectl
- Helm 3.0.0 or later
- minikube (for Milvus standalone)
- Docker 19.03 or later (for Milvus standalone)
- The release name should only contain letters, numbers and dashes. Dots are not allowed in the release name. +
- The default command line installs cluster version of Milvus while installing Milvus with Helm. Further setting is needed while installing Milvus standalone. +
- According to the deprecated API migration guide of Kubernetes, the policy/v1beta1 API version of PodDisruptionBudget is no longer served as of v1.25. You are suggested to migrate manifests and API clients to use the policy/v1 API version instead.
As a workaround for users who still use the policy/v1beta1 API version of PodDisruptionBudget on Kubernetes v1.25 and later, you can instead run the following command to install Milvus:
+helm install my-release milvus/milvus --set pulsar.bookkeeper.pdb.usePolicy=false,pulsar.broker.pdb.usePolicy=false,pulsar.proxy.pdb.usePolicy=false,pulsar.zookeeper.pdb.usePolicy=false
+ - See Milvus Helm Chart and Helm for more information. +
- Intel 2nd Gen Core CPU or higher
- Apple Silicon
- Standalone: 8 core or more
- Cluster: 16 core or more
- Intel 2nd Gen Core CPU or higher
- Apple Silicon
- Standalone: 4 core or more
- Cluster: 8 core or more
- SSE4.2
- AVX
- AVX2
- AVX-512
- SSE4.2
- AVX
- AVX2
- AVX-512
- Standalone: 16G
- Cluster: 64G
- Standalone: 32G
- Cluster: 128G
- Standalone: 8G
- Cluster: 32G
- Standalone: 16G
- Cluster: 128G
- `skip` refers to skipping data migration if the specified collection or partition already exists.
- `append` refers to appending data if the specified collection or partition already exists.
- `overwrite` refers to deleting existing data before insertion if the specified collection or partition already exists. | 'append' | -| `dest_collection_name` | Name of the collection to import data to. | 'test' | -| `dest_partition_name` (optional) | Name of the partition to import data to. | 'partition' | -| `collection_parameter` | Collection-specific information including vector dimension, index file size, and similarity metric. | "dimension: 512
- Billion-scale similarity search with little performance degradation
- Linear scalability when using multiple replicas -
- High-speed query +
- Very high-speed query
- Requires a recall rate as high as possible
- Large memory resources
- Low-dimensional vectors +
- Very high-speed query +
- Requires a recall rate as high as possible +
- Large memory resources
- Depends on relatively small datasets. +
- Requires perfect accuracy. +
- No compression applies. +
- Guarantee exact search results. +
- High-speed query +
- Requires a recall rate as high as possible +
- Depends on relatively small datasets. +
- Requires a 100% recall rate. +
- Weak-AND algorithm accelerated +
- Can get a significant speed improvement while only sacrificing a small amount of recall. +
- Relatively small dataset -
- Requires a 100% recall rate -
- High-speed query -
- Requires a recall rate as high as possible -
- Euclidean distance (L2)
- Inner product (IP)
- FLAT
- IVF_FLAT
- IVF_SQ8
- IVF_PQ
- HNSW
- IVF_HNSW
- RHNSW_FLAT
- RHNSW_SQ
- RHNSW_PQ
- ANNOY
- Euclidean distance (L2)
- Inner product (IP)
- Cosine similarity (COSINE)
- FLAT
- IVF_FLAT
- IVF_SQ8
- IVF_PQ
- GPU_IVF_FLAT
- GPU_IVF_PQ
- HNSW
- DISKANN
- Jaccard
- Tanimoto
- Hamming
- Jaccard
- Hamming
- BIN_FLAT
- BIN_IVF_FLAT
- Superstructure
- Substructure
- SPARSE_INVERTED_INDEX
- SPARSE_WAND
- Root prefix of the key to where Milvus stores data in etcd. -
- Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
- It is recommended to change this parameter before starting Milvus for the first time. -
- Set an easy-to-identify root key prefix for Milvus if etcd service already exists. +
- To share an etcd instance among multiple Milvus instances, consider changing this to a different value for each Milvus instance before you start them. For details, see Operation FAQs. +
- Set an easy-to-identify root path for Milvus if etcd service already exists. +
- Changing this for an already running Milvus instance may result in failures to read legacy data.
- TCP/IP address of index coordinator. -
- Index coordinator monitors all IPv4 addresses if this parameter is set as
0.0.0.0.
- - The maximum size of each RPC request that the index coord can receive. -
- Unit: Byte -
- The maximum size of each respond the index coord can send when receiving an RPC request. -
- Unit: Byte -
- The maximum size of each respond that the index coord can receive when sending an RPC request. -
- Unit: Byte -
- The maximum size of each RPC request that the index coord can send. -
- Unit: Byte -
- Producer Client ID of Kafka service. +
- Consumer Client ID of Kafka service. +
- List of broker addresses of Kafka service. +
- Each value is separated from each other by comma +
- Eg: localhost1:9092,localhost2:9092,localhost3:9092 +
- Username of Kafka service if
Simple Authentication and Security Layeris enabled.
+ - Password of Kafka service if
Simple Authentication and Security Layeris enabled.
+ - Mechanisms of Kafka service if
Simple Authentication and Security Layeris enabled.
+ - Valid values maybe: GSSAPI, PLAIN, SCRAM, OAUTHBEARER. +
- Security protocol of Kafka service. +
- Valid values maybe: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. +
- To share a Pulsar instance with multi-tenancy enabled among multiple Milvus instances, you need to change
pulsar.tenantorpulsar.namespaceto a unique value for each of the Milvus instances.
+ - To share a Pulsar instance with multi-tenancy disabled among multiple Milvus instances, you need to change
msgChannel.chanNamePrefix.clusterto a unique value for each of the Milvus instances.
+For details, refer to Operation FAQs.
+ - Root name prefix of the channel when a message channel is created. -
- Caution: Changing this parameter after using Milvus for a period of time will affect your access to old data.
- It is recommended to change this parameter before starting Milvus for the first time. +
- To share a Pulsar instance among multiple Milvus instances, consider changing this to a name rather than the default one for each Milvus instance before you start them. For details, see Operational FAQs.
- Access key ID that MinIO or S3 issues to user for authorized access. -
- Environment variable:
MINIO_ACCESS_KEY
+ - Environment variable:
MINIO_ACCESS_KEY_IDorminio.accessKeyID minio.accessKeyIDandminio.secretAccessKeytogether are used for identity authentication to access the MinIO or S3 service.
- - This configuration must be set identical to the environment variable
MINIO_ACCESS_KEY, which is necessary for starting MinIO or S3.
+ - This configuration must be set identical to the environment variable
MINIO_ACCESS_KEY_ID, which is necessary for starting MinIO or S3. - The default value applies to MinIO or S3 service that started with the default docker-compose.yml file.
- Secret key used to encrypt the signature string and verify the signature string on server. It must be kept strictly confidential and accessible only to the MinIO or S3 server and users. -
- Environment variable:
MINIO_SECRET_KEY
+ - Environment variable:
MINIO_SECRET_ACCESS_KEYorminio.secretAccessKey minio.accessKeyIDandminio.secretAccessKeytogether are used for identity authentication to access the MinIO or S3 service.
- - This configuration must be set identical to the environment variable
MINIO_SECRET_KEY, which is necessary for starting MinIO or S3.
+ - This configuration must be set identical to the environment variable
MINIO_SECRET_ACCESS_KEY, which is necessary for starting MinIO or S3. - The default value applies to MinIO or S3 service that started with the default docker-compose.yml file.
- Switch value to control if to access the MinIO or S3 service through SSL. -
- Milvus 2.0.0 does not support secure access to MinIO or S3 service. Future releases will support secure access to MinIO.
- Name of the bucket where Milvus stores data in MinIO or S3.
- Milvus 2.0.0 does not support storing data in multiple buckets.
- Bucket with this name will be created if it does not exist. If the bucket already exists and is accessible, it will be used directly. Otherwise, there will be an error. +
- To share an MinIO instance among multiple Milvus instances, consider changing this to a different value for each Milvus instance before you start them. For details, see Operation FAQs.
- The data will be stored in the local Docker if Docker is used to start the MinIO service locally. Ensure that there is sufficient storage space.
- A bucket name is globally unique in one MinIO or S3 instance.
Configurations only take effect after Milvus starts. See Start Milvus for more information.
+Configurations only take effect after Milvus starts. See Start Milvus for more information.
## Configure etcd on K8s
@@ -59,21 +59,29 @@ For Milvus clusters on K8s, you can configure etcd in the same command that star
The following table lists the keys for configuring etcd in the YAML file.
| Key | Description | Value |
| --------------------- | ------------------------------------ | ------------------------------------ |
-| externalEtcd.enabled | Enables or disables etcd. | true/false |
-| externalEtcd.endpoints | The endpoint to access etcd. | |
+| etcd.enabled | Enables or disables etcd. | true/false |
+| externalEtcd.enabled | Enables or disables external etcd. | true/false |
+| externalEtcd.endpoints | The endpoint to access etcd. | |
### Using the YAML file
-1. Configure the externaletcd section using your values in the values.yaml file.
+1. Configure the etcd section using your values in the values.yaml file.
+
+```yaml
+etcd:
+ enabled: false
+```
+
+2. Configure the externaletcd section using your values in the values.yaml file.
```yaml
externalEtcd:
enabled: true
## the endpoints of the external etcd
endpoints:
- - localhost:2379
+ - values.yaml file, run the following command to install Milvus that uses the etcd configurations.
@@ -86,7 +94,7 @@ helm install Configurations only take effect after Milvus starts. See Start Milvus for more information.
+Configurations only take effect after Milvus starts. See Start Milvus for more information.
-## Configure Pulsar on K8s
+## Configure Pulsar with Helm
-### Configure Pulsar on K8s
+For Milvus clusters on K8s, you can configure Pulsar in the same command that starts Milvus. Alternatively, you can configure Pulsar using the values.yml file on the /charts/milvus path in the [milvus-helm](https://github.com/milvus-io/milvus-helm) repository before you start Milvus.
-For Milvus clusters on K8s, you can configure Pulsar in the same command that starts Milvus. Alternatively, you can configure Pulsar using the values.yml file on the /charts/milvus path in the [milvus-helm](https://github.com/milvus-io/milvus-helm) repository before you start Milvus.
+For details on how to configure Milvus using Helm, refer to [Configure Milvus with Helm Charts](configure-helm.md). For details on Pulsar-related configuration items, refer to [Pulsar-related configurations](configure_pulsar.md).
+ |
+### Using the YAML file
- The following table lists the keys for configuring Pulsar in the YAML file.
-| Key | Description | Value |
-| --------------------- | ------------------------------------ | ------------------------------------ |
-| externalpulsar.enabled | Enables or disables Pulsar. | true/false |
-| externalpulsar.host | The endpoint to access Pulsar. | |
-| externalpulsar.port | The port to access Pulsar. | |
+1. Configure the externalConfigFiles section in the values.yaml file.
-#### Using the YAML file
+```yaml
+extraConfigFiles:
+ user.yaml: |+
+ pulsar:
+ address: localhost # Address of pulsar
+ port: 6650 # Port of Pulsar
+ webport: 80 # Web port of pulsar, if you connect direcly without proxy, should use 8080
+ maxMessageSize: 5242880 # 5 * 1024 * 1024 Bytes, Maximum size of each message in pulsar.
+ tenant: public
+ namespace: default
+```
-1. Configure the pulsar section in the values.yaml file.
+2. After configuring the preceding sections and saving the values.yaml file, run the following command to install Milvus which uses the Pulsar configurations.
-```yaml
-pulsar:
- enabled: false
+```shell
+helm install externalPulsar section using your values in the values.yaml file.
+## Configure Kafka with Helm
+
+For Milvus clusters on K8s, you can configure Kafka in the same command that starts Milvus. Alternatively, you can configure Kafka using the values.yml file on the /charts/milvus path in the [milvus-helm](https://github.com/milvus-io/milvus-helm) repository before you start Milvus.
+
+For details on how to configure Milvus using Helm, refer to [Configure Milvus with Helm Charts](configure-helm.md). For details on Pulsar-related configuration items, refer to [Kafka-related configurations](configure_kafka.md).
+
+### Using the YAML file
+
+1. Configure the externalConfigFiles section in the values.yaml file if you want to use Kafka as the message storage system.
```yaml
-externalPulsar:
- enabled: true
- host: values.yaml file, run the following command to install Milvus that uses the Pulsar configurations.
+2. After configuring the preceding sections and saving the values.yaml file, run the following command to install Milvus that uses the Kafka configurations.
```shell
helm install
-For Milvus clusters on K8s, you can configure Kafka in the same command that starts Milvus. Alternatively, you can configure Kafka using the
- The following table lists the keys for configuring Pulsar in the YAML file.
-| Key | Description | Value |
-| --------------------- | ------------------------------------ | ------------------------------------ |
-| values.yml file on the /charts/milvus path in the [milvus-helm](https://github.com/milvus-io/milvus-helm) repository before you start Milvus.
+Changing the message store is not recommended. If this is you want to do this, stop all DDL operations, then call the FlushAll API to flush all collections, and finally stop Milvus in the end before you actually change the message store.
+
+externalkafka.enabled | Enables or disables Kafka. | true/false |
-| externalkafka.brokerlist | The brokerlist to access Kafka. | |
+## Configure NATS with Helm
-The following table lists the mandatory configurations for external Kafka. Set them in Kafka configurations.
-| Key | Description | Value |
-| --------------------- | ------------------------------------ | ------------------------------------ |
-| `max.request.size` | The maximum size of a request in bytes. | `5242880` |
-| `message.max.bytes` | The largest record batch size allowed by Kafka | `10485760` |
-| `auto.create.topics.enable` | Enable auto creation of topic on the server | `true` |
-| `num.partitions` | The default number of log partitions per topic | `1` |
+NATS is an experimental message store alternative to RocksMQ. For detailed steps on how to configure Milvus with Helm, refer to [Configure Milvus with Helm Charts](configure-helm.md). For details on RocksMQ-related configuration items, refer to [NATS-related configurations](configure_nats.md).
-#### Using the YAML file
+- If you start Milvus with NATS and want to change its settings, you can run `helm upgrade -f ` with the changed settings in the following YAML file.
-1. Configure the kafka section in the values.yaml file.
+- If you have installed Milvus standalone with a message store other than NATS and want to change it to NATS, run `helm upgrade -f ` with the following YAML file after you flushed all collections and stopped Milvus.
```yaml
-kafka:
- enabled: false
- name: kafka
- replicaCount: 3
- image:
- repository: bitnami/kafka
- tag: 3.1.0-debian-10-r52
+extraConfigFiles:
+ user.yaml: |+
+ mq:
+ type: natsmq
+ natsmq:
+ # server side configuration for natsmq.
+ server:
+ # 4222 by default, Port for nats server listening.
+ port: 4222
+ # /var/lib/milvus/nats by default, directory to use for JetStream storage of nats.
+ storeDir: /var/lib/milvus/nats
+ # (B) 16GB by default, Maximum size of the 'file' storage.
+ maxFileStore: 17179869184
+ # (B) 8MB by default, Maximum number of bytes in a message payload.
+ maxPayload: 8388608
+ # (B) 64MB by default, Maximum number of bytes buffered for a connection applies to client connections.
+ maxPending: 67108864
+ # (√ms) 4s by default, waiting for initialization of natsmq finished.
+ initializeTimeout: 4000
+ monitor:
+ # false by default, If true enable debug log messages.
+ debug: false
+ # true by default, If set to false, log without timestamps.
+ logTime: true
+ # no log file by default, Log file path relative to.. .
+ logFile:
+ # (B) 0, unlimited by default, Size in bytes after the log file rolls over to a new one.
+ logSizeLimit: 0
+ retention:
+ # (min) 3 days by default, Maximum age of any message in the P-channel.
+ maxAge: 4320
+ # (B) None by default, How many bytes the single P-channel may contain. Removing oldest messages if the P-channel exceeds this size.
+ maxBytes:
+ # None by default, How many message the single P-channel may contain. Removing oldest messages if the P-channel exceeds this limit.
+ maxMsgs:
```
-2. Configure the externalKafka section using your values in the values.yaml file.
+
-```yaml
-externalKafka:
- enabled: false
- brokerList:
-```
+**Choose between RocksMQ and NATS?**
-3. After configuring the preceding sections and saving the milvus/milvus -f values.yaml
-```
-#### Using a command
+In the scenario where the data packet is smaller than 64 kb, RocksDB outperforms in terms of memory usage, CPU usage, and response time. On the other hand, if the data packet is greater than 64 kb, NATS excels in terms of response time with sufficient memory and ideal GC scheduling.
-To install Milvus and configure Kafka, run the following command using your values.
+Currently, you are advised to use NATS only for experiments.
-```shell
-helm install milvus/milvus --set cluster.enabled=true --set externalKafka.enabled=true --set externalKafka.brokerlist=''
-```
+
## What's next
diff --git a/preview/site/en/adminGuide/deploy_s3.md b/preview/site/en/adminGuide/deploy_s3.md
index ea08081ed..40058d582 100644
--- a/preview/site/en/adminGuide/deploy_s3.md
+++ b/preview/site/en/adminGuide/deploy_s3.md
@@ -27,13 +27,15 @@ minio:
```
See [MinIO/S3 Configurations](configure_minio.md) for more information.
+### 2. Refine docker-compose.yaml
+You'd also remove the `MINIO_ADDRESS` environment variable for milvus service at `docker-compose.yaml`. By default milvus will use local minio instead of external S3.
-### 2. Run Milvus
+### 3. Run Milvus
Run the following command to start Milvus that uses the S3 configurations.
```shell
-docker-compose up
+docker compose up
```
-values.yaml file, run the following command to install Milvus that uses the Kafka configurations.
+RockMQ uses CGO to interact with RocksDB and manages the memory by itself, while the pure-GO NATS embedded in the Milvus installation delegates its memory management to Go's garbage collector (GC).
-```shell
-helm install Configurations only take effect after Milvus starts. See Start Milvus for more information.
+Configurations only take effect after Milvus starts. See Start Milvus for more information.
## Configure S3 on K8s
@@ -42,13 +44,15 @@ For Milvus clusters on K8s, you can configure S3 in the same command that starts
The following table lists the keys for configuring S3 in the YAML file.
| Key | Description | Value |
| --------------------- | ------------------------------------ | ------------------------------------ |
-| externalS3.enabled | Enables or disables S3. | true/false |
-| externalS3.host | The endpoint to access S3. | |
-| externalS3.port | The port to access S3. | |
-| externalS3.accessKey | The access key ID for S3. | |
-| externalS3.secretKey | The secret access key for S3. | |
-| externalS3.bucketName | The name of the S3 bucket. | |
-| minio.enabled | Enables or disables MinIO. | true/false |
+| minio.enabled | Enables or disables MinIO. | true/false |
+| externalS3.enabled | Enables or disables S3. | true/false |
+| externalS3.host | The endpoint to access S3. | |
+| externalS3.port | The port to access S3. | |
+| externalS3.rootPath | The root path of the S3 storage. | An emtpy string by default. |
+| externalS3.accessKey | The access key ID for S3. | |
+| externalS3.secretKey | The secret access key for S3. | |
+| externalS3.bucketName | The name of the S3 bucket. | |
+| externalS3.useSSL | Whether to use SSL when connecting | The values defaults to false |
### Using the YAML file
@@ -83,7 +87,7 @@ helm install
+
+- Examples in this guide change the value of `proxy.minPasswordLength` to `8`. You can replace the key with the applicable ones listed in [Applicable configuration items](dynamic_config.md#Applicable-configuration-items).
+- Examples in this guide assume that the root path of your Milvus is `by-dev`. All configurations are listed under the path `by-dev/config`. The Milvus root path varies with the way you install it. For the instances installed using the Helm charts, the root path defaults to `by-dev`. If you do not know the root path, refer to [Connect to etcd](birdwatcher_usage_guides.md#Connect-to-etcd).
+
+
+
+## Change configurations
+
+On Milvus, `proxy.minPasswordLength` is set to `6` by default. To change this value, you can do as follows:
+
+```shell
+$ etcdctl put by-dev/config/proxy/minPasswordLength 8
+# or
+$ birdwatcher -olc "#connect --etcd 127.0.0.1:2379 --rootPath=by-dev,set config-etcd --key by-dev/config/proxy/minPasswordLength --value 8"
+```
+
+Then you can check the configurations as follows:
+
+```shell
+$ etcdctl get by-dev/config/proxy/minPasswordLength
+```
+
+## Roll back configurations
+
+Milvus also allows you to roll back your configurations in case the changed value no longer applies.
+
+```shell
+$ etcdctl del by-dev/config/proxy/minPasswordLength
+# or
+$ birdwatcher -olc "#connect --etcd 127.0.0.1:2379 --rootPath=by-dev,remove config-etcd --key by-dev/config/proxy/minPasswordLength"
+```
+
+Then you can check the configurations as follows:
+
+```shell
+$ etcdctl get by-dev/config/proxy/minPasswordLength
+```
+
+## View configurations
+
+Instead of viewing the value of a specific configuration item, you can also list all of them.
+
+```shell
+$ etcdctl get --prefix by-dev/config
+# or
+$ birdwatcher -olc "#connect --etcd 127.0.0.1:2379 --rootPath=by-dev,show config-etcd"
+```
+
+To view the configurations of a specific node:
+
+```shell
+Offline > connect --etcd ip:port
+Milvus(by-dev) > show session # List all nodes with their server ID
+Milvus(by-dev) > visit querycoord 1 # Visit a node by server ID
+QueryCoord-1(ip:port) > configuration # List the configuration of the node
+```
+
+## Applicable configuration items
+
+Currently, you can change the following configuration items on the fly.
+
+ | Configuration item | Default value |
+ |-------------------------------------------------------------------------|---------------------|
+ | pulsar.maxMessageSize | 5242880 |
+ | common.retentionDuration | 86400 |
+ | common.entityExpiration | -1 |
+ | common.gracefulTime | 5000 |
+ | common.gracefulStopTimeout | 30 |
+ | quotaAndLimits.ddl.enabled | FALSE |
+ | quotaAndLimits.indexRate.enabled | FALSE |
+ | quotaAndLimits.flushRate.enabled | FALSE |
+ | quotaAndLimits.compactionRate.enabled | FALSE |
+ | quotaAndLimits.dml.enabled | FALSE |
+ | quotaAndLimits.dql.enabled | FALSE |
+ | quotaAndLimits.limits.collection.maxNum | 64 |
+ | quotaAndLimits.limitWriting.forceDeny | FALSE |
+ | quotaAndLimits.limitWriting.ttProtection.enabled | FALSE |
+ | quotaAndLimits.limitWriting.ttProtection.maxTimeTickDelay | 9223372036854775807 |
+ | quotaAndLimits.limitWriting.memProtection.enabled | TRUE |
+ | quotaAndLimits.limitWriting.memProtection.dataNodeMemoryLowWaterLevel | 0.85 |
+ | quotaAndLimits.limitWriting.memProtection.dataNodeMemoryHighWaterLevel | 0.95 |
+ | quotaAndLimits.limitWriting.memProtection.queryNodeMemoryLowWaterLevel | 0.85 |
+ | quotaAndLimits.limitWriting.memProtection.queryNodeMemoryHighWaterLevel | 0.95 |
+ | quotaAndLimits.limitWriting.diskProtection.enabled | TRUE |
+ | quotaAndLimits.limitWriting.diskProtection.diskQuota | +INF |
+ | quotaAndLimits.limitReading.forceDeny | FALSE |
+ | quotaAndLimits.limitReading.queueProtection.enabled | FALSE |
+ | quotaAndLimits.limitReading.queueProtection.nqInQueueThreshold | 9223372036854775807 |
+ | quotaAndLimits.limitReading.queueProtection.queueLatencyThreshold | +INF |
+ | quotaAndLimits.limitReading.resultProtection.enabled | FALSE |
+ | quotaAndLimits.limitReading.resultProtection.maxReadResultRate | +INF |
+ | quotaAndLimits.limitReading.coolOffSpeed | 0.9 |
+ | autoIndex.enable | FALSE |
+ | autoIndex.params.build | "" |
+ | autoIndex.params.extra | "" |
+ | autoIndex.params.search | "" |
+ | proxy.maxNameLength | 255 |
+ | proxy.maxUsernameLength | 32 |
+ | proxy.minPasswordLength | 6 |
+ | proxy.maxPasswordLength | 256 |
+ | proxy.maxFieldNum | 64 |
+ | proxy.maxShardNum | 256 |
+ | proxy.maxDimension | 32768 |
+ | proxy.maxUserNum | 100 |
+ | proxy.maxRoleNum | 10 |
+ | queryNode.enableDisk | TRUE |
+ | dataCoord.segment.diskSegmentMaxSize | 2048 |
+ | dataCoord.compaction.enableAutoCompaction | TRUE |
+
+
+## What's next
+
+- Learn more about [System Configurations](system_configuration.md).
+- Learn how to configure Milvus installed using [Milvus Operator](configure_operator.md), [Helm charts](configure-helm.md), and [Docker](configure-docker.md).
diff --git a/preview/site/en/adminGuide/limit_collection_counts.md b/preview/site/en/adminGuide/limit_collection_counts.md
new file mode 100644
index 000000000..6438527a3
--- /dev/null
+++ b/preview/site/en/adminGuide/limit_collection_counts.md
@@ -0,0 +1,51 @@
+---
+id: limit_collection_counts.md
+title: Set Limits on Collection Number
+---
+
+# Limit Collection Counts
+
+A Milvus instance allows up to 65,536 collections. However, too many collections may result in performance issues. Therefore, it is recommended to limit the number of collections created in a Milvus instance.
+
+This guide provides instructions on how to set limits on the number of collections in a Milvus instance.
+
+Configuration varies with the way you install the Milvus instance.
+
+- For Milvus instances installed using Helm Charts
+
+ Add the configuration to the `values.yaml` file under the `config` section. For details, refer to [Configure Milvus with Helm Charts](configure-helm.md).
+
+- For Milvus instances installed using Docker Compose
+
+ Add the configuration to the `milvus.yaml` file you have used to start the Milvus instance. For details, refer to [Configure Milvus with Docker Compose](configure-docker.md).
+
+- For Milvus instances installed using Operator
+
+ Add the configuration to the `spec.components` section of the `Milvus` custom resource. For details, refer to [Configure Milvus with Operator](configure_operator.md).
+
+## Configuration options
+
+```yaml
+rootCoord:
+ maxGeneralCapacity: 1024
+```
+
+The `maxGeneralCapacity` parameter sets the maximum number of collections that the current Milvus instance can hold. The default value is `1024`.
+
+## Calculating the number of collections
+
+In a collection, you can set up multiple shards and partitions. Shards are logical units used to distribute data write operations among multiple data nodes. Partitions are logical units used to improve data retrieval efficiency by loading only a subset of collection data. When calculating the number of collections in the current Milvus instance, you also need to count the shards and partitions.
+
+For example, let's assume you have already created **100** collections, with **2** shards and **4** partitions in **60** of them and with **1** shard and **12** partitions in the rest **40** collections. The current number of collections can be calculated as:
+
+```
+60 (collections) x 2 (shards) x 4 (partitions) + 40 (collections) x 1 (shard) x 12 (partitions) = 960
+```
+
+In the above example, you have already used **960** out of the default limits. Now if you want to create a new collection with **4** shards and **20** partitions, you will receive the following error prompt because the total number of collections exceeds the maximum capacity:
+
+```shell
+failed checking constraint: sum_collections(parition*shard) exceeding the max general capacity:
+```
+
+To avoid this error, you can either reduce the number of shards or partitions in existing or new collections, delete some collections, or increase the `maxGeneralCapacity` value.
diff --git a/preview/site/en/adminGuide/m2m.md b/preview/site/en/adminGuide/m2m_legacy.md
similarity index 99%
rename from preview/site/en/adminGuide/m2m.md
rename to preview/site/en/adminGuide/m2m_legacy.md
index afa6a39cc..92ab8ee0e 100644
--- a/preview/site/en/adminGuide/m2m.md
+++ b/preview/site/en/adminGuide/m2m_legacy.md
@@ -1,8 +1,9 @@
---
-id: m2m.md
+id: m2m_legacy.md
title: Version Migration
related_key: version migration
summary: Use MilvusDM for version migration.
+deprecate: true
---
# Version Migration
diff --git a/preview/site/en/adminGuide/manage_dependencies.md b/preview/site/en/adminGuide/manage_dependencies.md
index ffa26e858..773c60475 100644
--- a/preview/site/en/adminGuide/manage_dependencies.md
+++ b/preview/site/en/adminGuide/manage_dependencies.md
@@ -1,6 +1,7 @@
---
id: manage_dependencies.md
title: Manage Dependencies
+deprecate: true
---
# Manage Dependencies
diff --git a/preview/site/en/adminGuide/message_storage_operator.md b/preview/site/en/adminGuide/message_storage_operator.md
index 3dfbd02fa..2f927a1b2 100644
--- a/preview/site/en/adminGuide/message_storage_operator.md
+++ b/preview/site/en/adminGuide/message_storage_operator.md
@@ -7,7 +7,7 @@ summary: Learn how to configure message storage with Milvus Operator.
# Configure Message Storage with Milvus Operator
-Milvus uses RocksMQ, Pulsar or Kafka for managing logs of recent changes, outputting stream logs, and providing log subscriptions. This topic introduces how to configure message storage dependencies when you install Milvus with Milvus Operator.
+Milvus uses RocksMQ, Pulsar or Kafka for managing logs of recent changes, outputting stream logs, and providing log subscriptions. This topic introduces how to configure message storage dependencies when you install Milvus with Milvus Operator. For more details, refer to [Configure Message Storage with Milvus Operator](https://github.com/zilliztech/milvus-operator/blob/main/docs/administration/manage-dependencies/message-storage.md) in the Milvus Operator repository.
This topic assumes that you have deployed Milvus Operator.
@@ -16,31 +16,34 @@ This topic assumes that you have deployed Milvus Operator.
You need to specify a configuration file for using Milvus Operator to start a Milvus cluster.
```YAML
-kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/config/samples/milvuscluster_default.yaml
+kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_cluster_default.yaml
```
-You only need to edit the code template in `milvuscluster_default.yaml` to configure third-party dependencies. The following sections introduce how to configure object storage, etcd, and Pulsar respectively.
+You only need to edit the code template in `milvus_cluster_default.yaml` to configure third-party dependencies. The following sections introduce how to configure object storage, etcd, and Pulsar respectively.
## Before you begin
-The table below shows whether RocksMQ, Pulsar, and Kafka are supported in Milvus standalone and cluster mode.
+The table below shows whether RocksMQ, NATS, Pulsar, and Kafka are supported in Milvus standalone and cluster mode.
-| | RocksMQ | Pulsar | Kafka |
-|:---------------:|:-------:|:------:|:-----:|
-| Standalone mode | ✔️ | ✔️ | ✔️ |
-| Cluster mode | ✖️ | ✔️ | ✔️ |
+| | RocksMQ | NATS | Pulsar | Kafka |
+|:---------------:|:-------:|:------:|:------:|:-----:|
+| Standalone mode | ✔️ | ✔️ | ✔️ | ✔️ |
+| Cluster mode | ✖️ | ✖️ | ✔️ | ✔️ |
There are also other limitations for specifying the message storage:
- Only one message storage for one Milvus instance is supported. However we still have backward compatibility with multiple message storages set for one instance. The priority is as follows:
- standalone mode: RocksMQ (default) > Pulsar > Kafka
- cluster mode: Pulsar (default) > Kafka
+ - Nats introduced in 2.3 do not participate in these priority rules for backward compatibility.
- The message storage cannot be changed while the Milvus system is running.
-- Only Kafka 2.x or 3.x verison is supported.
+- Only Kafka 2.x or 3.x verison is supported.
## Configure RocksMQ
RocksMQ is the default message storage in Milvus standalone.
+
Currently, you can only configure RocksMQ as the message storage for Milvus standalone with Milvus Operator.
+
#### Example
@@ -54,9 +57,84 @@ metadata:
name: milvus
spec:
dependencies: {}
+ components: {}
+ config: {}
+```
+
+## Configure NATS
+
+NATS is an alternative message storage for NATS.
+
+#### Example
+
+The following example configures a NATS service.
+
+```YAML
+apiVersion: milvus.io/v1alpha1
+kind: Milvus
+metadata:
+ name: milvus
+spec:
+ dependencies:
+ msgStreamType: 'natsmq'
+ natsmq:
+ # server side configuration for natsmq.
+ server:
+ # 4222 by default, Port for nats server listening.
+ port: 4222
+ # /var/lib/milvus/nats by default, directory to use for JetStream storage of nats.
+ storeDir: /var/lib/milvus/nats
+ # (B) 16GB by default, Maximum size of the 'file' storage.
+ maxFileStore: 17179869184
+ # (B) 8MB by default, Maximum number of bytes in a message payload.
+ maxPayload: 8388608
+ # (B) 64MB by default, Maximum number of bytes buffered for a connection applies to client connections.
+ maxPending: 67108864
+ # (√ms) 4s by default, waiting for initialization of natsmq finished.
+ initializeTimeout: 4000
+ monitor:
+ # false by default, If true enable debug log messages.
+ debug: false
+ # true by default, If set to false, log without timestamps.
+ logTime: true
+ # no log file by default, Log file path relative to.. .
+ logFile:
+ # (B) 0, unlimited by default, Size in bytes after the log file rolls over to a new one.
+ logSizeLimit: 0
+ retention:
+ # (min) 3 days by default, Maximum age of any message in the P-channel.
+ maxAge: 4320
+ # (B) None by default, How many bytes the single P-channel may contain. Removing oldest messages if the P-channel exceeds this size.
+ maxBytes:
+ # None by default, How many message the single P-channel may contain. Removing oldest messages if the P-channel exceeds this limit.
+ maxMsgs:
+ components: {}
config: {}
```
+To migrate the message storage from RocksMQ to NATS, do as follows:
+
+1. Stop all DDL operations.
+2. Call the FlushAll API and then stop Milvus once the API call finishes executing.
+3. Change `msgStreamType` to `natsmq` and make necessary changes to NATS settings in `spec.dependencies.natsmq`.
+4. Start Milvus again and check whether:
+
+ - A log entry that reads `mqType=natsmq` is present in the logs.
+ - A directory named `jetstream` is present in the directory specified in `spec.dependencies.natsmq.server.storeDir`.
+
+5. (Optional) Back up and clean up the data files in the RocksMQ storage directory.
+
+
+
+**Choose between RocksMQ and NATS?**
+
+RockMQ uses CGO to interact with RocksDB and manages the memory by itself, while the pure-GO NATS embedded in the Milvus installation delegates its memory management to Go's garbage collector (GC).
+
+In the scenario where the data packet is smaller than 64 kb, RocksDB outperforms in terms of memory usage, CPU usage, and response time. On the other hand, if the data packet is greater than 64 kb, NATS excels in terms of response time with sufficient memory and ideal GC scheduling.
+
+Currently, you are advised to use NATS only for experiments.
+
+
## Configure Pulsar
@@ -78,37 +156,21 @@ The following example configures an external Pulsar service.
```YAML
apiVersion: milvus.io/v1alpha1
-
kind: MilvusCluster
-
metadata:
-
name: my-release
-
labels:
-
app: milvus
-
spec:
-
dependencies: # Optional
-
pulsar: # Optional
-
# Whether (=true) to use an existed external pulsar as specified in the field endpoints or
-
# (=false) create a new pulsar inside the same kubernetes cluster for milvus.
-
external: true # Optional default=false
-
# The external pulsar endpoints if external=true
-
endpoints:
-
- 192.168.1.1:6650
-
components: {}
-
config: {}
```
@@ -122,77 +184,41 @@ The following example configures an internal Pulsar service.
```YAML
apiVersion: milvus.io/v1alpha1
-
kind: MilvusCluster
-
metadata:
-
name: my-release
-
labels:
-
app: milvus
-
spec:
-
dependencies:
-
pulsar:
-
inCluster:
-
values:
-
components:
-
autorecovery: false
-
zookeeper:
-
replicaCount: 1
-
bookkeeper:
-
replicaCount: 1
-
resoureces:
-
limit:
-
cpu: '4'
-
memory: 8Gi
-
requests:
-
cpu: 200m
-
memory: 512Mi
-
broker:
-
replicaCount: 1
-
configData:
-
## Enable `autoSkipNonRecoverableData` since bookkeeper is running
-
## without persistence
-
autoSkipNonRecoverableData: "true"
-
managedLedgerDefaultEnsembleSize: "1"
-
managedLedgerDefaultWriteQuorum: "1"
-
managedLedgerDefaultAckQuorum: "1"
-
proxy:
-
replicaCount: 1
-
components: {}
-
config: {}
```
@@ -225,15 +251,25 @@ Fields used to configure an external Kafka service include:
The following example configures an external Kafka service.
-```
+```yaml
apiVersion: milvus.io/v1alpha1
kind: MilvusCluster
metadata:
name: my-release
labels:
app: milvus
-spec:
+spec:
+ config:
+ kafka:
+ # securityProtocol supports: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL
+ securityProtocol: PLAINTEXT
+ # saslMechanisms supports: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512
+ saslMechanisms: PLAIN
+ saslUsername: ""
+ saslPassword: ""
+ # Omit other fields ...
dependencies:
+ # Omit other fields ...
msgStreamType: "kafka"
kafka:
external: true
@@ -241,9 +277,8 @@ spec:
- "kafkaBrokerAddr1:9092"
- "kafkaBrokerAddr2:9092"
# ...
- components: {}
- config: {}
```
+> SASL configurations are supported in operator v0.8.5 or higher version.
### Internal Kafka
diff --git a/preview/site/en/adminGuide/meta_storage_operator.md b/preview/site/en/adminGuide/meta_storage_operator.md
index dfc924c23..8001a0eb1 100644
--- a/preview/site/en/adminGuide/meta_storage_operator.md
+++ b/preview/site/en/adminGuide/meta_storage_operator.md
@@ -7,7 +7,7 @@ summary: Learn how to configure meta storage with Milvus Operator.
# Configure Meta Storage with Milvus Operator
-Milvus uses etcd for storing metadata. This topic introduces how to configure meta storage dependency when you install Milvus with Milvus Operator.
+Milvus uses etcd for storing metadata. This topic introduces how to configure meta storage dependency when you install Milvus with Milvus Operator. For more details, refer to [Configure Meta Storage with Milvus Operator](https://github.com/zilliztech/milvus-operator/blob/main/docs/administration/manage-dependencies/meta-storage.md) in the Milvus Operator repository.
This topic assumes that you have deployed Milvus Operator.
@@ -16,10 +16,10 @@ This topic assumes that you have deployed Milvus Operator.
You need to specify a configuration file for using Milvus Operator to start a Milvus cluster.
```YAML
-kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/config/samples/milvuscluster_default.yaml
+kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_cluster_default.yaml
```
-You only need to edit the code template in `milvuscluster_default.yaml` to configure third-party dependencies. The following sections introduce how to configure object storage, etcd, and Pulsar respectively.
+You only need to edit the code template in `milvus_cluster_default.yaml` to configure third-party dependencies. The following sections introduce how to configure object storage, etcd, and Pulsar respectively.
## Configure etcd
@@ -41,36 +41,20 @@ The following example configures an external etcd service.
```YAML
kind: MilvusCluster
-
metadata:
-
name: my-release
-
labels:
-
app: milvus
-
-
spec:
-
dependencies: # Optional
-
etcd: # Optional
-
# Whether (=true) to use an existed external etcd as specified in the field endpoints or
-
# (=false) create a new etcd inside the same kubernetes cluster for milvus.
-
external: true # Optional default=false
-
# The external etcd endpoints if external=true
-
endpoints:
-
- 192.168.1.1:2379
-
components: {}
-
config: {}
```
### Internal etcd
@@ -83,45 +67,25 @@ The following example configures an internal etcd service.
```YAML
apiVersion: milvus.io/v1alpha1
-
kind: MilvusCluster
-
metadata:
-
name: my-release
-
labels:
-
app: milvus
-
spec:
-
dependencies:
-
etcd:
-
inCluster:
-
values:
-
replicaCount: 5
-
resources:
-
limits:
-
cpu: '4'
-
memory: 8Gi
-
requests:
-
cpu: 200m
-
memory: 512Mi
-
components: {}
-
config: {}
```
diff --git a/preview/site/en/adminGuide/monitor/configure_grafana_loki.md b/preview/site/en/adminGuide/monitor/configure_grafana_loki.md
new file mode 100644
index 000000000..a1c1aa2b8
--- /dev/null
+++ b/preview/site/en/adminGuide/monitor/configure_grafana_loki.md
@@ -0,0 +1,149 @@
+---
+id: configure_grafana_loki.md
+title: Configure Grafana Loki
+summary: This topic describes how to collect logs using Loki and query logs for a Milvus cluster using Grafana.
+---
+
+# Configure Grafana Loki
+
+This guide provides instructions on how to configure Loki to collect logs and Grafana to query and display logs for a Milvus cluster.
+
+In this guide, you will learn how to:
+
+- Deploy [Loki](https://grafana.com/docs/loki/latest/get-started/overview/) and [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/) on a Milvus cluster using Helm.
+- Configure object storage for Loki.
+- Query logs using Grafana.
+
+## Prerequisites
+
+- You have [installed a Milvus cluster on K8s](install_cluster-helm.md).
+- You have installed necessary tools, including [Helm](https://helm.sh/docs/intro/install/) and [Kubectl](https://kubernetes.io/docs/tasks/tools/).
+
+## Deploy Loki
+
+Loki is a log aggregation system inspired by Prometheus. Deploy Loki using Helm to collect logs from your Milvus cluster.
+
+### 1. Add Grafana's Helm Chart Repository
+
+Add Grafana’s chart repository to Helm and update it:
+
+```
+helm repo add grafana https://grafana.github.io/helm-charts
+helm repo update
+```
+
+### 2. Configure Object Storage for Loki
+
+Choose one of the following storage options and create a `loki.yaml` configuration file:
+
+- Option 1: Using MinIO for storage
+
+ ```yaml
+ loki:
+ commonConfig:
+ replication_factor: 1
+ auth_enabled: false
+
+ minio:
+ enabled: true
+ ```
+
+- Option 2: Using AWS S3 for storage
+
+ In the following example, replace `There is a default `ChannelName` apart from the physical channels. | | Apply PK Latency | The average latency and the 99th percentile of primary key application latency by each proxy within the past two minutes. | p99:
``` histogram_quantile(0.99, sum by (le, pod, node_id) (rate(milvus_proxy_apply_pk_latency_bucket{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m]))) ```
avg:
``` sum(increase(milvus_proxy_apply_pk_latency_sum{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])) by (pod, node_id) / sum(increase(milvus_proxy_apply_pk_latency_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])) by (pod, node_id) ``` | `milvus_proxy_apply_pk_latency` | The latency of applying primary key. | | Apply Timestamp Latency | The average latency and the 99th percentile of timestamp application latency by each proxy within the past two minutes. | p99:
``` histogram_quantile(0.99, sum by (le, pod, node_id) (rate(milvus_proxy_apply_timestamp_latency_bucket{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m]))) ```
avg:
``` sum(increase(milvus_proxy_apply_timestamp_latency_sum{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])) by (pod, node_id) / sum(increase(milvus_proxy_apply_timestamp_latency_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])) by (pod, node_id) ``` | `milvus_proxy_apply_timestamp_latency` | The latency of applying timestamp. | -| Request Success Rate | The number of all types of successful requests received by each proxy with the statistics of each type displayed individually. Possible request types are DescribeCollection, DescribeIndex, GetCollectionStatistics, HasCollection, Search, Query, ShowPartitions, Insert, etc. | ```sum(increase(milvus_proxy_req_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace", status="success"}[2m])/120) by(function_name, pod, node_id)``` | `milvus_proxy_req_count` | The number of all types of receiving requests | -| Request Failed Rate | The number of all types of failed requests received by each proxy with the statistics of each type displayed individually. Possible request types are DescribeCollection, DescribeIndex, GetCollectionStatistics, HasCollection, Search, Query, ShowPartitions, Insert, etc. | ```sum(increase(milvus_proxy_req_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace", status="fail"}[2m])/120) by(function_name, pod, node_id)``` | `milvus_proxy_req_count` | The number of all types of receiving requests | +| Request Success Rate | The number of successful requests received per second by each proxy, with a detailed breakdown of each request type. Possible request types are DescribeCollection, DescribeIndex, GetCollectionStatistics, HasCollection, Search, Query, ShowPartitions, Insert, etc. + | ```sum(increase(milvus_proxy_req_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace", status="success"}[2m])/120) by(function_name, pod, node_id)``` | `milvus_proxy_req_count` | The number of all types of receiving requests | +| Request Failed Rate | The number of failed requests received per second by each proxy, with a detailed breakdown of each request type. Possible request types are DescribeCollection, DescribeIndex, GetCollectionStatistics, HasCollection, Search, Query, ShowPartitions, Insert, etc. + | ```sum(increase(milvus_proxy_req_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace", status="fail"}[2m])/120) by(function_name, pod, node_id)``` | `milvus_proxy_req_count` | The number of all types of receiving requests | | Request Latency | The average latency and the 99th percentile of the latency of all types of receiving requests by each proxy | p99:
```histogram_quantile(0.99, sum by (le, pod, node_id, function_name) (rate(milvus_proxy_req_latency_bucket{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])))```
avg:
```sum(increase(milvus_proxy_req_latency_sum{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])) by (pod, node_id, function_name) / sum(increase(milvus_proxy_req_latency_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])) by (pod, node_id, function_name)``` | `milvus_proxy_req_latency` | The latency of all types of receiving requests | | Insert/Delete Request Byte Rate | The number of bytes of insert and delete requests received per second by proxy within the past two minutes. | ``` sum(increase(milvus_proxy_receive_bytes_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])/120) by(pod, node_id) ``` | `milvus_proxy_receive_bytes_count` | The count of insert and delete requests. | | Send Byte Rate | The number of bytes per second sent back to the client while each proxy is responding to search and query requests within the past two minutes. | ``` sum(increase(milvus_proxy_send_bytes_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])/120) by(pod, node_id) ``` | `milvus_proxy_send_bytes_count` | The number of bytes sent back to the client while each proxy is responding to search and query requests. | @@ -155,9 +157,9 @@ We recommend reading [Milvus monitoring framework overview](monitor_overview.md) | Panel | Panel description | PromQL (Prometheus query language) | The Milvus metrics used | Milvus metrics description | |---|---|---|---|---| -| Index Request Rate | The average number of index building requests received per second by index coord within the past two minutes. | ``` sum(increase(milvus_indexcoord_indexreq_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])/120) by (status) ``` | `milvus_indexcoord_indexreq_count` | The number of index building requests received by index coord. | -| Index Task Count | The count of all indexing tasks recorded by index coord in index metadata. | ``` sum(milvus_indexcoord_indextask_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}) by (index_task_status) ``` | `milvus_indexcoord_indextask_count` | The count of all indexing tasks recorded by index coord in index metadata. | -| Index Node Num | The number of index nodes managed by index coord. | ``` sum(milvus_indexcoord_indexnode_num{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}) by (app_kubernetes_io_instance) ``` | `milvus_indexcoord_indexnode_num` | The number of index nodes managed by index coord. | +| Index Request Rate | The average number of index building requests received per second within the past two minutes. | ``` sum(increase(milvus_indexcoord_indexreq_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}[2m])/120) by (status) ``` | `milvus_indexcoord_indexreq_count` | The number of index building requests received. | +| Index Task Count | The count of all indexing tasks recorded in index metadata. | ``` sum(milvus_indexcoord_indextask_count{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}) by (index_task_status) ``` | `milvus_indexcoord_indextask_count` | The count of all indexing tasks recorded in index metadata. | +| Index Node Num | The number of managed index nodes. | ``` sum(milvus_indexcoord_indexnode_num{app_kubernetes_io_instance=~"$instance", app_kubernetes_io_name="$app_name", namespace="$namespace"}) by (app_kubernetes_io_instance) ``` | `milvus_indexcoord_indexnode_num` | The number of managed index nodes. | diff --git a/preview/site/en/adminGuide/monitor/monitor.md b/preview/site/en/adminGuide/monitor/monitor.md index 4d8eac5c2..60f209106 100644 --- a/preview/site/en/adminGuide/monitor/monitor.md +++ b/preview/site/en/adminGuide/monitor/monitor.md @@ -32,10 +32,21 @@ Before deploying monitoring services, you need to create a monitoring stack by u ``` $ git clone https://github.com/prometheus-operator/kube-prometheus.git -$ cd # to the local path of the repo -$ kubectl create -f manifests/setup -$ until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done -$ kubectl create -f manifests/ +$ cd kube-prometheus +$ kubectl apply --server-side -f manifests/setup +$ kubectl wait \ + --for condition=Established \ + --all CustomResourceDefinition \ + --namespace=monitoring +$ kubectl apply -f manifests/ +``` + +
+The default prometheus-k8s clusterrole can not capture milvus' metrics, need to patch:
+
+
+```bash
+kubectl patch clusterrole prometheus-k8s --type=json -p='[{"op": "add", "path": "/rules/-", "value": {"apiGroups": [""], "resources": ["pods", "services", "endpoints"], "verbs": ["get", "watch", "list"]}}]'
```
To delete a stack, run `kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup`.
@@ -44,19 +55,19 @@ To delete a stack, run `kubectl delete --ignore-not-found=true -f manifests/ -f
### 1. Access the dashboards
-You can access Prometheus via `http://localhost:9090`, and Grafana at `http://localhost:3000`.
+Forward the Prometheus service to port `9090`, and Grafana service to port `3000`.
```
-$ kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
-$ kubectl --namespace monitoring port-forward svc/grafana 3000
+$ kubectl --namespace monitoring --address 0.0.0.0 port-forward svc/prometheus-k8s 9090
+$ kubectl --namespace monitoring --address 0.0.0.0 port-forward svc/grafana 3000
```
### 2. Enable ServiceMonitor
-The ServiceMonitor is not enabled for Milvus Helm by default. After installing the Prometheus Operator in the Kubernetes cluster, you can enable it by adding the parameter `metrics.serviceMontior.enabled=true`.
+The ServiceMonitor is not enabled for Milvus Helm by default. After installing the Prometheus Operator in the Kubernetes cluster, you can enable it by adding the parameter `metrics.serviceMonitor.enabled=true`.
```
-$ helm install my-release milvus/milvus --set metrics.serviceMonitor.enabled=true
+$ helm upgrade my-release milvus/milvus --set metrics.serviceMonitor.enabled=true --reuse-values
```
When the installation completes, use `kubectl` to check the ServiceMonitor resource.
@@ -78,4 +89,4 @@ my-release-milvus 54s
- If you are looking for information about how to scale a Milvus cluster:
- Learn [scale a Milvus cluster](scaleout.md)
- If you are interested in upgrading the Milvus version,
- - Read the [upgrading guide](upgrade.md)
+ - Read the [guide for upgrading Milvus cluster](upgrade_milvus_cluster-operator.md) and [that for upgrade Milvus standalone](upgrade_milvus_standalone-operator.md).
diff --git a/preview/site/en/adminGuide/monitor/monitor_and_alert.md b/preview/site/en/adminGuide/monitor/monitor_and_alert.md
deleted file mode 100644
index de81b99b1..000000000
--- a/preview/site/en/adminGuide/monitor/monitor_and_alert.md
+++ /dev/null
@@ -1,18 +0,0 @@
----
-id: monitor_and_alert.md
-title: Monitor and Alert
----
-
-# Monitor and Alert
-
-Milvus uses Prometheus to monitor metrics and Grafana to visualize metrics and create alerts.
-
-- [Monitoring Architecture](monitor_overview.md): An overview of the Milvus monitoring framework.
-
-- [Deploy Monitoring Services](monitor.md): Use Prometheus to deploy monitoring services on Kubernetes.
-
-- [Visualize Milvus Metrics](visualize.md): Visualize Milvus metrics using Grafana.
-
-- [Milvus Metrics Dashboard](metrics_dashboard.md): A detailed list of important metrics including search QPS and latency that are newly added to the Milvus metrics dashboard.
-
-- [Create an Alert](alert.md): Learn about why, when, and how to create alerts in Milvus.
diff --git a/preview/site/en/adminGuide/monitor/monitor_overview.md b/preview/site/en/adminGuide/monitor/monitor_overview.md
index 7c59a9ea9..0d03d786a 100644
--- a/preview/site/en/adminGuide/monitor/monitor_overview.md
+++ b/preview/site/en/adminGuide/monitor/monitor_overview.md
@@ -1,6 +1,6 @@
---
id: monitor_overview.md
-title: Overview
+title: Monitor Overview
related_key: monitor, alert
summary: Learn how Prometheus and Grafana are used in Milvus for montoring and alerting services.
---
diff --git a/preview/site/en/adminGuide/monitor/visualize.md b/preview/site/en/adminGuide/monitor/visualize.md
index 70907dec1..cc1738c6f 100644
--- a/preview/site/en/adminGuide/monitor/visualize.md
+++ b/preview/site/en/adminGuide/monitor/visualize.md
@@ -26,7 +26,7 @@ This tutorial uses Grafana, an open-source platform for time-series analytics, t
Download and import Milvus dashboard from the JSON file.
```
-wget https://raw.githubusercontent.com/milvus-io/milvus/2.1.0/deployments/monitor/grafana/milvus-dashboard.json
+wget https://raw.githubusercontent.com/milvus-io/milvus/2.2.0/deployments/monitor/grafana/milvus-dashboard.json
```
@@ -47,4 +47,4 @@ Select the Milvus instance you want to monitor. Then you can see the Milvus comp
- Adjust your [resource allocation](allocate.md)
- [Scale out or scale in a Milvus cluster](scaleout.md)
- If you are interested in upgrading the Milvus version,
- - Read the [upgrading guide](upgrade.md)
+ - Read the [guide for upgrading Milvus cluster](upgrade_milvus_cluster-operator.md) and [that for upgrade Milvus standalone](upgrade_milvus_standalone-operator.md).
diff --git a/preview/site/en/adminGuide/object_storage_operator.md b/preview/site/en/adminGuide/object_storage_operator.md
index 1dfb42909..99503fc85 100644
--- a/preview/site/en/adminGuide/object_storage_operator.md
+++ b/preview/site/en/adminGuide/object_storage_operator.md
@@ -7,7 +7,7 @@ summary: Learn how to configure object storage with Milvus Operator.
# Configure Object Storage with Milvus Operator
-Milvus uses MinIO or S3 as object storage to persist large-scale files, such as index files and binary logs. This topic introduces how to configure object storage dependencies when you install Milvus with Milvus Operator.
+Milvus uses MinIO or S3 as object storage to persist large-scale files, such as index files and binary logs. This topic introduces how to configure object storage dependencies when you install Milvus with Milvus Operator. For more details, refer to [Configure Object Storage with Milvus Operator](https://github.com/zilliztech/milvus-operator/blob/main/docs/administration/manage-dependencies/object-storage.md) in the Milvus Operator repository.
This topic assumes that you have deployed Milvus Operator.
@@ -16,105 +16,221 @@ This topic assumes that you have deployed Milvus Operator.
You need to specify a configuration file for using Milvus Operator to start a Milvus cluster.
```YAML
-kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/config/samples/milvuscluster_default.yaml
+kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_cluster_default.yaml
```
-You only need to edit the code template in `milvuscluster_default.yaml` to configure third-party dependencies. The following sections introduce how to configure object storage, etcd, and Pulsar respectively.
+You only need to edit the code template in `milvus_cluster_default.yaml` to configure third-party dependencies. The following sections introduce how to configure object storage, etcd, and Pulsar respectively.
## Configure object storage
-A Milvus cluster uses MinIO or S3 as object storage to persist large-scale files, such as index files and binary logs. Add required fields under `spec.dependencies.storage` to configure object storage.
-
-`storage` supports `external` and `inCluster`.
-
-### External object storage
-
-`external` indicates using an external object storage service.
-
-Fields used to configure an external object storage service include:
-
-- `external`: A `true` value indicates that Milvus uses an external storage service.
-- `type`: Specifies Milvus uses whether S3 or MinIO as object storage.
-- `secretRef`: The secret reference that the object storage service uses.
-- `endpoint`: The endpoint of the object storage service.
-
-#### Example
-
-The following example configures an external object storage service.
-
-```YAML
-kind: MilvusCluster
-metadata:
- name: my-release
- labels:
- app: milvus
-spec:
- dependencies: # Optional
- storage: # Optional
- # Whether (=true) to use an existed external storage as specified in the field endpoints or
- # (=false) create a new storage inside the same kubernetes cluster for milvus.
- external: true # Optional default=false
- type: "MinIO" # Optional ("MinIO", "S3") default:="MinIO"
- # Secret reference of the storage if it has
- secretRef: mySecret # Optional
- # The external storage endpoint if external=true
- endpoint: "storageEndpoint"
- components: {}
- config: {}
-```
+A Milvus cluster uses MinIO or S3 as object storage to persist large-scale files, such as index files and binary logs. Add required fields under `spec.dependencies.storage` to configure object storage, possible options are `external` and `inCluster`.
### Internal object storage
-`inCluster` indicates when a Milvus cluster starts, a MinIO service starts automatically in the cluster.
-
-A Milvus cluster only supports using MinIO as the internal object storage service.
-
-#### Example
-
-The following example configures an internal MinIO service.
+By default, Milvus Operator deploys an in-cluster MinIO for Milvus. The following is an example configuration to demonstrate how to use this MinIO as an internal object storage.
```YAML
-apiVersion: milvus.io/v1alpha1
-kind: MilvusCluster
+apiVersion: milvus.io/v1beta1
+kind: Milvus
metadata:
name: my-release
labels:
app: milvus
spec:
+ # Omit other fields ...
dependencies:
- storage: #
- external: false
- type: "MinIO" # Optional ("MinIO", "S3") default:="MinIO"
- inCluster:
- # deletionPolicy of storage when the milvus cluster is deleted
- deletionPolicy: Retain # Optional ("Delete", "Retain") default="Retain"
- # When deletionPolicy="Delete" whether the PersistantVolumeClaim shoud be deleted when the storage is deleted
- pvcDeletion: false
+ # Omit other fields ...
+ storage:
+ inCluster:
values:
+ mode: standalone
resources:
- limits:
- cpu: '2'
- memory: 6Gi
requests:
- cpu: 100m
- memory: 512Mi
- statefulset:
- replicaCount: 6
- components: {}
- config: {}
+ memory: 100Mi
+ deletionPolicy: Delete # Delete | Retain, default: Retain
+ pvcDeletion: true # default: false
```
-In this example,
+After the above configuration applies, the in-cluster MinIO will run in standalone mode with a memory limit of up to 100Mi. Note that
-inCluster.deletionPolicy defines a deleletion policy for data. inCluster.values.resources defines the compute resources that MinIO uses. inCluster.values.statefulset.replicaCount defines the number of replicas of MinIO on each drive.Find the complete configuration items to configure an internal MinIO service in values.yaml. Add configuration items as needed under
+- The `deletionPolicy` field specifies the deletion policy of the in-cluster MinIO. It defaults to `Delete` and has `Retain` as the alternative option.
-Assuming that the configuration file is named `milvuscluster.yaml`, run the following command to apply the configuration.
+ - `Delete` indicates that the in-cluster object storage is deleted when you stop your Milvus instance.
+ - `Retain` indicates that the in-cluster object storage is retained as the dependency service for later startups of your Milvus instance.
-```Shell
-kubectl apply -f milvuscluster.yaml
-```
+- The `pvcDeletion` field specifies whether to delete the PVC(Persistent Volume Claim) when the in-cluster MinIO is deleted.
+
+The fields under `inCluster.values` are the same as those in Milvus Helm Chart, and you can find them [here](https://github.com/milvus-io/milvus-helm/blob/master/charts/minio/values.yaml).
-storage.inCluster.values as shown in the preceding example.If
+### External object storage
+
+Using `external` in the template YAML file indicates using an external object storage service. To use an external object storage, you need to properly set fields under `spec.dependencies.storage` and `spec.config.minio` in the Milvus CRD.
+
+#### Use Amazon Web Service (AWS) S3 as external object storage
+
+- Configure AWS S3 Access by AK/SK
+
+ An S3 bucket can usually be accessed by a pair of an access key and an access secret key. You can create a `Secret` object to store them in your Kubernetes as follows:
+
+ ```YAML
+ # # change the my-release is an existing Milvus cluster, milvuscluster.yaml overwrites its configuration. Otherwise, a new Milvus cluster is created.See Deploy Milvus Operator for more information.
+See Deploy Milvus Operator for more information.
You need to specify a configuration file for using Milvus Operator to start a Milvus cluster.
```YAML
-kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/config/samples/milvuscluster_default.yaml
+kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvuscluster_default.yaml
```
You only need to edit the code template in `milvuscluster_default.yaml` to configure third-party dependencies. The following sections introduce how to configure object storage, etcd, and Pulsar respectively.
@@ -148,7 +149,7 @@ spec:
config: {}
```
-In this example,
+inCluster.deletionPolicy defines a deleletion policy for data. inCluster.values.resources defines the compute resources that MinIO uses. inCluster.values.statefulset.replicaCount defines the number of replicas of MinIO on each drive.In this example,
inCluster.deletionPolicy defines a deletion policy for data. inCluster.values.resources defines the compute resources that MinIO uses. inCluster.values.statefulset.replicaCount defines the number of replicas of MinIO on each drive.Find the complete configuration items to configure an internal MinIO service in values.yaml. Add configuration items as needed under
@@ -265,7 +266,7 @@ spec:
storage.inCluster.values as shown in the preceding example.The preceding example specifies the number of replicas as
-5 and limits the compute resources for etcd.Find the complete configuration items to configure an internal etcd service in values.yaml. Add configuration items as needed under
+etcd.inCluster.values as shown in the preceding example.Find the complete configuration items to configure an internal etcd service in values.yaml. Add configuration items as needed under
Assuming that the configuration file is named `milvuscluster.yaml`, run the following command to apply the configuration.
@@ -412,7 +413,7 @@ spec:
etcd.inCluster.values as shown in the preceding example.This example specifies the numbers of replicas of each component of Pulsar, the compute resources of Pulsar BookKeeper, and other configurations.
-Find the complete configuration items to configure an internal Pulsar service in values.yaml. Add configuration items as needed under
+pulsar.inCluster.values as shown in the preceding example.Find the complete configuration items to configure an internal Pulsar service in values.yaml. Add configuration items as needed under
Assuming that the configuration file is named `milvuscluster.yaml`, run the following command to apply the configuration.
diff --git a/preview/site/en/adminGuide/rbac.md b/preview/site/en/adminGuide/rbac.md
index 3a576cb2d..4583626ad 100644
--- a/preview/site/en/adminGuide/rbac.md
+++ b/preview/site/en/adminGuide/rbac.md
@@ -2,6 +2,7 @@
id: rbac.md
related_key: enable RBAC
summary: Learn how to manage users, roles, and privileges.
+title: Enable RBAC
---
# Enable RBAC
@@ -10,102 +11,148 @@ By enabling RBAC, you can control access to specific Milvus resources (Eg. a col
This topic describes how to enable RBAC and manage [users and roles](users_and_roles.md).
-## 1. Create a user
+pulsar.inCluster.values as shown in the preceding example.
+The code snippets on this page use new MilvusClient (Python) to interact with Milvus. New MilvusClient SDKs for other languages will be released in future updates.
+
+
+
+## 1. Initiate a Milvus client to establish a connection
+
+After you enable [user authentication](authenticate.md), connect to your Milvus instance using `token` that consists of a username and a password. By default, Milvus uses the `root` user with the password `Milvus`.
+
+```python
+from pymilvus import MilvusClient
+
+client = MilvusClient(
+ uri='http://localhost:19530', # replace with your own Milvus server address
+ token='root:Milvus' # replace with your own Milvus server token
+)
```
-utility.create_user(user, password, using="default")
+
+## 2. Create a user
+
+Create a user named `user_1` with the password `P@ssw0rd`:
+
+```python
+client.create_user(
+ user_name='user_1',
+ password='P@ssw0rd'
+)
```
After creating a user, you can:
- Update a user password. You need to provide both the original and the new password.
-```
-utility.update_password(user, old_password, new_password, using="default")
+```python
+client.update_password(
+ user_name='user_1',
+ old_password='P@ssw0rd',
+ new_password='P@ssw0rd123'
+)
```
- List all users.
-```
-utility.list_usernames(using="default")
+```python
+client.list_users()
+
+# output:
+# ['root', 'user_1']
```
- Check the role of a particular user.
-```
-utility.list_user(username, include_role_info, using="default")
-```
+```python
+client.describe_user(user_name='user_1')
-- Check the roles of all users.
-
-```
-utility.list_users(include_role_info, using="default")
+# output:
+# {'user_name': 'user_1', 'roles': ()}
```
-## 2. Create a role
+## 3. Create a role
The following example creates a role named `roleA`.
-```
-role_name = "roleA"
-role = Role(role_name, using=_CONNECTION)
-role.create()
+```python
+client.create_role(
+ role_name="roleA",
+)
```
After creating a role, you can:
-- Check if a role exists.
-
-```
-role.is_exist("roleA")
-```
-
- List all roles.
-```
-utility.list_roles(include_user_info, using="default")
+```python
+client.list_roles()
+
+# output:
+# ['admin', 'public', 'roleA']
```
-## 3. Grant a privilege to a role
+## 4. Grant a privilege to a role
The following example demonstrates how to grant the permission of searching all collections to the role named `roleA`. See [Users and Roles](users_and_roles.md) for other types of privileges you can grant.
-```
-role.grant("Collection", "*", "Search")
+Before managing role privileges, make sure you have enabled user authentication. Otherwise, an error may occur. For information on how to enable user authentication, refer to [Authenticate User Access](authenticate.md).
+
+```python
+# grant privilege to a role
+
+client.grant_privilege(
+ role_name='roleA',
+ object_type='User',
+ object_name='SelectUser',
+ privilege='SelectUser'
+)
```
After granting a privilege to a role, you can:
-- List certain privileges to an object granted to a role.
-
-```
-role.list_grant("Collection","CollectionA")
-```
+- View the privileges granted to a role.
-- List all privileges granted to a role.
+```python
+client.describe_role(
+ role_name='roleA'
+)
-```
-role.list_grants()
+# output:
+# {'role': 'roleA',
+# 'privileges': [{'object_type': 'User',
+# 'object_name': 'SelectUser',
+# 'db_name': 'default',
+# 'role_name': 'roleA',
+# 'privilege': 'SelectUser',
+# 'grantor_name': 'root'}]}
```
+## 5. Grant a role to a user
-## 4. Bind a role to a user
+Grant the role to a user so that this user can inherit all the privileges of the role.
-Bind the role to a user so that this user can inherit all the privileges of the role.
+```python
+# grant a role to a user
-```
-role.add_user("roleA", username)
+client.grant_role(
+ user_name='user_1',
+ role_name='roleA'
+)
```
-After binding a role to a user, you can:
+After granting the role, verity that it has been granted:
-- List all users bind to a role
+```python
+client.describe_user(
+ user_name='user_1'
+)
-```
-role.get_users("roleA")
+# output:
+# {'user_name': 'user_1', 'roles': ('roleA',)}
```
-## 5. Deny access or privileges
+## 6. Revoke privileges
@@ -113,28 +160,36 @@ Exercise caution when performing the following operations because these operatio
-- Remove a privilege from a role.
+- Remove a privilege from a role. If you revoke a privilege that has not been granted to the role, an error will occur.
-```
-role.revoke("Collection","*","Search")
+```python
+client.revoke_privilege(
+ role_name='roleA',
+ object_type='User',
+ object_name='SelectUser',
+ privilege='SelectUser'
+)
```
-- Remove a user from a role
+- Remove a user from a role. If you revoke a role that has not been granted to the user, an error will occur.
-```
-role.remove_user(username)
+```python
+client.revoke_role(
+ user_name='user_1',
+ role_name='roleA'
+)
```
-- Delete a role
+- Drop a role.
-```
-role.drop("roleA"):
+```python
+client.drop_role(role_name='roleA')
```
-- Delete a user
+- Drop a user.
-```
-utility.delete_user(user, using="default")
+```python
+client.drop_user(user_name='user_1')
```
## What's next
diff --git a/preview/site/en/adminGuide/resource_group.md b/preview/site/en/adminGuide/resource_group.md
index 350538692..75c94d357 100644
--- a/preview/site/en/adminGuide/resource_group.md
+++ b/preview/site/en/adminGuide/resource_group.md
@@ -2,30 +2,54 @@
id: resource_group.md
related_key: Manage Resource Groups
summary: Learn how to manage resource groups.
+title: Manage Resource Groups
---
-## Manage Resource Groups
+# Manage Resource Groups
In Milvus, you can use a resource group to physically isolate certain query nodes from others. This guide walks you through how to create and manage custom resource groups as well as transfer nodes between them.
-### What is a resource group
+## What is a resource group
-A resource group can hold several or all of the query nodes in a Milvus instance. You decide how you want to allocate query nodes among resource groups based on what makes the most sense for you. For example, in a multi-tenant scenario, you can allocate an appropriate number of query nodes to each tenant so that the operations within each resource group are physically independent of those in other resource groups.
+A resource group can hold several or all of the query nodes in a Milvus cluster. You decide how you want to allocate query nodes among resource groups based on what makes the most sense for you. For example, in a multi-collection scenario, you can allocate an appropriate number of query nodes to each resource group and load collections into different resource group, so that the operations within each collection are physically independent of those in other collections.
-Note that a Milvus instance maintains a default resource group to hold all the query nodes at the start-up and names it **__default_resource_group**. You can move some nodes from the default resource group to the one you create.
+Note that a Milvus instance maintains a default resource group to hold all the query nodes at the start-up and names it **__default_resource_group**.
-### Manage resource groups
+Starting from version 2.4.1, Milvus provides a declarative resource group API, while the old resource group API has been deprecated. The new declarative API enables users to achieve idempotency, to do secondary development in cloud-native environments easilier.
-
-
-All code samples on this page are in PyMilvus 2.2.2. Upgrade your PyMilvus installation before running them.
+## Concepts of resource group
-To upgrade PyMilvus, run the following:
+A resource group is described by a resource group config:
-```Shell
-python -m pip install --upgrade pymilvus
+```json
+{
+ "requests": { "nodeNum": 1 },
+ "limits": { "nodeNum": 1 },
+ "transfer_from": [{ "resource_group": "rg1" }],
+ "transfer_to": [{ "resource_group": "rg2" }]
+}
```
+- The **requests** attribute specifies the conditions that a resource group must meet.
+- The **limits** attribute specifies the maximum limits for a resource group.
+- The **transfer_from** and **transfer_to** attributes describe from which resource groups a resource group should preferably acquire resources and to which resource groups it should transfer resources, respectively.
+
+Once the configuration of a resource group changes, the Milvus will adjust the current Query Node resources as much as possible according to the new configuration, ensuring that all resource groups eventually meet the following condition:
+
+`.requests.nodeNum < nodeNumOfResourceGroup < .limits.nodeNum.`
+
+Except in the following cases:
+- When the number of QueryNodes in the Milvus cluster is insufficient, i.e., `NumOfQueryNode < sum(.requests.nodeNum)`, there will always be resource groups without enough QueryNodes.
+- When the number of QueryNodes in the Milvus cluster is excessive, i.e., `NumOfQueryNode > sum(.limits.nodeNum)`, the redundant QueryNodes will always be placed in the **__default_resource_group** first.
+
+Of course, if the number of QueryNodes in the cluster changes, the Milvus will continuously attempt to adjust to meet the final conditions. Therefore, you can first apply the resource group configuration changes and then perform QueryNode scaling.
+
+## Use declarative api to manage resource group
+
+, // string, rg name
+ # , // string, rg name
# , // int, num_node which has been transfer to this rg
# , // int, available node_num, some node may shutdown
# , // map[string]int, from collection_name to loaded replica of each collecion in this rg
@@ -80,19 +107,31 @@ python -m pip install --upgrade pymilvus
4. Transfer nodes between resource groups.
You may notice that the described resource group does not have any query node yet. Move some nodes from the default resource group to the one you create as follows:
+ Assuming there are currently 1 QueryNodes in the **__default_resource_group** of the cluster, and we want to transfer one node into created **rg**.
+ `update_resource_groups` ensures atomicity for multiple configuration changes, so no intermediate states will be visible to Milvus.
```Python
source = '__default_resource_group'
target = 'rg'
- num_nodes = 1
+ expected_num_nodes_in_default = 0
+ expected_num_nodes_in_rg = 1
try:
- utility.transfer_node(source, target, num_nodes, using="default")
- print(f"Succeeded in moving {num_node} node(s) from {source} to {target}.")
+ utility.update_resource_groups({
+ source: ResourceGroupConfig(
+ requests={"node_num": expected_num_nodes_in_default},
+ limits={"node_num": expected_num_nodes_in_default},
+ ),
+ target: ResourceGroupConfig(
+ requests={"node_num": expected_num_nodes_in_rg},
+ limits={"node_num": expected_num_nodes_in_rg},
+ )
+ }, using="default")
+ print(f"Succeeded in move 1 node(s) from {source} to {target}.")
except Exception:
print("Something went wrong while moving nodes.")
- # Succeeded in moving 1 node(s) from __default_resource_group to rg.
+ # After a while, succeeded in moving 1 node(s) from __default_resource_group to rg.
```
5. Load collections and partitions to a resource group.
@@ -108,8 +147,9 @@ python -m pip install --upgrade pymilvus
collection.load(replica_number=2)
# Or, you can ask Milvus load the collection to the desired resource group.
+ # make sure that query nodes num should be greater or equal to replica_number
resource_groups = ['rg']
- collection.load(replica_number=2, _resource_group=resource_groups)
+ collection.load(replica_number=2, _resource_groups=resource_groups)
```
Also, you can just load a partition into a resource group and have its replicas distributed among several resource groups. The following assumes that a collection named `Books` already exists and it has a partition named `Novels`.
@@ -118,14 +158,14 @@ python -m pip install --upgrade pymilvus
collection = Collection("Books")
# Use the load method of a collection to load one of its partition
- collection.load(["Novels"], replica_number=2, _resource_group=resource_groups)
+ collection.load(["Novels"], replica_number=2, _resource_groups=resource_groups)
# Or, you can use the load method of a partition directly
partition = Partition(collection, "Novels")
- partition.load(replica_number=2, _resource_group=resource_groups)
+ partition.load(replica_number=2, _resource_groups=resource_groups)
```
- Note that `_resource_group` is an optional parameter, and leaving it unspecified have Milvus load the replicas onto the query nodes in the default resource group.
+ Note that `_resource_groups` is an optional parameter, and leaving it unspecified have Milvus load the replicas onto the query nodes in the default resource group.
To have Milus load each replica of a collection in a separate resource group, ensure that the number of resource groups equals the number of replicas.
@@ -150,24 +190,133 @@ python -m pip install --upgrade pymilvus
7. Drop a resource group.
- You can drop a resource group at any time only if there are no query nodes in the resource group. In this guide, resource group `rg` now has one query node. You need to move it to another resource group before you drop this resource group.
+ You can drop a resource group that hold no query node (`limits.node_num = 0`) at any time. In this guide, resource group `rg` now has one query node. You need to change the configuration `limits.node_num` of resource group into zero first.
```Python
- source = 'rg'
- target = '__default_resource_group'
- num_nodes = 1
-
try:
- utility.transfer_node(source, target, num_nodes, using="default")
- utility.drop_resource_group(source, using="default")
+ utility.update_resource_groups({
+ "rg": utility.ResourceGroupConfig(
+ requests={"node_num": 0},
+ limits={"node_num": 0},
+ ),
+ }, using="default")
+ utility.drop_resource_group("rg", using="default")
print(f"Succeeded in dropping {source}.")
except Exception:
print(f"Something went wrong while dropping {source}.")
```
-## What's next
+For more details, please refer to the [relevant examples in pymilvus](https://github.com/milvus-io/pymilvus/blob/v2.4.3/examples/resource_group_declarative_api.py)
+
+## A good practice to manage cluster scaling
+
+Currently, Milvus cannot independently scale in and out in cloud-native environments. However, by using the **Declarative Resource Group API** in conjunction with container orchestration, Milvus can easily achieve resource isolation and management for QueryNodes.
+Here is a good practice for managing QueryNodes in a cloud environment:
+
+1. By default, Milvus creates a **__default_resource_group**. This resource group cannot be deleted and also serves as the default loading resource group for all collections and redundant QueryNodes are always assigned to it. Therefore, we can create a pending resource group to hold unusing QueryNode resources, preventing QueryNode resources from being occupied by the **__default_resource_group**.
+
+ Additionally, if we strictly enforce the constraint `sum(.requests.nodeNum) <= queryNodeNum`, we can precisely control the assignment of QueryNodes in the cluster. Let's assume there is currently only one QueryNode in the cluster and initialize the cluster.
+ Here is an example setup:
+
+ ```Python
+ from pymilvus import utility
+ from pymilvus.client.types import ResourceGroupConfig
+
+ _PENDING_NODES_RESOURCE_GROUP="__pending_nodes"
+
+ def init_cluster(node_num: int):
+ print(f"Init cluster with {node_num} nodes, all nodes will be put in default resource group")
+ # create a pending resource group, which can used to hold the pending nodes that do not hold any data.
+ utility.create_resource_group(name=_PENDING_NODES_RESOURCE_GROUP, config=ResourceGroupConfig(
+ requests={"node_num": 0}, # this resource group can hold 0 nodes, no data will be load on it.
+ limits={"node_num": 10000}, # this resource group can hold at most 10000 nodes
+ ))
+
+ # update default resource group, which can used to hold the nodes that all initial node in it.
+ utility.update_resource_groups({
+ "__default_resource_group": ResourceGroupConfig(
+ requests={"node_num": node_num},
+ limits={"node_num": node_num},
+ transfer_from=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}], # recover missing node from pending resource group at high priority.
+ transfer_to=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}], # recover redundant node to pending resource group at low priority.
+ )})
+ utility.create_resource_group(name="rg1", config=ResourceGroupConfig(
+ requests={"node_num": 0},
+ limits={"node_num": 0},
+ transfer_from=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}],
+ transfer_to=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}],
+ ))
+ utility.create_resource_group(name="rg2", config=ResourceGroupConfig(
+ requests={"node_num": 0},
+ limits={"node_num": 0},
+ transfer_from=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}],
+ transfer_to=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}],
+ ))
+
+ init_cluster(1)
+ ```
+
+ Using the example code above, we create a resource group named **__pending_nodes** to hold additional QueryNodes. We also create two user-specific Resource Groups named **rg1** and **rg2**. Additionally, we ensure that the other resource group prioritizes recovering missing or redundant QueryNodes from **__pending_nodes**.
+
+2. Cluster scale out
+
+ Assuming we have the following scaling function:
+ ```Python
+
+ def scale_to(node_num: int):
+ # scale the querynode number in Milvus into node_num.
+ pass
+ ```
+
+ We can use the API to scale a specific resource group to a designated number of QueryNodes without affecting any other resource groups.
+ ```Python
+ # scale rg1 into 3 nodes, rg2 into 1 nodes
+ utility.update_resource_groups({
+ "rg1": ResourceGroupConfig(
+ requests={"node_num": 3},
+ limits={"node_num": 3},
+ transfer_from=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}],
+ transfer_to=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}],
+ ),
+ "rg2": ResourceGroupConfig(
+ requests={"node_num": 1},
+ limits={"node_num": 1},
+ transfer_from=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}],
+ transfer_to=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}],
+ ),
+ })
+ scale_to(5)
+ # rg1 has 3 nodes, rg2 has 1 node, __default_resource_group has 1 node.
+ ```
+
+3. Cluster scale in
+
+ Similarly, we can establish scaling-in rules that prioritize selecting QueryNodes from **__pending_nodes** resource group. This information can be obtained through the `describe_resource_group` API. Achieving the goal of scaling-in specified resource group.
+
+ ```Python
+ # scale rg1 from 3 nodes into 2 nodes
+ utility.update_resource_groups({
+ "rg1": ResourceGroupConfig(
+ requests={"node_num": 2},
+ limits={"node_num": 2},
+ transfer_from=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}],
+ transfer_to=[{"resource_group": _PENDING_NODES_RESOURCE_GROUP}],
+ ),
+ })
+
+ # rg1 has 2 nodes, rg2 has 1 node, __default_resource_group has 1 node, __pending_nodes has 1 node.
+ scale_to(4)
+ # scale the node in __pending_nodes
+ ```
+
+## How resource groups interacts with multiple replicas
+
+- The replicas of a single collection and resource groups have an N-to-N relationship.
+- When multiple replicas of a single collection are loaded into the one resource group, the QueryNodes of that resource group are evenly distributed among the replicas, ensuring that the difference in the number of QueryNodes each replica has does not exceed 1.
+
+# What's next
To deploy a multi-tenant Milvus instance, read the following:
- [Enable RBAC](rbac.md)
-- [Users and roles](users_and_roles.md)
\ No newline at end of file
+- [Users and roles](users_and_roles.md)
diff --git a/preview/site/en/adminGuide/scale-dependencies.md b/preview/site/en/adminGuide/scale-dependencies.md
new file mode 100644
index 000000000..092e653b7
--- /dev/null
+++ b/preview/site/en/adminGuide/scale-dependencies.md
@@ -0,0 +1,220 @@
+---
+id: scale-dependencies.md
+title: Scale Dependencies
+---
+
+# Scale Milvus Dependencies
+
+Milvus relies on various dependencies such as MinIO, Kafka, Pulsar, and etcd. Scaling these components can enhance Milvus's adaptability to different requirements.
+
+For Milvus Operator users, please also refer to [Manage Dependencies For Milvus Operator](manage_dependencies.md)
+
+## Scale MinIO
+
+### Increase resources per MinIO pod
+
+MinIO, an object storage system used by Milvus, can have its CPU and memory resources increased for each pod.
+
+```yaml
+# new-values.yaml
+minio:
+ resources:
+ limits:
+ cpu: 2
+ memory: 8Gi
+```
+
+After saving the file, apply the changes with the following command:
+
+```shell
+helm upgrade --reuse-values -f new-values.yaml milvus/milvus
+```
+
+You can also increase the disk capacity for the MioIO cluster by manually changing the value of `spec.resources.requests.storage` for each MioIO Persistent Volume Claim (PVC). Note that your default storage class should allow volume expansion.
+
+### Add an extra MinIO server pool (Recommended)
+
+You are advised to add an extra MioIO server pool for your Milvus instance.
+
+```yaml
+# new-values.yam;
+minio:
+ zones: 2
+```
+
+After saving the file, apply the changes with the following command:
+
+```shell
+helm upgrade --reuse-values -f new-values.yaml milvus/milvus
+```
+
+This adds an additional server pool to your MinIO cluster, allowing Milvus to write to the MinIO server pool based on the free disk capacity of each server pool. For example, if a group of three pools has a total of 10 TiB free space distributed across the pools as follows:
+
+| | Free space | Write possibility |
+|--------|------------|------------------|
+| Pool A | 3 TiB | 30% (3/10) |
+| Pool B | 2 TiB | 20% (2/10) |
+| Pool C | 5 TiB | 50% (5/10) |
+
+ --reuse-values -f new-values.yaml milvus/milvus
+```
+
+You can also increase the disk capacity for the Kafka cluster by manually changing the value of `spec.resources.requests.storage` for each Kafka Persistent Volume Claim (PVC). Ensure your default storage class allows volume expansion.
+
+## Add an extra Kafka broker pool (Recommended)
+
+You are advised to add an extra Kafka server pool for your Milvus instance.
+
+```yaml
+# new-values.yaml
+kafka:
+ replicaCount: 4
+```
+
+After saving the file, apply the changes with the following command:
+
+```shell
+helm upgrade --reuse-values -f new-values.yaml milvus/milvus
+```
+
+This will add an extra broker to your Kafka cluster.
+
+ --reuse-values -f new-values.yaml milvus/milvus
+```
+
+## Increase resources per Pulsar bookie pod
+
+```yaml
+# new-values.yaml
+pulsar:
+ bookkeeper:
+ resources:
+ limits:
+ cpu: 4
+ memory: 16Gi
+```
+
+After saving the file, apply the changes with the following command:
+
+```shell
+helm upgrade --reuse-values -f new-values.yaml milvus/milvus
+```
+
+You can also increase the disk capacity for the Pulsar cluster by manually changing the value of `spec.resources.requests.storage` for each Pulsar bookie's Persistent Volume Claim (PVC). Note that your default storage class should allow volume expansion.
+
+A Pulsar bookie pod has two types of storage: `journal` and `legers`. For the `journal` type of storage, consider using `ssd` or `gp3` as the storage class.
+
+### Add an extra Pulsar broker pod
+
+```yaml
+# new-values.yaml
+pulsar:
+ broker:
+ replicaCount: 3
+```
+
+After saving the file, apply the changes with the following command:
+
+```shell
+helm upgrade --reuse-values -f new-values.yaml milvus/milvus
+```
+
+
+### Add an extra Pulsar bookie pod (Recommended)
+
+```yaml
+# new-values.yaml
+pulsar:
+ bookkeeper:
+ replicaCount: 3
+```
+
+After saving the file, apply the changes with the following command:
+
+```shell
+helm upgrade --reuse-values -f new-values.yaml milvus/milvus
+```
+
+## etcd
+
+### Increase resources per etcd pod (recommended)
+
+```yaml
+# new-values.yaml
+etcd:
+ resources:
+ limits:
+ cpu: 2
+ memory: 8Gi
+```
+
+After saving the file, apply the changes with the following command:
+
+```shell
+helm upgrade --reuse-values -f new-values.yaml milvus/milvus
+```
+
+### Add extra etcd pods
+
+The total number of etcd pods should be in odd numbers.
+
+```yaml
+# new-values.yaml
+etcd:
+ replicaCount: 5
+```
+
+After saving the file, apply the changes with the following command:
+
+```shell
+helm upgrade --reuse-values -f new-values.yaml milvus/milvus
+```
diff --git a/preview/site/en/adminGuide/scaleout.md b/preview/site/en/adminGuide/scaleout.md
index 9349e4f82..8596d6dba 100644
--- a/preview/site/en/adminGuide/scaleout.md
+++ b/preview/site/en/adminGuide/scaleout.md
@@ -2,6 +2,7 @@
id: scaleout.md
related_key: scale Milvus cluster
summary: Learn how to manually or automatically scale out and scale in a Milvus cluster.
+title: Scale a Milvus Cluster
---
# Scale a Milvus Cluster
diff --git a/preview/site/en/adminGuide/security.md b/preview/site/en/adminGuide/security.md
index 706a70f9d..d974e22d4 100644
--- a/preview/site/en/adminGuide/security.md
+++ b/preview/site/en/adminGuide/security.md
@@ -1,6 +1,7 @@
---
id: security.md
title: Security
+deprecate: true
---
# Security
diff --git a/preview/site/en/adminGuide/tls.md b/preview/site/en/adminGuide/tls.md
index c7966bcbe..dc0ace3fa 100644
--- a/preview/site/en/adminGuide/tls.md
+++ b/preview/site/en/adminGuide/tls.md
@@ -10,6 +10,12 @@ TLS (Transport Layer Security) is an encryption protocol to ensure communication
This topic describes how to enable TLS proxy in Milvus.
+
@@ -263,12 +269,6 @@ authorityKeyIdentifier=keyid,issuer
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
-subjectAltName = @alt_names
-
-[ alt_names ]
-DNS.1 = localhost
-DNS.2 = *.ronething.cn
-DNS.3 = *.ronething.com
[ v3_ca ]
@@ -403,6 +403,8 @@ ess_cert_id_chain = no # Must the ESS cert id chain be included?
The `openssl.cnf` file is a default OpenSSL configuration file. See [manual page](https://www.openssl.org/docs/manmaster/man5/config.html) for more information. The `gen.sh` file generates relevant certificate files. You can modify the `gen.sh` file for different purposes such as changing the validity period of the certificate file, the length of the certificate key or the certificate file names.
+It is necessary to configure the `CommonName` in the `gen.sh` file. The `CommonName` refers to the server name that the client should specify while connecting.
+
```shell
@@ -423,7 +425,7 @@ openssl req -new -x509 -key ca.key -out ca.pem -days 3650 -subj "/C=$Country/ST=
echo "generate server SAN certificate"
openssl genpkey -algorithm RSA -out server.key
-openssl req -new -nodes -key server.key -out server5.csr -days 3650 -subj "/C=$Country/O=$Organization/OU=$Organizational/CN=$CommonName" -config ./openssl.cnf -extensions v3_req
+openssl req -new -nodes -key server.key -out server.csr -days 3650 -subj "/C=$Country/O=$Organization/OU=$Organizational/CN=$CommonName" -config ./openssl.cnf -extensions v3_req
openssl x509 -req -days 3650 -in server.csr -out server.pem -CA ca.pem -CAkey ca.key -CAcreateserial -extfile ./openssl.cnf -extensions v3_req
echo "generate client SAN certificate"
@@ -497,50 +499,117 @@ Open the `server.csr`, the `ca.key` and the `ca.pem` files to sign the certifica
openssl x509 -req -days 3650 -in server.csr -out server.pem -CA ca.pem -CAkey ca.key -CAcreateserial -extfile ./openssl.cnf -extensions v3_req
```
+## Set up a Milvus server with TLS
-## Modify Milvus server configurations
+This section outlines the steps to configure a Milvus server with TLS encryption.
-Set `tlsEnabled` to `true` and configure the file paths of `server.pem`, `server.key`, and `ca.pem` for the server in `config/milvus.yaml`.
+
-```
+ ```
+ docker stop
+ ```
2. Prepare the configuration file `migrate.yaml` for meta migration.
-```
-# migration.yaml
-cmd:
- # Option: run/backup/rollback
- type: run
- runWithBackup: true
-config:
- sourceVersion: 2.1.4 # Specify your milvus version
- targetVersion: 2.2.0
- backupFilePath: /tmp/migration.bak
-metastore:
- type: etcd
-etcd:
- endpoints:
- - milvus-etcd:2379 # Use the etcd container name
- rootPath: by-dev # The root path where data is stored in etcd
- metaSubPath: meta
- kvSubPath: kv
-```
+ ```yaml
+ # migration.yaml
+ cmd:
+ # Option: run/backup/rollback
+ type: run
+ runWithBackup: true
+ config:
+ sourceVersion: 2.1.4 # Specify your milvus version
+ targetVersion: 2.4.5
+ backupFilePath: /tmp/migration.bak
+ metastore:
+ type: etcd
+ etcd:
+ endpoints:
+ - milvus-etcd:2379 # Use the etcd container name
+ rootPath: by-dev # The root path where data is stored in etcd
+ metaSubPath: meta
+ kvSubPath: kv
+ ```
3. Run the migration container.
-```
-# Suppose your docker-compose run with the default milvus network,
-# and you put migration.yaml in the same directory with docker-compose.yaml.
-docker run --rm -it --network milvus -v $(pwd)/migration.yaml:/milvus/configs/migration.yaml milvus/meta-migration:v2.2.0 /milvus/bin/meta-migration -config=/milvus/configs/migration.yaml
-```
+ ```
+ # Suppose your docker-compose run with the default milvus network,
+ # and you put migration.yaml in the same directory with docker-compose.yaml.
+ docker run --rm -it --network milvus -v $(pwd)/migration.yaml:/milvus/configs/migration.yaml milvus/meta-migration:v2.2.0 /milvus/bin/meta-migration -config=/milvus/configs/migration.yaml
+ ```
4. Start Milvus components again with the new Milvus image.
-```
-Update the milvus image tag in the docker-compose.yaml
-docker-compose down
-docker-compose up -d
-```
-
-
-
-
+ ```
+ Update the milvus image tag in the docker-compose.yaml
+ docker compose down
+ docker compose up -d
+ ```
## What's next
- You might also want to learn how to:
diff --git a/preview/site/en/adminGuide/upgrade_milvus_cluster-helm.md b/preview/site/en/adminGuide/upgrade_milvus_cluster-helm.md
index 8a7ef125b..d5f9f1eff 100644
--- a/preview/site/en/adminGuide/upgrade_milvus_cluster-helm.md
+++ b/preview/site/en/adminGuide/upgrade_milvus_cluster-helm.md
@@ -5,9 +5,10 @@ order: 1
group: upgrade_milvus_cluster-operator.md
related_key: upgrade Milvus Cluster
summary: Learn how to upgrade Milvus cluster with Helm Chart.
+title: Upgrade Milvus Cluster with Helm Chart
---
-
+
# Upgrade Milvus Cluster with Helm Chart
@@ -19,34 +20,113 @@ Run the following commands to check new Milvus versions.
```
$ helm repo update
-$ helm search repo milvus --versions
+$ helm search repo zilliztech/milvus --versions
```
+
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0 -r by-dev -o migrate -w
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5 -r by-dev -o rollback -w milvusdb/milvus:v2.1.1
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5 -r by-dev -o migrate -w milvusdb/milvus:v2.4.5
```
-
-
-## Conduct a rolling upgrade
-
-Since Milvus 2.2.3, you can configure Milvus coordinators to work in active-standby mode and enable the rolling upgrade feature for them, so that Milvus can respond to incoming requests during the coordinator upgrades. In previous releases, coordinators are to be removed and then created during an upgrade, which may introduce certain downtime of the service.
-
-Rolling upgrades requires coordinators to work in active-standby mode. You can use [the script](https://raw.githubusercontent.com/milvus-io/milvus/master/deployments/upgrade/rollingUpdate.sh) we provide to configure the coordinators to work in active-standby mode and start the rolling upgrade.
-
-Based on the rolling update capabilities provided by Kubernetes, the above script enforces an ordered update of the deployments according to their dependencies. In addition, Milvus implements a mechanism to ensure that its components remain compatible with those depending on them during the upgrade, significantly reducing potential service downtime.
-
-The script applies only to the upgrade of Milvus installed with Helm. The following table lists the command flags available in the scripts.
-
-| Parameters | Description | Default value | Required |
-| ------------ | ----------------------------------------------------------| -------------------------------- | ----------------------- |
-| `i` | Milvus instance name | `None` | True |
-| `n` | Namespace that Milvus is installed in | `default` | False |
-| `t` | Target Milvus version | `None` | True |
-| `w` | New Milvus image tag | `milvusdb/milvus:v2.2.3` | True |
-| `o` | Operation | `update` | False |
-
-Once you have ensured that all deployments in your Milvus instance are in their normal status. You can run the following command to upgrade the Milvus instance to 2.2.3.
-
-```shell
-sh rollingUpdate.sh -n default -i my-release -o update -t 2.2.3 -w 'milvusdb/milvus:v2.2.3'
-```
-
-
+ image: milvusdb/milvus:v2.4.5
```
In this above configuration file, set `spec.components.enableRollingUpdate` to `true` and set `spec.components.image` to the desired Milvus version.
@@ -121,7 +63,7 @@ spec:
components:
enableRollingUpdate: true
imageUpdateMode: all
- image: milvusdb/milvus:
+ image: milvusdb/milvus:v2.4.5
```
You can set `spec.components.imageUpdateMode` to `rollingDowngrade` to have Milvus replace coordinator pod images with a lower version.
@@ -142,4 +84,86 @@ Then save your configuration as a YAML file (for example, `milvusupgrade.yml`) a
```shell
kubectl apply -f milvusupgrade.yml
-```
\ No newline at end of file
+```
+
+
+## Upgrade Milvus by changing its image
+
+In normal cases, you can simply update your Milvus to the latest by changing its image. However, note that there will be a certain downtime when upgrading Milvus in this way.
+
+Compose a configuration file as follows and save it as **milvusupgrade.yaml**:
+
+```yaml
+apiVersion: milvus.io/v1beta1
+kind: Milvus
+metadata:
+ name: my-release
+spec:
+ # Omit other fields ...
+ components:
+ image: milvusdb/milvus:v2.4.5
+```
+
+Then run the following to perform the upgrade:
+
+```shell
+kubectl apply -f milvusupgrade.yaml
+```
+
+## Migrate the metadata
+
+Since Milvus 2.2.0, the metadata is incompatible with that in previous releases. The following example snippets assume an upgrade from Milvus 2.1.4 to Milvus 2.4.5.
+
+### 1. Create a `.yaml` file for metadata migration
+
+Create a metadata migration file. The following is an example. You need to specify the `name`, `sourceVersion`, and `targetVersion` in the configuration file. The following example sets the `name` to `my-release-upgrade`, `sourceVersion` to `v2.1.4`, and `targetVersion` to `v2.4.5`. This means that your Milvus cluster will be upgraded from v2.1.4 to v2.4.5.
+
+```
+apiVersion: milvus.io/v1beta1
+kind: MilvusUpgrade
+metadata:
+ name: my-release-upgrade
+spec:
+ milvus:
+ namespace: default
+ name: my-release
+ sourceVersion: "v2.1.4"
+ targetVersion: "v2.4.5"
+ # below are some omit default values:
+ # targetImage: "milvusdb/milvus:v2.4.5"
+ # toolImage: "milvusdb/meta-migration:v2.2.0"
+ # operation: upgrade
+ # rollbackIfFailed: true
+ # backupPVC: ""
+ # maxRetry: 3
+```
+
+
+
+### 2. Apply the new configuration
+
+Run the following command to apply the new configuration.
+
+```
+$ kubectl apply -f https://github.com/zilliztech/milvus-operator/blob/main/config/samples/beta/milvusupgrade.yaml
+```
+
+
+
+### 3. Check the status of metadata migration
+
+Run the following command to check the status of your metadata migration.
+
+```
+kubectl describe milvus release-name
+```
+
+The status of `ready` in the output means that the metadata migration is successful.
+
+Or, you can also run `kubectl get pod` to check all the pods. If all the pods are `ready`, the metadata migration is successful.
+
+
+
+### 4. Delete `my-release-upgrade`
+
+When the upgrade is successful, delete `my-release-upgrade` in the YAML file.
diff --git a/preview/site/en/adminGuide/upgrade_milvus_embedded.md b/preview/site/en/adminGuide/upgrade_milvus_embedded.md
index c0fe59f83..76eabccbf 100644
--- a/preview/site/en/adminGuide/upgrade_milvus_embedded.md
+++ b/preview/site/en/adminGuide/upgrade_milvus_embedded.md
@@ -2,6 +2,8 @@
id: upgrade_milvus_embedded.md
related_key: upgrade Embedded Milvus
summary: Learn how to upgrade embedded Milvus.
+title: Upgrade Embedded Milvus
+deprecate: true
---
# Upgrade Embedded Milvus
\ No newline at end of file
diff --git a/preview/site/en/adminGuide/upgrade_milvus_standalone-docker.md b/preview/site/en/adminGuide/upgrade_milvus_standalone-docker.md
index fcfab5e1f..b08193735 100644
--- a/preview/site/en/adminGuide/upgrade_milvus_standalone-docker.md
+++ b/preview/site/en/adminGuide/upgrade_milvus_standalone-docker.md
@@ -1,67 +1,92 @@
---
id: upgrade_milvus_standalone-docker.md
label: Docker Compose
-order: 2
-group: upgrade_milvus_standalone-helm.md
+order: 1
+group: upgrade_milvus_standalone-operator.md
related_key: upgrade Milvus Standalone
summary: Learn how to upgrade Milvus standalone with Docker Compose.
+title: Upgrade Milvus Standalone with Docker Compose
---
-
+
# Upgrade Milvus Standalone with Docker Compose
-A major change in Milvus 2.2 is the meta structure of segment indexes. This topic describes how to use Docker Compose to migrate the meta and upgrade Milvus from v2.1.x to v2.2.0.
+This topic describes how to upgrade your Milvus using Docker Compose.
-## Upgrade Milvus from v2.1.x to v2.2.0
+In normal cases, you can [upgrade Milvus by changing its image](#Upgrade-Milvus-by-changing-its-image). However, you need to [migrate the metadata](#Migrate-the-metadata) before any upgrade from v2.1.x to v2.4.5.
-1. Stop all Milvus components.
+
+ ```
+2. Prepare the configuration file `migration.yaml` for meta migration.
+ ```yaml
+ # migration.yaml
+ cmd:
+ # Option: run/backup/rollback
+ type: run
+ runWithBackup: true
+ config:
+ sourceVersion: 2.1.4 # Specify your milvus version
+ targetVersion: 2.4.5
+ backupFilePath: /tmp/migration.bak
+ metastore:
+ type: etcd
+ etcd:
+ endpoints:
+ - milvus-etcd:2379 # Use the etcd container name
+ rootPath: by-dev # The root path where data is stored in etcd
+ metaSubPath: meta
+ kvSubPath: kv
+ ```
+3. Run the migration container.
+
+ ```
+ # Suppose your docker-compose run with the default milvus network,
+ # and you put migration.yaml in the same directory with docker-compose.yaml.
+ docker run --rm -it --network milvus -v $(pwd)/migration.yaml:/milvus/configs/migration.yaml milvusdb/meta-migration:v2.2.0 /milvus/bin/meta-migration -config=/milvus/configs/migration.yaml
+ ```
+
+4. Start Milvus components again with the new Milvus image.
+ ```shell
+ // Run the following only after update the milvus image tag in the docker-compose.yaml
+ docker compose down
+ docker compose up -d
+ ```
## What's next
- You might also want to learn how to:
diff --git a/preview/site/en/adminGuide/upgrade_milvus_standalone-helm.md b/preview/site/en/adminGuide/upgrade_milvus_standalone-helm.md
index 527dad2eb..899fa9ac3 100644
--- a/preview/site/en/adminGuide/upgrade_milvus_standalone-helm.md
+++ b/preview/site/en/adminGuide/upgrade_milvus_standalone-helm.md
@@ -5,9 +5,10 @@ order: 1
group: upgrade_milvus_standalone-operator.md
related_key: upgrade Milvus Standalone
summary: Learn how to upgrade Milvus standalone with Helm Chart.
+title: Upgrade Milvus Standalone with Helm Chart
---
-
+
# Upgrade Milvus Standalone with Helm Chart
@@ -20,34 +21,114 @@ Run the following commands to check new Milvus versions.
```
$ helm repo update
-$ helm search repo milvus --versions
+$ helm search repo zilliztech/milvus --versions
```
+
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0 -r by-dev -o migrate -w
- ```
-
-## Conduct a rolling upgrade
-
-Since Milvus 2.2.3, you can configure Milvus coordinators to work in active-standby mode and enable the rolling upgrade feature for them, so that Milvus can respond to incoming requests during the coordinator upgrades. In previous releases, coordinators are to be removed and then created during an upgrade, which may introduce certain downtime of the service.
-
-Rolling upgrades requires coordinators to work in active-standby mode. You can use [the script](https://raw.githubusercontent.com/milvus-io/milvus/master/deployments/upgrade/rollingUpdate.sh) we provide to configure the coordinators to work in active-standby mode and start the rolling upgrade.
-
-Based on the rolling update capabilities provided by Kubernetes, the above script enforces an ordered update of the deployments according to their dependencies. In addition, Milvus implements a mechanism to ensure that its components remain compatible with those depending on them during the upgrade, significantly reducing potential service downtime.
-
-The script applies only to the upgrade of Milvus installed with Helm. The following table lists the command flags available in the scripts.
-
-| Parameters | Description | Default value | Required |
-| ------------ | ----------------------------------------------------------| -------------------------------- | ----------------------- |
-| `i` | Milvus instance name | `None` | True |
-| `n` | Namespace that Milvus is installed in | `default` | False |
-| `t` | Target Milvus version | `None` | True |
-| `w` | New Milvus image tag | `milvusdb/milvus:v2.2.3` | True |
-| `o` | Operation | `update` | False |
-
-Once you have ensured that all deployments in your Milvus instance are in their normal status. You can run the following command to upgrade the Milvus instance to 2.2.3.
-
-```shell
-sh rollingUpdate.sh -n default -i my-release -o update -t 2.2.3 -w 'milvusdb/milvus:v2.2.3'
-```
-
-
+ image: milvusdb/milvus:v2.4.5
```
In this above configuration file, set `spec.components.enableRollingUpdate` to `true` and set `spec.components.image` to the desired Milvus version.
-By default, Milvus performs rolling upgrade for coordinators in an ordered way, in which it replaces the coordinator pod images one after another. To reduce the upgrade time, consider setting `spec.components.imageUpdateMode` to `all` so that Milvus replaces all pod images at the same time.
+By default, Milvus performs a rolling upgrade for coordinators in an ordered way, in which it replaces the coordinator pod images one after another. To reduce the upgrade time, consider setting `spec.components.imageUpdateMode` to `all` so that Milvus replaces all pod images at the same time.
```yaml
apiVersion: milvus.io/v1beta1
@@ -121,7 +63,7 @@ spec:
components:
enableRollingUpdate: true
imageUpdateMode: all
- image: milvusdb/milvus:
+ image: milvusdb/milvus:v2.4.5
```
You can set `spec.components.imageUpdateMode` to `rollingDowngrade` to have Milvus replace coordinator pod images with a lower version.
@@ -143,3 +85,84 @@ Then save your configuration as a YAML file (for example, `milvusupgrade.yml`) a
```shell
kubectl apply -f milvusupgrade.yml
```
+
+## Upgrade Milvus by changing its image
+
+In normal cases, you can simply update your Milvus to the latest by changing its image. However, note that there will be a certain downtime when upgrading Milvus in this way.
+
+Compose a configuration file as follows and save it as **milvusupgrade.yaml**:
+
+```yaml
+apiVersion: milvus.io/v1beta1
+kind: Milvus
+metadata:
+ name: my-release
+labels:
+ app: milvus
+spec:
+ # Omit other fields ...
+ components:
+ image: milvusdb/milvus:v2.4.5
+```
+
+Then run the following to perform the upgrade:
+
+```shell
+kubectl apply -f milvusupgrade.yaml
+```
+
+## Migrate the metadata
+
+Since Milvus 2.2.0, the metadata is incompatible with that in previous releases. The following example snippets assume an upgrade from Milvus 2.1.4 to Milvus v2.4.5.
+
+### 1. Create a `.yaml` file for metadata migration
+
+Create a metadata migration file. The following is an example. You need to specify the `name`, `sourceVersion`, and `targetVersion` in the configuration file. The following example sets the `name` to `my-release-upgrade`, `sourceVersion` to `v2.1.4`, and `targetVersion` to `v2.4.5`. This means that your Milvus instance will be upgraded from v2.1.4 to v2.4.5.
+
+```
+apiVersion: milvus.io/v1beta1
+kind: MilvusUpgrade
+metadata:
+ name: my-release-upgrade
+spec:
+ milvus:
+ namespace: default
+ name: my-release
+ sourceVersion: "v2.1.4"
+ targetVersion: "v2.4.5"
+ # below are some omit default values:
+ # targetImage: "milvusdb/milvus:v2.4.5"
+ # toolImage: "milvusdb/meta-migration:v2.2.0"
+ # operation: upgrade
+ # rollbackIfFailed: true
+ # backupPVC: ""
+ # maxRetry: 3
+```
+
+### 2. Apply the new configuration
+
+Run the following command to apply the new configuration.
+
+```
+$ kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvusupgrade.yaml
+```
+
+
+
+### 3. Check the status of metadata migration
+
+Run the following command to check the status of your metadata migration.
+
+```
+kubectl describe milvus release-name
+```
+
+The status of `ready` in the output means that the metadata migration is successful.
+
+Or, you can also run `kubectl get pod` to check all the pods. If all the pods are `ready`, the metadata migration is successful.
+
+
+
+### 4. Delete `my-release-upgrade`
+
+When the upgrade is successful, delete `my-release-upgrade` in the YAML file.
diff --git a/preview/site/en/api.md b/preview/site/en/api.md
deleted file mode 100644
index a7bfa7dfa..000000000
--- a/preview/site/en/api.md
+++ /dev/null
@@ -1,14 +0,0 @@
----
-id: api.md
-title: API References
----
-
-# API References
-
-Choose from the language below and check out the detailed API references for the Milvus vector database.
-
-- [Python](https://milvus.io/api-reference/pymilvus/v2.1.1/About.md)
-- [Java](https://milvus.io/api-reference/java/v2.1.0/About.md)
-- [Go](https://milvus.io/api-reference/go/v2.1.1/About.md)
-- [Node.js](https://milvus.io/api-reference/node/v2.1.2/About.md)
-
diff --git a/preview/site/en/embeddings/embed-with-bgm-m3.md b/preview/site/en/embeddings/embed-with-bgm-m3.md
new file mode 100644
index 000000000..189810f5a
--- /dev/null
+++ b/preview/site/en/embeddings/embed-with-bgm-m3.md
@@ -0,0 +1,103 @@
+---
+id: embed-with-bgm-m3.md
+order: 4
+summary: BGE-M3 is named for its capabilities in Multi-Linguality, Multi-Functionality, and Multi-Granularity.
+title: BGE M3
+---
+
+# BGE M3
+
+[BGE-M3](https://arxiv.org/abs/2402.03216) is named for its capabilities in Multi-Linguality, Multi-Functionality, and Multi-Granularity. Capable of supporting over 100 languages, BGE-M3 sets new benchmarks in multi-lingual and cross-lingual retrieval tasks. Its unique ability to perform dense retrieval, multi-vector retrieval, and sparse retrieval within a single framework makes it an ideal choice for a wide range of information retrieval (IR) applications.
+
+Milvus integrates with the BGE M3 model using the __BGEM3EmbeddingFunction__ class. This class handles the computation of embeddings and returns them in a format compatible with Milvus for indexing and searching. To use this feature, FlagEmbedding must be installed.
+
+To use this feature, install the necessary dependencies:
+
+```bash
+pip install --upgrade pymilvus
+pip install "pymilvus[model]"
+```
+
+Then, instantiate the __BGEM3EmbeddingFunction__:
+
+```python
+from pymilvus.model.hybrid import BGEM3EmbeddingFunction
+
+bge_m3_ef = BGEM3EmbeddingFunction(
+ model_name='BAAI/bge-m3', # Specify the model name
+ device='cpu', # Specify the device to use, e.g., 'cpu' or 'cuda:0'
+ use_fp16=False # Specify whether to use fp16. Set to `False` if `device` is `cpu`.
+)
+```
+
+__Parameters__:
+
+- __model_name__ (_string_)
+
+ The name of the model to use for encoding. The value defaults to __BAAI/bge-m3__.
+
+- __device__ (_string_)
+
+ The device to use, with __cpu__ for the CPU and __cuda:n__ for the nth GPU device.
+
+- __use_fp16__ (_bool_)
+
+ Whether to utilize 16-bit floating-point precision (fp16). Specify __False__ when __device__ is __cpu__.
+
+To create embeddings for documents, use the __encode_documents()__ method:
+
+```python
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+docs_embeddings = bge_m3_ef.encode_documents(docs)
+
+# Print embeddings
+print("Embeddings:", docs_embeddings)
+# Print dimension of dense embeddings
+print("Dense document dim:", bge_m3_ef.dim["dense"], docs_embeddings["dense"][0].shape)
+# Since the sparse embeddings are in a 2D csr_array format, we convert them to a list for easier manipulation.
+print("Sparse document dim:", bge_m3_ef.dim["sparse"], list(docs_embeddings["sparse"])[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: {'dense': [array([-0.02505937, -0.00142193, 0.04015467, ..., -0.02094924,
+ 0.02623661, 0.00324098], dtype=float32), array([ 0.00118463, 0.00649292, -0.00735763, ..., -0.01446293,
+ 0.04243685, -0.01794822], dtype=float32), array([ 0.00415287, -0.0101492 , 0.0009811 , ..., -0.02559666,
+ 0.08084674, 0.00141647], dtype=float32)], 'sparse': <3x250002 sparse array of type ''
+ with 43 stored elements in Compressed Sparse Row format>}
+Dense document dim: 1024 (1024,)
+Sparse document dim: 250002 (1, 250002)
+```
+
+To create embeddings for queries, use the __encode_queries()__ method:
+
+```python
+queries = ["When was artificial intelligence founded",
+ "Where was Alan Turing born?"]
+
+query_embeddings = bge_m3_ef.encode_queries(queries)
+
+# Print embeddings
+print("Embeddings:", query_embeddings)
+# Print dimension of dense embeddings
+print("Dense query dim:", bge_m3_ef.dim["dense"], query_embeddings["dense"][0].shape)
+# Since the sparse embeddings are in a 2D csr_array format, we convert them to a list for easier manipulation.
+print("Sparse query dim:", bge_m3_ef.dim["sparse"], list(query_embeddings["sparse"])[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: {'dense': [array([-0.02024024, -0.01514386, 0.02380808, ..., 0.00234648,
+ -0.00264978, -0.04317448], dtype=float32), array([ 0.00648045, -0.0081542 , -0.02717067, ..., -0.00380103,
+ 0.04200587, -0.01274772], dtype=float32)], 'sparse': <2x250002 sparse array of type ''
+ with 14 stored elements in Compressed Sparse Row format>}
+Dense query dim: 1024 (1024,)
+Sparse query dim: 250002 (1, 250002)
+```
diff --git a/preview/site/en/embeddings/embed-with-bm25.md b/preview/site/en/embeddings/embed-with-bm25.md
new file mode 100644
index 000000000..037cb32a8
--- /dev/null
+++ b/preview/site/en/embeddings/embed-with-bm25.md
@@ -0,0 +1,129 @@
+---
+id: embed-with-bm25.md
+order: 5
+summary: BM25 is a ranking function used in information retrieval to estimate the relevance of documents to a given search query.
+title: BM25
+---
+
+# BM25
+
+[BM25](https://en.wikipedia.org/wiki/Okapi_BM25) is a ranking function used in information retrieval to estimate the relevance of documents to a given search query. It enhances the basic term frequency approach by incorporating document length normalization and term frequency saturation. BM25 can generate sparse embeddings by representing documents as vectors of term importance scores, allowing for efficient retrieval and ranking in sparse vector spaces.
+
+Milvus integrates with the BM25 model using the __BM25EmbeddingFunction__ class. This class handles the computation of embeddings and returns them in a format compatible with Milvus for indexing and searching. Essential to this process is building an analyzer for tokenization.
+
+To use this feature, install the necessary dependencies:
+
+```bash
+pip install --upgrade pymilvus
+pip install "pymilvus[model]"
+```
+
+To easily create a tokenizer, Milvus offers a default analyzer that only requires specifying the language of the text.
+
+
+__Example__:
+
+```python
+from pymilvus.model.sparse.bm25.tokenizers import build_default_analyzer
+from pymilvus.model.sparse import BM25EmbeddingFunction
+
+# there are some built-in analyzers for several languages, now we use 'en' for English.
+analyzer = build_default_analyzer(language="en")
+
+corpus = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+# analyzer can tokenize the text into tokens
+tokens = analyzer(corpus[0])
+print("tokens:", tokens)
+```
+
+__Parameters__:
+
+- __language__ (_string_)
+
+ The language of the text to be tokenized. Valid options are __en__ (English), __de__ (German), __fr__ (French), __ru__ (Russian), __sp__ (Spanish), __it__ (Italian), __pt__ (Portuguese), __zh__ (Chinese), __jp__ (Japanese), __kr__ (Korean).
+
+The expected output is similar to the following:
+
+```python
+tokens: ['artifici', 'intellig', 'found', 'academ', 'disciplin', '1956']
+```
+
+The BM25 algorithm processes text by first breaking it into tokens using a built-in analyzer, as shown with English language tokens like __'artifici'__, __'intellig'__, and __'academ'__. It then gathers statistics on these tokens, evaluating their frequency and distribution across documents. The core of BM25 calculates the relevance score of each token based on its importance, with rarer tokens receiving higher scores. This concise process enables effective ranking of documents by relevance to a query.
+
+To gather statistics on the corpus, use the __fit()__ method:
+
+```python
+# Use the analyzer to instantiate the BM25EmbeddingFunction
+bm25_ef = BM25EmbeddingFunction(analyzer)
+
+# Fit the model on the corpus to get the statstics of the corpus
+bm25_ef.fit(corpus)
+```
+
+Then, use __encode_documents()__ to create embeddings for documents:
+
+```python
+docs = [
+ "The field of artificial intelligence was established as an academic subject in 1956.",
+ "Alan Turing was the pioneer in conducting significant research in artificial intelligence.",
+ "Originating in Maida Vale, London, Turing grew up in the southern regions of England.",
+ "In 1956, artificial intelligence emerged as a scholarly field.",
+ "Turing, originally from Maida Vale, London, was brought up in the south of England."
+]
+
+# Create embeddings for the documents
+docs_embeddings = bm25_ef.encode_documents(docs)
+
+# Print embeddings
+print("Embeddings:", docs_embeddings)
+# Since the output embeddings are in a 2D csr_array format, we convert them to a list for easier manipulation.
+print("Sparse dim:", bm25_ef.dim, list(docs_embeddings)[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: (0, 0) 1.0208816705336425
+ (0, 1) 1.0208816705336425
+ (0, 3) 1.0208816705336425
+...
+ (4, 16) 0.9606986899563318
+ (4, 17) 0.9606986899563318
+ (4, 20) 0.9606986899563318
+Sparse dim: 21 (1, 21)
+```
+
+To create embeddings for queries, use the __encode_queries()__ method:
+
+```python
+queries = ["When was artificial intelligence founded",
+ "Where was Alan Turing born?"]
+
+query_embeddings = bm25_ef.encode_queries(queries)
+
+# Print embeddings
+print("Embeddings:", query_embeddings)
+# Since the output embeddings are in a 2D csr_array format, we convert them to a list for easier manipulation.
+print("Sparse dim:", bm25_ef.dim, list(query_embeddings)[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: (0, 0) 0.5108256237659907
+ (0, 1) 0.5108256237659907
+ (0, 2) 0.5108256237659907
+ (1, 6) 0.5108256237659907
+ (1, 7) 0.11554389108992644
+ (1, 14) 0.5108256237659907
+Sparse dim: 21 (1, 21)
+```
+
+__Notes:__
+
+When using __BM25EmbeddingFunction__, note that __encoding_queries()__ and __encoding_documents()__ operations cannot be interchanged mathematically. Therefore, there is no implemented __bm25_ef(texts)__ available.
\ No newline at end of file
diff --git a/preview/site/en/embeddings/embed-with-cohere.md b/preview/site/en/embeddings/embed-with-cohere.md
new file mode 100644
index 000000000..c83dbc10a
--- /dev/null
+++ b/preview/site/en/embeddings/embed-with-cohere.md
@@ -0,0 +1,114 @@
+---
+id: embed-with-cohere.md
+order: 9
+summary: This article describes how to use the CohereEmbeddingFunction to encode documents and queries using the Cohere embedding model.
+title: Embed Cohere
+---
+
+# Cohere
+
+Cohere's embedding models are used to generate text embeddings, which are lists of floating-point numbers that capture semantic information about the text. These embeddings can be used for tasks like text classification and semantic search.
+
+Milvus integrates with Cohere's embedding models using the `CohereEmbeddingFunction` class. This class handles the computation of embeddings and returns them in a format compatible with Milvus for indexing and searching.
+
+To use this feature, install the necessary dependencies:
+
+```bash
+pip install --upgrade pymilvus
+pip install "pymilvus[model]"
+```
+
+Then, instantiate the `CohereEmbeddingFunction`:
+
+```python
+cohere_ef = CohereEmbeddingFunction(
+ model_name="embed-english-light-v3.0",
+ api_key="YOUR_COHERE_API_KEY",
+ input_type="search_document",
+ embedding_types=["float"]
+)
+```
+
+__Parameters__:
+
+- `model_name` (*string*)
+
+ The name of the Cohere embedding model to use for encoding. You can specify any of the available Cohere embedding model names, for example, `embed-english-v3.0`, `embed-multilingual-v3.0`, etc. If you leave this parameter unspecified, `embed-english-light-v3.0` will be used. For a list of available models, refer to [Embed](https://docs.cohere.com/docs/models#embed).
+
+- `api_key` (*string*)
+
+ The API key for accessing the Cohere API.
+
+- `input_type` (*string*)
+
+ The type of input passed to the model. Required for embedding models v3 and higher.
+
+ - `"search_document"`: Used for embeddings stored in a vector database for search use-cases.
+ - `"search_query"`: Used for embeddings of search queries run against a vector DB to find relevant documents.
+ - `"classification"`: Used for embeddings passed through a text classifier.
+ - `"clustering"`: Used for the embeddings run through a clustering algorithm.
+
+- `embedding_types` (*List[str]*)
+
+ The type of embeddings you want to get back. Not required and default is None, which returns the Embed Floats response type. Currently, you can only specify a single value for this parameter. Possible values:
+
+ - `"float"`: Use this when you want to get back the default float embeddings. Valid for all models.
+ - `"binary"`: Use this when you want to get back signed binary embeddings. Valid for only v3 models.
+ - `"ubinary"`: Use this when you want to get back unsigned binary embeddings. Valid for only v3 models.
+
+To create embeddings for documents, use the `encode_documents()` method:
+
+```python
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+docs_embeddings = cohere_ef.encode_documents(docs)
+
+# Print embeddings
+print("Embeddings:", docs_embeddings)
+# Print dimension and shape of embeddings
+print("Dim:", cohere_ef.dim, docs_embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([ 3.43322754e-02, 1.16252899e-03, -5.25207520e-02, 1.32846832e-03,
+ -6.80541992e-02, 6.10961914e-02, -7.06176758e-02, 1.48925781e-01,
+ 1.54174805e-01, 1.98516846e-02, 2.43835449e-02, 3.55224609e-02,
+ 1.82952881e-02, 7.57446289e-02, -2.40783691e-02, 4.40063477e-02,
+...
+ 0.06359863, -0.01971436, -0.02253723, 0.00354195, 0.00222015,
+ 0.00184727, 0.03408813, -0.00777817, 0.04919434, 0.01519775,
+ -0.02862549, 0.04760742, -0.07891846, 0.0124054 ], dtype=float32)]
+Dim: 384 (384,)
+```
+
+To create embeddings for queries, use the `encode_queries()` method:
+
+```python
+queries = ["When was artificial intelligence founded",
+ "Where was Alan Turing born?"]
+
+query_embeddings = cohere_ef.encode_queries(queries)
+
+print("Embeddings:", query_embeddings)
+print("Dim", cohere_ef.dim, query_embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([-1.33361816e-02, 9.79423523e-04, -7.28759766e-02, -1.93786621e-02,
+ -9.71679688e-02, 4.34875488e-02, -9.81445312e-02, 1.16882324e-01,
+ 5.89904785e-02, -4.19921875e-02, 4.95910645e-02, 5.83496094e-02,
+ 3.47595215e-02, -5.87463379e-03, -7.30514526e-03, 2.92816162e-02,
+...
+ 0.00749969, -0.01192474, 0.02719116, 0.03347778, 0.07696533,
+ 0.01409149, 0.00964355, -0.01681519, -0.0073204 , 0.00043154,
+ -0.04577637, 0.03591919, -0.02807617, -0.04812622], dtype=float32)]
+Dim 384 (384,)
+```
diff --git a/preview/site/en/embeddings/embed-with-jina.md b/preview/site/en/embeddings/embed-with-jina.md
new file mode 100644
index 000000000..2db58a5fd
--- /dev/null
+++ b/preview/site/en/embeddings/embed-with-jina.md
@@ -0,0 +1,97 @@
+---
+id: embed-with-jina.md
+order: 8
+summary: This article describes how to use the JinaEmbeddingFunction to encode documents and queries using the Jina AI embedding model.
+title: Jina AI - Embed
+---
+
+# Jina AI
+
+Jina AI's embedding models are high-performance text embedding models that can translate textual inputs into numerical representations, capturing the semantics of the text. These models excel in applications like dense retrieval, semantic textual similarity, and multilingual understanding.
+
+Milvus integrates with Jina AI's embedding models via the `JinaEmbeddingFunction` class. This class provides methods for encoding documents and queries using the Jina AI embedding models and returning the embeddings as dense vectors compatible with Milvus indexing. To utilize this functionality, obtain an API key from [Jina AI](https://jina.ai/embeddings/).
+
+To use this feature, install the necessary dependencies:
+
+```bash
+pip install --upgrade pymilvus
+pip install "pymilvus[model]"
+```
+
+Then, instantiate the `JinaEmbeddingFunction`:
+
+```python
+from pymilvus.model.dense import JinaEmbeddingFunction
+
+jina_ef = JinaEmbeddingFunction(
+ model_name="jina-embeddings-v2-base-en", # Defaults to `jina-embeddings-v2-base-en`
+ api_key=JINAAI_API_KEY # Provide your Jina AI API key
+)
+```
+
+__Parameters__:
+
+- `model_name` (*string*)
+
+ The name of the Jina AI embedding model to use for encoding. You can specify any of the available Jina AI embedding model names, for example, `jina-embeddings-v2-base-en`, `jina-embeddings-v2-small-en`, etc. If you leave this parameter unspecified, `jina-embeddings-v2-base-en` will be used. For a list of available models, refer to [Jina Embeddings](https://jina.ai/embeddings).
+
+- `api_key` (*string*)
+
+ The API key for accessing the Jina AI API.
+
+To create embeddings for documents, use the `encode_documents()` method:
+
+```python
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+docs_embeddings = jina_ef.encode_documents(docs)
+
+# Print embeddings
+print("Embeddings:", docs_embeddings)
+# Print dimension and shape of embeddings
+print("Dim:", jina_ef.dim, docs_embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([-4.88487840e-01, -4.28095880e-01, 4.90086500e-01, -1.63274320e-01,
+ 3.43437800e-01, 3.21476880e-01, 2.83173790e-02, -3.10403670e-01,
+ 4.76985040e-01, -1.77410420e-01, -3.84803180e-01, -2.19224200e-01,
+ -2.52898000e-01, 6.62411900e-02, -8.58173100e-01, 1.05221800e+00,
+...
+ -2.04462400e-01, 7.14229800e-01, -1.66823000e-01, 8.72551440e-01,
+ 5.53560140e-01, 8.92506300e-01, -2.39408610e-01, -4.22413560e-01,
+ -3.19551350e-01, 5.59153850e-01, 2.44338100e-01, -8.60452100e-01])]
+Dim: 768 (768,)
+```
+
+To create embeddings for queries, use the `encode_queries()` method:
+
+```python
+queries = ["When was artificial intelligence founded",
+ "Where was Alan Turing born?"]
+
+query_embeddings = jina_ef.encode_queries(queries)
+
+print("Embeddings:", query_embeddings)
+print("Dim", jina_ef.dim, query_embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([-5.99164660e-01, -3.49827350e-01, 8.22405160e-01, -1.18632730e-01,
+ 5.78107540e-01, 1.09789170e-01, 2.91604200e-01, -3.29306450e-01,
+ 2.93779640e-01, -2.17880800e-01, -6.84535440e-01, -3.79752000e-01,
+ -3.47541800e-01, 9.20846100e-02, -6.13804400e-01, 6.31312800e-01,
+...
+ -1.84993740e-02, 9.38629150e-01, 2.74858470e-02, 1.09396360e+00,
+ 3.96270750e-01, 7.44445800e-01, -1.95404050e-01, -6.08383200e-01,
+ -3.75076300e-01, 3.87512200e-01, 8.11889650e-01, -3.76407620e-01])]
+Dim 768 (768,)
+```
diff --git a/preview/site/en/embeddings/embed-with-openai.md b/preview/site/en/embeddings/embed-with-openai.md
new file mode 100644
index 000000000..d47be7402
--- /dev/null
+++ b/preview/site/en/embeddings/embed-with-openai.md
@@ -0,0 +1,100 @@
+---
+id: embed-with-openai.md
+order: 2
+summary: Milvus integrates with OpenAI's models via the OpenAIEmbeddingFunction class.
+title: OpenAI
+---
+
+# OpenAI
+
+Milvus integrates with OpenAI's models via the __OpenAIEmbeddingFunction__ class. This class provides methods for encoding documents and queries using the pretrained OpenAI models and returning the embeddings as dense vectors compatible with Milvus indexing. To utilize this functionality, obtain an API key from [OpenAI](https://openai.com/api/) by creating an account on their platform.
+
+To use this feature, install the necessary dependencies:
+
+```bash
+pip install --upgrade pymilvus
+pip install "pymilvus[model]"
+```
+
+Then, instantiate the __OpenAIEmbeddingFunction__:
+
+```python
+from pymilvus import model
+
+openai_ef = model.dense.OpenAIEmbeddingFunction(
+ model_name='text-embedding-3-large', # Specify the model name
+ api_key='YOUR_API_KEY', # Provide your OpenAI API key
+ dimensions=512 # Set the embedding dimensionality
+)
+```
+
+__Parameters__:
+
+- __model_name__ (_string_)
+
+ The name of the OpenAI model to use for encoding. Valid options are __text-embedding-3-small__, __text-embedding-3-large__, and __text-embedding-ada-002__ (default).
+
+- __api_key__ (_string_)
+
+ The API key for accessing the OpenAI API.
+
+- __dimensions__ (_int_)
+
+ The number of dimensions the resulting output embeddings should have. Only supported in __text-embedding-3__ and later models.
+
+To create embeddings for documents, use the __encode_documents()__ method:
+
+```python
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+docs_embeddings = openai_ef.encode_documents(docs)
+
+# Print embeddings
+print("Embeddings:", docs_embeddings)
+# Print dimension and shape of embeddings
+print("Dim:", openai_ef.dim, docs_embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([ 1.76741909e-02, -2.04964578e-02, -1.09788161e-02, -5.27223349e-02,
+ 4.23139781e-02, -6.64533582e-03, 4.21088142e-03, 1.04644023e-01,
+ 5.10009527e-02, 5.32827862e-02, -3.26061808e-02, -3.66494283e-02,
+...
+ -8.93232748e-02, 6.68255147e-03, 3.55093405e-02, -5.09071983e-02,
+ 3.74144339e-03, 4.72541340e-02, 2.11916920e-02, 1.00753829e-02,
+ -5.76633997e-02, 9.68257990e-03, 4.62721288e-02, -4.33261096e-02])]
+Dim: 512 (512,)
+```
+
+To create embeddings for queries, use the __encode_queries()__ method:
+
+```python
+queries = ["When was artificial intelligence founded",
+ "Where was Alan Turing born?"]
+
+query_embeddings = openai_ef.encode_queries(queries)
+
+# Print embeddings
+print("Embeddings:", query_embeddings)
+# Print dimension and shape of embeddings
+print("Dim", openai_ef.dim, query_embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([ 0.00530251, -0.01907905, -0.01672608, -0.05030033, 0.01635982,
+ -0.03169853, -0.0033602 , 0.09047844, 0.00030747, 0.11853652,
+ -0.02870182, -0.01526102, 0.05505067, 0.00993909, -0.07165466,
+...
+ -9.78106782e-02, -2.22669560e-02, 1.21873049e-02, -4.83198799e-02,
+ 5.32377362e-02, -1.90469325e-02, 5.62430918e-02, 1.02650477e-02,
+ -6.21757433e-02, 7.88027793e-02, 4.91846527e-04, -1.51633881e-02])]
+Dim 512 (512,)
+```
diff --git a/preview/site/en/embeddings/embed-with-sentence-transform.md b/preview/site/en/embeddings/embed-with-sentence-transform.md
new file mode 100644
index 000000000..351bc22c5
--- /dev/null
+++ b/preview/site/en/embeddings/embed-with-sentence-transform.md
@@ -0,0 +1,96 @@
+---
+id: embed-with-sentence-transform.md
+order: 3
+summary: This article demonstrates how to use Sentence Transformers in Milvus to encode documents and queries into dense vectors.
+title: Sentence Transformers
+---
+
+# Sentence Transformers
+
+Milvus integrates with [Sentence Transformer](https://www.sbert.net/docs/pretrained_models.html#model-overview) pre-trained models via the __SentenceTransformerEmbeddingFunction__ class. This class provides methods for encoding documents and queries using the pretrained Sentence Transformer models and returning the embeddings as dense vectors compatible with Milvus indexing.
+
+To use this feature, install the necessary dependencies:
+
+```bash
+pip install --upgrade pymilvus
+pip install "pymilvus[model]"
+```
+
+Then, instantiate the __SentenceTransformerEmbeddingFunction__:
+
+```python
+from pymilvus import model
+
+sentence_transformer_ef = model.dense.SentenceTransformerEmbeddingFunction(
+ model_name='all-MiniLM-L6-v2', # Specify the model name
+ device='cpu' # Specify the device to use, e.g., 'cpu' or 'cuda:0'
+)
+```
+
+__Parameters__:
+
+- __model_name__ (_string_)
+
+ The name of the Sentence Transformer model to use for encoding. The value defaults to __all-MiniLM-L6-v2__. You can use any of Sentence Transformers' pre-trained models. For a list of available models, refer to [Pretrained models](https://www.sbert.net/docs/pretrained_models.html).
+
+- __device__ (_string_)
+
+ The device to use, with __cpu__ for the CPU and __cuda:n__ for the nth GPU device.
+
+To create embeddings for documents, use the __encode_documents()__ method:
+
+```python
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+docs_embeddings = sentence_transformer_ef.encode_documents(docs)
+
+# Print embeddings
+print("Embeddings:", docs_embeddings)
+# Print dimension and shape of embeddings
+print("Dim:", sentence_transformer_ef.dim, docs_embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([-3.09392996e-02, -1.80662833e-02, 1.34775648e-02, 2.77156215e-02,
+ -4.86349640e-03, -3.12581174e-02, -3.55921760e-02, 5.76934684e-03,
+ 2.80773244e-03, 1.35783911e-01, 3.59678417e-02, 6.17732145e-02,
+...
+ -4.61330153e-02, -4.85207550e-02, 3.13997865e-02, 7.82178566e-02,
+ -4.75336798e-02, 5.21207601e-02, 9.04406682e-02, -5.36676683e-02],
+ dtype=float32)]
+Dim: 384 (384,)
+```
+
+To create embeddings for queries, use the __encode_queries()__ method:
+
+```python
+queries = ["When was artificial intelligence founded",
+ "Where was Alan Turing born?"]
+
+query_embeddings = sentence_transformer_ef.encode_queries(queries)
+
+# Print embeddings
+print("Embeddings:", query_embeddings)
+# Print dimension and shape of embeddings
+print("Dim:", sentence_transformer_ef.dim, query_embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([-2.52114702e-02, -5.29330298e-02, 1.14570223e-02, 1.95571519e-02,
+ -2.46500354e-02, -2.66519729e-02, -8.48201662e-03, 2.82961670e-02,
+ -3.65092754e-02, 7.50745758e-02, 4.28900979e-02, 7.18822703e-02,
+...
+ -6.76431581e-02, -6.45996556e-02, -4.67132553e-02, 4.78532910e-02,
+ -2.31596199e-03, 4.13446948e-02, 1.06935494e-01, -1.08258888e-01],
+ dtype=float32)]
+Dim: 384 (384,)
+```
+
diff --git a/preview/site/en/embeddings/embed-with-splade.md b/preview/site/en/embeddings/embed-with-splade.md
new file mode 100644
index 000000000..928c01946
--- /dev/null
+++ b/preview/site/en/embeddings/embed-with-splade.md
@@ -0,0 +1,97 @@
+---
+id: embed-with-splade.md
+order: 6
+summary: This article describes how to use the SpladeEmbeddingFunction to encode documents and queries using the SPLADE model.
+title: SPLADE
+---
+
+# SPLADE
+
+[SPLADE](https://arxiv.org/abs/2109.10086) embedding is a model that offers highly sparse representations for documents and queries, inheriting desirable properties from bag-of-words (BOW) models such as exact term matching and efficiency.
+
+Milvus integrates with the SPLADE model via the __SpladeEmbeddingFunction__ class. This class provides methods for encoding documents and queries and returning the embeddings as sparse vectors compatible with Milvus indexing.
+
+To use this feature, install the necessary dependencies:
+
+```bash
+pip install --upgrade pymilvus
+pip install "pymilvus[model]"
+```
+
+To instantiate the __SpladeEmbeddingFunction__, use the command:
+
+```python
+from pymilvus import model
+
+splade_ef = model.sparse.SpladeEmbeddingFunction(
+ model_name="naver/splade-cocondenser-selfdistil",
+ device="cpu"
+)
+```
+
+__Parameters__:
+
+- __model_name__ (_string_)
+
+ The name of the SPLADE model to use for encoding. Valid options are __naver/splade-cocondenser-ensembledistil__ (default), __naver/splade_v2_max__, __naver/splade_v2_distil__, and __naver/splade-cocondenser-selfdistil__. For more information, refer to [Play with models](https://github.com/naver/splade?tab=readme-ov-file#playing-with-the-model).
+
+- __device__ (_string_)
+
+ The device to use, with __cpu__ for the CPU and __cuda:n__ for the nth GPU device.
+
+To create embeddings for documents, use the __encode_documents()__ method:
+
+```python
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+docs_embeddings = splade_ef.encode_documents(docs)
+
+# Print embeddings
+print("Embeddings:", docs_embeddings)
+# since the output embeddings are in a 2D csr_array format, we convert them to a list for easier manipulation.
+print("Sparse dim:", splade_ef.dim, list(docs_embeddings)[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: (0, 2001) 0.6392706036567688
+ (0, 2034) 0.024093208834528923
+ (0, 2082) 0.3230178654193878
+...
+ (2, 23602) 0.5671860575675964
+ (2, 26757) 0.5770265460014343
+ (2, 28639) 3.1990697383880615
+Sparse dim: 30522 (1, 30522)
+```
+
+To create embeddings for queries, use the __encode_queries()__ method:
+
+```python
+queries = ["When was artificial intelligence founded",
+ "Where was Alan Turing born?"]
+
+query_embeddings = splade_ef.encode_queries(queries)
+
+# Print embeddings
+print("Embeddings:", query_embeddings)
+# since the output embeddings are in a 2D csr_array format, we convert them to a list for easier manipulation.
+print("Sparse dim:", splade_ef.dim, list(query_embeddings)[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: (0, 2001) 0.6353746056556702
+ (0, 2194) 0.015553371049463749
+ (0, 2301) 0.2756537199020386
+...
+ (1, 18522) 0.1282549500465393
+ (1, 23602) 0.13133203983306885
+ (1, 28639) 2.8150033950805664
+Sparse dim: 30522 (1, 30522)
+```
diff --git a/preview/site/en/embeddings/embed-with-voyage.md b/preview/site/en/embeddings/embed-with-voyage.md
new file mode 100644
index 000000000..169733c87
--- /dev/null
+++ b/preview/site/en/embeddings/embed-with-voyage.md
@@ -0,0 +1,83 @@
+---
+id: embed-with-voyage.md
+order: 7
+summary: This article describes how to use the VoyageEmbeddingFunction to encode documents and queries using the Voyage model.
+title: Embed Voyage
+---
+
+# Voyage
+
+Milvus integrates with Voyage's models via the VoyageEmbeddingFunction class. This class provides methods for encoding documents and queries using the Voyage models and returning the embeddings as dense vectors compatible with Milvus indexing. To utilize this functionality, obtain an API key from [Voyage](https://docs.voyageai.com/docs/api-key-and-installation) by creating an account on their platform.
+
+To use this feature, install the necessary dependencies:
+
+```bash
+pip install --upgrade pymilvus
+pip install "pymilvus[model]"
+```
+
+Then, instantiate the `VoyageEmbeddingFunction`:
+
+```python
+from pymilvus.model.dense import VoyageEmbeddingFunction
+
+voyage_ef = VoyageEmbeddingFunction(
+ model_name="voyage-lite-02-instruct", # Defaults to `voyage-2`
+ api_key=VOYAGE_API_KEY # Provide your Voyage API key
+)
+```
+
+__Parameters__:
+
+- `model_name` (string)
+ The name of the Voyage model to use for encoding. You can specify any of the available Voyage model names, for example, `voyage-law-2`, `voyage-code-2`, etc. If you leave this parameter unspecified, `voyage-2` will be used. For a list of available models, refer to [Voyage official documentation](https://docs.voyageai.com/docs/embeddings).
+- `api_key` (string)
+ The API key for accessing the Voyage API. For information on how to create an API key, refer to [API Key and Python Client](https://docs.voyageai.com/docs/api-key-and-installation).
+
+To create embeddings for documents, use the `encode_documents()` method:
+
+```python
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+docs_embeddings = voyage_ef.encode_documents(docs)
+
+# Print embeddings
+print("Embeddings:", docs_embeddings)
+# Print dimension and shape of embeddings
+print("Dim:", voyage_ef.dim, docs_embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([ 0.02582654, -0.00907086, -0.04604037, ..., -0.01227521,
+ 0.04420955, -0.00038829]), array([ 0.03844212, -0.01597065, -0.03728884, ..., -0.02118733,
+ 0.03349845, 0.0065346 ]), array([ 0.05143557, -0.01096631, -0.02690451, ..., -0.02416254,
+ 0.07658645, 0.03064499])]
+Dim: 1024 (1024,)
+```
+
+To create embeddings for queries, use the `encode_queries()` method:
+
+```python
+queries = ["When was artificial intelligence founded",
+ "Where was Alan Turing born?"]
+
+query_embeddings = voyage_ef.encode_queries(queries)
+
+print("Embeddings:", query_embeddings)
+print("Dim", voyage_ef.dim, query_embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([ 0.01733501, -0.0230672 , -0.05208827, ..., -0.00957995,
+ 0.04493361, 0.01485138]), array([ 0.05937521, -0.00729363, -0.02184347, ..., -0.02107683,
+ 0.05706626, 0.0263358 ])]
+Dim 1024 (1024,)
+```
\ No newline at end of file
diff --git a/preview/site/en/embeddings/embeddings.md b/preview/site/en/embeddings/embeddings.md
new file mode 100644
index 000000000..2d140b48c
--- /dev/null
+++ b/preview/site/en/embeddings/embeddings.md
@@ -0,0 +1,162 @@
+---
+id: embeddings.md
+order: 1
+summary: Learn how to generate embeddings for your data.
+title: Embedding Overview
+---
+
+# Embedding Overview
+
+Embedding is a machine learning concept for mapping data into a high-dimensional space, where data of similar semantic are placed close together. Typically being a Deep Neural Network from BERT or other Transformer families, the embedding model can effectively represent the semantics of text, images, and other data types with a series of numbers known as vectors. A key feature of these models is that the mathematical distance between vectors in the high-dimensional space can indicate the similarity of the semantics of original text or images. This property unlocks many information retrieval applications, such as web search engines like Google and Bing, product search and recommendations on e-commerce sites, and the recently popular Retrieval Augmented Generation (RAG) paradigm in generative AI.
+
+There are two main categories of embeddings, each producing a different type of vector:
+
+- __Dense embedding__: Most embedding models represent information as a floating point vector of hundreds to thousands of dimensions. The output is called "dense" vectors as most dimensions have non-zero values. For instance, the popular open-source embedding model BAAI/bge-base-en-v1.5 outputs vectors of 768 floating point numbers (768-dimension float vector).
+
+- __Sparse embedding__: In contrast, the output vectors of sparse embeddings has most dimensions being zero, namely "sparse" vectors. These vectors often have much higher dimensions (tens of thousands or more) which is determined by the size of the token vocabulary. Sparse vectors can be generated by Deep Neural Networks or statistical analysis of text corpora. Due to their interpretability and observed better out-of-domain generalization capabilities, sparse embeddings are increasingly adopted by developers as a complement to dense embeddings.
+
+Milvus is a vector database designed for vector data management, storage, and retrieval. By integrating mainstream embedding and [reranking](https://milvus.io/docs/rerankers-overview.md) models, you can easily transform original text into searchable vectors or rerank the results using powerful models to achieve more accurate results for RAG. This integration simplifies text transformation and eliminates the need for additional embedding or reranking components, thereby streamlining RAG development and validation.
+
+To create embeddings in action, refer to [Using PyMilvus's Model To Generate Text Embeddings](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/model/embedding_functions.ipynb).
+
+| Embedding Function | Type | API or Open-sourced |
+| ------------------------------------------------------------------------------------- | ------- | -------------------- |
+| [openai](https://milvus.io/api-reference/pymilvus/v2.4.x/EmbeddingModels/OpenAIEmbeddingFunction/OpenAIEmbeddingFunction.md) | Dense | API |
+| [sentence-transformer](https://milvus.io/api-reference/pymilvus/v2.4.x/EmbeddingModels/SentenceTransformerEmbeddingFunction/SentenceTransformerEmbeddingFunction.md) | Dense | Open-sourced |
+| [bm25](https://milvus.io/api-reference/pymilvus/v2.4.x/EmbeddingModels/BM25EmbeddingFunction/BM25EmbeddingFunction.md) | Sparse | Open-sourced |
+| [Splade](https://milvus.io/api-reference/pymilvus/v2.4.x/EmbeddingModels/SpladeEmbeddingFunction/SpladeEmbeddingFunction.md) | Sparse | Open-sourced |
+| [bge-m3](https://milvus.io/api-reference/pymilvus/v2.4.x/EmbeddingModels/BGEM3EmbeddingFunction/BGEM3EmbeddingFunction.md) | Hybrid | Open-sourced |
+| [voyageai](https://milvus.io/api-reference/pymilvus/v2.4.x/EmbeddingModels/VoyageEmbeddingFunction/VoyageEmbeddingFunction.md) | Dense | API |
+| [jina](https://milvus.io/api-reference/pymilvus/v2.4.x/EmbeddingModels/JinaEmbeddingFunction/JinaEmbeddingFunction.md) | Dense | API |
+| [cohere](https://milvus.io/api-reference/pymilvus/v2.4.x/EmbeddingModels/CohereEmbeddingFunction/CohereEmbeddingFunction.md) | Dense | API |
+
+## Example 1: Use default embedding function to generate dense vectors
+
+To use embedding functions with Milvus, first install the PyMilvus client library with the `model` subpackage that wraps all the utilities for embedding generation.
+
+```python
+pip install "pymilvus[model]"
+```
+
+The `model` subpackage supports various embedding models, from [OpenAI](https://milvus.io/docs/embed-with-openai.md), [Sentence Transformers](https://milvus.io/docs/embed-with-sentence-transform.md), [BGE M3](https://milvus.io/docs/embed-with-bgm-m3.md), [BM25](https://milvus.io/docs/embed-with-bm25.md), to [SPLADE](https://milvus.io/docs/embed-with-splade.md) pretrained models. For simpilicity, this example uses the `DefaultEmbeddingFunction` which is __all-MiniLM-L6-v2__ sentence transformer model, the model is about 70MB and it will be downloaded during first use:
+
+```python
+from pymilvus import model
+
+# This will download "all-MiniLM-L6-v2", a light weight model.
+ef = model.DefaultEmbeddingFunction()
+
+# Data from which embeddings are to be generated
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+embeddings = ef.encode_documents(docs)
+
+# Print embeddings
+print("Embeddings:", embeddings)
+# Print dimension and shape of embeddings
+print("Dim:", ef.dim, embeddings[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Embeddings: [array([-3.09392996e-02, -1.80662833e-02, 1.34775648e-02, 2.77156215e-02,
+ -4.86349640e-03, -3.12581174e-02, -3.55921760e-02, 5.76934684e-03,
+ 2.80773244e-03, 1.35783911e-01, 3.59678417e-02, 6.17732145e-02,
+...
+ -4.61330153e-02, -4.85207550e-02, 3.13997865e-02, 7.82178566e-02,
+ -4.75336798e-02, 5.21207601e-02, 9.04406682e-02, -5.36676683e-02],
+ dtype=float32)]
+Dim: 384 (384,)
+```
+
+## Example 2: Generate dense and sparse vectors in one call with BGE M3 model
+
+In this example, we use [BGE M3](https://milvus.io/docs/embed-with-bgm-m3.md) hybrid model to embed text into both dense and sparse vectors and use them to retrieve relevant documents. The overall steps are as follows:
+
+1. Embed the text as dense and sparse vectors using BGE-M3 model;
+
+1. Set up a Milvus collection to store the dense and sparse vectors;
+
+1. Insert the data to Milvus;
+
+1. Search and inspect the result.
+
+First, we need to install the necessary dependencies.
+
+```python
+from pymilvus.model.hybrid import BGEM3EmbeddingFunction
+from pymilvus import (
+ utility,
+ FieldSchema, CollectionSchema, DataType,
+ Collection, AnnSearchRequest, RRFRanker, connections,
+)
+```
+
+Use BGE M3 to encode docs and queries for embedding retrieval.
+
+```python
+# 1. prepare a small corpus to search
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+query = "Who started AI research?"
+
+# BGE-M3 model can embed texts as dense and sparse vectors.
+# It is included in the optional `model` module in pymilvus, to install it,
+# simply run "pip install pymilvus[model]".
+
+bge_m3_ef = BGEM3EmbeddingFunction(use_fp16=False, device="cpu")
+
+docs_embeddings = bge_m3_ef(docs)
+query_embeddings = bge_m3_ef([query])
+```
+
+## Example 3: Generate sparse vectors using BM25 model
+
+BM25 is a well-known method that uses word occurrence frequencies to determine the relevance between queries and documents. In this example, we will show how to use `BM25EmbeddingFunction` to generate sparse embeddings for both queries and documents.
+
+First, import the __BM25EmbeddingFunction__ class.
+
+```xml
+from pymilvus.model.sparse import BM25EmbeddingFunction
+```
+
+In BM25, it's important to calculate the statistics in your documents to obtain the IDF (Inverse Document Frequency), which can represent the pattern in your documents. The IDF is a measure of how much information a word provides, that is, whether it's common or rare across all documents.
+
+```python
+# 1. prepare a small corpus to search
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+query = "Where was Turing born?"
+bm25_ef = BM25EmbeddingFunction()
+
+# 2. fit the corpus to get BM25 model parameters on your documents.
+bm25_ef.fit(docs)
+
+# 3. store the fitted parameters to disk to expedite future processing.
+bm25_ef.save("bm25_params.json")
+
+# 4. load the saved params
+new_bm25_ef = BM25EmbeddingFunction()
+new_bm25_ef.load("bm25_params.json")
+
+docs_embeddings = new_bm25_ef.encode_documents(docs)
+query_embeddings = new_bm25_ef.encode_queries([query])
+print("Dim:", new_bm25_ef.dim, list(docs_embeddings)[0].shape)
+```
+
+The expected output is similar to the following:
+
+```python
+Dim: 21 (1, 21)
+```
diff --git a/preview/site/en/faq/faq.md b/preview/site/en/faq/faq.md
deleted file mode 100644
index fbc433ecd..000000000
--- a/preview/site/en/faq/faq.md
+++ /dev/null
@@ -1,16 +0,0 @@
----
-id: faq.md
-title: FAQs
----
-
-# FAQs
-
-Still having questions when using Milvus? Read the most frequently asked questions about the Milvus vector database.
-
-- [Performance FAQs](performance_faq.md): Questions about Milvus' performance.
-
-- [Product FAQs](product_faq.md): Questions about Milvus' cost, architecture, features, and more.
-
-- [Operational FAQs](operational_faq.md): Questions about running and using Milvus.
-
-- [Troubleshooting](troubleshooting.md): Common issues and possible troubleshooting tips.
diff --git a/preview/site/en/faq/operational_faq.md b/preview/site/en/faq/operational_faq.md
index d819b9fe0..5e1e8e568 100644
--- a/preview/site/en/faq/operational_faq.md
+++ b/preview/site/en/faq/operational_faq.md
@@ -1,6 +1,7 @@
---
id: operational_faq.md
summary: Find answers to commonly asked questions about operations in Milvus.
+title: Operational FAQ
---
# Operational FAQ
@@ -46,8 +47,7 @@ Milvus does not support modification to configuration files during runtime. You
If Milvus is started using Docker Compose, run `docker ps` to observe how many Docker containers are running and check if Milvus services started correctly.
-- For Milvus standalone, you should be able to observe at least three running Docker containers, one being the Milvus service and the other two being etcd management and storage service. For more information, see [Installing Milvus Standalone](install_standalone-docker.md).
-- For Milvus Cluster, you should be able to observe at least twelve running Docker containers, nine for the Milvus service and three for basic services. For more information, see [Installing Milvus Cluster](install_cluster-docker.md).
+For Milvus standalone, you should be able to observe at least three running Docker containers, one being the Milvus service and the other two being etcd management and storage service. For more information, see [Installing Milvus Standalone](install_standalone-docker.md).
#### Why is the time in the log files different from the system time?
@@ -89,7 +89,7 @@ It is not recommended to install PyMilvus on Windows. But if you have to install
#### Can I deploy Milvus when disconnected from the Internet?
-Yes. You can install Milvus in an offline environment. See [Install Milvus Offline](install_offline-docker.md) for more information.
+Yes. You can install Milvus in an offline environment. See [Install Milvus Offline](install_offline-helm.md) for more information.
#### Where can I find the logs generated by Milvus?
@@ -102,6 +102,21 @@ If you didn't change the config, using kubectl logs or docker logs CO
Yes, you can. But we recommend inserting data in batches, each of which should not exceed 256 MB, before indexing each segment.
+#### Can I share an etcd instance among multiple Milvus instances?
+
+Yes, you can share an etcd instance among multiple Milvus instances. To do so, you need to change `etcd.rootPath` to a separate value for each Milvus instance in the configuration files of each before starting them.
+
+#### Can I share a Pulsar instance among multiple Milvus instances?
+
+Yes, you can share a Pulsar instance among multiple Milvus instances. To do so, you can
+
+- If multi-tenancy is enabled on your Pulsar instance, consider allocating a separate tenant or namespace for each Milvus instance. To do so, you need to change `pulsar.tenant` or `pulsar.namespace` in the configuration files of your Milvus instances to a unique value for each before starting them.
+- If you do not plan on enabling multi-tenancy on your Pulsar instance, consider changing `msgChannel.chanNamePrefix.cluster` in the configuration files of your Milvus instances to a unique value for each before starting them.
+
+#### Can I share a MinIO instance among multiple Milvus instances?
+
+Yes, you can share a MinIO instance among multiple Milvus instances. To do so, you need to change `minio.rootPath` to a unique value for each Milvus instance in the configuration files of each before starting them.
+
#### Still have questions?
You can:
diff --git a/preview/site/en/faq/performance_faq.md b/preview/site/en/faq/performance_faq.md
index c980811f9..05458f1f8 100644
--- a/preview/site/en/faq/performance_faq.md
+++ b/preview/site/en/faq/performance_faq.md
@@ -1,6 +1,7 @@
---
id: performance_faq.md
summary: Find answers to frequently asked questions about search performance, performance enhancements, and other performance related issues.
+title: Performance FAQ
---
# Performance FAQ
diff --git a/preview/site/en/faq/product_faq.md b/preview/site/en/faq/product_faq.md
index d42cf3360..cdcb9b8fc 100644
--- a/preview/site/en/faq/product_faq.md
+++ b/preview/site/en/faq/product_faq.md
@@ -1,6 +1,7 @@
---
id: product_faq.md
summary: Find answers to frequently asked questions about the world's most advanced vector database.
+title: Product FAQ
---
# Product FAQ
@@ -37,7 +38,7 @@ Theoretically, the maximum dataset size Milvus can handle is determined by the h
Milvus deals with two types of data, inserted data and metadata.
-Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage (for now MinIO only) as incremental log.
+Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).
Metadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.
@@ -45,10 +46,6 @@ Metadata are generated within Milvus. Each Milvus module has its own metadata th
etcd stores Milvus module metadata; MinIO stores entities.
-#### Does Milvus' Python SDK have a connection pool?
-
-Python SDKs for Milvus v0.9.0 or higher have a connection pool. The number of connections in a connection pool has no upper limit.
-
#### Does Milvus support inserting and searching data simultaneously?
Yes. Insert operations and query operations are handled by two separate modules that are mutually independent. From the client’s perspective, an insert operation is complete when the inserted data enters the message queue. However, inserted data are unsearchable until they are loaded to the query node. If the segment size does not reach the index-building threshold (512 MB by default), Milvus resorts to brute-force search and query performance may be diminished.
@@ -85,9 +82,7 @@ No. Milvus has varied behavior. Data must be loaded to memory before searching.
#### Can indexes be created after inserting vectors?
-Yes. If `create_index()` is called, Milvus builds an index for subsequently inserted vectors. However, Milvus does not build an index until the newly inserted vectors fill an entire segment and the newly created index file is separate from the previous one.
-
-
+Yes. If an index has been built for a collection by `create_index()` before, Milvus will automatically build an index for subsequently inserted vectors. However, Milvus does not build an index until the newly inserted vectors fill an entire segment and the newly created index file is separate from the previous one.
#### How are the FLAT and IVF_FLAT indexes different?
@@ -112,7 +107,14 @@ See [Wikipedia](https://en.wikipedia.org/wiki/Unit_vector) for more information.
For normalized vectors, Euclidean distance (L2) is mathematically equivalent to inner product (IP). If these similarity metrics return different results, check to see if your vectors are normalized
#### Is there a limit to the total number of collections and partitions in Milvus?
-There is no limit on the number of collections. However, the number of partitions in each collection must not exceed the value set by the parameter `master.maxPartitionNum`.
+
+Yes. You can create up to 65,535 collections in a Milvus instance. When calculating the number of existing collections, Milvus counts all collections with shards and partitions in them.
+
+For example, let's assume you have already created 100 collections, with 2 shards and 4 partitions in 60 of them and with 1 shard and 12 partitions in the rest 40 collections. The current number of collections can be calculated as:
+
+```
+60 * 2 * 4 + 40 * 1 * 12 = 960
+```
#### Why do I get fewer than k vectors when searching for `topk` vectors?
@@ -126,7 +128,7 @@ See [Vector Index](index.md) for more information.
#### What is the maximum vector dimension supported in Milvus?
-Milvus can manage vectors with up to 32,768 dimensions.
+Milvus can manage vectors with up to 32,768 dimensions by default. You can increase the value of `Proxy.maxDimension` to allow for a larger dimension vector.
#### Does Milvus support Apple M1 CPU?
@@ -134,7 +136,7 @@ Current Milvus release does not support Apple M1 CPU.
#### What data types does Milvus support on the primary key field?
-In current release, Milvus support both INT64 and string.
+In current release, Milvus supports both INT64 and string.
#### Is Milvus scalable?
@@ -156,13 +158,21 @@ Data in MinIO is designed to remain for a certain period of time for the conveni
Yes. Kafka is supported in Milvus 2.1.0.
-#### What's the diference between a search and a query?
+#### What's the difference between a search and a query?
In Milvus, a vector similarity search retrieves vectors based on similarity calculation and vector index acceleration. Unlike a vector similarity search, a vector query retrieves vectors via scalar filtering based on a boolean expression. The boolean expression filters on scalar fields or the primary key field, and it retrieves all results that match the filters. In a query, neither similarity metrics nor vector index is involved.
#### Why does a float vector value have a precision of 7 decimal digits in Milvus?
-Milvus stores vectors as Float32 arrays. A Float32 value has a precision of 7 decimal digits. Even with a Float64 value, such as 1.3476964684980388, Milvus stores it as 1.347696. Therefore, when you retrieve such a vector from Milvus, the precision of the Float64 value is lost.
+Milvus supports storing vectors as Float32 arrays. A Float32 value has a precision of 7 decimal digits. Even with a Float64 value, such as 1.3476964684980388, Milvus stores it as 1.347696. Therefore, when you retrieve such a vector from Milvus, the precision of the Float64 value is lost.
+
+#### How does Milvus handle vector data types and precision?
+
+Milvus supports Binary, Float32, Float16, and BFloat16 vector types.
+
+- Binary vectors: Store binary data as sequences of 0s and 1s, used in image processing and information retrieval.
+- Float32 vectors: Default storage with a precision of about 7 decimal digits. Even Float64 values are stored with Float32 precision, leading to potential precision loss upon retrieval.
+- Float16 and BFloat16 vectors: Offer reduced precision and memory usage. Float16 is suitable for applications with limited bandwidth and storage, while BFloat16 balances range and efficiency, commonly used in deep learning to reduce computational requirements without significantly impacting accuracy.
#### Still have questions?
diff --git a/preview/site/en/faq/troubleshooting.md b/preview/site/en/faq/troubleshooting.md
index 91e600b9e..a4ac66b96 100644
--- a/preview/site/en/faq/troubleshooting.md
+++ b/preview/site/en/faq/troubleshooting.md
@@ -1,6 +1,7 @@
---
id: troubleshooting.md
summary: Learn about common issues you may encounter with Milvus and how to overcome them.
+title: Troubleshooting
---
# Troubleshooting
This page lists common issues that may occur when running Milvus, as well as possible troubleshooting tips. Issues on this page fall into the following categories:
diff --git a/preview/site/en/getstarted/cluster/install_cluster-ansible.md b/preview/site/en/getstarted/cluster/install_cluster-ansible.md
deleted file mode 100644
index 910df125f..000000000
--- a/preview/site/en/getstarted/cluster/install_cluster-ansible.md
+++ /dev/null
@@ -1,376 +0,0 @@
----
-id: install_cluster-ansible.md
-label: Ansible
-related_key: Docker
-order: 3
-group: install_cluster-milvusoperator.md
-summary: Learn how to install Milvus cluster with Ansible Controller.
----
-
-
-
-# Install Milvus Cluster with Ansible
-
-This topic introduces how to deploy a Milvus cluster with Ansible. We provide the Docker Host for creating an Ansible playbook and the Container for running the Milvus cluster. The script in this topic is only for running on Ubuntu 20.04 LTS systems. Modify the script commands and parameters if you want to run the Milvus cluster on other versions of operating systems.
-
-
- - -*Congratulations! You have started Milvus standalone and performed your first vector similarity search.* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_NL.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_NL.md deleted file mode 100644 index 701e13363..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_NL.md +++ /dev/null @@ -1,159 +0,0 @@ ---- -id: example_code_NL.md ---- - -# Voer Milvus uit met Python - -Dit onderwerp beschrijft hoe je Milvus kan uitvoeren met Python. - -## 1. Installeer PyMilvus - -```Python -pip3 install pymilvus==2.2.2 -``` -
- -*Gefeliciteerd! Je heb Milvus standalone opgestart en je eerste vector vergelijkbaarheids zoekopdracht uitgevoerd.* \ No newline at end of file diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_bn.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_bn.md deleted file mode 100644 index 37d5eab0f..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_bn.md +++ /dev/null @@ -1,163 +0,0 @@ ---- -id: example_code_bn.md ---- - -# পাইথন ব্যবহার করে মিলভাস চালান - -এই টপিকটি বর্ণনা করে কিভাবে পাইথন ব্যবহার করে মিলভাস চালানো যায়। - -## 1. PyMilvus ইনস্টল করুন - -```Python -pip3 install pymilvus==2.2.2 -``` -
- - -*অভিনন্দন! আপনি মিলভাস স্ট্যান্ড-অ্যালোনভাবে শুরু করেছেন এবং আপনার প্রথম ভেক্টর সাদৃশ্য অনুসন্ধান করেছেন।* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_de.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_de.md deleted file mode 100644 index 56edcf93d..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_de.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -id: example_code_de.md ---- - -# Milvus mit Python ausführen - -Dieser Abschnitt beschreibt, wie man Milvus mit Python ausführt. - -## 1. PyMilvus installieren - -```Python -pip3 install pymilvus==2.2.2 -``` -
- - -*Herzlichen Glückwunsch! Sie haben es geschafft, Milvus unabhängig zu starten und Ihre erste Vektorähnlichkeitssuche durchzuführen.* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_es.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_es.md deleted file mode 100644 index 8293e119c..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_es.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -id: example_code_es.md ---- - -# Ejecuta Milvus usando Python - -Este tema describe cómo ejecutar Milvus usando Python. - -## 1. Instala PyMilvus - -```Python -pip3 install pymilvus==2.2.2 -``` -
- - -*¡Felicidades! Has iniciado Milvus standalone y ejecutado tu primera búsqueda por similitud de vector.* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_fr.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_fr.md deleted file mode 100644 index a6913dc3e..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_fr.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -id: example_code_fr.md ---- - -# Exécuter Milvus avec Python - -Ce topic décrit comment exécuter Milvus avec Python. - -## 1. Installer PyMilvus - -```Python -pip3 install pymilvus==2.2.2 -``` -
- - -*Félicitations ! Vous avez démarré Milvus standalone et exécuté votre première recherche de similitude entre vecteurs.* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_id.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_id.md deleted file mode 100644 index 2e9e7dbee..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_id.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -id: example_code_id.md ---- - -# Menjalankan Milvus menggunkan Python - -Topik ini menjelaskan bagaimana cara menjalankan Milvus menggunkan Python. - -## 1. Instal PyMilvus - -```Python -pip3 install pymilvus==2.2.2 -``` -
- - -*Selamat! Kamu baru saja memulai Milvus mandiri dan melakukan pencarian persamaan vector pertama.* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_it.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_it.md deleted file mode 100644 index 2a75060a5..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_it.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -id: example_code_it.md ---- - -# Esegui Milvus usando Python - -Questa sezione descrive come eseguire Milvus usando Python. - -## 1. Installa PyMilvus - -```Python -pip3 install pymilvus==2.2.2 -``` -
- - -*Congratulazioni! Sei riuscito ad avviare Milvus indipendentemente e hai eseguito la tua prima ricerca di somiglianza vettoriale.* - diff --git a/preview/site/en/getstarted/example_code_node.md b/preview/site/en/getstarted/example_code_node.md deleted file mode 100644 index db1870cf5..000000000 --- a/preview/site/en/getstarted/example_code_node.md +++ /dev/null @@ -1,213 +0,0 @@ ---- -id: example_code_node.md -label: Node.js -order: 1 -group: example_code.md -summary: Get started with Milvus faster using this Node.js example code. ---- - - - -# Run Milvus using Node.js - -This topic describes how to run Milvus using Node.js. - - -## 1. Initialize a Node.js Project -```bash -npm init -``` - -
- - -*Congratulations! You have successfully booted Milvus Standalone and created your first collection.* diff --git a/preview/site/en/getstarted/get_started.md b/preview/site/en/getstarted/get_started.md deleted file mode 100644 index befacf1f6..000000000 --- a/preview/site/en/getstarted/get_started.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -id: get_started.md -title: Get Started ---- - -# Get Started - -New to Milvus? No sweat! Follow the topics below for instant onboarding with Milvus. - -- [Prerequisites](prerequisite-docker.md): Check the installation requirements for your hardware and software. - -- [Install Milvus](install_milvus.md): Learn about the various and flexible ways to either install Milvus standalone or deploy a Milvus cluster. - -- [Install SDKs](install-pymilvus.md): Choose the install the Milvus SDK in the language that you prefer. - -- [Hello Milvus](example_code.md): Run the example code to get a quick preview of what Milvus is capable of. - diff --git a/preview/site/en/getstarted/install_SDKs/install-go.md b/preview/site/en/getstarted/install_SDKs/install-go.md index b58977871..5ccff3c81 100644 --- a/preview/site/en/getstarted/install_SDKs/install-go.md +++ b/preview/site/en/getstarted/install_SDKs/install-go.md @@ -3,6 +3,7 @@ id: install-go.md label: Install GO SDK related_key: SDK summary: Learn how to install the GO SDK of Milvus. +title: Install Milvus Go SDK --- # Install Milvus Go SDK @@ -13,7 +14,7 @@ Current version of Milvus supports SDKs in Python, Node.js, GO, and Java. ## Requirement -GO (1.15 or later) is required. +GO (1.17 or later) is required. ## Install Milvus GO SDK @@ -28,10 +29,10 @@ $ go get -u github.com/milvus-io/milvus-sdk-go/v2 Having installed Milvus GO SDK, you can: - Learn the basic operations of Milvus: - - [Connect to Milvus server](manage_connection.md) - - [Create a collection](create_collection.md) - - [Create a partition](create_partition.md) - - [Insert data](insert_data.md) - - [Conduct a vector search](search.md) + - [Manage Collections](manage-collections.md) + - [Manage Partitions](manage-partitions.md) + - [Insert, Upsert & Delete](insert-update-delete.md) + - [Single-Vector Search](single-vector-search.md) + - [Hybrid Search](multi-vector-search.md) diff --git a/preview/site/en/getstarted/install_SDKs/install-java.md b/preview/site/en/getstarted/install_SDKs/install-java.md index 06ac50329..38f378759 100644 --- a/preview/site/en/getstarted/install_SDKs/install-java.md +++ b/preview/site/en/getstarted/install_SDKs/install-java.md @@ -3,6 +3,7 @@ id: install-java.md label: Install Java SDK related_key: SDK summary: Learn how to install the Java SDK of Milvus. +title: Install Milvus Java SDK --- # Install Milvus Java SDK @@ -26,14 +27,14 @@ Run the following command to install Milvus Java SDK.
io.milvus
milvus-sdk-java
- 2.2.1
+ 2.4.1
```
- Gradle/Grails
```
-compile 'io.milvus:milvus-sdk-java:2.2.1'
+implementation 'io.milvus:milvus-sdk-java:2.4.1'
```
## What's next
@@ -41,11 +42,11 @@ compile 'io.milvus:milvus-sdk-java:2.2.1'
Having installed Milvus Java SDK, you can:
- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
-- Explore [Milvus Java API reference](/api-reference/java/v2.2.1/About.md)
+- Explore [Milvus Java API reference](/api-reference/java/v2.4.x/About.md)
diff --git a/preview/site/en/getstarted/install_SDKs/install-node.md b/preview/site/en/getstarted/install_SDKs/install-node.md
index c66a4fbdd..63287ec34 100644
--- a/preview/site/en/getstarted/install_SDKs/install-node.md
+++ b/preview/site/en/getstarted/install_SDKs/install-node.md
@@ -3,36 +3,54 @@ id: install-node.md
label: Install Node.js SDK
related_key: SDK
summary: Learn how to install the Node.js SDK of Milvus.
+title: Install Milvus Nodejs SDK
---
-# Install Milvu Nodejs SDK
+# Install Milvus Nodejs SDK
This topic describes how to install Milvus Node.js SDK for Milvus.
-Current version of Milvus supports SDKs in Python, Node.js, GO, and Java.
+## Compatibility
+
+The following collection shows Milvus versions and recommended @zilliz/milvus2-sdk-node versions:
+
+| Milvus version | Recommended @zilliz/milvus2-sdk-node version |
+| :------------: | :------------------------------------------: |
+| 2.4.x | 2.4.x |
+| 2.3.x | 2.3.x |
+| 2.2.x | 2.2.x |
+| 2.1.x | 2.1.x |
+| 2.0.1 | 2.0.0, 2.0.1 |
+| 2.0.0 | 2.0.0 |
## Requirement
-Node.js (12.0.0 or later) is required.
+Node.js v18+
-## Install Milvus Node.js SDK
+## Installation
-Run the following command to install Milvus Node.js SDK.
+The recommended way to get started using the Milvus node.js client is by using npm (Node package manager) to install the dependency in your project.
-```
-$ npm install @zilliz/milvus2-sdk-node
+```javascript
+npm install @zilliz/milvus2-sdk-node
+# or ...
+yarn add @zilliz/milvus2-sdk-node
```
+This will download the Milvus node.js sdk and add a dependency entry in your package.json file.
+
## What's next
Having installed Milvus Node.js SDK, you can:
+
+- View [quick start of milvus node.js sdk](https://github.com/milvus-io/milvus-sdk-node)
- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
-- Explore [Milvus Node.js API reference](/api-reference/node/v2.2.x/About.md)
+- Explore [Milvus Node.js API reference](/api-reference/node/v2.4.x/About.md)
diff --git a/preview/site/en/getstarted/install_SDKs/install-pymilvus.md b/preview/site/en/getstarted/install_SDKs/install-pymilvus.md
index e30131764..d25ef6e17 100644
--- a/preview/site/en/getstarted/install_SDKs/install-pymilvus.md
+++ b/preview/site/en/getstarted/install_SDKs/install-pymilvus.md
@@ -3,6 +3,7 @@ id: install-pymilvus.md
label: Install PyMilvus
related_key: SDK
summary: Learn how to install the Python SDK of Milvus.
+title: Install Milvus Python SDK
---
# Install Milvus Python SDK
@@ -13,7 +14,7 @@ Current version of Milvus supports SDKs in Python, Node.js, GO, and Java.
## Requirements
-- Python 3.6 or later is required.
+- Python 3.7 or later is required.
- Google protobuf is installed. You can install it with the command `pip3 install protobuf==3.20.0`.
- grpcio-tools is installed. You can install it with the command `pip3 install grpcio-tools`.
@@ -26,7 +27,7 @@ It is recommended to install a PyMilvus version that matches the version of the
```
-$ python3 -m pip install pymilvus==2.2.2
+$ python3 -m pip install pymilvus==2.4.4
```
## Verify installation
@@ -44,11 +45,11 @@ $ python3 -c "from pymilvus import Collection"
Having installed PyMilvus, you can:
- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
-- Explore [PyMilvus API reference](/api-reference/pymilvus/v2.2.2/About.md)
+- Explore [PyMilvus API reference](/api-reference/pymilvus/v2.4.x/About.md)
diff --git a/preview/site/en/getstarted/install_embedded_milvus.md b/preview/site/en/getstarted/install_embedded_milvus.md
deleted file mode 100644
index 2780e3c42..000000000
--- a/preview/site/en/getstarted/install_embedded_milvus.md
+++ /dev/null
@@ -1,212 +0,0 @@
----
-id: install_embedded_milvus.md
-related_key: installation
-title: Install Embedded Milvus
-summary: Learn how to install embedded Milvus.
----
-
-# Install Embedded Milvus
-
-
-This topic describes how to install [embedded Milvus](https://github.com/milvus-io/embd-milvus).
-
-
+
+All code samples on this page are in PyMilvus 2.4.4. Upgrade your PyMilvus installation before running them.
+
1. Create a resource group.
@@ -34,17 +58,20 @@ python -m pip install --upgrade pymilvus
```Python
import pymilvus
-
+
# A resource group name should be a string of 1 to 255 characters, starting with a letter or an underscore (_) and containing only numbers, letters, and underscores (_).
name = "rg"
-
+ node_num = 0
+
+ # create a resource group that exactly hold no query node.
try:
- utility.create_resource_group(name, using='default')
+ utility.create_resource_group(name, config=utility.ResourceGroupConfig(
+ requests={"node_num": node_num},
+ limits={"node_num": node_num},
+ ), using='default')
print(f"Succeeded in creating resource group {name}.")
except Exception:
print("Failed to create the resource group.")
-
- # Succeeded in creating resource group rg.
```
2. List resource groups.
@@ -69,7 +96,7 @@ python -m pip install --upgrade pymilvus
print(f"Resource group description: {info}")
# Resource group description:
- #
+
+MinIO does not automatically rebalance objects across new server pools. You can manually initiate a rebalance procedure with `mc admin rebalance` if needed.
+
+
+
+## Kafka
+
+### Increase resource per Kafka broker pod
+
+Enhance the Kafka broker capacity by adjusting the CPU and memory resources for each broker pod.
+
+```yaml
+# new-values.yaml
+kafka:
+ resources:
+ limits:
+ cpu: 2
+ memory: 12Gi
+```
+
+After saving the file, apply the changes with the following command:
+
+```bash
+helm upgrade
+
+Kafka does not automatically rebalance topics across all brokers. Manually rebalance topics/partitions across all Kafka brokers using `bin/kafka-reassign-partitions.sh` after logging into each Kafka broker pod if needed.
+
+
+
+## Pulsar
+
+Pulsar separates computation and storage. You can independently increase the capacity of Pulsar brokers (computation) and Pulsar bookies (storage).
+
+## Increase resources per Pulsar broker pod
+
+```yaml
+# new-values.yaml
+pulsar:
+ broker:
+ resources:
+ limits:
+ cpu: 4
+ memory: 16Gi
+```
+
+After saving the file, apply the changes with the following command:
+
+```shell
+helm upgrade
+
+TLS and user authentication are two distinct security approaches. If you have enabled both user authentication and TLS in your Milvus system, you will need to provide a username, password, and certificate file paths. For information on how to enable user authentication, refer to [Authenticate User Access](authenticate.md).
+
+
+
## Create your own certificate
### Prerequisites
@@ -35,7 +41,7 @@ mkdir cert && cd cert
touch openssl.cnf gen.sh
```
-2. Copy the following configurations into the files. Configuration of `CommonName` is required. `CommonName` refers to the server name that the client needs to specify when connecting.
+2. Copy the following configurations into the files respectively.
openssl.cnf
@@ -263,12 +269,6 @@ authorityKeyIdentifier=keyid,issuer
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
-subjectAltName = @alt_names
-
-[ alt_names ]
-DNS.1 = localhost
-DNS.2 = *.ronething.cn
-DNS.3 = *.ronething.com
[ v3_ca ]
@@ -403,6 +403,8 @@ ess_cert_id_chain = no # Must the ESS cert id chain be included?
The `openssl.cnf` file is a default OpenSSL configuration file. See [manual page](https://www.openssl.org/docs/manmaster/man5/config.html) for more information. The `gen.sh` file generates relevant certificate files. You can modify the `gen.sh` file for different purposes such as changing the validity period of the certificate file, the length of the certificate key or the certificate file names.
+It is necessary to configure the `CommonName` in the `gen.sh` file. The `CommonName` refers to the server name that the client should specify while connecting.
+
gen.sh
```shell
@@ -423,7 +425,7 @@ openssl req -new -x509 -key ca.key -out ca.pem -days 3650 -subj "/C=$Country/ST=
echo "generate server SAN certificate"
openssl genpkey -algorithm RSA -out server.key
-openssl req -new -nodes -key server.key -out server5.csr -days 3650 -subj "/C=$Country/O=$Organization/OU=$Organizational/CN=$CommonName" -config ./openssl.cnf -extensions v3_req
+openssl req -new -nodes -key server.key -out server.csr -days 3650 -subj "/C=$Country/O=$Organization/OU=$Organizational/CN=$CommonName" -config ./openssl.cnf -extensions v3_req
openssl x509 -req -days 3650 -in server.csr -out server.pem -CA ca.pem -CAkey ca.key -CAcreateserial -extfile ./openssl.cnf -extensions v3_req
echo "generate client SAN certificate"
@@ -497,50 +499,117 @@ Open the `server.csr`, the `ca.key` and the `ca.pem` files to sign the certifica
openssl x509 -req -days 3650 -in server.csr -out server.pem -CA ca.pem -CAkey ca.key -CAcreateserial -extfile ./openssl.cnf -extensions v3_req
```
+## Set up a Milvus server with TLS
-## Modify Milvus server configurations
+This section outlines the steps to configure a Milvus server with TLS encryption.
-Set `tlsEnabled` to `true` and configure the file paths of `server.pem`, `server.key`, and `ca.pem` for the server in `config/milvus.yaml`.
+
-```
+Due to current limitations with milvus-helm and milvus-operator regarding certificate file path configurations, this guide will focus on deployment using Docker Compose.
+
+
+
+### 1. Modify the Milvus server configuration
+
+To enable TLS, set `common.security.tlsMode` in `milvus.yaml` to `1` (for one-way TLS) or `2` (for two-way TLS).
+
+```yaml
tls:
- serverPemPath: configs/cert/server.pem
- serverKeyPath: configs/cert/server.key
- caPemPath: configs/cert/ca.pem
+ serverPemPath: /milvus/tls/server.pem
+ serverKeyPath: /milvus/tls/server.key
+ caPemPath: /milvus/tls/ca.pem
common:
security:
- tlsMode: 2
- ```
+ tlsMode: 1
+```
+
+Parameters:
-### One-way authentication
+- `serverPemPath`: The path to the server certificate file.
+- `serverKeyPath`: The path to the server key file.
+- `caPemPath`: The path to the CA certificate file.
+- `tlsMode`: The TLS mode for encryption. Valid values:
+ - `1`: One-way authentication, where only the server requires a certificate and the client verifies it. This mode requires `server.pem` and `server.key` from the server side, and `server.pem` from the client side.
+ - `2`: Two-way authentication, where both the server and the client require certificates to establish a secure connection. This mode requires `server.pem`, `server.key`, and `ca.pem` from the server side, and `client.pem`, `client.key`, and `ca.pem` from the client side.
-Server need server.pem and server.key. Client-side need server.pem.
+### 2. Map certificate files to the container
-### Two-way authentication
+#### Prepare certificate files
-Server-side need server.pem, server.key and ca.pem. Client-side need client.pem, client.key, ca.pem.
+Create a new folder named `tls` in the same directory as your `docker-compose.yaml`. Copy the `server.pem`, `server.key`, and `ca.pem` into the `tls` folder. Place them in a directory structure as follows:
+
+```
+├── docker-compose.yml
+├── milvus.yaml
+└── tls
+ ├── server.pem
+ ├── server.key
+ └── ca.pem
+```
+
+#### Update Docker Compose configuration
+
+Edit the `docker-compose.yaml` file to map the certificate file paths inside the container as shown below:
+
+```yaml
+ standalone:
+ container_name: milvus-standalone
+ image: milvusdb/milvus:latest
+ command: ["milvus", "run", "standalone"]
+ security_opt:
+ - seccomp:unconfined
+ environment:
+ ETCD_ENDPOINTS: etcd:2379
+ MINIO_ADDRESS: minio:9000
+ volumes:
+ - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
+ - ${DOCKER_VOLUME_DIRECTORY:-.}/tls:/milvus/tls
+ - ${DOCKER_VOLUME_DIRECTORY:-.}/milvus.yaml:/milvus/configs/milvus.yaml
+```
+
+#### Deploy Milvus using Docker Compose
+
+Execute the following command to deploy Milvus:
+
+```bash
+sudo docker compose up -d
+```
## Connect to the Milvus server with TLS
-Configure the file paths of `client.pem`, `client.key`, and `ca.pem` for the client when using the Milvus SDK.
+For SDK interactions, use the following setups depending on the TLS mode.
+
+### One-way TLS connection
-The following example uses the Milvus Python SDK.
+Provide the path to `server.pem` and ensure the `server_name` matches the `CommonName` configured in the certificate.
+
+```python
+from pymilvus import MilvusClient
+client = MilvusClient(
+ uri="http://localhost:19530",
+ secure=True,
+ server_pem_path="path_to/server.pem",
+ server_name="localhost"
+)
```
-from pymilvus import connections
-_HOST = '127.0.0.1'
-_PORT = '19530'
+### Two-way TLS connection
-print(f"\nCreate connection...")
-connections.connect(host=_HOST, port=_PORT, secure=True, client_pem_path="cert/client.pem",
- client_key_path="cert/client.key",
- ca_pem_path="cert/ca.pem", server_name="localhost")
-print(f"\nList connections:")
-print(connections.list_connections())
+Provide paths to `client.pem`, `client.key`, and `ca.pem`, and ensure the `server_name` matches the `CommonName` configured in the certificate.
+
+```python
+from pymilvus import MilvusClient
+
+client = MilvusClient(
+ uri="http://localhost:19530",
+ secure=True,
+ client_pem_path="path_to/client.pem",
+ client_key_path="path_to/client.key",
+ ca_pem_path="path_to/ca.pem",
+ server_name="localhost"
+)
```
See [example_tls1.py](https://github.com/milvus-io/pymilvus/blob/master/examples/example_tls1.py) and [example_tls2.py](https://github.com/milvus-io/pymilvus/blob/master/examples/example_tls2.py) for more information.
-
-
diff --git a/preview/site/en/adminGuide/upgrade.md b/preview/site/en/adminGuide/upgrade.md
deleted file mode 100644
index 84295824c..000000000
--- a/preview/site/en/adminGuide/upgrade.md
+++ /dev/null
@@ -1,263 +0,0 @@
----
-id: upgrade.md
-related_key: upgrade Milvus
-summary: Learn how to upgrade Milvus.
----
-
-# Upgrade Milvus Using Helm Chart
-
-This topic describes how to upgrade Milvus 2.x with Helm Chart using the example of upgrading from Milvus v2.0.2 to v2.1.0.
-
-
-Helm Chart does not support upgrading from Milvus 2.0 standalone to Milvus 2.0 cluster or vice versa. Milvus 2.0.0-RC7 is not compatible with earlier RC versions. Therefore, you cannot upgrade from prior versions to 2.0.0-RC7.
-
-
-## Upgrade Milvus standalone
-
-### Step 1. Check the Milvus version
-
-Run `$ helm list` to check your Milvus app version. You can see the `APP VERSION` is 2.0.2.
-
-```
-NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
-my-release default 1 2022-07-28 15:28:12.32068 +0800 CST deployed milvus-3.0.29 2.0.2
-```
-
-### Step 2. Check the running pods
-
-Run `$ kubectl get pods` to check the running pods. You can see the following output.
-
-```
-NAME READY STATUS RESTARTS AGE
-my-release-etcd-0 1/1 Running 0 84s
-my-release-milvus-standalone-75c599fffc-6rwlj 1/1 Running 0 84s
-my-release-minio-744dd9586f-qngzv 1/1 Running 0 84s
-```
-
-### Step 3. Check the image tag
-
-Check the image tag for the pod `my-release-milvus-standalone-75c599fffc-6rwlj`. You can see the release of your Milvus standalone is v2.0.2.
-
-```
-$ kubectl get pods my-release-milvus-standalone-75c599fffc-6rwlj -o=jsonpath='{$.spec.containers[0].image}'
-```
-
-```
-milvusdb/milvus:v2.0.2
-```
-
-
-### Step 4. Check new Milvus standalone versions
-
-Run the following commands to check new Milvus versions. You can see there are several new versions after v2.0.2.
-
-```
-$ helm repo update
-$ helm search repo milvus --versions
-```
-
-```
-NAME CHART VERSION APP VERSION DESCRIPTION
-milvus/milvus 3.1.2 2.1.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.1.1 2.1.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.1.0 2.1.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.29 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.28 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.27 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.26 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.25 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.24 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.23 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.21 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.20 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.19 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.18 2.0.2 Milvus is an open-source vector database built ...
-```
-
-### Step 5. Upgrade
-
-1. Run the following commands to upgrade your Milvus standalone from v2.0.2 to v2.1.0.
-
-```
-$ helm repo update
-$ helm upgrade my-release milvus/milvus --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsar.enabled=false
-```
-
-2. Run `$ helm list` again to check your Milvus app version. You can see your Milvus standalone has been upgraded to v2.1.0.
-
-```
-NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
-my-release default 2 2022-07-28 15:40:18.22437 +0800 CST deployed milvus-3.1.2 2.1.0
-```
-
-3. Run `$ kubectl get pods` to check the new pods. You can see the following output.
-
-```
-NAME READY STATUS RESTARTS AGE
-my-release-etcd-0 1/1 Running 0 3m32s
-my-release-milvus-standalone-6967454987-72r55 1/1 Running 0 22s
-my-release-minio-744dd9586f-qngzv 1/1 Running 0 3m32s
-```
-
-
-When upgrading your Milvus standalone, old pods will be deleted. Therefore, the service may be offline for a short period of time.
-
-
-4. Run the following command to check the new image version. You can see it is v2.0.0-rc8 now.
-
-```
-$ kubectl get pods my-release-milvus-standalone-6967454987-72r55 -o=jsonpath='{$.spec.containers[0].image}'
-```
-
-```
-milvusdb/milvus:v2.1.0
-```
-
-## Upgrade Milvus cluster
-
-### Step 1. Check the Milvus version
-
-Run `$ helm list` to check your Milvus app version. You can see the `APP VERSION` is 2.0.2.
-
-```
-NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
-my-release default 1 2022-07-28 15:50:43.21188 +0800 CST deployed milvus-3.0.29 2.0.2
-```
-
-### Step 2. Check the running pods
-
-Run `$ kubectl get pods` to check the running pods. You can see the following output.
-
-```
-NAME READY STATUS RESTARTS AGE
-my-release-etcd-0 1/1 Running 0 5m40s
-my-release-etcd-1 1/1 Running 0 5m40s
-my-release-etcd-2 1/1 Running 0 5m40s
-my-release-milvus-datacoord-c99d7dfdf-mjghl 1/1 Running 0 5m40s
-my-release-milvus-datanode-69cccf85d8-9r8ph 1/1 Running 0 5m40s
-my-release-milvus-indexcoord-64f7d548fb-46hn8 1/1 Running 0 5m40s
-my-release-milvus-indexnode-57b96d9cc7-gvmvl 1/1 Running 0 5m40s
-my-release-milvus-proxy-6664d564f9-pwqn9 1/1 Running 0 5m40s
-my-release-milvus-querycoord-59767cb88c-n54l6 1/1 Running 0 5m40s
-my-release-milvus-querynode-847ccdf855-78mnz 1/1 Running 0 5m40s
-my-release-milvus-rootcoord-597bd9f565-2jgzq 1/1 Running 0 5m40s
-my-release-minio-0 1/1 Running 0 5m40s
-my-release-minio-1 1/1 Running 0 5m40s
-my-release-minio-2 1/1 Running 0 5m40s
-my-release-minio-3 1/1 Running 0 5m40s
-my-release-pulsar-autorecovery-869bffb7b8-g4cbh 1/1 Running 0 5m40s
-my-release-pulsar-bastion-7c659df966-86b5s 1/1 Running 0 5m40s
-my-release-pulsar-bookkeeper-0 1/1 Running 0 5m40s
-my-release-pulsar-bookkeeper-1 1/1 Running 0 3m54s
-my-release-pulsar-broker-864775f5ff-zlnfx 1/1 Running 0 5m40s
-my-release-pulsar-proxy-86bcdbbb4c-24kcj 2/2 Running 0 5m40s
-my-release-pulsar-zookeeper-0 1/1 Running 0 5m40s
-my-release-pulsar-zookeeper-1 1/1 Running 0 5m20s
-my-release-pulsar-zookeeper-2 1/1 Running 0 5m5s
-my-release-pulsar-zookeeper-metadata-hw5xt 0/1 Completed 0 5m40s
-```
-
-### Step 3. Check the image tag
-
-Check the image tag for the pod `my-release-milvus-proxy-6664d564f9-pwqn9`. You can see the release of your Milvus cluster is v2.0.2.
-
-```
-$ kubectl get pods my-release-milvus-proxy-6664d564f9-pwqn9 -o=jsonpath='{$.spec.containers[0].image}'
-```
-
-```
-milvusdb/milvus:v2.0.2
-```
-
-### Step 4. Check new Milvus cluster versions
-
-Run the following commands to check new Milvus versions. You can see there are several new versions after 2.0.2.
-
-```
-$ helm repo update
-$ helm search repo milvus --versions
-```
-
-```
-NAME CHART VERSION APP VERSION DESCRIPTION
-milvus/milvus 3.1.2 2.1.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.1.1 2.1.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.1.0 2.1.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.29 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.28 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.27 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.26 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.25 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.24 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.23 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.21 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.20 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.19 2.0.2 Milvus is an open-source vector database built ...
-milvus/milvus 3.0.18 2.0.2 Milvus is an open-source vector database built ...
-```
-
-### Step 5. Upgrade
-
-1. Run the following commands to upgrade your Milvus cluster from v2.0.2 to v2.1.0.
-
-```
-$ helm repo update
-$ helm upgrade my-release milvus/milvus
-```
-
-2. Run `$ helm list` again to check your Milvus version. You can see your Milvus cluster has been upgraded to v2.1.0.
-
-```
-NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
-my-release default 2 2022-07-28 16:05:20.11326 +0800 CST deployed milvus-3.1.2 2.1.0
-```
-
-3. Run `$ kubectl get pods` to check the new pods. You can see the following output.
-
-```
-NAME READY STATUS RESTARTS AGE
-my-release-etcd-0 1/1 Running 0 71s
-my-release-etcd-1 1/1 Running 0 2m34s
-my-release-etcd-2 1/1 Running 0 3m41s
-my-release-milvus-datacoord-76d55548b6-zl4kj 1/1 Running 0 3m45s
-my-release-milvus-datanode-5b9774cc75-dhn7j 1/1 Running 0 3m45s
-my-release-milvus-indexcoord-96549bfff-r9m99 1/1 Running 0 3m45s
-my-release-milvus-indexnode-f7c9b444b-vjqnm 1/1 Running 0 3m44s
-my-release-milvus-proxy-5685bbc546-v6scq 1/1 Running 0 3m44s
-my-release-milvus-querycoord-5fcd65544-8m6lb 1/1 Running 0 3m44s
-my-release-milvus-querynode-5b76d575f6-2szfj 1/1 Running 0 3m44s
-my-release-milvus-rootcoord-8668f8c46b-9nss2 1/1 Running 0 3m44s
-my-release-minio-0 1/1 Running 0 11m
-my-release-minio-1 1/1 Running 0 11m
-my-release-minio-2 1/1 Running 0 11m
-my-release-minio-3 1/1 Running 0 11m
-my-release-pulsar-autorecovery-869bffb7b8-g4cbh 1/1 Running 0 11m
-my-release-pulsar-bastion-7c659df966-86b5s 1/1 Running 0 11m
-my-release-pulsar-bookkeeper-0 1/1 Running 0 11m
-my-release-pulsar-bookkeeper-1 1/1 Running 0 9m55s
-my-release-pulsar-broker-864775f5ff-zlnfx 1/1 Running 0 11m
-my-release-pulsar-proxy-86bcdbbb4c-24kcj 2/2 Running 0 11m
-my-release-pulsar-zookeeper-0 1/1 Running 0 11m
-my-release-pulsar-zookeeper-1 1/1 Running 0 11m
-my-release-pulsar-zookeeper-2 1/1 Running 0 11m
-```
-
-4. Run the following command to check the new image version. You can see it is v2.1.0 now.
-
-```
-$ kubectl get pods my-release-milvus-proxy-5685bbc546-v6scq -o=jsonpath='{$.spec.containers[0].image}'
-```
-
-```
-milvusdb/milvus:v2.1.0
-```
-
-
-## What's next
-- You might also want to learn how to:
- - [Scale a Milvus cluster](scaleout.md)
-- If you are ready to deploy your cluster on clouds:
- - Learn how to [Deploy Milvus on AWS with Terraform and Ansible](aws.md)
- - Learn how to [Deploy Milvus on Amazon EKS with Terraform](eks.md)
- - Learn how to [Deploy Milvus Cluster on GCP with Kubernetes](gcp.md)
- - Learn how to [Deploy Milvus on Microsoft Azure With Kubernetes](azure.md)
diff --git a/preview/site/en/adminGuide/upgrade_milvus_cluster-docker.md b/preview/site/en/adminGuide/upgrade_milvus_cluster-docker.md
index 3d22d57ca..3cb372d23 100644
--- a/preview/site/en/adminGuide/upgrade_milvus_cluster-docker.md
+++ b/preview/site/en/adminGuide/upgrade_milvus_cluster-docker.md
@@ -1,67 +1,112 @@
---
id: upgrade_milvus_cluster-docker.md
-label: Docker Compose
-order: 2
-group: upgrade_milvus_cluster-operator.md
-related_key: upgrade Milvus Cluster
summary: Learn how to upgrade Milvus cluster with Docker Compose.
+title: Upgrade Milvus Cluster with Docker Compose
---
-
+
# Upgrade Milvus Cluster with Docker Compose
-A major change in Milvus 2.2 is the meta structure of segment indexes. This topic describes how to use Docker Compose to migrate the meta and upgrade Milvus from v2.1.x to v2.2.0.
-
-## Upgrade Milvus from v2.1.x to v2.2.0
+This topic describes how to upgrade your Milvus using Docker Compose.
+
+In normal cases, you can [upgrade Milvus by changing its image](#Upgrade-Milvus-by-changing-its-image). However, you need to [migrate the metadata](#Migrate-the-metadata) before any upgrade from v2.1.x to v2.4.5.
+
+## Upgrade Milvus by changing its image
+
+In normal cases, you can upgrade Milvus as follows:
+
+1. Change the Milvus image tags in `docker-compose.yaml`.
+
+ Note that you need to change the image tags for the Proxy, all coordinators, and all worker nodes.
+
+ ```yaml
+ ...
+ rootcoord:
+ container_name: milvus-rootcoord
+ image: milvusdb/milvus:v2.4.5
+ ...
+ proxy:
+ container_name: milvus-proxy
+ image: milvusdb/milvus:v2.4.5
+ ...
+ querycoord:
+ container_name: milvus-querycoord
+ image: milvusdb/milvus:v2.4.5
+ ...
+ querynode:
+ container_name: milvus-querynode
+ image: milvusdb/milvus:v2.4.5
+ ...
+ indexcoord:
+ container_name: milvus-indexcoord
+ image: milvusdb/milvus:v2.4.5
+ ...
+ indexnode:
+ container_name: milvus-indexnode
+ image: milvusdb/milvus:v2.4.5
+ ...
+ datacoord:
+ container_name: milvus-datacoord
+ image: milvusdb/milvus:v2.4.5
+ ...
+ datanode:
+ container_name: milvus-datanode
+ image: milvusdb/milvus:v2.4.5
+ ```
+
+2. Run the following commands to perform the upgrade.
+
+ ```shell
+ docker compose down
+ docker compose up -d
+ ```
+
+## Migrate the metadata
1. Stop all Milvus components.
-```
-docker stop
+
+The Milvus Helm Charts repo at `https://milvus-io.github.io/milvus-helm/` has been archived and you can get further updates from `https://zilliztech.github.io/milvus-helm/` as follows:
+
+```shell
+helm repo add zilliztech https://zilliztech.github.io/milvus-helm
+helm repo update
+# upgrade existing helm release
+helm upgrade my-release zilliztech/milvus
+```
+
+The archived repo is still available for the charts up to 4.0.31. For later releases, use the new repo instead.
+
+
+
+```
+NAME CHART VERSION APP VERSION DESCRIPTION
+zilliztech/milvus 4.1.34 2.4.5 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.33 2.4.4 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.32 2.4.3 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.31 2.4.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.30 2.4.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.29 2.4.0 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.24 2.3.11 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.23 2.3.10 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.22 2.3.10 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.21 2.3.10 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.20 2.3.10 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.18 2.3.10 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.18 2.3.9 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.17 2.3.8 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.16 2.3.7 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.15 2.3.5 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.14 2.3.6 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.13 2.3.5 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.12 2.3.5 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.11 2.3.4 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.10 2.3.3 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.9 2.3.3 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.8 2.3.2 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.7 2.3.2 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.6 2.3.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.5 2.3.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.4 2.3.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.3 2.3.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.2 2.3.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.1 2.3.0 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.0 2.3.0 Milvus is an open-source vector database built ...
```
-NAME CHART VERSION APP VERSION DESCRIPTION
-milvus/milvus 4.0.8 2.2.3 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.7 2.2.3 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.6 2.2.2 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.5 2.2.2 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.4 2.2.2 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.3 2.2.2 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.2 2.2.2 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.1 2.2.1 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.0 2.2.1 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.6 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.5 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.4 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.3 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.2 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.1 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.0 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.2.18 2.1.4 Milvus is an open-source vector database built ...
+
+You can choose the upgrade path for your Milvus as follows:
+
+
+
+- [Upgrade Milvus using Helm](#Upgrade-Milvus-using-Helm) for an upgrade from a minor release before v2.2.3 to v2.4.5.
+
+- [Migrate the metadata](#Migrate-the-metadata) before the upgrade from Milvus v2.1.x to v2.4.5.
+
+
+
+## Upgrade Milvus using Helm
-- [Migrate the metadata](#Migrate-the-metadata)
-- [Conduct a rolling upgrade](#Conduct-a-rolling-upgrade)
+To upgrade Milvus from a minor release before v2.2.3 to the latest, run the following commands:
+
+```shell
+helm repo update
+helm upgrade my-release zilliztech/milvus --reuse-values --version=4.1.24 # use the helm chart version here
+```
+
+Use the Helm chart version in the preceding command. For details on how to obtain the Helm chart version, refer to [Check the Milvus version](#Check-the-Milvus-version).
## Migrate the metadata
@@ -99,15 +179,13 @@ my-release-pulsar-zookeeper-2 1/1 Running 0 2
Check the image tag for the pod `my-release-milvus-proxy-6c548f787f-scspp`. You can see the release of your Milvus cluster is v2.1.4.
-```
+```shell
$ kubectl get pods my-release-milvus-proxy-6c548f787f-scspp -o=jsonpath='{$.spec.containers[0].image}'
-```
-
-```
-milvusdb/milvus:v2.1.4
+# milvusdb/milvus:v2.1.4
```
### 4. Migrate the metadata
+
A major change in Milvus 2.2 is the metadata structure of segment indexes. Therefore, you need to use Helm to migrate the metadata while upgrading Milvus from v2.1.x to v2.2.0. Here is [a script](https://github.com/milvus-io/milvus/blob/master/deployments/migrate-meta/migrate.sh) for you to safely migrate your metadata.
This script only applies to Milvus installed on a K8s cluster. Roll back to the previous version with the rollback operation first if an error occurs during the process.
@@ -136,74 +214,43 @@ The following table lists the operations you can do for meta migration.
4. Migrate the Milvus metadata.
5. Start Milvus components with a new image.
-#### 2. Upgrade Milvus from v2.1.x to v2.2.0
+#### 2. Upgrade Milvus from v2.1.x to 2.4.5
+
+The following commands assume that you upgrade Milvus from v2.1.4 to 2.4.5. Change them to the versions that fit your needs.
1. Specify Milvus instance name, source Milvus version, and target Milvus version.
```
- ./migrate.sh -i my-release -s 2.1.1 -t 2.2.0
+ ./migrate.sh -i my-release -s 2.1.4 -t 2.4.5
```
2. Specify the namespace with `-n` if your Milvus is not installed in the default K8s namespace.
```
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5
```
3. Specify the root path with `-r` if your Milvus is installed with the custom `rootpath`.
```
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0 -r by-dev
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5 -r by-dev
```
4. Specify the image tag with `-w` if your Milvus is installed with a custom `image`.
```
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0 -r by-dev -w milvusdb/milvus:master-20221016-15878781
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5 -r by-dev -w milvusdb/milvus:v2.4.5
```
5. Set `-d true` if you want to automatically remove the migration pod after the migration is completed.
```
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0 -w milvusdb/milvus:master-20221016-15878781 -d true
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5 -w milvusdb/milvus:v2.4.5 -d true
```
6. Rollback and migrate again if the migration fails.
```
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0 -r by-dev -o rollback -w
-
-1. The script hard-codes the upgrade order of the deployments and cannot be changed.
-2. The script uses `kubectl patch` to update the deployments and `kubectl rollout status` to watch their status.
-3. The script uses `kubectl patch` to update the `app.kubernetes.io/version` label of the deployments to the one specified after the `-t` flag in the command.
-
-
\ No newline at end of file
diff --git a/preview/site/en/adminGuide/upgrade_milvus_cluster-operator.md b/preview/site/en/adminGuide/upgrade_milvus_cluster-operator.md
index fa16ecc4f..ceb919f38 100644
--- a/preview/site/en/adminGuide/upgrade_milvus_cluster-operator.md
+++ b/preview/site/en/adminGuide/upgrade_milvus_cluster-operator.md
@@ -5,88 +5,30 @@ order: 0
group: upgrade_milvus_cluster-operator.md
related_key: upgrade Milvus Cluster
summary: Learn how to upgrade Milvus cluster with Milvus Operator.
+title: Upgrade Milvus Cluster with Milvus Operator
---
-
+
# Upgrade Milvus Cluster with Milvus Operator
-This guide describes how to ugrade your Milvus cluster with Milvus operator.
+This guide describes how to upgrade your Milvus cluster with Milvus operator.
## Upgrade your Milvus operator
-Run the following command to upgrade the version of your Milvus Operator to v0.7.7.
+Run the following command to upgrade the version of your Milvus Operator to v0.9.17.
```
-helm repo add milvus-operator https://milvus-io.github.io/milvus-operator/
-helm repo update milvus-operator
-helm -n milvus-operator upgrade milvus-operator milvus-operator/milvus-operator
+helm repo add zilliztech-milvus-operator https://zilliztech.github.io/milvus-operator/
+helm repo update zilliztech-milvus-operator
+helm -n milvus-operator upgrade milvus-operator zilliztech-milvus-operator/milvus-operator
```
Once you have upgraded your Milvus operator to the latest version, you have the following choices:
-- [Migrate the metadata](#Migrate-the-metadata)
-- [Conduct a rolling upgrade](#Conduct-a-rolling-upgrade)
-
-## Migrate the metadata
-
-Since Milvus 2.2.0, the metadata is incompatible with that in previous releases. The following example snippets assume an upgrade from Milvus 2.1.4 to Milvus 2.2.0.
-
-### 1. Create a `.yaml` file for metadata migration
-
-Create a metadata migration file. The following is an example. You need to specify the `name`, `sourceVersion`, and `targetVersion` in the configuration file. The following example sets the `name` to `my-release-upgrade`, `sourceVersion` to `v2.1.4`, and `targetVersion` to `v2.2.0`. This means that your Milvus cluster will be upgraded from v2.1.4 to v2.2.0.
-
-```
-apiVersion: milvus.io/v1beta1
-kind: MilvusUpgrade
-metadata:
- name: my-release-upgrade
-spec:
- milvus:
- namespace: default
- name: my-release
- sourceVersion: "v2.1.4"
- targetVersion: "v2.2.0"
- # below are some omit default values:
- # targetImage: "milvusdb/milvus:v2.2.0"
- # toolImage: "milvusdb/meta-migration:v2.2.0"
- # operation: upgrade
- # rollbackIfFailed: true
- # backupPVC: ""
- # maxRetry: 3
-```
-
-
-
-### 2. Apply the new configuration
-
-Run the following command to apply the new configuration.
-
-```
-$ kubectl apply -f https://github.com/milvus-io/milvus-operator/blob/main/config/samples/beta/milvusupgrade.yaml
-```
-
-
-
-### 3. Check the status of metadata migration
-
-Run the following command to check the status of your metadata migration.
-
-```
-kubectl describe milvus release-name
-```
-
-The status of `ready` in the output means that the metadata migration is successful.
-
-Or, you can also run `kubectl get pod` to check all the pods. If all the pods are `ready`, the metadata migration is successful.
-
-
-
-### 4. Delete `my-release-upgrade`
-
-When the upgrade is successful, delete `my-release-upgrade` in the YAML file.
-
-
+- To upgrade Milvus from v2.2.3 or later releases to 2.4.5, you can [conduct a rolling upgrade](#Conduct-a-rolling-upgrade).
+- To upgrade Milvus from a minor release before v2.2.3 to 2.4.5, you are advised to [upgrade Milvus by changing its image version](#Upgrade-Milvus-by-changing-its-image).
+- To upgrade Milvus from v2.1.x to 2.4.5, you need to [migrate the metadata](#Migrate-the-metadata) before the actual upgrade.
## Conduct a rolling upgrade
@@ -105,7 +47,7 @@ spec:
components:
enableRollingUpdate: true
imageUpdateMode: rollingUpgrade # Default value, can be omitted
- image: milvusdb/milvus:
-```
-docker stop
-```
+Due to security concerns, Milvus upgrades its MinIO to RELEASE.2023-03-20T20-16-18Z with the release of v2.2.5. Before any upgrades from previous Milvus Standalone releases installed using Docker Compose, you should create a Single-Node Single-Drive MinIO deployment and migrate existing MinIO settings and content to the new deployment. For details, refer to [this guide](https://min.io/docs/minio/linux/operations/install-deploy-manage/migrate-fs-gateway.html#id2).
-2. Prepare the configuration file `migration.yaml` for meta migration.
+
-```
-# migration.yaml
-cmd:
- # Option: run/backup/rollback
- type: run
- runWithBackup: true
-config:
- sourceVersion: 2.1.4 # Specify your milvus version
- targetVersion: 2.2.0
- backupFilePath: /tmp/migration.bak
-metastore:
- type: etcd
-etcd:
- endpoints:
- - milvus-etcd:2379 # Use the etcd container name
- rootPath: by-dev # The root path where data is stored in etcd
- metaSubPath: meta
- kvSubPath: kv
-```
+## Upgrade Milvus by changing its image
-3. Run the migration container.
+In normal cases, you can upgrade Milvus as follows:
-```
-# Suppose your docker-compose run with the default milvus network,
-# and you put migration.yaml in the same directory with docker-compose.yaml.
-docker run --rm -it --network milvus -v $(pwd)/migration.yaml:/milvus/configs/migration.yaml milvusdb/meta-migration:v2.2.0 /milvus/bin/meta-migration -config=/milvus/configs/migration.yaml
-```
+1. Change the Milvus image tag in `docker-compose.yaml`.
-4. Start Milvus components again with the new Milvus image.
+ ```yaml
+ ...
+ standalone:
+ container_name: milvus-standalone
+ image: milvusdb/milvus:v2.4.5
+ ```
+
+2. Run the following commands to perform the upgrade.
+
+ ```shell
+ docker compose down
+ docker compose up -d
+ ```
+
+## Migrate the metadata
-```
-Update the milvus image tag in the docker-compose.yaml
-docker-compose down
-docker-compose up -d
-```
+1. Stop all Milvus components.
+
+ ```
+ docker stop
+
+The Milvus Helm Charts repo at `https://milvus-io.github.io/milvus-helm/` has been archived and you can get further updates from `https://zilliztech.github.io/milvus-helm/` as follows:
+
+```shell
+helm repo add zilliztech https://zilliztech.github.io/milvus-helm
+helm repo update
+# upgrade existing helm release
+helm upgrade my-release zilliztech/milvus
```
-NAME CHART VERSION APP VERSION DESCRIPTION
-milvus/milvus 4.0.8 2.2.3 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.7 2.2.3 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.6 2.2.2 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.5 2.2.2 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.4 2.2.2 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.3 2.2.2 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.2 2.2.2 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.1 2.2.1 Milvus is an open-source vector database built ...
-milvus/milvus 4.0.0 2.2.1 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.6 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.5 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.4 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.3 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.2 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.1 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.3.0 2.2.0 Milvus is an open-source vector database built ...
-milvus/milvus 3.2.18 2.1.4 Milvus is an open-source vector database built ...
+
+The archived repo is still available for the charts up to 4.0.31. For later releases, use the new repo instead.
+
+
+
+```
+NAME CHART VERSION APP VERSION DESCRIPTION
+zilliztech/milvus 4.1.34 2.4.5 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.33 2.4.4 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.32 2.4.3 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.31 2.4.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.30 2.4.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.29 2.4.0 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.24 2.3.11 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.23 2.3.10 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.22 2.3.10 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.21 2.3.10 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.20 2.3.10 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.18 2.3.10 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.18 2.3.9 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.17 2.3.8 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.16 2.3.7 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.15 2.3.5 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.14 2.3.6 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.13 2.3.5 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.12 2.3.5 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.11 2.3.4 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.10 2.3.3 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.9 2.3.3 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.8 2.3.2 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.7 2.3.2 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.6 2.3.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.5 2.3.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.4 2.3.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.3 2.3.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.2 2.3.1 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.1 2.3.0 Milvus is an open-source vector database built ...
+zilliztech/milvus 4.1.0 2.3.0 Milvus is an open-source vector database built ...
```
-In normal cases, you can upgrade your Milvus instance using `helm upgrade` except you want to:
+You can choose the upgrade path for your Milvus as follows:
+
+
+
+- [Upgrade Milvus using Helm](#Upgrade-Milvus-using-Helm) for an upgrade from a minor release before v2.2.3 to v2.4.5.
+
+- [Migrate the metadata](#Migrate-the-metadata) before the upgrade from Milvus v2.1.x to v2.4.5.
+
+
+
+## Upgrade Milvus using Helm
+
+To upgrade Milvus from a minor release before v2.2.3 to the latest, run the following commands:
-- [Migrate the metadata](#Migrate-the-metadata)
-- [Conduct a rolling upgrade](#Conduct-a-rolling-upgrade)
+```shell
+helm repo update
+helm upgrade my-release milvus/milvus --reuse-values --version=4.1.24 # use the helm chart version here
+```
+
+Use the Helm chart version in the preceding command. For details on how to obtain the Helm chart version, refer to [Check the Milvus version](#Check-the-Milvus-version).
## Migrate the metadata
@@ -75,18 +156,16 @@ my-release-minio-744dd9586f-qngzv 1/1 Running 0 84s
### 3. Check the image tag
-Check the image tag for the pod `my-release-milvus-standalone-75c599fffc-6rwlj`. You can see the release of your Milvus standalone is v2.1.4.
-
-```
-$ kubectl get pods my-release-milvus-standalone-75c599fffc-6rwlj -o=jsonpath='{$.spec.containers[0].image}'
-```
+Check the image tag for the pod `my-release-milvus-proxy-6c548f787f-scspp`. You can see the release of your Milvus cluster is v2.1.4.
-```
-milvusdb/milvus:v2.1.4
+```shell
+$ kubectl get pods my-release-milvus-proxy-6c548f787f-scspp -o=jsonpath='{$.spec.containers[0].image}'
+# milvusdb/milvus:v2.1.4
```
### 4. Migrate the metadata
-A major change in Milvus 2.2 is the metadata structure of segment indexes. Therefore, you need to use Helm to migrate the meta while upgrading Milvus from v2.1.x to v2.2.0. Here is [a script](https://github.com/milvus-io/milvus/blob/master/deployments/migrate-meta/migrate.sh) for you to safely migrate your metadata.
+
+A major change in Milvus 2.2 is the metadata structure of segment indexes. Therefore, you need to use Helm to migrate the metadata while upgrading Milvus from v2.1.x to v2.2.0. Here is [a script](https://github.com/milvus-io/milvus/blob/master/deployments/migrate-meta/migrate.sh) for you to safely migrate your metadata.
This script only applies to Milvus installed on a K8s cluster. Roll back to the previous version with the rollback operation first if an error occurs during the process.
@@ -100,88 +179,56 @@ The following table lists the operations you can do for meta migration.
| `t` | The target Milvus version. | `None` | True |
| `r` | The root path of Milvus meta. | `by-dev` | False |
| `w` | The new Milvus image tag. | `milvusdb/milvus:v2.2.0` | False |
-| `m` | The meta migration image tag. | `harbor.milvus.io/milvus/meta-migration:20221025-e54b6181b` | False |
+| `m` | The meta migration image tag. | `milvusdb/meta-migration:v2.2.0` | False |
| `o` | The meta migration operation. | `migrate` | False |
| `d` | Whether to delete migration pod after the migration is completed. | `false` | False |
| `c` | The storage class for meta migration pvc. | `default storage class` | False |
| `e` | The etcd enpoint used by milvus. | `etcd svc installed with milvus` | False |
-#### 1. Migrate meta
+#### 1. Migrate the metadata
1. Download the [migration script](https://github.com/milvus-io/milvus/blob/master/deployments/migrate-meta/migrate.sh).
-2. Stop the Milvus components. Any live session in the Milvus etcd can cause the migration to fail.
+2. Stop the Milvus components. Any live session in the Milvus etcd can cause a migration failure.
3. Create a backup for the Milvus metadata.
4. Migrate the Milvus metadata.
5. Start Milvus components with a new image.
-#### 2. Upgrade Milvus from v2.1.x to v2.2.0
+#### 2. Upgrade Milvus from v2.1.x to 2.4.5
+
+The following commands assume that you upgrade Milvus from v2.1.4 to 2.4.5. Change them to the versions that fit your needs.
1. Specify Milvus instance name, source Milvus version, and target Milvus version.
```
- ./migrate.sh -i my-release -s 2.1.1 -t 2.2.0
+ ./migrate.sh -i my-release -s 2.1.4 -t 2.4.5
```
2. Specify the namespace with `-n` if your Milvus is not installed in the default K8s namespace.
```
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5
```
3. Specify the root path with `-r` if your Milvus is installed with the custom `rootpath`.
```
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0 -r by-dev
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5 -r by-dev
```
4. Specify the image tag with `-w` if your Milvus is installed with a custom `image`.
```
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0 -r by-dev -w milvusdb/milvus:master-20221016-15878781
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5 -r by-dev -w milvusdb/milvus:v2.4.5
```
5. Set `-d true` if you want to automatically remove the migration pod after the migration is completed.
```
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0 -w milvusdb/milvus:master-20221016-15878781 -d true
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5 -w milvusdb/milvus:v2.4.5 -d true
```
6. Rollback and migrate again if the migration fails.
```
- ./migrate.sh -i my-release -n milvus -s 2.1.1 -t 2.2.0 -r by-dev -o rollback -w
-
-1. The script **does not apply** to the Milvus instance installed with **RocksMQ**.
-1. The script hard-codes the upgrade order of the deployments and cannot be changed.
-2. The script uses `kubectl patch` to update the deployments and `kubectl rollout status` to watch their status.
-3. The script uses `kubectl patch` to update the `app.kubernetes.io/version` label of the deployments to the one specified after the `-t` flag in the command.
-
-
\ No newline at end of file
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5 -r by-dev -o rollback -w milvusdb/milvus:v2.1.1
+ ./migrate.sh -i my-release -n milvus -s 2.1.4 -t 2.4.5 -r by-dev -o migrate -w milvusdb/milvus:v2.4.5
diff --git a/preview/site/en/adminGuide/upgrade_milvus_standalone-operator.md b/preview/site/en/adminGuide/upgrade_milvus_standalone-operator.md
index 54a90e418..97e700242 100644
--- a/preview/site/en/adminGuide/upgrade_milvus_standalone-operator.md
+++ b/preview/site/en/adminGuide/upgrade_milvus_standalone-operator.md
@@ -5,88 +5,30 @@ order: 0
group: upgrade_milvus_standalone-operator.md
related_key: upgrade Milvus Standalone
summary: Learn how to upgrade Milvus standalone with Milvus operator.
+title: Upgrade Milvus Standalone with Milvus Operator
---
-
+
# Upgrade Milvus Standalone with Milvus Operator
-This guide describes how to ugrade your Milvus standalone with Milvus operator.
+This guide describes how to upgrade your Milvus standalone with Milvus operator.
## Upgrade your Milvus operator
-Run the following command to upgrade the version of your Milvus operator to v0.7.7.
+Run the following command to upgrade the version of your Milvus operator to v0.9.17.
```
-helm repo add milvus-operator https://milvus-io.github.io/milvus-operator/
-helm repo update milvus-operator
-helm -n milvus-operator upgrade milvus-operator milvus-operator/milvus-operator
+helm repo add zilliztech-milvus-operator https://zilliztech.github.io/milvus-operator/
+helm repo update zilliztech-milvus-operator
+helm -n milvus-operator upgrade milvus-operator zilliztech-milvus-operator/milvus-operator
```
Once you have upgraded your Milvus operator to the latest version, you have the following choices:
-- [Migrate the metadata](#Migrate-the-metadata)
-- [Conduct a rolling upgrade](#Conduct-a-rolling-upgrade)
-
-## Migrate the metadata
-
-Since Milvus 2.2.0, the metadata is incompatible with that in previous releases. The following example snippets assume an upgrade from Milvus 2.1.4 to Milvus 2.2.0.
-
-### 1. Create a `.yaml` file for metadata migration
-
-Create a metadata migration file. The following is an example. You need to specify the `name`, `sourceVersion`, and `targetVersion` in the configuration file. The following example sets the `name` to `my-release-upgrade`, `sourceVersion` to `v2.1.4`, and `targetVersion` to `v2.2.0`. This means that your Milvus instance will be upgraded from v2.1.4 to v2.2.0.
-
-```
-apiVersion: milvus.io/v1beta1
-kind: MilvusUpgrade
-metadata:
- name: my-release-upgrade
-spec:
- milvus:
- namespace: default
- name: my-release
- sourceVersion: "v2.1.4"
- targetVersion: "v2.2.0"
- # below are some omit default values:
- # targetImage: "milvusdb/milvus:v2.2.0"
- # toolImage: "milvusdb/meta-migration:v2.2.0"
- # operation: upgrade
- # rollbackIfFailed: true
- # backupPVC: ""
- # maxRetry: 3
-```
-
-
-
-### 2. Apply the new configuration
-
-Run the following command to apply the new configuration.
-
-```
-$ kubectl apply -f https://github.com/milvus-io/milvus-operator/blob/main/config/samples/beta/milvusupgrade.yaml
-```
-
-
-
-### 3. Check the status of metadata migration
-
-Run the following command to check the status of your metadata migration.
-
-```
-kubectl describe milvus release-name
-```
-
-The status of `ready` in the output means that the metadata migration is successful.
-
-Or, you can also run `kubectl get pod` to check all the pods. If all the pods are `ready`, the metadata migration is successful.
-
-
-
-### 4. Delete `my-release-upgrade`
-
-When the upgrade is successful, delete `my-release-upgrade` in the YAML file.
-
-
+- To upgrade Milvus from v2.2.3 or later releases to 2.4.5, you can [conduct a rolling upgrade](#Conduct-a-rolling-upgrade).
+- To upgrade Milvus from a minor release before v2.2.3 to 2.4.5, you are advised to [upgrade Milvus by changing its image version](#Upgrade-Milvus-by-changing-its-image).
+- To upgrade Milvus from v2.1.x to 2.4.5, you need to [migrate the metadata](#Migrate-the-metadata) before the actual upgrade.
## Conduct a rolling upgrade
@@ -105,12 +47,12 @@ spec:
components:
enableRollingUpdate: true
imageUpdateMode: rollingUpgrade # Default value, can be omitted
- image: milvusdb/milvus:
-Ansible can only be used in test environments. It is not recommended that you deploy Milvus distributed clusters in this way in production environments.
-
-
-## Prerequisites
-
-- Ensure that your CPU and RAM meet the requirements in [Environment Checklist](prerequisite-docker.md).
-- Hardware: Four servers including three Docker hosts, each with at least four cores of CPU and 8GB of RAM, and one Ansible controller
-- Operating system: Ubuntu 20.04 LTS
-- Software: [Ansible Controller](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html)
-
-## Set up Ansible admin controller
-
-We recommend creating a new Ansible controller on the Ubuntu operating system. Make sure system resources are sufficient for running Ansible tasks. See [Installing Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) for more information.
-
-## Download Ansible Milvus node deployment Playbook
-
-Clone Milvus repository from GitHub to download the Ansible Milvus node deployment Playbook.
-
-```
-$ git clone https://github.com/milvus-io/milvus.git
-```
-
-## Configure installation files
-
-The `inventory.ini` and `ansible.cfg` files are used to control the environment variables and log-in verification methods in Ansible playbook. In the `inventory.ini` file, the `dockernodes` section defines all the servers of docker engines. The `ansible.cfg` section defines all the servers of Milvus coordinators. The `node` section defines all the servers of Milvus nodes.
-
-Enter the local path to the Playbook and configure the installation files.
-
-```shell
-$ cd ./milvus/deployments/docker/cluster-distributed-deployment
-```
-
-#### `inventory.ini`
-
-Configure `inventory.ini` to divide hosts in groups in accordance with their roles in the Milvus system.
-
-Add host names, and define `docker` group and `vars`.
-
-```
-[dockernodes] #Add docker host names.
-dockernode01
-dockernode02
-dockernode03
-
-[admin] #Add Ansible controller name.
-ansible-controller
-
-[coords] #Add the host names of Milvus coordinators.
-; Take note the IP of this host VM, and replace 10.170.0.17 with it.
-dockernode01
-
-[nodes] #Add the host names of Milvus nodes.
-dockernode02
-
-[dependencies] #Add the host names of Milvus dependencies.
-; dependencies node will host etcd, minio, pulsar, these 3 roles are the foundation of Milvus.
-; Take note the IP of this host VM, and replace 10.170.0.19 with it.
-dockernode03
-
-[docker:children]
-dockernodes
-coords
-nodes
-dependencies
-
-[docker:vars]
-ansible_python_interpreter= /usr/bin/python3
-StrictHostKeyChecking= no
-
-; Setup variables to controll what type of network to use when creating containers.
-dependencies_network= host
-nodes_network= host
-
-; Setup varibale to controll what version of Milvus image to use.
-image= milvusdb/milvus-dev:master-20220412-4781db8a
-
-; Setup static IP addresses of the docker hosts as variable for container environment variable config.
-; Before running the playbook, below 4 IP addresses need to be replaced with the IP of your host VM
-; on which the etcd, minio, pulsar, coordinators will be hosted.
-etcd_ip= 10.170.0.19
-minio_ip= 10.170.0.19
-pulsar_ip= 10.170.0.19
-coords_ip= 10.170.0.17
-
-; Setup container environment which later will be used in container creation.
-ETCD_ENDPOINTS= {{etcd_ip}}:2379
-MINIO_ADDRESS= {{minio_ip}}:9000
-PULSAR_ADDRESS= pulsar://{{pulsar_ip}}:6650
-QUERY_COORD_ADDRESS= {{coords_ip}}:19531
-DATA_COORD_ADDRESS= {{coords_ip}}:13333
-ROOT_COORD_ADDRESS= {{coords_ip}}:53100
-INDEX_COORD_ADDRESS= {{coords_ip}}:31000
-```
-
-#### `ansible.cfg`
-
-`ansible.cfg` controls the action of the playbook, for example, SSH key, etc. Do not set up passphrase via the SSH key on docker hosts. Otherwise, the Ansible SSH connection will fail. We recommend setting up the same username and SSH key on the three hosts and setting up the new user account to execute sudo without a password. Otherwise, you will receive errors that the user name does not match the password or you are not granted elevated privileges when running Ansible playbook.
-
-
-```
-[defaults]
-host_key_checking = False
-inventory = inventory.ini # Specify the Inventory file
-private_key_file=~/.my_ssh_keys/gpc_sshkey # Specify the SSH key that Ansible uses to access Docker host
-```
-
-#### `deploy-docker.yml`
-
-`deploy-docker.yml` defines the tasks during the installation of Docker. See the code comments in the file for details.
-
-```yaml
----
-- name: setup pre-requisites # Install prerequisite
- hosts: all
- become: yes
- become_user: root
- roles:
- - install-modules
- - configure-hosts-file
-
-- name: install docker
- become: yes
- become_user: root
- hosts: dockernodes
- roles:
- - docker-installation
-```
-
-## Test Ansible connectivity
-
-Test the connectivity to Ansible.
-
-```shell
-$ ansible all -m ping
-```
-
-Add `-i` in the command to specify the path to the inventory file if you did not specify it in `ansible.cfg`, otherwise Ansible uses `/etc/ansible/hosts`.
-
-The terminal returns as follow:
-
-```
-dockernode01 | SUCCESS => {
-"changed": false,
-"ping": "pong"
-}
-ansible-controller | SUCCESS => {
- "ansible_facts": {
- "discovered_interpreter_python": "/usr/bin/python3"
- },
- "changed": false,
- "ping": "pong"
-}
-dockernode03 | SUCCESS => {
- "changed": false,
- "ping": "pong"
-}
-dockernode02 | SUCCESS => {
- "changed": false,
- "ping": "pong"
-}
-```
-
-## Check the Playbook Syntax
-
-Check the syntax of the Playbook.
-
-```shell
-$ ansible-playbook deploy-docker.yml --syntax-check
-```
-
-Normally, the terminal returns as follow:
-
-```
-playbook: deploy-docker.yml
-```
-
-## Install Docker
-
-Install Docker with the Playbook.
-
-```shell
-$ ansible-playbook deploy-docker.yml
-```
-
-If Docker is successfully installed on the three hosts, the terminal returns as follow:
-
-```
-TASK [docker-installation : Install Docker-CE] *******************************************************************
-ok: [dockernode01]
-ok: [dockernode03]
-ok: [dockernode02]
-
-TASK [docker-installation : Install python3-docker] **************************************************************
-ok: [dockernode01]
-ok: [dockernode02]
-ok: [dockernode03]
-
-TASK [docker-installation : Install docker-compose python3 library] **********************************************
-changed: [dockernode01]
-changed: [dockernode03]
-changed: [dockernode02]
-
-PLAY RECAP *******************************************************************************************************
-ansible-controller : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
-dockernode01 : ok=10 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
-dockernode02 : ok=10 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
-dockernode03 : ok=10 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
-```
-
-## Verify the installation
-
-Log in to the three hosts with the SSH key, and verify the installation on the hosts.
-
-- For root host:
-
-```shell
-$ docker -v
-```
-
-- For non-root hosts:
-
-```shell
-$ sudo docker -v
-```
-
-Normally, the terminal returns as follow:
-
-```
-Docker version 20.10.14, build a224086
-```
-
-Check the running status of the containers.
-
-```shell
-$ docker ps
-```
-
-## Check the Syntax
-
-Check the Syntax of `deploy-milvus.yml`.
-
-```shell
-$ ansible-playbook deploy-milvus.yml --syntax-check
-```
-
-Normally, the terminal returns as follow:
-
-```
-playbook: deploy-milvus.yml
-```
-
-## Create Milvus container
-
-The tasks to create Milvus container are defined in `deploy-milvus.yml`.
-
-```shell
-$ ansible-playbook deploy-milvus.yml
-```
-
-The terminal returns:
-
-```
-PLAY [Create milvus-etcd, minio, pulsar] *****************************************************************
-
-TASK [Gathering Facts] ********************************************************************************************
-ok: [dockernode03]
-
-TASK [etcd] *******************************************************************************************************
-changed: [dockernode03]
-
-TASK [pulsar] *****************************************************************************************************
-changed: [dockernode03]
-
-TASK [minio] ******************************************************************************************************
-changed: [dockernode03]
-
-PLAY [Create milvus nodes] ****************************************************************************************
-
-TASK [Gathering Facts] ********************************************************************************************
-ok: [dockernode02]
-
-TASK [querynode] **************************************************************************************************
-changed: [dockernode02]
-
-TASK [datanode] ***************************************************************************************************
-changed: [dockernode02]
-
-TASK [indexnode] **************************************************************************************************
-changed: [dockernode02]
-
-PLAY [Create milvus coords] ***************************************************************************************
-
-TASK [Gathering Facts] ********************************************************************************************
-ok: [dockernode01]
-
-TASK [rootcoord] **************************************************************************************************
-changed: [dockernode01]
-
-TASK [datacoord] **************************************************************************************************
-changed: [dockernode01]
-
-TASK [querycoord] *************************************************************************************************
-changed: [dockernode01]
-
-TASK [indexcoord] *************************************************************************************************
-changed: [dockernode01]
-
-TASK [proxy] ******************************************************************************************************
-changed: [dockernode01]
-
-PLAY RECAP ********************************************************************************************************
-dockernode01 : ok=6 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
-dockernode02 : ok=4 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
-dockernode03 : ok=4 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
-```
-
-Now you have Milvus deployed on the three hosts.
-
-## Scale out Milvus nodes
-
-If you need to scale out Milvus nodes, you can follow the steps below to add new docker hosts directly in Ansible playbook.
-
-1. Prepare host resources as requested in the prerequisites.
-2. Ensure connectivity to the network and host names can be resolved.
-3. Add the new host to the corresponding section in the `inventory.ini` file. More specifically, add the host name of the new node in the node section as shown in the example below.
-
-```
-[nodes] #Add host names of Milvus nodes.
-dockernode02
-dockernode04 #This is the host name of the new node.
-```
-
-
-
-The new configuration is automatically applied when running the playbook and does not affect the original host configurations. Ansible playbook records all deployments, and when a new host is introduced, a new deployment is made on the new host in order not to affect the original deployment.
-
-
-
-4. After deployment, run the following command to check the running status of the new host.
-
-```
-Docker ps -a
-```
-
-
-## What's next
-
-Having installed Milvus, you can:
-
-- Check [Hello Milvus](example_code.md) to run an example code with different SDKs to see what Milvus can do.
-
-- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
-
-- Explore [MilvusDM](migrate_overview.md), an open-source tool designed for importing and exporting data in Milvus.
-- [Monitor Milvus with Prometheus](monitor.md).
\ No newline at end of file
diff --git a/preview/site/en/getstarted/cluster/install_cluster-docker.md b/preview/site/en/getstarted/cluster/install_cluster-docker.md
deleted file mode 100644
index b63938da5..000000000
--- a/preview/site/en/getstarted/cluster/install_cluster-docker.md
+++ /dev/null
@@ -1,112 +0,0 @@
----
-id: install_cluster-docker.md
-label: Docker Compose
-related_key: Docker
-order: 2
-group: install_cluster-milvusoperator.md
-summary: Learn how to install Milvus cluster with Docker Compose.
----
-
-
-
-# Install Milvus Cluster with Docker Compose
-
-This topic introduces how to deploy a Milvus cluster with Docker Compose.
-
-
-Docker Compose cannot deploy Milvus distributed clusters across machines, and can only be used in test environments. It is not recommended that you deploy Milvus distributed clusters in this way in production environments.
-
-
-## Prerequisites
-
-Check [the requirements](prerequisite-docker.md) for hardware and software prior to your installation.
-
-## Download the `YAML` file
-
-[Download](https://github.com/milvus-io/milvus/releases/download/v2.2.3/milvus-cluster-docker-compose.yml) `milvus-cluster-docker-compose.yml` and save it as `docker-compose.yml` manually, or with the following command.
-
-```
-$ wget https://github.com/milvus-io/milvus/releases/download/v2.2.3/milvus-cluster-docker-compose.yml -O docker-compose.yml
-```
-
-
-## Start Milvus
-
-In the same directory as the `docker-compose.yml` file, start up Milvus by running:
-
-```Shell
-$ sudo docker-compose up -d
-```
-
-
-If your system has Docker Compose V2 installed instead of V1, use
-
-```Text
-Creating milvus-etcd ... done
-Creating milvus-minio ... done
-Creating milvus-pulsar ... done
-Creating milvus-proxy ... done
-Creating milvus-rootcoord ... done
-Creating milvus-indexcoord ... done
-Creating milvus-querycoord ... done
-Creating milvus-datacoord ... done
-Creating milvus-querynode ... done
-Creating milvus-indexnode ... done
-Creating milvus-datanode ... done
-```
-
-Check the status of the containers.
-
-```
-$ sudo docker ps
-```
-
-After Milvus cluster starts, 11 running docker containers appear including three dependencies and eight Milvus services.
-
-```
- Name Command State Ports
---------------------------------------------------------------------------------------------------------------------
-milvus-datacoord /tini -- milvus run datacoord Up
-milvus-datanode /tini -- milvus run datanode Up
-milvus-etcd etcd -advertise-client-url ... Up 2379/tcp, 2380/tcp
-milvus-indexcoord /tini -- milvus run indexcoord Up
-milvus-indexnode /tini -- milvus run indexnode Up
-milvus-minio /usr/bin/docker-entrypoint ... Up (healthy) 0.0.0.0:9000->9000/tcp, 0.0.0.0:9001->9001/tcp
-milvus-proxy /tini -- milvus run proxy Up 0.0.0.0:19530->19530/tcp, 0.0.0.0:9091->9091/tcp
-milvus-pulsar /bin/bash -c Up
- bin/apply-co ...
-milvus-querycoord /tini -- milvus run querycoord Up
-milvus-querynode /tini -- milvus run querynode Up
-milvus-rootcoord /tini -- milvus run rootcoord Up
-```
-
-## Stop Milvus
-
-To stop Milvus cluster, run:
-
-```shell
-$ sudo docker-compose down
-```
-
-To delete data after stopping Milvus, run:
-
-```shell
-$ sudo rm -rf volumes
-```
-
-## What's next
-
-Having installed Milvus, you can:
-
-- Check [Hello Milvus](example_code.md) to run an example code with different SDKs to see what Milvus can do.
-
-- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
-
-- Explore [MilvusDM](migrate_overview.md), an open-source tool designed for importing and exporting data in Milvus.
-- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/cluster/install_cluster-helm.md b/preview/site/en/getstarted/cluster/install_cluster-helm.md
deleted file mode 100644
index d6687cd19..000000000
--- a/preview/site/en/getstarted/cluster/install_cluster-helm.md
+++ /dev/null
@@ -1,195 +0,0 @@
----
-id: install_cluster-helm.md
-label: Helm
-related_key: Kubernetes
-order: 1
-group: install_cluster-milvusoperator.md
-summary: Learn how to install Milvus cluster on Kubernetes.
----
-
-
-
-# Install Milvus Cluster with Helm
-
-This topic introduces how to deploy a Milvus cluster with Helm on Kubernetes (K8s).
-
-## Prerequisites
-
-Check [the requirements](prerequisite-helm.md) for hardware and software prior to your installation.
-
-## Create a K8s Cluster
-
-If you have already deployed a K8s cluster for production, you can skip this step and proceed directly to [Install Helm Chart for Milvus](install_cluster-helm.md#Install-Helm-Chart-for-Milvus). If not, you can follow the steps below to quickly create a K8s for testing, and then use it to deploy a Milvus cluster with Helm.
-
-### Create a K8s cluster using minikube
-
-We recommend installing Milvus on K8s with [minikube](https://minikube.sigs.k8s.io/docs/), a tool that allows you to run K8s locally.
-
- docker compose instead of docker-compose . Check if this is the case with $ docker compose version . Read here for more information.
-
-minikube can only be used in test environments. It is not recommended that you deploy Milvus distributed clusters in this way in production environments.
-
-
-#### 1. Install minikube
-
-See [install minikube](https://minikube.sigs.k8s.io/docs/start/) for more information.
-
-#### 2. Start a K8s cluster using minikube
-
-After installing minikube, run the following command to start a K8s cluster.
-
-```
-$ minikube start
-```
-
-#### 3. Check the K8s cluster status
-
-Run `$ kubectl cluster-info` to check the status of the K8s cluster you just created. Ensure that you can access the K8s cluster via `kubectl`. If you have not installed `kubectl` locally, see [Use kubectl inside minikube](https://minikube.sigs.k8s.io/docs/handbook/kubectl/).
-
-
-minikube has a dependency on default StorageClass when installed. Check the dependency by running the following command. Other installation methods require manual configuration of the StorageClass. See [Change the default StorageClass](https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/) for more information.
-
-```
-$ kubectl get sc
-```
-
-```
-NAME PROVISIONER RECLAIMPOLICY VOLUMEBIINDINGMODE ALLOWVOLUMEEXPANSION AGE
-standard (default) k8s.io/minikube-hostpath Delete Immediate false 3m36s
-```
-
-## Install Helm Chart for Milvus
-
-Helm is a K8s package manager that can help you deploy Milvus quickly.
-
-1. Add Milvus Helm repository.
-
-```
-$ helm repo add milvus https://milvus-io.github.io/milvus-helm/
-```
-
-2. Update charts locally.
-
-```
-$ helm repo update
-```
-
-## Start Milvus
-
-Start Milvus with Helm by specifying the release name, the chart, and parameters you expect to change. This topic uses my-release as the release name. To use a different release name, replace my-release in the command.
-
-```
-$ helm install my-release milvus/milvus
-```
-
-
-
-
-Check the status of the running pods.
-
-```
-$ kubectl get pods
-```
-
-After Milvus starts, the `READY` column displays `1/1` for all pods.
-
-```
-NAME READY STATUS RESTARTS AGE
-my-release-etcd-0 1/1 Running 0 3m23s
-my-release-etcd-1 1/1 Running 0 3m23s
-my-release-etcd-2 1/1 Running 0 3m23s
-my-release-milvus-datacoord-6fd4bd885c-gkzwx 1/1 Running 0 3m23s
-my-release-milvus-datanode-68cb87dcbd-4khpm 1/1 Running 0 3m23s
-my-release-milvus-indexcoord-5bfcf6bdd8-nmh5l 1/1 Running 0 3m23s
-my-release-milvus-indexnode-5c5f7b5bd9-l8hjg 1/1 Running 0 3m24s
-my-release-milvus-proxy-6bd7f5587-ds2xv 1/1 Running 0 3m24s
-my-release-milvus-querycoord-579cd79455-xht5n 1/1 Running 0 3m24s
-my-release-milvus-querynode-5cd8fff495-k6gtg 1/1 Running 0 3m24s
-my-release-milvus-rootcoord-7fb9488465-dmbbj 1/1 Running 0 3m23s
-my-release-minio-0 1/1 Running 0 3m23s
-my-release-minio-1 1/1 Running 0 3m23s
-my-release-minio-2 1/1 Running 0 3m23s
-my-release-minio-3 1/1 Running 0 3m23s
-my-release-pulsar-autorecovery-86f5dbdf77-lchpc 1/1 Running 0 3m24s
-my-release-pulsar-bookkeeper-0 1/1 Running 0 3m23s
-my-release-pulsar-bookkeeper-1 1/1 Running 0 98s
-my-release-pulsar-broker-556ff89d4c-2m29m 1/1 Running 0 3m23s
-my-release-pulsar-proxy-6fbd75db75-nhg4v 1/1 Running 0 3m23s
-my-release-pulsar-zookeeper-0 1/1 Running 0 3m23s
-my-release-pulsar-zookeeper-metadata-98zbr 0/1 Completed 0 3m24s
-```
-
-## Connect to Milvus
-
-Verify which local port the Milvus server is listening on. Replace the pod name with your own.
-
-```bash
-$ kubectl get pod my-release-milvus-proxy-6bd7f5587-ds2xv --template
-='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
-19530
-```
-
-Open a new terminal and run the following command to forward a local port to the port that Milvus uses. Optionally, omit the designated port and use `:19530` to let `kubectl` allocate a local port for you so that you don't have to manage port conflicts.
-
-```bash
-$ kubectl port-forward service/my-release-milvus 27017:19530
-Forwarding from 127.0.0.1:27017 -> 19530
-```
-
-## Uninstall Milvus
-
-Run the following command to uninstall Milvus.
-
-```bash
-$ helm uninstall my-release
-```
-
-## Stop the K8s cluster
-
-Stop the cluster and the minikube VM without deleting the resources you created.
-
-```bash
-$ minikube stop
-```
-
-Run `minikube start` to restart the cluster.
-
-
-## Delete the K8s cluster
-
--
-
-Run
-
-Delete the cluster, the minikube VM, and all resources you created including persistent volumes.
-
-```bash
-$ minikube delete
-```
-
-## What's next
-
-Having installed Milvus, you can:
-
-- Check [Hello Milvus](example_code.md) to run an example code with different SDKs to see what Milvus can do.
-
-- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
-
-- [Upgrade Milvus Using Helm Chart](upgrade.md).
-- [Scale your Milvus cluster](scaleout.md).
-- Deploy your Milvu cluster on clouds:
- - [Amazon EC2](aws.md)
- - [Amazon EKS](eks.md)
-- Explore [MilvusDM](migrate_overview.md), an open-source tool designed for importing and exporting data in Milvus.
-- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/cluster/install_cluster-milvusoperator.md b/preview/site/en/getstarted/cluster/install_cluster-milvusoperator.md
deleted file mode 100644
index 4fd3b90da..000000000
--- a/preview/site/en/getstarted/cluster/install_cluster-milvusoperator.md
+++ /dev/null
@@ -1,501 +0,0 @@
----
-id: install_cluster-milvusoperator.md
-label: Milvus Operator
-related_key: Kubernetes
-order: 0
-group: install_cluster-milvusoperator.md
-summary: Learn how to install Milvus cluster on Kubernetes using Milvus Operator
----
-
-
-
-# Install Milvus Cluster with Milvus Operator
-
-Milvus Operator is a solution that helps you deploy and manage a full Milvus service stack to target Kubernetes (K8s) clusters. The stack includes all Milvus components and relevant dependencies like etcd, Pulsar and MinIO. This topic introduces how to deploy a Milvus cluster with Milvus Operator on K8s.
-
-## Prerequisites
-[Check the requirements for hardware and software](prerequisite-helm.md) prior to your installation.
-
-## Create a K8s Cluster
-
-If you have already deployed a K8s cluster for production, you can skip this step and proceed directly to [deploy Milvus Operator](install_cluster-milvusoperator.md#Deploy-Milvus-Operator). If not, you can follow the steps below to quickly create a K8s for testing, and then use it to deploy a Milvus cluster with Milvus Operator.
-
-### Create a K8s cluster using minikube
-
-We recommend installing Milvus on K8s with [minikube](https://minikube.sigs.k8s.io/docs/), a tool that allows you to run K8s locally.
-
-$ kubectl logs `pod_name` to get the stderr log of the pod before deleting the cluster and all resources.
-
-minikube can only be used in test environments. It is not recommended that you deploy Milvus distributed clusters in this way in production environments.
-
-
-#### 1. Install minikube
-
-See [install minikube](https://minikube.sigs.k8s.io/docs/start/) for more information.
-
-#### 2. Start a K8s cluster using minikube
-
-After installing minikube, run the following command to start a K8s cluster.
-
-```
-$ minikube start
-```
-
-#### 3. Check the K8s cluster status
-
-Run `$ kubectl cluster-info` to check the status of the K8s cluster you just created. Ensure that you can access the K8s cluster via `kubectl`. If you have not installed `kubectl` locally, see [Use kubectl inside minikube](https://minikube.sigs.k8s.io/docs/handbook/kubectl/).
-
-
-## Deploy Milvus Operator
-
-Milvus Operator defines a Milvus cluster custom resources on top of [Kubernetes Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). When the custom resources are defined, you can use K8s APIs in a declarative way and manage Milvus deployment stack to ensure its scalability and high-availability.
-
-### Prerequisites
-
-- Ensure that you can access the K8s cluster via `kubectl` or `helm`.
-- Ensure the StorageClass dependency is installed as Milvus clusters depend on default StorageClass for data persistence. minikube has a dependency on default StorageClass when installed. Check the dependency by running the command `kubectl get sc`. If StorageClass is installed, you will see the following output. If not, see [Change the Default StorageClass](https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/) for more information.
-
-```
-NAME PROVISIONER RECLAIMPOLICY VOLUMEBIINDINGMODE ALLOWVOLUMEEXPANSION AGE
-standard (default) k8s.io/minikube-hostpath Delete Immediate false 3m36s
-```
-
-### 1. Install cert-manager
-
-
-You can install Milvus Operator with Helm or `kubectl` command. If you choose to use Helm, you can skip this step and proceed directly to Install by Helm command.
-
-
-Milvus Operator uses [cert-manager](https://cert-manager.io/docs/installation/supported-releases/) to provide certificate for webhook server. Run the following command to install cert-manager.
-
-```
-$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml
-```
-
-If cert-manager is installed, you can see the following output.
-
-```
-customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
-customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
-customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
-customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
-customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
-customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
-namespace/cert-manager created
-serviceaccount/cert-manager-cainjector created
-serviceaccount/cert-manager created
-serviceaccount/cert-manager-webhook created
-clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
-clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
-clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
-clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
-clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
-clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
-clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
-clusterrole.rbac.authorization.k8s.io/cert-manager-view created
-clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
-clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
-clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
-clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
-clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
-clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
-clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
-clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
-clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
-clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
-clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
-clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
-clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
-clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
-role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
-role.rbac.authorization.k8s.io/cert-manager:leaderelection created
-role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
-rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
-rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
-rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
-service/cert-manager created
-service/cert-manager-webhook created
-deployment.apps/cert-manager-cainjector created
-deployment.apps/cert-manager created
-deployment.apps/cert-manager-webhook created
-mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
-validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
-```
-
-
-cert-manager version 1.1.3 or later is required.
-
-
-Run `$ kubectl get pods -n cert-manager` to check if cert-manager is running. You can see the following output if all the pods are running.
-
-```
-NAME READY STATUS RESTARTS AGE
-cert-manager-848f547974-gccz8 1/1 Running 0 70s
-cert-manager-cainjector-54f4cc6b5-dpj84 1/1 Running 0 70s
-cert-manager-webhook-7c9588c76-tqncn 1/1 Running 0 70s
-```
-
-### 2. Install Milvus Operator
-
-There are two ways to install Milvus Operator on K8s:
-
-- with helm chart
-- with `kubectl` command directly with raw manifests
-
-#### Install by Helm command
-
-```
-helm install milvus-operator \
- -n milvus-operator --create-namespace \
- --wait --wait-for-jobs \
- https://github.com/milvus-io/milvus-operator/releases/download/v0.7.7/milvus-operator-0.7.7.tgz
-```
-
-If Milvus Operator is installed, you can see the following output.
-```
-NAME: milvus-operator
-LAST DEPLOYED: Thu Jul 7 13:18:40 2022
-NAMESPACE: milvus-operator
-STATUS: deployed
-REVISION: 1
-TEST SUITE: None
-NOTES:
-Milvus Operator Is Starting, use `kubectl get -n milvus-operator deploy/milvus-operator` to check if its successfully installed
-If Operator not started successfully, check the checker's log with `kubectl -n milvus-operator logs job/milvus-operator-checker`
-Full Installation doc can be found in https://github.com/milvus-io/milvus-operator/blob/main/docs/installation/installation.md
-Quick start with `kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/config/samples/milvus_minimum.yaml`
-More samples can be found in https://github.com/milvus-io/milvus-operator/tree/main/config/samples
-CRD Documentation can be found in https://github.com/milvus-io/milvus-operator/tree/main/docs/CRD
-```
-
-#### Install by `kubectl` command
-
-```
-$ kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/deploy/manifests/deployment.yaml
-```
-
-If Milvus Operator is installed, you can see the following output.
-
-```
-namespace/milvus-operator created
-customresourcedefinition.apiextensions.k8s.io/milvusclusters.milvus.io created
-serviceaccount/milvus-operator-controller-manager created
-role.rbac.authorization.k8s.io/milvus-operator-leader-election-role created
-clusterrole.rbac.authorization.k8s.io/milvus-operator-manager-role created
-clusterrole.rbac.authorization.k8s.io/milvus-operator-metrics-reader created
-clusterrole.rbac.authorization.k8s.io/milvus-operator-proxy-role created
-rolebinding.rbac.authorization.k8s.io/milvus-operator-leader-election-rolebinding created
-clusterrolebinding.rbac.authorization.k8s.io/milvus-operator-manager-rolebinding created
-clusterrolebinding.rbac.authorization.k8s.io/milvus-operator-proxy-rolebinding created
-configmap/milvus-operator-manager-config created
-service/milvus-operator-controller-manager-metrics-service created
-service/milvus-operator-webhook-service created
-deployment.apps/milvus-operator-controller-manager created
-certificate.cert-manager.io/milvus-operator-serving-cert created
-issuer.cert-manager.io/milvus-operator-selfsigned-issuer created
-mutatingwebhookconfiguration.admissionregistration.k8s.io/milvus-operator-mutating-webhook-configuration created
-validatingwebhookconfiguration.admissionregistration.k8s.io/milvus-operator-validating-webhook-configuration created
-```
-
-Run `$ kubectl get pods -n milvus-operator` to check if Milvus Operator is running. You can see the following output if Milvus Operator is running.
-
-```
-NAME READY STATUS RESTARTS AGE
-milvus-operator-5fd77b87dc-msrk4 1/1 Running 0 46s
-```
-
-## Install a Milvus cluster
-
-This tutorial uses the default configuration to install a Milvus cluster. All Milvus cluster components are enabled with multiple replicas, which consumes many resources.
-
-
-If you have very limited local resources, you can install a Milvus cluster using the minimum configuration).
-
-
-### 1. Deploy a Milvus cluster
-
-When Milvus Operator starts, run the following command to deploy a Milvus cluster.
-
-```
-$ kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/config/samples/milvus_cluster_default.yaml
-```
-
-When the cluster is deployed, you can see the following output.
-
-```
-milvuscluster.milvus.io/my-release created
-```
-
-### 2. Check the Milvus cluster status
-
-Run the following command to check the status of the Milvus cluster you just deployed.
-
-```
-$ kubectl get milvus my-release -o yaml
-```
-
-You can confirm the current status of Milvus cluster from the `status` field in the output. When the Milvus cluster is still under creation, the `status` shows `Unhealthy`.
-
-```
-apiVersion: milvus.io/v1alpha1
-kind: MilvusCluster
-metadata:
-...
-status:
- conditions:
- - lastTransitionTime: "2021-11-02T02:52:04Z"
- message: 'Get "http://my-release-minio.default:9000/minio/admin/v3/info": dial
- tcp 10.96.78.153:9000: connect: connection refused'
- reason: ClientError
- status: "False"
- type: StorageReady
- - lastTransitionTime: "2021-11-02T02:52:04Z"
- message: connection error
- reason: PulsarNotReady
- status: "False"
- type: PulsarReady
- - lastTransitionTime: "2021-11-02T02:52:04Z"
- message: All etcd endpoints are unhealthy
- reason: EtcdNotReady
- status: "False"
- type: EtcdReady
- - lastTransitionTime: "2021-11-02T02:52:04Z"
- message: Milvus Dependencies is not ready
- reason: DependencyNotReady
- status: "False"
- type: MilvusReady
- endpoint: my-release-milvus.default:19530
- status: Unhealthy
-```
-
-Run the following command to check the current status of Milvus pods.
-
-```
-$ kubectl get pods
-```
-
-```
-NAME READY STATUS RESTARTS AGE
-my-release-etcd-0 0/1 Running 0 16s
-my-release-etcd-1 0/1 ContainerCreating 0 16s
-my-release-etcd-2 0/1 ContainerCreating 0 16s
-my-release-minio-0 1/1 Running 0 16s
-my-release-minio-1 1/1 Running 0 16s
-my-release-minio-2 0/1 Running 0 16s
-my-release-minio-3 0/1 ContainerCreating 0 16s
-my-release-pulsar-bookie-0 0/1 Pending 0 15s
-my-release-pulsar-bookie-1 0/1 Pending 0 15s
-my-release-pulsar-bookie-init-h6tfz 0/1 Init:0/1 0 15s
-my-release-pulsar-broker-0 0/1 Init:0/2 0 15s
-my-release-pulsar-broker-1 0/1 Init:0/2 0 15s
-my-release-pulsar-proxy-0 0/1 Init:0/2 0 16s
-my-release-pulsar-proxy-1 0/1 Init:0/2 0 15s
-my-release-pulsar-pulsar-init-d2t56 0/1 Init:0/2 0 15s
-my-release-pulsar-recovery-0 0/1 Init:0/1 0 16s
-my-release-pulsar-toolset-0 1/1 Running 0 16s
-my-release-pulsar-zookeeper-0 0/1 Pending 0 16s
-```
-
-
-### 3. Enable Milvus components
-
-Milvus Operator first creates all dependencies like etcd, Pulsar, and MinIO, and then continues to create Milvus components. Therefore, you can only see the pods of etcd, Pulsar, and MinIO now. Once all denependencies are enabled, Milvus Operator will start all Milvus components. The status of the Milvus cluster is shown as in the following output.
-
-```
-...
-status:
- conditions:
- - lastTransitionTime: "2021-11-02T05:59:41Z"
- reason: StorageReady
- status: "True"
- type: StorageReady
- - lastTransitionTime: "2021-11-02T06:06:23Z"
- message: Pulsar is ready
- reason: PulsarReady
- status: "True"
- type: PulsarReady
- - lastTransitionTime: "2021-11-02T05:59:41Z"
- message: Etcd endpoints is healthy
- reason: EtcdReady
- status: "True"
- type: EtcdReady
- - lastTransitionTime: "2021-11-02T06:06:24Z"
- message: '[datacoord datanode indexcoord indexnode proxy querycoord querynode
- rootcoord] not ready'
- reason: MilvusComponentNotHealthy
- status: "False"
- type: MilvusReady
- endpoint: my-release-milvus.default:19530
- status: Unhealthy
-```
-
-Check the status of the Milvus pods again.
-
-```
-$ kubectl get pods
-```
-
-```
-NAME READY STATUS RESTARTS AGE
-my-release-etcd-0 1/1 Running 0 6m49s
-my-release-etcd-1 1/1 Running 0 6m49s
-my-release-etcd-2 1/1 Running 0 6m49s
-my-release-milvus-datacoord-6c7bb4b488-k9htl 0/1 ContainerCreating 0 16s
-my-release-milvus-datanode-5c686bd65-wxtmf 0/1 ContainerCreating 0 16s
-my-release-milvus-indexcoord-586b9f4987-vb7m4 0/1 Running 0 16s
-my-release-milvus-indexnode-5b9787b54-xclbx 0/1 ContainerCreating 0 16s
-my-release-milvus-proxy-84f67cdb7f-pg6wf 0/1 ContainerCreating 0 16s
-my-release-milvus-querycoord-865cc56fb4-w2jmn 0/1 Running 0 16s
-my-release-milvus-querynode-5bcb59f6-nhqqw 0/1 ContainerCreating 0 16s
-my-release-milvus-rootcoord-fdcccfc84-9964g 0/1 Running 0 16s
-my-release-minio-0 1/1 Running 0 6m49s
-my-release-minio-1 1/1 Running 0 6m49s
-my-release-minio-2 1/1 Running 0 6m49s
-my-release-minio-3 1/1 Running 0 6m49s
-my-release-pulsar-bookie-0 1/1 Running 0 6m48s
-my-release-pulsar-bookie-1 1/1 Running 0 6m48s
-my-release-pulsar-bookie-init-h6tfz 0/1 Completed 0 6m48s
-my-release-pulsar-broker-0 1/1 Running 0 6m48s
-my-release-pulsar-broker-1 1/1 Running 0 6m48s
-my-release-pulsar-proxy-0 1/1 Running 0 6m49s
-my-release-pulsar-proxy-1 1/1 Running 0 6m48s
-my-release-pulsar-pulsar-init-d2t56 0/1 Completed 0 6m48s
-my-release-pulsar-recovery-0 1/1 Running 0 6m49s
-my-release-pulsar-toolset-0 1/1 Running 0 6m49s
-my-release-pulsar-zookeeper-0 1/1 Running 0 6m49s
-my-release-pulsar-zookeeper-1 1/1 Running 0 6m
-my-release-pulsar-zookeeper-2 1/1 Running 0 6m26s
-```
-
-When all components are enabled, the `status` of the Milvus cluster is shown as `Healthy`.
-
-```
-...
-status:
- conditions:
- - lastTransitionTime: "2021-11-02T05:59:41Z"
- reason: StorageReady
- status: "True"
- type: StorageReady
- - lastTransitionTime: "2021-11-02T06:06:23Z"
- message: Pulsar is ready
- reason: PulsarReady
- status: "True"
- type: PulsarReady
- - lastTransitionTime: "2021-11-02T05:59:41Z"
- message: Etcd endpoints is healthy
- reason: EtcdReady
- status: "True"
- type: EtcdReady
- - lastTransitionTime: "2021-11-02T06:12:36Z"
- message: All Milvus components are healthy
- reason: MilvusClusterHealthy
- status: "True"
- type: MilvusReady
- endpoint: my-release-milvus.default:19530
- status: Healthy
-```
-
-Check the status of the Milvus pods again. You can see all the pods are running now.
-
-```
-$ kubectl get pods
-NAME READY STATUS RESTARTS AGE
-my-release-etcd-0 1/1 Running 0 14m
-my-release-etcd-1 1/1 Running 0 14m
-my-release-etcd-2 1/1 Running 0 14m
-my-release-milvus-datacoord-6c7bb4b488-k9htl 1/1 Running 0 6m
-my-release-milvus-datanode-5c686bd65-wxtmf 1/1 Running 0 6m
-my-release-milvus-indexcoord-586b9f4987-vb7m4 1/1 Running 0 6m
-my-release-milvus-indexnode-5b9787b54-xclbx 1/1 Running 0 6m
-my-release-milvus-proxy-84f67cdb7f-pg6wf 1/1 Running 0 6m
-my-release-milvus-querycoord-865cc56fb4-w2jmn 1/1 Running 0 6m
-my-release-milvus-querynode-5bcb59f6-nhqqw 1/1 Running 0 6m
-my-release-milvus-rootcoord-fdcccfc84-9964g 1/1 Running 0 6m
-my-release-minio-0 1/1 Running 0 14m
-my-release-minio-1 1/1 Running 0 14m
-my-release-minio-2 1/1 Running 0 14m
-my-release-minio-3 1/1 Running 0 14m
-my-release-pulsar-bookie-0 1/1 Running 0 14m
-my-release-pulsar-bookie-1 1/1 Running 0 14m
-my-release-pulsar-bookie-init-h6tfz 0/1 Completed 0 14m
-my-release-pulsar-broker-0 1/1 Running 0 14m
-my-release-pulsar-broker-1 1/1 Running 0 14m
-my-release-pulsar-proxy-0 1/1 Running 0 14m
-my-release-pulsar-proxy-1 1/1 Running 0 14m
-my-release-pulsar-pulsar-init-d2t56 0/1 Completed 0 14m
-my-release-pulsar-recovery-0 1/1 Running 0 14m
-my-release-pulsar-toolset-0 1/1 Running 0 14m
-my-release-pulsar-zookeeper-0 1/1 Running 0 14m
-my-release-pulsar-zookeeper-1 1/1 Running 0 13m
-my-release-pulsar-zookeeper-2 1/1 Running 0 13m
-```
-
-When the Milvus cluster is installed, you can learn how to [Connect to Milvus server](manage_connection.md).
-
-## Uninstall the Milvus cluster
-
-Run the following command to uninstall the Milvus cluster.
-
-```
-$ kubectl delete milvus my-release
-```
-
-
-
-
-## Uninstall Milvus Operator
-
-There are also two ways to uninstall Milvus Operator on K8s:
-
-### Uninstall Milvus Operator by Helm command
-
-```
-$ helm -n milvus-operator uninstall milvus-operator
-```
-
-### Uninstall Milvus Operator by `kubectl` command
-
-```
-$ kubectl delete -f https://raw.githubusercontent.com/milvus-io/milvus-operator/v0.7.7/deploy/manifests/deployment.yaml
-```
-
-## Delete the K8s cluster
-
-When you no longer need the K8s cluster in the test environment, you can run `$ minikube delete` to delete it.
-
-## What's next
-
-Having installed Milvus, you can:
-- Check [Hello Milvus](example_code.md) to run an example code with different SDKs to see what Milvus can do.
-- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
-- [Upgrade Milvus Using Helm Chart](upgrade.md)
-- [Scale your Milvus cluster](scaleout.md)
-- Deploy your Milvu cluster on clouds:
- - [Amazon EC2](aws.md)
- - [Amazon EKS](eks.md)
-- Explore [MilvusDM](migrate_overview.md), an open-source tool designed for importing and exporting data in Milvus.
-- [Monitor Milvus with Prometheus](monitor.md)
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/preview/site/en/getstarted/example_code.md b/preview/site/en/getstarted/example_code.md
deleted file mode 100644
index 9322e80a0..000000000
--- a/preview/site/en/getstarted/example_code.md
+++ /dev/null
@@ -1,197 +0,0 @@
----
-id: example_code.md
-related_key: pymilvus
-label: Python
-order: 0
-group: example_code.md
-summary: Get started with Milvus faster using this Python example code.
----
-
-
-
-# Run Milvus using Python
-
-This topic describes how to run Milvus using Python.
-
-Through running the example code we provided, you will have a primary understanding of what Milvus is capable of.
-
-## Preparations
-
-- [Milvus 2.2.3](install_standalone-docker.md)
-- Python 3 (3.71 or later)
-- [PyMilvus 2.2.2](install-pymilvus.md)
-
-
-## Download example code
-
-[Download](https://raw.githubusercontent.com/milvus-io/pymilvus/v2.2.2/examples/hello_milvus.py) `hello_milvus.py` directly or with the following command.
-
-```bash
-$ wget https://raw.githubusercontent.com/milvus-io/pymilvus/v2.2.2/examples/hello_milvus.py
-```
-
-
-## Scan the example code
-
-The example code performs the following steps.
-
-- Imports a PyMilvus package:
-```Python
-from pymilvus import (
- connections,
- utility,
- FieldSchema,
- CollectionSchema,
- DataType,
- Collection,
-)
-```
-
-- Connects to a server:
-```Python
-connections.connect("default", host="localhost", port="19530")
-```
-
-- Creates a collection:
-```Python
-fields = [
- FieldSchema(name="pk", dtype=DataType.INT64, is_primary=True, auto_id=False),
- FieldSchema(name="random", dtype=DataType.DOUBLE),
- FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=8)
-]
-schema = CollectionSchema(fields, "hello_milvus is the simplest demo to introduce the APIs")
-hello_milvus = Collection("hello_milvus", schema)
-```
-
-- Inserts vectors in the collection:
-```Python
-import random
-entities = [
- [i for i in range(3000)], # field pk
- [float(random.randrange(-20, -10)) for _ in range(3000)], # field random
- [[random.random() for _ in range(8)] for _ in range(3000)], # field embeddings
-]
-insert_result = hello_milvus.insert(entities)
-# After final entity is inserted, it is best to call flush to have no growing segments left in memory
-hello_milvus.flush()
-```
-
-- Builds indexes on the entities:
-```Python
-index = {
- "index_type": "IVF_FLAT",
- "metric_type": "L2",
- "params": {"nlist": 128},
-}
-hello_milvus.create_index("embeddings", index)
-```
-
-- Loads the collection to memory and performs a vector similarity search:
-```Python
-hello_milvus.load()
-vectors_to_search = entities[-1][-2:]
-search_params = {
- "metric_type": "L2",
- "params": {"nprobe": 10},
-}
-result = hello_milvus.search(vectors_to_search, "embeddings", search_params, limit=3, output_fields=["random"])
-```
-
-
-- Performs a vector query:
-
-```Python
-result = hello_milvus.query(expr="random > -14", output_fields=["random", "embeddings"])
-```
-
-- Performs a hybrid search:
-
-```Python
-result = hello_milvus.search(vectors_to_search, "embeddings", search_params, limit=3, expr="random > -12", output_fields=["random"])
-```
-
-- Deletes entities by their primary keys:
-
-```Python
-expr = f"pk in [{ids[0]}, {ids[1]}]"
-hello_milvus.delete(expr)
-```
-
-- Drops the collection:
-
-```Python
-utility.drop_collection("hello_milvus")
-```
-
-## Run the example code
-
-Execute the following command to run the example code.
-
-```Python
-$ python3 hello_milvus.py
-```
-
-*The returned results and query latency are shown as follows:*
-
-```
-=== start connecting to Milvus ===
-
-Does collection hello_milvus exist in Milvus: False
-
-=== Create collection `hello_milvus` ===
-
-
-=== Start inserting entities ===
-
-Number of entities in Milvus: 3000
-
-=== Start Creating index IVF_FLAT ===
-
-
-=== Start loading ===
-
-
-=== Start searching based on vector similarity ===
-
-hit: (distance: 0.0, id: 2998), random field: -11.0
-hit: (distance: 0.11455299705266953, id: 1581), random field: -18.0
-hit: (distance: 0.1232629269361496, id: 2647), random field: -13.0
-hit: (distance: 0.0, id: 2999), random field: -11.0
-hit: (distance: 0.10560893267393112, id: 2430), random field: -18.0
-hit: (distance: 0.13938161730766296, id: 377), random field: -14.0
-search latency = 0.2796s
-
-=== Start querying with `random > -14` ===
-
-query result:
--{'pk': 9, 'random': -13.0, 'embeddings': [0.298433, 0.931987, 0.949756, 0.598713, 0.290125, 0.094323, 0.064444, 0.306993]}
-search latency = 0.2970s
-
-=== Start hybrid searching with `random > -12` ===
-
-hit: (distance: 0.0, id: 2998), random field: -11.0
-hit: (distance: 0.15773043036460876, id: 472), random field: -11.0
-hit: (distance: 0.3273330628871918, id: 2146), random field: -11.0
-hit: (distance: 0.0, id: 2999), random field: -11.0
-hit: (distance: 0.15844076871871948, id: 2218), random field: -11.0
-hit: (distance: 0.1622171700000763, id: 1403), random field: -11.0
-search latency = 0.3028s
-
-=== Start deleting with expr `pk in [0, 1]` ===
-
-query before delete by expr=`pk in [0, 1]` -> result:
--{'pk': 0, 'random': -18.0, 'embeddings': [0.142279, 0.414248, 0.378628, 0.971863, 0.535941, 0.107011, 0.207052, 0.98182]}
--{'pk': 1, 'random': -15.0, 'embeddings': [0.57512, 0.358512, 0.439131, 0.862369, 0.083284, 0.294493, 0.004961, 0.180082]}
-
-query after delete by expr=`pk in [0, 1]` -> result: []
-
-
-=== Drop collection `hello_milvus` ===
-```
-
-
-- - -*Congratulations! You have started Milvus standalone and performed your first vector similarity search.* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_NL.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_NL.md deleted file mode 100644 index 701e13363..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_NL.md +++ /dev/null @@ -1,159 +0,0 @@ ---- -id: example_code_NL.md ---- - -# Voer Milvus uit met Python - -Dit onderwerp beschrijft hoe je Milvus kan uitvoeren met Python. - -## 1. Installeer PyMilvus - -```Python -pip3 install pymilvus==2.2.2 -``` -
-Python 3.6 of later is vereist. Zie Python downloaden voor meer informatie.
-
-
-## 2. Download voorbeeld code
-
-```bash
-$ wget https://raw.githubusercontent.com/milvus-io/pymilvus/v2.2.2/examples/hello_milvus.py
-```
-
-## 3. Scan het voorbeeld
-De voorbeeld code voert de volgende stappen uit.
-
-- Importeer een PyMilvus package:
-```Python
-from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection
-```
-
-- Verbind met een server:
-```Python
-connections.connect(host='localhost', port='19530')
-```
-
-- Maakt een collectie aan:
-```Python
-dim = 128
-default_fields = [
- FieldSchema(name="count", dtype=DataType.INT64, is_primary=True),
- FieldSchema(name="random_value", dtype=DataType.DOUBLE),
- FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
-]
-default_schema = CollectionSchema(fields=default_fields, description="test collection")
-
-print(f"\nCreate collection...")
-collection = Collection(name="hello_milvus", schema=default_schema)
-```
-
-- voert vectors in in collectie:
-```Python
-import random
-nb = 3000
-vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
-collection.insert(
- [
- [i for i in range(nb)],
- [float(random.randrange(-20,-10)) for _ in range(nb)],
- vectors
- ]
-)
-```
-
-- Bouwt indexes en laadt de collectie:
-```Python
-default_index = {"index_type": "IVF_FLAT", "params": {"nlist": 128}, "metric_type": "L2"}
-collection.create_index(field_name="float_vector", index_params=default_index)
-collection.load()
-```
-
-- Voert vector vergelijkbaarheids zoekopdracht uit:
-```Python
-topK = 5
-search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
-# define output_fields of search result
-res = collection.search(
- vectors[-2:], "float_vector", search_params, topK,
- "count > 100", output_fields=["count", "random_value"]
-)
-```
-Om zoekresultaten weer te printen op basis van ID en afstand, voer de volgende opdracht uit.
-```Python
-for raw_result in res:
- for result in raw_result:
- id = result.id # result id
- distance = result.distance
- print(id, distance)
-```
-Zie [API Reference](/api-reference/pymilvus/v2.2.2/results.html) voor meer informatie.
-
-- Voert hybride zoekopdracht uit：
-
- Het volgende voorbeeld voert een geschatte zoekopdracht uit op entiteiten met
-
-```Python
-from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
-import random
-connections.connect()
-schema = CollectionSchema([
- FieldSchema("film_id", DataType.INT64, is_primary=True),
- FieldSchema("films", dtype=DataType.FLOAT_VECTOR, dim=2)
-])
-collection = Collection("test_collection_search", schema)
-# insert
-data = [
- [i for i in range(10)],
- [[random.random() for _ in range(2)] for _ in range(10)],
-]
-collection.insert(data)
-collection.num_entities
-10
-collection.load()
-# search
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "films",
- "param": {"metric_type": "L2"},
- "limit": 2,
- "expr": "film_id in [2,4,6,8]",
-}
-res = collection.search(**search_param)
-assert len(res) == 1
-hits = res[0]
-assert len(hits) == 2
-print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
-- Total hits: 2, hits ids: [2, 4]
-print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")
-- Top1 hit id: 2, distance: 0.10143111646175385, score: 0.101431116461
-
-```
-
-## 4. Voer voorbeeld uit
-```Python
-$ python3 hello_milvus.py
-```
-
-*De teruggegeven resultaten en query latentie zijn als volgt laten zien:*
-
-film_id range binnen [2,4,6,8].
-
-
-
-
-Search...
-(distance: 0.0, id: 2998) -20.0
-(distance: 13.2614107131958, id: 989) -11.0
-(distance: 14.489648818969727, id: 1763) -19.0
-(distance: 15.295698165893555, id: 968) -20.0
-(distance: 15.34445571899414, id: 2049) -19.0
-(distance: 0.0, id: 2999) -12.0
-(distance: 14.63361930847168, id: 1259) -13.0
-(distance: 15.421361923217773, id: 2530) -15.0
-(distance: 15.427900314331055, id: 600) -14.0
-(distance: 15.538337707519531, id: 637) -19.0
-search latency = 0.0549s
-- -*Gefeliciteerd! Je heb Milvus standalone opgestart en je eerste vector vergelijkbaarheids zoekopdracht uitgevoerd.* \ No newline at end of file diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_bn.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_bn.md deleted file mode 100644 index 37d5eab0f..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_bn.md +++ /dev/null @@ -1,163 +0,0 @@ ---- -id: example_code_bn.md ---- - -# পাইথন ব্যবহার করে মিলভাস চালান - -এই টপিকটি বর্ণনা করে কিভাবে পাইথন ব্যবহার করে মিলভাস চালানো যায়। - -## 1. PyMilvus ইনস্টল করুন - -```Python -pip3 install pymilvus==2.2.2 -``` -
-পাইথন 3.6 বা তার পরের ভার্সন প্রয়োজন। দেখুন পাইথন ডাউনলোড করা হচ্ছে
- আরও তথ্যের জন্য.
-
-
-## 2. নমুনা কোড ডাউনলোড করুন
-
-```bash
-$ wget https://raw.githubusercontent.com/milvus-io/pymilvus/v2.2.2/examples/hello_milvus.py
-```
-
-## 3. নমুনাটি স্ক্যান করুন
-নমুনা কোডটি নিম্নলিখিত ধাপগুলি সম্পাদন করে।
-
-- একটি PyMilvus প্যাকেজ আমদানি করে:
-```Python
-from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection
-```
-
-- একটি সার্ভারের সাথে সংযোগ স্থাপন করে:
-```Python
-connections.connect(host='localhost', port='19530')
-```
-
-- একটি কালেকশন তৈরি করে:
-```Python
-dim = 128
-default_fields = [
- FieldSchema(name="count", dtype=DataType.INT64, is_primary=True),
- FieldSchema(name="random_value", dtype=DataType.DOUBLE),
- FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
-]
-default_schema = CollectionSchema(fields=default_fields, description="পরীক্ষামূলক সংগ্রহ")
-
-print(f"\nCreate collection...")
-collection = Collection(name="hello_milvus", schema=default_schema)
-```
-
-- কাল্কেশনে ভেক্টর প্রবেশ করে:
-```Python
-import random
-nb = 3000
-vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
-collection.insert(
- [
- [i for i in range(nb)],
- [float(random.randrange(-20,-10)) for _ in range(nb)],
- vectors
- ]
-)
-```
-
-- সূচী তৈরি করে এবং কালেকশনটি লোড করে:
-```Python
-default_index = {"index_type": "IVF_FLAT", "params": {"nlist": 128}, "metric_type": "L2"}
-collection.create_index(field_name="float_vector", index_params=default_index)
-collection.load()
-```
-
-- ভেক্টর সাদৃশ্য অনুসন্ধান করে:
-```Python
-topK = 5
-search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
-# define output_fields of search result
-res = collection.search(
- vectors[-2:], "float_vector", search_params, topK,
- "count > 100", output_fields=["count", "random_value"]
-)
-```
-
-আইডি এবং দূরত্ব অনুসারে অনুসন্ধান ফলাফলগুলি মুদ্রণ করতে, নিম্নলিখিত কমান্ডটি চালান।
-```Python
-for raw_result in res:
- for result in raw_result:
- id = result.id # result id
- distance = result.distance
- print(id, distance)
-```
-আরও তথ্য পেতে [API Reference](/api-reference/pymilvus/v2.2.2/results.html) দেখুন।
-
-- একটি হাইব্রিড অনুসন্ধান করে
-
- নিম্নলিখিত উদাহরণটি
-
-```Python
-from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
-import random
-connections.connect()
-schema = CollectionSchema([
- FieldSchema("film_id", DataType.INT64, is_primary=True),
- FieldSchema("films", dtype=DataType.FLOAT_VECTOR, dim=2)
-])
-collection = Collection("test_collection_search", schema)
-# insert
-data = [
- [i for i in range(10)],
- [[random.random() for _ in range(2)] for _ in range(10)],
-]
-collection.insert(data)
-collection.num_entities
-10
-collection.load()
-# search
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "films",
- "param": {"metric_type": "L2"},
- "limit": 2,
- "expr": "film_id in [2,4,6,8]",
-}
-res = collection.search(**search_param)
-assert len(res) == 1
-hits = res[0]
-assert len(hits) == 2
-print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
-- Total hits: 2, hits ids: [2, 4]
-print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")
-- Top1 hit id: 2, distance: 0.10143111646175385, score: 0.101431116461
-
-```
-
-## 4. নমুনাটি রান করুন
-```Python
-$ python3 hello_milvus.py
-```
-
-*ফিরে আসা ফলাফল এবং ক্যোয়ারী এর বিলম্ব নিম্নে দেয়া হলো:*
-
-film_id এর সাহায্যে [২,৪,৬,৮] এর পরিসরে এন্টিটিগুলোর উপর আনুমানিক অনুসন্ধান করে
-
-
-
-
-Search...
-(distance: 0.0, id: 2998) -20.0
-(distance: 13.2614107131958, id: 989) -11.0
-(distance: 14.489648818969727, id: 1763) -19.0
-(distance: 15.295698165893555, id: 968) -20.0
-(distance: 15.34445571899414, id: 2049) -19.0
-(distance: 0.0, id: 2999) -12.0
-(distance: 14.63361930847168, id: 1259) -13.0
-(distance: 15.421361923217773, id: 2530) -15.0
-(distance: 15.427900314331055, id: 600) -14.0
-(distance: 15.538337707519531, id: 637) -19.0
-search latency = 0.0549s
-- - -*অভিনন্দন! আপনি মিলভাস স্ট্যান্ড-অ্যালোনভাবে শুরু করেছেন এবং আপনার প্রথম ভেক্টর সাদৃশ্য অনুসন্ধান করেছেন।* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_de.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_de.md deleted file mode 100644 index 56edcf93d..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_de.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -id: example_code_de.md ---- - -# Milvus mit Python ausführen - -Dieser Abschnitt beschreibt, wie man Milvus mit Python ausführt. - -## 1. PyMilvus installieren - -```Python -pip3 install pymilvus==2.2.2 -``` -
-Python 3.6 oder höher ist erforderlich. Siehe Downloading Python für weitere Informationen.
-
-
-## 2. Laden Sie den Beispielcode herunter
-
-```bash
-$ wget https://raw.githubusercontent.com/milvus-io/pymilvus/v2.2.2/examples/hello_milvus.py
-```
-
-## 3. Scannen Sie die Probe
-Der Beispielcode führt die folgenden Schritte aus.
-
-- Importieren Sie das PyMilvus-Paket:
-```Python
-from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection
-```
-
-- Stellt eine Verbindung zu einem Server her:
-```Python
-connections.connect(host='localhost', port='19530')
-```
-
-- Erstellen Sie eine Sammlung:
-```Python
-dim = 128
-default_fields = [
- FieldSchema(name="count", dtype=DataType.INT64, is_primary=True),
- FieldSchema(name="random_value", dtype=DataType.DOUBLE),
- FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
-]
-default_schema = CollectionSchema(fields=default_fields, description="test collection")
-
-print(f"\nCreate collection...")
-collection = Collection(name="hello_milvus", schema=default_schema)
-```
-
-- Fügt Vektoren in die Sammlung ein:
-```Python
-import random
-nb = 3000
-vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
-collection.insert(
- [
- [i for i in range(nb)],
- [float(random.randrange(-20,-10)) for _ in range(nb)],
- vectors
- ]
-)
-```
-
-- Es erstellt Indizes und lädt die Sammlung:
-```Python
-default_index = {"index_type": "IVF_FLAT", "params": {"nlist": 128}, "metric_type": "L2"}
-collection.create_index(field_name="float_vector", index_params=default_index)
-collection.load()
-```
-
-- Führt eine Vektorähnlichkeitssuche durch:
-```Python
-topK = 5
-search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
-# define output_fields of search result
-res = collection.search(
- vectors[-2:], "float_vector", search_params, topK,
- "count > 100", output_fields=["count", "random_value"]
-)
-```
-Um die Ergebnisse der Suche nach ID und Entfernung auszudrucken, führen Sie den folgenden Befehl aus.
-```Python
-for raw_result in res:
- for result in raw_result:
- id = result.id # result id
- distance = result.distance
- print(id, distance)
-```
-Siehe [API Reference](/api-reference/pymilvus/v2.2.2/results.html) für weitere Informationen.
-
-- Führt eine hybride Suche durch：
-
- Das folgende Beispiel führt eine ungefähre Suche in Entitäten mit
-
-```Python
-from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
-import random
-connections.connect()
-schema = CollectionSchema([
- FieldSchema("film_id", DataType.INT64, is_primary=True),
- FieldSchema("films", dtype=DataType.FLOAT_VECTOR, dim=2)
-])
-collection = Collection("test_collection_search", schema)
-# insert
-data = [
- [i for i in range(10)],
- [[random.random() for _ in range(2)] for _ in range(10)],
-]
-collection.insert(data)
-collection.num_entities
-10
-collection.load()
-# search
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "films",
- "param": {"metric_type": "L2"},
- "limit": 2,
- "expr": "film_id in [2,4,6,8]",
-}
-res = collection.search(**search_param)
-assert len(res) == 1
-hits = res[0]
-assert len(hits) == 2
-print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
-- Total hits: 2, hits ids: [2, 4]
-print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")
-- Top1 hit id: 2, distance: 0.10143111646175385, score: 0.101431116461
-
-```
-
-## 4. Führen Sie die Probe aus
-```Python
-$ python3 hello_milvus.py
-```
-
-*Die zurückgegebenen Ergebnisse und die Latenzzeit der Abfrage werden wie folgt angezeigt:*
-
-film_id gleich [2,4,6,8] durch.
-
-
-
-
-Search...
-(distance: 0.0, id: 2998) -20.0
-(distance: 13.2614107131958, id: 989) -11.0
-(distance: 14.489648818969727, id: 1763) -19.0
-(distance: 15.295698165893555, id: 968) -20.0
-(distance: 15.34445571899414, id: 2049) -19.0
-(distance: 0.0, id: 2999) -12.0
-(distance: 14.63361930847168, id: 1259) -13.0
-(distance: 15.421361923217773, id: 2530) -15.0
-(distance: 15.427900314331055, id: 600) -14.0
-(distance: 15.538337707519531, id: 637) -19.0
-search latency = 0.0549s
-- - -*Herzlichen Glückwunsch! Sie haben es geschafft, Milvus unabhängig zu starten und Ihre erste Vektorähnlichkeitssuche durchzuführen.* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_es.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_es.md deleted file mode 100644 index 8293e119c..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_es.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -id: example_code_es.md ---- - -# Ejecuta Milvus usando Python - -Este tema describe cómo ejecutar Milvus usando Python. - -## 1. Instala PyMilvus - -```Python -pip3 install pymilvus==2.2.2 -``` -
-Python 3.6 o más reciente es requerido. Lee Descargando Python para más información.
-
-
-## 2. Descarga el código muestra
-
-```bash
-$ wget https://raw.githubusercontent.com/milvus-io/pymilvus/v2.2.2/examples/hello_milvus.py
-```
-
-## 3. Escanea la muestra
-El código de muestra funciona siguiendo estos pasos.
-
-- Importa un paquete PyMilvus:
-```Python
-from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection
-```
-
-- Se conecta a un servidor:
-```Python
-connections.connect(host='localhost', port='19530')
-```
-
-- Crea una colección:
-```Python
-dim = 128
-default_fields = [
- FieldSchema(name="count", dtype=DataType.INT64, is_primary=True),
- FieldSchema(name="random_value", dtype=DataType.DOUBLE),
- FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
-]
-default_schema = CollectionSchema(fields=default_fields, description="test collection")
-
-print(f"\nCreate collection...")
-collection = Collection(name="hello_milvus", schema=default_schema)
-```
-
-- Inserta vectores en la colección:
-```Python
-import random
-nb = 3000
-vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
-collection.insert(
- [
- [i for i in range(nb)],
- [float(random.randrange(-20,-10)) for _ in range(nb)],
- vectors
- ]
-)
-```
-
-- Construye índices y carga la colección:
-```Python
-default_index = {"index_type": "IVF_FLAT", "params": {"nlist": 128}, "metric_type": "L2"}
-collection.create_index(field_name="float_vector", index_params=default_index)
-collection.load()
-```
-
-- Ejecuta una búsuqeda de similitud de vector:
-```Python
-topK = 5
-search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
-# define output_fields of search result
-res = collection.search(
- vectors[-2:], "float_vector", search_params, topK,
- "count > 100", output_fields=["count", "random_value"]
-)
-```
-Para mostrar los resultados de la búsqueda por ID y distancia, ejecuta el siguiente comando.
-```Python
-for raw_result in res:
- for result in raw_result:
- id = result.id # result id
- distance = result.distance
- print(id, distance)
-```
-Revisa la [documentación de API](/api-reference/pymilvus/v2.2.2/results.html) para más información.
-
-- Ejecuta una búsqueda híbrida：
-
- El siguiente ejemplo ejecuta una búsqueda aproximada en entidades con
-
-```Python
-from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
-import random
-connections.connect()
-schema = CollectionSchema([
- FieldSchema("film_id", DataType.INT64, is_primary=True),
- FieldSchema("films", dtype=DataType.FLOAT_VECTOR, dim=2)
-])
-collection = Collection("test_collection_search", schema)
-# insert
-data = [
- [i for i in range(10)],
- [[random.random() for _ in range(2)] for _ in range(10)],
-]
-collection.insert(data)
-collection.num_entities
-10
-collection.load()
-# search
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "films",
- "param": {"metric_type": "L2"},
- "limit": 2,
- "expr": "film_id in [2,4,6,8]",
-}
-res = collection.search(**search_param)
-assert len(res) == 1
-hits = res[0]
-assert len(hits) == 2
-print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
-- Total hits: 2, hits ids: [2, 4]
-print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")
-- Top1 hit id: 2, distance: 0.10143111646175385, score: 0.101431116461
-
-```
-
-## 4. Ejecuta la muestra
-```Python
-$ python3 hello_milvus.py
-```
-
-*Los resultados y latencia de la petición se muestran de la siguiente forma:*
-
-film_id dentro del rango [2,4,6,8].
-
-
-
-
-Search...
-(distance: 0.0, id: 2998) -20.0
-(distance: 13.2614107131958, id: 989) -11.0
-(distance: 14.489648818969727, id: 1763) -19.0
-(distance: 15.295698165893555, id: 968) -20.0
-(distance: 15.34445571899414, id: 2049) -19.0
-(distance: 0.0, id: 2999) -12.0
-(distance: 14.63361930847168, id: 1259) -13.0
-(distance: 15.421361923217773, id: 2530) -15.0
-(distance: 15.427900314331055, id: 600) -14.0
-(distance: 15.538337707519531, id: 637) -19.0
-search latency = 0.0549s
-- - -*¡Felicidades! Has iniciado Milvus standalone y ejecutado tu primera búsqueda por similitud de vector.* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_fr.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_fr.md deleted file mode 100644 index a6913dc3e..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_fr.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -id: example_code_fr.md ---- - -# Exécuter Milvus avec Python - -Ce topic décrit comment exécuter Milvus avec Python. - -## 1. Installer PyMilvus - -```Python -pip3 install pymilvus==2.2.2 -``` -
-Il est nécessaire d'utiliser la version 3.6 ou une version plus récente de Python. Plus d'informations sont disponibles dans le guide d'installation de Python.
-
-
-## 2. Télécharger un exemple code source
-
-```bash
-$ wget https://raw.githubusercontent.com/milvus-io/pymilvus/v2.2.2/examples/hello_milvus.py
-```
-
-## 3. Scanner le code source
-L'exemple fourni exécute les étapes suivantes.
-
-- Import du package PyMilvus :
-```Python
-from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection
-```
-
-- Connexion à un serveur :
-```Python
-connections.connect(host='localhost', port='19530')
-```
-
-- Création d'une collection :
-```Python
-dim = 128
-default_fields = [
- FieldSchema(name="count", dtype=DataType.INT64, is_primary=True),
- FieldSchema(name="random_value", dtype=DataType.DOUBLE),
- FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
-]
-default_schema = CollectionSchema(fields=default_fields, description="test collection")
-
-print(f"\nCreate collection...")
-collection = Collection(name="hello_milvus", schema=default_schema)
-```
-
-- Insertion des vecteurs dans la collection :
-```Python
-import random
-nb = 3000
-vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
-collection.insert(
- [
- [i for i in range(nb)],
- [float(random.randrange(-20,-10)) for _ in range(nb)],
- vectors
- ]
-)
-```
-
-- Construction des index et chargement de la collection en mémoire :
-```Python
-default_index = {"index_type": "IVF_FLAT", "params": {"nlist": 128}, "metric_type": "L2"}
-collection.create_index(field_name="float_vector", index_params=default_index)
-collection.load()
-```
-
-- Exécute une recherche de similitude entre vecteurs :
-```Python
-topK = 5
-search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
-# define output_fields of search result
-res = collection.search(
- vectors[-2:], "float_vector", search_params, topK,
- "count > 100", output_fields=["count", "random_value"]
-)
-```
-Pour afficher les résultats classés par ID et distance, exécutez la commande suivante.
-```Python
-for raw_result in res:
- for result in raw_result:
- id = result.id # result id
- distance = result.distance
- print(id, distance)
-```
-Se référer à [la documentation de l'API](/api-reference/pymilvus/v2.2.2/results.html) pour plus d'informations.
-
-- Exécute une recherche hybride :
-
- L'exemple suivant fait une recherche approximative sur entités où la variable
-
-```Python
-from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
-import random
-connections.connect()
-schema = CollectionSchema([
- FieldSchema("film_id", DataType.INT64, is_primary=True),
- FieldSchema("films", dtype=DataType.FLOAT_VECTOR, dim=2)
-])
-collection = Collection("test_collection_search", schema)
-# insert
-data = [
- [i for i in range(10)],
- [[random.random() for _ in range(2)] for _ in range(10)],
-]
-collection.insert(data)
-collection.num_entities
-10
-collection.load()
-# search
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "films",
- "param": {"metric_type": "L2"},
- "limit": 2,
- "expr": "film_id in [2,4,6,8]",
-}
-res = collection.search(**search_param)
-assert len(res) == 1
-hits = res[0]
-assert len(hits) == 2
-print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
-- Total hits: 2, hits ids: [2, 4]
-print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")
-- Top1 hit id: 2, distance: 0.10143111646175385, score: 0.101431116461
-
-```
-
-## 4. Exécuter le code source
-```Python
-$ python3 hello_milvus.py
-```
-
-*Les résultats ainsi que le temps de latence des requêtes sont affichés de cette manière :*
-
-film_id prend les valeurs [2,4,6,8].
-
-
-
-
-Search...
-(distance: 0.0, id: 2998) -20.0
-(distance: 13.2614107131958, id: 989) -11.0
-(distance: 14.489648818969727, id: 1763) -19.0
-(distance: 15.295698165893555, id: 968) -20.0
-(distance: 15.34445571899414, id: 2049) -19.0
-(distance: 0.0, id: 2999) -12.0
-(distance: 14.63361930847168, id: 1259) -13.0
-(distance: 15.421361923217773, id: 2530) -15.0
-(distance: 15.427900314331055, id: 600) -14.0
-(distance: 15.538337707519531, id: 637) -19.0
-search latency = 0.0549s
-- - -*Félicitations ! Vous avez démarré Milvus standalone et exécuté votre première recherche de similitude entre vecteurs.* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_id.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_id.md deleted file mode 100644 index 2e9e7dbee..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_id.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -id: example_code_id.md ---- - -# Menjalankan Milvus menggunkan Python - -Topik ini menjelaskan bagaimana cara menjalankan Milvus menggunkan Python. - -## 1. Instal PyMilvus - -```Python -pip3 install pymilvus==2.2.2 -``` -
-Python 3.6 atau yang lebih baru dibutuhkan. Lihat Mengunduh Python untuk informasi lebih lanjut.
-
-
-## 2. Unduh kode sampel
-
-```bash
-$ wget https://raw.githubusercontent.com/milvus-io/pymilvus/v2.2.2/examples/hello_milvus.py
-```
-
-## 3. Pindai sampel
-Langkah-langkah yang dilakukan untuk membuat kode sampel.
-
-- Impor paket PyMilvus :
-```Python
-from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection
-```
-
-- Koneksikan ke server:
-```Python
-connections.connect(host='localhost', port='19530')
-```
-
-- Buat koleksi:
-```Python
-dim = 128
-default_fields = [
- FieldSchema(name="count", dtype=DataType.INT64, is_primary=True),
- FieldSchema(name="random_value", dtype=DataType.DOUBLE),
- FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
-]
-default_schema = CollectionSchema(fields=default_fields, description="test collection")
-
-print(f"\nCreate collection...")
-collection = Collection(name="hello_milvus", schema=default_schema)
-```
-
-- Tambahkan vector ke dalam koleksi:
-```Python
-import random
-nb = 3000
-vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
-collection.insert(
- [
- [i for i in range(nb)],
- [float(random.randrange(-20,-10)) for _ in range(nb)],
- vectors
- ]
-)
-```
-
-- Bangun indeks dan muat koleksi:
-```Python
-default_index = {"index_type": "IVF_FLAT", "params": {"nlist": 128}, "metric_type": "L2"}
-collection.create_index(field_name="float_vector", index_params=default_index)
-collection.load()
-```
-
-- Lakukan pencarian persamaan vektor::
-```Python
-topK = 5
-search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
-# define output_fields of search result
-res = collection.search(
- vectors[-2:], "float_vector", search_params, topK,
- "count > 100", output_fields=["count", "random_value"]
-)
-```
-Untuk mencetak hasil pencarian berdasarkan ID dan jarak, jalankan perintah berikut.
-```Python
-for raw_result in res:
- for result in raw_result:
- id = result.id # result id
- distance = result.distance
- print(id, distance)
-```
-Baca [Referensi API](/api-reference/pymilvus/v2.2.2/results.html) untuk informasi lebih lanjut.
-
-- Lakukan pencarian hibrid：
-
- Contoh berikut melakukan pencarian perkiraan pada entitas dengan
-
-```Python
-from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
-import random
-connections.connect()
-schema = CollectionSchema([
- FieldSchema("film_id", DataType.INT64, is_primary=True),
- FieldSchema("films", dtype=DataType.FLOAT_VECTOR, dim=2)
-])
-collection = Collection("test_collection_search", schema)
-# insert
-data = [
- [i for i in range(10)],
- [[random.random() for _ in range(2)] for _ in range(10)],
-]
-collection.insert(data)
-collection.num_entities
-10
-collection.load()
-# search
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "films",
- "param": {"metric_type": "L2"},
- "limit": 2,
- "expr": "film_id in [2,4,6,8]",
-}
-res = collection.search(**search_param)
-assert len(res) == 1
-hits = res[0]
-assert len(hits) == 2
-print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
-- Total hits: 2, hits ids: [2, 4]
-print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")
-- Top1 hit id: 2, distance: 0.10143111646175385, score: 0.101431116461
-
-```
-
-## 4. Jalankan sampel
-```Python
-$ python3 hello_milvus.py
-```
-
-*Hasil yang dikembalikan dan latensi kueri ditampilkan sebagai berikut:*
-
-film_id berkisar [2,4,6,8].
-
-
-
-
-Search...
-(distance: 0.0, id: 2998) -20.0
-(distance: 13.2614107131958, id: 989) -11.0
-(distance: 14.489648818969727, id: 1763) -19.0
-(distance: 15.295698165893555, id: 968) -20.0
-(distance: 15.34445571899414, id: 2049) -19.0
-(distance: 0.0, id: 2999) -12.0
-(distance: 14.63361930847168, id: 1259) -13.0
-(distance: 15.421361923217773, id: 2530) -15.0
-(distance: 15.427900314331055, id: 600) -14.0
-(distance: 15.538337707519531, id: 637) -19.0
-search latency = 0.0549s
-- - -*Selamat! Kamu baru saja memulai Milvus mandiri dan melakukan pencarian persamaan vector pertama.* - diff --git a/preview/site/en/getstarted/example_code_multi_lang/example_code_it.md b/preview/site/en/getstarted/example_code_multi_lang/example_code_it.md deleted file mode 100644 index 2a75060a5..000000000 --- a/preview/site/en/getstarted/example_code_multi_lang/example_code_it.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -id: example_code_it.md ---- - -# Esegui Milvus usando Python - -Questa sezione descrive come eseguire Milvus usando Python. - -## 1. Installa PyMilvus - -```Python -pip3 install pymilvus==2.2.2 -``` -
-E' richiesto Python 3.6 o versioni successive. Vedi Downloading Python per maggiori informazioni.
-
-
-## 2. Scarica il codice di esempio
-
-```bash
-$ wget https://raw.githubusercontent.com/milvus-io/pymilvus/v2.2.2/examples/hello_milvus.py
-```
-
-## 3. Scannerizza il campione
-Il codice di esempio esegue i seguenti passaggi.
-
-- Importa il pacchetto PyMilvus:
-```Python
-from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection
-```
-
-- Si connette ad un server:
-```Python
-connections.connect(host='localhost', port='19530')
-```
-
-- Crea una collezione:
-```Python
-dim = 128
-default_fields = [
- FieldSchema(name="count", dtype=DataType.INT64, is_primary=True),
- FieldSchema(name="random_value", dtype=DataType.DOUBLE),
- FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
-]
-default_schema = CollectionSchema(fields=default_fields, description="test collection")
-
-print(f"\nCreate collection...")
-collection = Collection(name="hello_milvus", schema=default_schema)
-```
-
-- Inserisce dei vettori nella collezione:
-```Python
-import random
-nb = 3000
-vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
-collection.insert(
- [
- [i for i in range(nb)],
- [float(random.randrange(-20,-10)) for _ in range(nb)],
- vectors
- ]
-)
-```
-
-- Crea degli indici e carica la collezione:
-```Python
-default_index = {"index_type": "IVF_FLAT", "params": {"nlist": 128}, "metric_type": "L2"}
-collection.create_index(field_name="float_vector", index_params=default_index)
-collection.load()
-```
-
-- Esegue una ricerca di somiglianza vettoriale:
-```Python
-topK = 5
-search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
-# define output_fields of search result
-res = collection.search(
- vectors[-2:], "float_vector", search_params, topK,
- "count > 100", output_fields=["count", "random_value"]
-)
-```
-Per stampare i risultati della ricerca per ID e per distanza, esegui il seguente comando.
-```Python
-for raw_result in res:
- for result in raw_result:
- id = result.id # result id
- distance = result.distance
- print(id, distance)
-```
-Vedi [API Reference](/api-reference/pymilvus/v2.2.2/results.html) per maggiori informazioni.
-
-- Esegue una ricerca ibrida：
-
- Il seguente esempio esegue una ricerca approssimata nelle entità con
-
-```Python
-from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
-import random
-connections.connect()
-schema = CollectionSchema([
- FieldSchema("film_id", DataType.INT64, is_primary=True),
- FieldSchema("films", dtype=DataType.FLOAT_VECTOR, dim=2)
-])
-collection = Collection("test_collection_search", schema)
-# insert
-data = [
- [i for i in range(10)],
- [[random.random() for _ in range(2)] for _ in range(10)],
-]
-collection.insert(data)
-collection.num_entities
-10
-collection.load()
-# search
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "films",
- "param": {"metric_type": "L2"},
- "limit": 2,
- "expr": "film_id in [2,4,6,8]",
-}
-res = collection.search(**search_param)
-assert len(res) == 1
-hits = res[0]
-assert len(hits) == 2
-print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
-- Total hits: 2, hits ids: [2, 4]
-print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")
-- Top1 hit id: 2, distance: 0.10143111646175385, score: 0.101431116461
-
-```
-
-## 4. Esegui il campione
-```Python
-$ python3 hello_milvus.py
-```
-
-*I risultati ritornati e la latenza della query vengono mostrati come segue:*
-
-film_id che spazia tra [2,4,6,8].
-
-
-
-
-Search...
-(distance: 0.0, id: 2998) -20.0
-(distance: 13.2614107131958, id: 989) -11.0
-(distance: 14.489648818969727, id: 1763) -19.0
-(distance: 15.295698165893555, id: 968) -20.0
-(distance: 15.34445571899414, id: 2049) -19.0
-(distance: 0.0, id: 2999) -12.0
-(distance: 14.63361930847168, id: 1259) -13.0
-(distance: 15.421361923217773, id: 2530) -15.0
-(distance: 15.427900314331055, id: 600) -14.0
-(distance: 15.538337707519531, id: 637) -19.0
-search latency = 0.0549s
-- - -*Congratulazioni! Sei riuscito ad avviare Milvus indipendentemente e hai eseguito la tua prima ricerca di somiglianza vettoriale.* - diff --git a/preview/site/en/getstarted/example_code_node.md b/preview/site/en/getstarted/example_code_node.md deleted file mode 100644 index db1870cf5..000000000 --- a/preview/site/en/getstarted/example_code_node.md +++ /dev/null @@ -1,213 +0,0 @@ ---- -id: example_code_node.md -label: Node.js -order: 1 -group: example_code.md -summary: Get started with Milvus faster using this Node.js example code. ---- - - - -# Run Milvus using Node.js - -This topic describes how to run Milvus using Node.js. - - -## 1. Initialize a Node.js Project -```bash -npm init -``` - -
-Node.js version 12 or later is required. View Node.js Beginners Guide for information about installing the correct version for your system.
-
-
-
-
-## 2. Install TypeScript and Node Milvus SDK and its dependencies
-
-```bash
-npm install @zilliz/milvus2-sdk-node typescript --save
-```
-
-
-## 3. Download sample code HelloMilvus.ts
-```bash
-$ wget https://raw.githubusercontent.com/milvus-io/milvus-sdk-node/main/example/HelloMilvus.ts
-```
-
-## 4. Scan HelloMilvus.ts
-
-This sample code does the following:
-
-- Imports the Node.js SDK package:
-```ts
-import { MilvusClient } from "@zilliz/milvus2-sdk-node"
-import { DataType } from "@zilliz/milvus2-sdk-node/dist/milvus/types/Common";
-import { InsertReq } from "@zilliz/milvus2-sdk-node/dist/milvus/types/Insert";
-```
-
-- Connects to the Milvus server:
-```ts
-const milvusClient = new MilvusClient("localhost:19530");
-const collectionManager = milvusClient.collectionManager;
-```
-
-- Creates a collection:
-```ts
-const collectionName = "hello_milvus";
- const dim = "4";
- const createRes = await collectionManager.createCollection(
- {
- collection_name: collectionName,
- fields: [
- {
- name: "count",
- data_type: DataType.VarChar,
- is_primary_key: true,
- type_params: {
- max_length: '100',
- },
- },
- {
- name: "random_value",
- data_type: DataType.Double,
- description: "",
- },
- {
- name: "float_vector",
- data_type: DataType.FloatVector,
- description: "",
- type_params: {
- dim
- }
- }
- ]
- }
- );
-
-
- console.log("--- Create collection ---", createRes, collectionName);
-```
-
-
-- Inserts vectors in the new collection:
-```ts
-const generateInsertData = function generateInsertData(
- fields: { isVector: boolean; dim?: number; name: string; isBool?: boolean }[],
- count: number) {
- const results = [];
- while (count > 0) {
- let value: any = {};
-
- fields.forEach((v) => {
- const { isVector, dim, name, isBool } = v;
- value[name] = isVector
- ? [...Array(dim)].map(() => Math.random() * 10)
- : isBool
- ? count % 2 === 0
- : count;
- });
-
- value["count"] = count;
- results.push(value);
- count--;
- }
- return results;
-}
-
- const fields = [
- {
- isVector: true,
- dim: 4,
- name: "float_vector",
- },
- {
- isVector: false,
- name: "random_value",
- },
- ];
- const vectorsData = generateInsertData(fields, 1000);
-
- const params: InsertReq = {
- collection_name: collectionName,
- fields_data: vectorsData,
- partition_name: "test",
- };
-
- await milvusClient.dataManager.insert(params);
- console.log("--- Insert Data to Collection ---");
-```
-
-- Loads the collection and builds index on it:
-``` ts
- await milvusClient.indexManager.createIndex({
- collection_name: collectionName,
- field_name: "float_vector",
- extra_params: {
- index_type: "IVF_FLAT",
- metric_type: "L2",
- params: JSON.stringify({ nlist: 10 }),
- },
- });
- console.log("--- Create Index in Collection ---");
-```
-
-- Searches the collection:
-```ts
- // need load collection before search
- const loadCollectionRes = await collectionManager.loadCollectionSync({
- collection_name: collectionName,
- });
- console.log("--- Load collection (" + collectionName + ") ---", loadCollectionRes);
-
-
- const result = await milvusClient.dataManager.search({
- collection_name: collectionName,
- vectors: [vectorsData[0]["float_vector"]],
- search_params: {
- anns_field: "float_vector",
- topk: "4",
- metric_type: "L2",
- params: JSON.stringify({ nprobe: 1024 }),
- round_decimal: 4,
- },
- output_fields: ["count"],
- vector_type: DataType.FloatVector,
- });
-
- console.log("--- Search collection (" + collectionName + ") ---", result);
-```
-
-- Releases the collection:
-```ts
- const releaseRes = await collectionManager.releaseCollection({
- collection_name: collectionName,
- });
- console.log("--- Release Collection ---", releaseRes);
-```
-
-- Drops the collection:
-```tw
- const dropRes = await collectionManager.dropCollection({
- collection_name: collectionName,
- });
- console.log("--- Drop Collection ---", dropRes);
-```
-
-## 5. Compile the file
-```bash
-tsc MilvusHello.ts
-```
-
-
-## 6. Run the example
-```bash
-node MilvusHello.ts
-```
-
-
-- - -*Congratulations! You have successfully booted Milvus Standalone and created your first collection.* diff --git a/preview/site/en/getstarted/get_started.md b/preview/site/en/getstarted/get_started.md deleted file mode 100644 index befacf1f6..000000000 --- a/preview/site/en/getstarted/get_started.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -id: get_started.md -title: Get Started ---- - -# Get Started - -New to Milvus? No sweat! Follow the topics below for instant onboarding with Milvus. - -- [Prerequisites](prerequisite-docker.md): Check the installation requirements for your hardware and software. - -- [Install Milvus](install_milvus.md): Learn about the various and flexible ways to either install Milvus standalone or deploy a Milvus cluster. - -- [Install SDKs](install-pymilvus.md): Choose the install the Milvus SDK in the language that you prefer. - -- [Hello Milvus](example_code.md): Run the example code to get a quick preview of what Milvus is capable of. - diff --git a/preview/site/en/getstarted/install_SDKs/install-go.md b/preview/site/en/getstarted/install_SDKs/install-go.md index b58977871..5ccff3c81 100644 --- a/preview/site/en/getstarted/install_SDKs/install-go.md +++ b/preview/site/en/getstarted/install_SDKs/install-go.md @@ -3,6 +3,7 @@ id: install-go.md label: Install GO SDK related_key: SDK summary: Learn how to install the GO SDK of Milvus. +title: Install Milvus Go SDK --- # Install Milvus Go SDK @@ -13,7 +14,7 @@ Current version of Milvus supports SDKs in Python, Node.js, GO, and Java. ## Requirement -GO (1.15 or later) is required. +GO (1.17 or later) is required. ## Install Milvus GO SDK @@ -28,10 +29,10 @@ $ go get -u github.com/milvus-io/milvus-sdk-go/v2 Having installed Milvus GO SDK, you can: - Learn the basic operations of Milvus: - - [Connect to Milvus server](manage_connection.md) - - [Create a collection](create_collection.md) - - [Create a partition](create_partition.md) - - [Insert data](insert_data.md) - - [Conduct a vector search](search.md) + - [Manage Collections](manage-collections.md) + - [Manage Partitions](manage-partitions.md) + - [Insert, Upsert & Delete](insert-update-delete.md) + - [Single-Vector Search](single-vector-search.md) + - [Hybrid Search](multi-vector-search.md) diff --git a/preview/site/en/getstarted/install_SDKs/install-java.md b/preview/site/en/getstarted/install_SDKs/install-java.md index 06ac50329..38f378759 100644 --- a/preview/site/en/getstarted/install_SDKs/install-java.md +++ b/preview/site/en/getstarted/install_SDKs/install-java.md @@ -3,6 +3,7 @@ id: install-java.md label: Install Java SDK related_key: SDK summary: Learn how to install the Java SDK of Milvus. +title: Install Milvus Java SDK --- # Install Milvus Java SDK @@ -26,14 +27,14 @@ Run the following command to install Milvus Java SDK.
-Do not use embedded Milvus in production environment or if you require high performance.
-
-
-
-Embedded Milvus is suitable for the following scenarios:
-- You want to use Milvus directly without having it installed using [Docker Compose, Helm, etc.](install_standalone-docker.md).
-- You do not want to use any containers like Docker.
-- You want to use Milvus without keeping a long-running Milvus process in your machine.
-
-## Prerequisites
-
-- Python 3.6 or later
-- Supported operating systems include:
- - Ubuntu 18.04
- - Mac x86_64 >= 10.4
- - Mac M1 >= 11.0
-
-## Install embedded Milvus
-
-1. Run the following command to install embedded Milvus.
-
-```
-$ python3 -m pip install milvus
-```
-
-
-You can also install a specific version of embedded Milvus. The following example installs the 2.1.0 version of embedded Milvus.
-
-```
-$ python3 -m pip install milvus==2.1.0
-```
-
-2. Create data folder for embedded Milvus under `/var/bin/e-milvus`.
-
-```
-$ sudo mkdir -p /var/bin/e-milvus
-$ sudo chmod -R 777 /var/bin/e-milvus
-```
-
-## Start embedded Milvus
-
-1. When running embedded Milvus for the first time, import Milvus and run `milvus.before()` to set up embedded Milvus.
-
-```
-$ python3
-Python 3.9.10 (main, Jan 15 2022, 11:40:53)
-[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
-Type "help", "copyright", "credits" or "license" for more information.
->>> import milvus
---- if you are running Milvus for the first time, type milvus.before() for pre-run instructions ---
---- otherwise, type milvus.start() ---
->>>
->>> milvus.before()
-please do the following if you have not already done so:
-1. install required dependencies: bash /var/bin/e-milvus/lib/install_deps.sh
-2. (Linux system only) export LD_PRELOAD=/Users/yuchengao/Documents/GitHub/soothing-rain/embd-milvus/milvus/bin/embd-milvus.so
-3. (on Linux systems) export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib:/usr/local/lib:/var/bin/e-milvus/lib/
- (on MacOS systems) export DYLD_FALLBACK_LIBRARY_PATH=DYLD_FALLBACK_LIBRARY_PATH:/usr/lib:/usr/local/lib:/var/bin/e-milvus/lib/
->>>
-```
-
-2. Install required dependencies.
-
-```
-# exit() python interactive mode first
-# Note that this must be done AFTER `import milvus`
-$ bash /var/bin/e-milvus/lib/install_deps.sh
-```
-
-3. Set the environment variable.
-
-```
-# exit() python interactive mode first
-# Note that this must be done AFTER `import milvus`
-$ (Linux system only) export LD_PRELOAD=/Users/yuchengao/Documents/GitHub/soothing-rain/embd-milvus/milvus/bin/embd-milvus.so
-(on Linux systems) $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib:/usr/local/lib:/var/bin/e-milvus/lib/
-(on MacOS systems) $ export DYLD_FALLBACK_LIBRARY_PATH=DYLD_FALLBACK_LIBRARY_PATH:/usr/lib:/usr/local/lib:/var/bin/e-milvus/lib/
-```
-
-4. Start embedded Milvus.
-
-```
-$ python3
-Python 3.9.10 (main, Jan 15 2022, 11:40:53)
-[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
-Type "help", "copyright", "credits" or "license" for more information.
->>> import milvus
---- if you are running Milvus for the first time, type milvus.before() for pre-run instructions ---
---- otherwise, type milvus.start() ---
->>>
->>> milvus.start()
----Milvus Proxy successfully initialized and ready to serve!---
->>>
-```
-
-## Play with embedded Milvus
-
-There are two ways to play with embedded Milvus.
-
-1. Start another terminal window and run your Milvus client script.The following is an example.
-
-```
-# Download hello_milvus script
-$ wget https://raw.githubusercontent.com/milvus-io/pymilvus/v2.1.0/examples/hello_milvus.py
-# Run Hello Milvus
-$ python3 hello_milvus.py
-```
-
-2. In the same terminal window, import and run PyMilvus script immediately after running `milvus.start()`.
-
-
-This method is not suggested in Linux systems.
-
-
-```python
-$ python3
-Python 3.9.10 (main, Jan 15 2022, 11:40:53)
-[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
-Type "help", "copyright", "credits" or "license" for more information.
->>> import milvus
---- if you are running Milvus for the first time, type milvus.before() for pre-run instructions ---
---- otherwise, type milvus.start() ---
->>>
->>> milvus.start()
----Milvus Proxy successfully initialized and ready to serve!---
->>>
->>>
->>> import random
->>> from pymilvus import (
-... connections,
-... utility,
-... FieldSchema, CollectionSchema, DataType,
-... Collection,
-... )
->>> connections.connect("default", host="localhost", port="19530")
->>> has = utility.has_collection("hello_milvus")
->>> fields = [
-... FieldSchema(name="pk", dtype=DataType.INT64, is_primary=True, auto_id=False),
-... FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=8)
-... ]
->>> schema = CollectionSchema(fields, "hello_milvus is the simplest demo to introduce the APIs")
->>> hello_milvus = Collection("hello_milvus", schema, consistency_level="Strong")
->>> num_entities = 3000
->>> entities = [
-... [i for i in range(num_entities)], # provide the pk field because `auto_id` is set to False
-... [[random.random() for _ in range(8)] for _ in range(num_entities)], # field embeddings
-... ]
->>> insert_result = hello_milvus.insert(entities)
->>> index = {
-... "index_type": "IVF_FLAT",
-... "metric_type": "L2",
-... "params": {"nlist": 128},
-... }
->>> hello_milvus.create_index("embeddings", index)
->>> hello_milvus.load()
->>> vectors_to_search = entities[-1][-2:]
->>> search_params = {
-... "metric_type": "l2",
-... "params": {"nprobe": 10},
-... }
->>> result = hello_milvus.search(vectors_to_search, "embeddings", search_params, limit=3)
->>> for hits in result:
-... for hit in hits:
-... print(f"hit: {hit}")
-...
-hit: (distance: 0.0, id: 2998)
-hit: (distance: 0.1088758111000061, id: 2345)
-hit: (distance: 0.12012234330177307, id: 1172)
-hit: (distance: 0.0, id: 2999)
-hit: (distance: 0.0297045037150383, id: 2000)
-hit: (distance: 0.16927233338356018, id: 560)
->>> utility.drop_collection("hello_milvus")
->>>
-```
-
-
-
-If embedded Milvus quits with segmentation error on Linux systems, start another terminal window and run your Milvus client. Setting the environment variable `LD_PRELOAD` in Linux sysmtems might cause conflicts and segmentation error.
-
-
-
-
-## Stop embedded Milvus and clean up the data
-
-It is highly recommended that you stop embedded Milvus gracefully and use exit() or Ctrl-D (i.e. EOF) to exit when you finish using embedded Milvus.
-
-```python
->>> milvus.stop()
-to clean up, run:
-(Linux system only) export LD_PRELOAD=
-(on Linux) export LD_LIBRARY_PATH=
-(on MacOS) export DYLD_FALLBACK_LIBRARY_PATH=
->>>
->>> exit()
-```
-
-## What's next
-
-If you encounter any problems when installing or using embedded Milvus, [file an issue here](https://github.com/milvus-io/embd-milvus/issues/new).
diff --git a/preview/site/en/getstarted/install_milvus.md b/preview/site/en/getstarted/install_milvus.md
deleted file mode 100644
index 6467aaeb6..000000000
--- a/preview/site/en/getstarted/install_milvus.md
+++ /dev/null
@@ -1,17 +0,0 @@
----
-id: install_milvus.md
-title: Install Milvus
----
-
-# Install Milvus
-
-Milvus provides various and flexible ways of installation. Learn how to install Milvus standalone or deploy a Milvus cluster.
-
-- [Milvus Standalone](install_standalone-docker.md): Install Milvus standalone with Docker Compose, Helm, or APT/YUM.
-
-- [Milvus Cluster](install_cluster-docker.md): Install Milvus cluster with Docker Compose, Helm, Milvus Operator, or Ansible.
-
-- [Install Offline](install_offline-docker.md): Install Milvus in an offline environment to avoid image loading errors.
-
-- [Install Embedded Milvus](install_embedded_milvus.md): Use embedded Milvus to instantly install and run Milvus with Python.
-
diff --git a/preview/site/en/getstarted/milvus_lite.md b/preview/site/en/getstarted/milvus_lite.md
new file mode 100644
index 000000000..5d74979f8
--- /dev/null
+++ b/preview/site/en/getstarted/milvus_lite.md
@@ -0,0 +1,223 @@
+---
+id: milvus_lite.md
+summary: Get started with Milvus Lite.
+title: Run Milvus Lite Locally
+---
+
+# Run Milvus Lite Locally
+
+This page illustrates how to run Milvus Lite locally for development and testing purposes.
+
+## Overview
+
+Milvus Lite is the lightweight version of [Milvus](https://github.com/milvus-io/milvus), an open-source vector database that powers AI applications with vector embeddings and similarity search.
+
+Milvus Lite can be imported into your Python application, providing the core vector search functionality of Milvus. Milvus Lite is included in the [Python SDK of Milvus](https://github.com/milvus-io/pymilvus), thus it can be simply deployed with `pip install pymilvus`. This repo contains the core components of Milvus Lite.
+
+Milvus Lite shares the same API and covers most of the features of Milvus. Together, they provide a consistent user experience across different types of environments, fitting use cases of different size. With the same client-side code, you can run a quick demo of less than a million vectors with Milvus Lite, or a small scale app with Milvus Docker container hosted on a single machine, and eventually to a large scale production deployment on Kubenetes serving billions of vectors at thousands of QPS.
+
+## Prerequisites
+
+Milvus Lite supports the following OS distributions and sillicon types:
+
+- Ubuntu >= 20.04 (x86_64)
+- MacOS >= 11.0 (Apple Silicon and x86_64)
+
+Please note that Milvus Lite is good for getting started with vector search or building demos and prototypes. For a production use case, we recommend using Milvus on [Docker](install_standalone-docker.md) and [Kubenetes](install_cluster-milvusoperator.md), or considering the fully-managed Milvus on [Zilliz Cloud](https://zilliz.com/cloud).
+
+## Set up Milvus Lite
+
+Milvus Lite has been packed along with pymilvus, the Python SDK library of Milvus. To set up Milvus Lite, run the following command in the terminal.
+
+```
+pip install "pymilvus>=2.4.2"
+```
+
+## Connect to Milvus Lite
+
+You can connect to Milvus Lite as follows.
+
+```python
+from pymilvus import MilvusClient
+
+client = MilvusClient("milvus_demo.db")
+```
+
+After running the above code snippet, a database file named **milvus_demo.db** will be generated in the current folder.
+
+## Limits
+
+When running Milvus Lite, note that some features are not supported. The following tables summarize the usage limits on Milvus Lite.
+
+### Collection
+
+| Method / Parameter | Supported in Milvus Lite |
+|----------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|
+| [create_collection()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md) | Support with limited parameters |
+| `collection_name` | Y |
+| `dimension` | Y |
+| `primary_field_name` | Y |
+| `id_type` | Y |
+| `vector_field_name` | Y |
+| `metric_type` | Y |
+| `auto_id` | Y |
+| `schema` | Y |
+| `index_params` | Y |
+| `enable_dynamic_field` | Y |
+| `num_shards` | N |
+| `partition_key_field` | N |
+| `num_partitions` | N |
+| `consistency_level` | N (Only supports `Strong`; Any configuration will be treated as `Strong`.) |
+| [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md) | Supports getting collection statistics. |
+| `collection_name` | Y |
+| `timeout` | Y |
+| [describe_collection()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/describe_collection.md) | `num_shards`, `consistency_level`, and `collection_id` in response are invalid. |
+| `timeout` | Y |
+| [has_collection()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/has_collection.md) | Supports checking if a collection exists. |
+| `collection_name` | Y |
+| `timeout` | Y |
+| [list_collections()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/list_collections.md) | Supports listing all collections. |
+| [drop_collection()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/drop_collection.md) | Supports dropping a collection. |
+| `collection_name` | Y |
+| `timeout` | Y |
+| [rename_collection()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/rename_collection.md) | Renaming a collection is not supported. |
+
+### Field & Schema
+
+| Method / Parameter | Supported in Milvus Lite |
+|--------------------------------------------------------------------------------------------------------------|---------------------------------|
+| [create_schema()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md) | Support with limited parameters |
+| `auto_id` | Y |
+| `enable_dynamic_field` | Y |
+| `primary_field` | Y |
+| `partition_key_field` | N |
+| [add_field()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/CollectionSchema/add_field.md) | Support with limited parameters |
+| `field_name` | Y |
+| `datatype` | Y |
+| `is_primary` | Y |
+| `max_length` | Y |
+| `element_type` | Y |
+| `max_capacity` | Y |
+| `dim` | Y |
+| `is_partition_key` | N |
+
+### Insert & Search
+
+| Method / Parameter | Supported in Milvus Lite |
+|-------------------------------------------------------------------------------------------|---------------------------------|
+| [search()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/search.md) | Support with limited parameters |
+| `collection_name` | Y |
+| `data` | Y |
+| `filter` | Y |
+| `limit` | Y |
+| `output_fields` | Y |
+| `search_params` | Y |
+| `timeout` | Y |
+| `partition_names` | N |
+| `anns_field` | Y |
+| [query()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/query.md) | Support with limited parameters |
+| `collection_name` | Y |
+| `filter` | Y |
+| `output_fields` | Y |
+| `timeout` | Y |
+| `ids` | Y |
+| `partition_names` | N |
+| [get()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/get.md) | Support with limited parameters |
+| `collection_name` | Y |
+| `ids` | Y |
+| `output_fields` | Y |
+| `timeout` | Y |
+| `partition_names` | N |
+| [delete()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/delete.md) | Support with limited parameters |
+| `collection_name` | Y |
+| `ids` | Y |
+| `timeout` | Y |
+| `filter` | Y |
+| `partition_name` | N |
+| [insert()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/insert.md) | Support with limited parameters |
+| `collection_name` | Y |
+| `data` | Y |
+| `timeout` | Y |
+| `partition_name` | N |
+| [upsert()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/upsert.md) | Support with limited parameters |
+| `collection_name` | Y |
+| `data` | Y |
+| `timeout` | Y |
+| `partition_name` | N |
+
+### Load & Release
+
+| Method / Parameter | Supported in Milvus Lite |
+|-----------------------------------------------------------------------------------------------------------------------|--------------------------|
+| [load_collection()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/load_collection.md) | Y |
+| `collection_name` | Y |
+| `timeout` | Y |
+| [release_collection()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/release_collection.md) | Y |
+| `collection_name` | Y |
+| `timeout` | Y |
+| [get_load_state()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/get_load_state.md) | Getting load status is not supported. |
+| [refresh_load()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/refresh_load.md) | Loading the unloaded data of a loaded collection is not supported. |
+| [close()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/close.md) | Y |
+
+### Index
+
+| Method / Parameter | Supported in Milvus Lite |
+|----------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|
+| [list_indexes()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/list_collections.md) | Listing indexes is supported. |
+| `collection_name` | Y |
+| `field_name` | Y |
+| [create_index()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/create_index.md) | Only supports partial index types: `FLAT`, `HNSW`, `BIN_FLAT`, `SPARSE_INVERTED_INDEX`, `SPARSE_WAND`. |
+| `index_params` | Y |
+| `timeout` | Y |
+| [drop_index()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/drop_index.md) | Dropping indexes is supported. |
+| `collection_name` | Y |
+| `index_name` | Y |
+| `timeout` | Y |
+| [describe_index()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/describe_index.md) | Describing indexes is supported. |
+| `collection_name` | Y |
+| `index_name` | Y |
+| `timeout` | Y |
+
+### Partition
+
+Milvus Lite does not support partitions and partition-related methods.
+
+### Users & Roles
+
+Milvus Lite does not support users and roles and related methods.
+
+### Alias
+
+Milvus Lite does not support aliases and alias-related methods.
+
+### Others
+
+For other methods not listed in the above tables, Milvus Lite does not support them.
+
+## What's next
+
+Having connected to Milvus Lite, you can:
+
+- Check [Quickstart](quickstart.md) to see what Milvus can do.
+
+- Learn the basic operations of Milvus:
+ - [Manage Databases](manage_databases.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
+
+- [Upgrade Milvus Using Helm Chart](upgrade_milvus_cluster-helm.md).
+- [Scale your Milvus cluster](scaleout.md).
+- Deploy your Milvus cluster on clouds:
+ - [Amazon EC2](aws.md)
+ - [Amazon EKS](eks.md)
+ - [Google Cloud](gcp.md)
+ - [Google Cloud Storage](gcs.md)
+ - [Microsoft Azure](azure.md)
+ - [Microsoft Azure Blob Storage](abs.md)
+- Explore [Milvus Backup](milvus_backup_overview.md), an open-source tool for Milvus data backups.
+- Explore [Birdwatcher](birdwatcher_overview.md), an open-source tool for debugging Milvus and dynamic configuration updates.
+- Explore [Attu](https://milvus.io/docs/attu.md), an open-source GUI tool for intuitive Milvus management.
+- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/offline/install_offline-docker.md b/preview/site/en/getstarted/offline/install_offline-docker.md
deleted file mode 100644
index 6ad85cb38..000000000
--- a/preview/site/en/getstarted/offline/install_offline-docker.md
+++ /dev/null
@@ -1,93 +0,0 @@
----
-id: install_offline-docker.md
-label: Docker Compose
-order: 1
-group: install_offline-helm.md
-related_key: offline
-summary: Learn how to install Milvus with Docker Compose offline.
----
-
-
-
-# Install Milvus Offline with Docker Compose
-
-This topic describes how to install Milvus with Docker Compose in an offline environment.
-
-Installation of Milvus might fail due to image loading errors. You can install Milvus in an offline environment to avoid such problem.
-
-## Download files and images
-
-To install Milvus offline, you need to pull and save all images in an online environment first, and then transfer them to the target host and load them manually.
-
-1. Download an installation file.
-
-- For Milvus standalone:
-
-```
-$ wget https://github.com/milvus-io/milvus/releases/download/v2.2.3/milvus-standalone-docker-compose.yml -O docker-compose.yml
-```
-
-- For Milvus cluster:
-
-```
-$ wget https://github.com/milvus-io/milvus/releases/download/v2.2.3/milvus-cluster-docker-compose.yml -O docker-compose.yml
-
-```
-
-2. Download requirement and script files.
-
-```
-$ wget https://raw.githubusercontent.com/milvus-io/milvus/master/deployments/offline/requirements.txt
-$ wget https://raw.githubusercontent.com/milvus-io/milvus/master/deployments/offline/save_image.py
-```
-
-3. Pull and save images.
-
-```bash
-pip3 install -r requirements.txt
-python3 save_image.py --manifest docker-compose.yml
-```
-
-
- The images are stored in the
-
-
-4. Load the images.
-
-```bash
-cd images/; for image in $(find . -type f -name "*.tar.gz") ; do gunzip -c $image | docker load; done
-```
-
-## Install Milvus offline
-
-Having transferred the images to the target host, run the following command to install Milvus offline.
-
-```bash
-docker-compose -f docker-compose.yml up -d
-```
-
-## Uninstall Milvus
-
-To uninstall Milvus, run the following command.
-
-```bash
-docker-compose -f docker-compose.yml down
-```
-
-## What's next
-
-Having installed Milvus, you can:
-
-- Check [Hello Milvus](example_code.md) to run an example code with different SDKs to see what Milvus can do.
-
-- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
-
-- [Scale your Milvus cluster](scaleout.md).
-- Explore [MilvusDM](migrate_overview.md), an open-source tool designed for importing and exporting data in Milvus.
-- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/offline/install_offline-helm.md b/preview/site/en/getstarted/offline/install_offline-helm.md
index 631cf760c..87c21a014 100644
--- a/preview/site/en/getstarted/offline/install_offline-helm.md
+++ b/preview/site/en/getstarted/offline/install_offline-helm.md
@@ -1,14 +1,10 @@
---
id: install_offline-helm.md
-label: Kubernetes
-order: 0
-group: install_offline-helm.md
-related_key: offline
summary: Learn how to install Milvus on Kubernetes offline.
+title: Install Milvus Offline with Helm Charts
+deprecate: true
---
-
-
# Install Milvus Offline with Helm Charts
This topic describes how to install Milvus with Helm charts in an offline environment.
@@ -22,7 +18,7 @@ To install Milvus offline, you need to pull and save all images in an online env
1. Add and update Milvus Helm repository locally.
```
-helm repo add milvus https://milvus-io.github.io/milvus-helm/
+helm repo add milvus https://zilliztech.github.io/milvus-helm/
helm repo update
```
@@ -91,16 +87,16 @@ kubectl delete -f milvus_manifest.yaml
Having installed Milvus, you can:
-- Check [Hello Milvus](example_code.md) to run an example code with different SDKs to see what Milvus can do.
+- Check [Hello Milvus](quickstart.md) to run an example code with different SDKs to see what Milvus can do.
- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
-- [Upgrade Milvus Using Helm Chart](upgrade.md).
+- [Upgrade Milvus Using Helm Chart](upgrade_milvus_cluster-helm.md).
- [Scale your Milvus cluster](scaleout.md).
- Explore [MilvusDM](migrate_overview.md), an open-source tool designed for importing and exporting data in Milvus.
- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/quickstart.md b/preview/site/en/getstarted/quickstart.md
new file mode 100644
index 000000000..a07931818
--- /dev/null
+++ b/preview/site/en/getstarted/quickstart.md
@@ -0,0 +1,291 @@
+---
+id: quickstart.md
+summary: Get started with Milvus.
+title: Quickstart
+---
+
+# Quickstart with Milvus Lite
+
+/images folder.
-
+
+Vectors, the output data format of Neural Network models, can effectively encode information and serve a pivotal role in AI applications such as knowledge base, semantic search, Retrieval Augmented Generation (RAG) and more.
+
+Milvus is an open-source vector database that suits AI applications of every size from running a demo chatbot in Jupyter notebook to building web-scale search that serves billions of users. In this guide, we will walk you through how to set up Milvus locally within minutes and use the Python client library to generate, store and search vectors.
+
+## Install Milvus
+In this guide we use Milvus Lite, a python library included in `pymilvus` that can be embedded into the client application. Milvus also supports deployment on [Docker](https://milvus.io/docs/install_standalone-docker.md) and [Kubernetes](https://milvus.io/docs/install_cluster-milvusoperator.md) for production use cases.
+
+Before starting, make sure you have Python 3.8+ available in the local environment. Install `pymilvus` which contains both the python client library and Milvus Lite:
+
+
+```python
+$ pip install -U pymilvus
+```
+
+
+> If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**. (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+## Set Up Vector Database
+To create a local Milvus vector database, simply instantiate a `MilvusClient` by specifying a file name to store all data, such as "milvus_demo.db".
+
+
+```python
+from pymilvus import MilvusClient
+
+client = MilvusClient("milvus_demo.db")
+```
+
+## Create a Collection
+In Milvus, we need a collection to store vectors and their associated metadata. You can think of it as a table in traditional SQL databases. When creating a collection, you can define schema and index params to configure vector specs such as dimensionality, index types and distant metrics. There are also complex concepts to optimize the index for vector search performance. For now, let's just focus on the basics and use default for everything possible. At minimum, you only need to set the collection name and the dimension of the vector field of the collection.
+
+
+```python
+if client.has_collection(collection_name="demo_collection"):
+ client.drop_collection(collection_name="demo_collection")
+client.create_collection(
+ collection_name="demo_collection",
+ dimension=768, # The vectors we will use in this demo has 768 dimensions
+)
+```
+
+In the above setup,
+- The primary key and vector fields use their default names ("id" and "vector").
+- The metric type (vector distance definition) is set to its default value ([COSINE](https://milvus.io/docs/metric.md#Cosine-Similarity)).
+- The primary key field accepts integers and does not automatically increments (namely not using [auto-id feature](https://milvus.io/docs/schema.md))
+Alternatively, you can formally define the schema of the collection by following this [instruction](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md).
+
+## Prepare Data
+In this guide, we use vectors to perform semantic search on text. We need to generate vectors for text by downloading embedding models. This can be easily done by using the utility functions from `pymilvus[model]` library.
+
+## Represent text with vectors
+First, install the model library. This package includes essential ML tools such as PyTorch. The package download may take some time if your local environment has never installed PyTorch.
+
+
+```python
+$ pip install "pymilvus[model]"
+```
+
+Generate vector embeddings with default model. Milvus expects data to be inserted organized as a list of dictionaries, where each dictionary represents a data record, termed as an entity.
+
+
+```python
+from pymilvus import model
+
+# If connection to https://huggingface.co/ failed, uncomment the following path
+# import os
+# os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
+
+# This will download a small embedding model "paraphrase-albert-small-v2" (~50MB).
+embedding_fn = model.DefaultEmbeddingFunction()
+
+# Text strings to search from.
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+vectors = embedding_fn.encode_documents(docs)
+# The output vector has 768 dimensions, matching the collection that we just created.
+print("Dim:", embedding_fn.dim, vectors[0].shape) # Dim: 768 (768,)
+
+# Each entity has id, vector representation, raw text, and a subject label that we use
+# to demo metadata filtering later.
+data = [
+ {"id": i, "vector": vectors[i], "text": docs[i], "subject": "history"}
+ for i in range(len(vectors))
+]
+
+print("Data has", len(data), "entities, each with fields: ", data[0].keys())
+print("Vector dim:", len(data[0]["vector"]))
+```
+
+ Dim: 768 (768,)
+ Data has 3 entities, each with fields: dict_keys(['id', 'vector', 'text', 'subject'])
+ Vector dim: 768
+
+
+## [Alternatively] Use fake representation with random vectors
+If you couldn't download the model due to network issues, as a walkaround, you can use random vectors to represent the text and still finish the example. Just note that the search result won't reflect semantic similarity as the vectors are fake ones.
+
+
+```python
+import random
+
+# Text strings to search from.
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+# Use fake representation with random vectors (768 dimension).
+vectors = [[random.uniform(-1, 1) for _ in range(768)] for _ in docs]
+data = [
+ {"id": i, "vector": vectors[i], "text": docs[i], "subject": "history"}
+ for i in range(len(vectors))
+]
+
+print("Data has", len(data), "entities, each with fields: ", data[0].keys())
+print("Vector dim:", len(data[0]["vector"]))
+```
+
+ Data has 3 entities, each with fields: dict_keys(['id', 'vector', 'text', 'subject'])
+ Vector dim: 768
+
+
+## Insert Data
+Let's insert the data into the collection:
+
+
+```python
+res = client.insert(collection_name="demo_collection", data=data)
+
+print(res)
+```
+
+ {'insert_count': 3, 'ids': [0, 1, 2], 'cost': 0}
+
+
+## Semantic Search
+Now we can do semantic searches by representing the search query text as vector, and conduct vector similarity search on Milvus.
+
+### Vector search
+Milvus accepts one or multiple vector search requests at the same time. The value of the query_vectors variable is a list of vectors, where each vector is an array of float numbers.
+
+
+```python
+query_vectors = embedding_fn.encode_queries(["Who is Alan Turing?"])
+# If you don't have the embedding function you can use a fake vector to finish the demo:
+# query_vectors = [ [ random.uniform(-1, 1) for _ in range(768) ] ]
+
+res = client.search(
+ collection_name="demo_collection", # target collection
+ data=query_vectors, # query vectors
+ limit=2, # number of returned entities
+ output_fields=["text", "subject"], # specifies fields to be returned
+)
+
+print(res)
+```
+
+ data: ["[{'id': 2, 'distance': 0.5859944820404053, 'entity': {'text': 'Born in Maida Vale, London, Turing was raised in southern England.', 'subject': 'history'}}, {'id': 1, 'distance': 0.5118255615234375, 'entity': {'text': 'Alan Turing was the first person to conduct substantial research in AI.', 'subject': 'history'}}]"] , extra_info: {'cost': 0}
+
+
+The output is a list of results, each mapping to a vector search query. Each query contains a list of results, where each result contains the entity primary key, the distance to the query vector, and the entity details with specified `output_fields`.
+
+## Vector Search with Metadata Filtering
+You can also conduct vector search while considering the values of the metadata (called "scalar" fields in Milvus, as scalar refers to non-vector data). This is done with a filter expression specifying certain criteria. Let's see how to search and filter with the `subject` field in the following example.
+
+
+```python
+# Insert more docs in another subject.
+docs = [
+ "Machine learning has been used for drug design.",
+ "Computational synthesis with AI algorithms predicts molecular properties.",
+ "DDR1 is involved in cancers and fibrosis.",
+]
+vectors = embedding_fn.encode_documents(docs)
+data = [
+ {"id": 3 + i, "vector": vectors[i], "text": docs[i], "subject": "biology"}
+ for i in range(len(vectors))
+]
+
+client.insert(collection_name="demo_collection", data=data)
+
+# This will exclude any text in "history" subject despite close to the query vector.
+res = client.search(
+ collection_name="demo_collection",
+ data=embedding_fn.encode_queries(["tell me AI related information"]),
+ filter="subject == 'biology'",
+ limit=2,
+ output_fields=["text", "subject"],
+)
+
+print(res)
+```
+
+ data: ["[{'id': 4, 'distance': 0.27030569314956665, 'entity': {'text': 'Computational synthesis with AI algorithms predicts molecular properties.', 'subject': 'biology'}}, {'id': 3, 'distance': 0.16425910592079163, 'entity': {'text': 'Machine learning has been used for drug design.', 'subject': 'biology'}}]"] , extra_info: {'cost': 0}
+
+
+By default, the scalar fields are not indexed. If you need to perform metadata filtered search in large dataset, you can consider using fixed schema and also turn on the [index](https://milvus.io/docs/scalar_index.md) to improve the search performance.
+
+In addition to vector search, you can also perform other types of searches:
+
+### Query
+A query() is an operation that retrieves all entities matching a cretria, such as a [filter expression](https://milvus.io/docs/boolean.md) or matching some ids.
+
+For example, retrieving all entities whose scalar field has a particular value:
+
+
+```python
+res = client.query(
+ collection_name="demo_collection",
+ filter="subject == 'history'",
+ output_fields=["text", "subject"],
+)
+```
+
+Directly retrieve entities by primary key:
+
+
+```python
+res = client.query(
+ collection_name="demo_collection",
+ ids=[0, 2],
+ output_fields=["vector", "text", "subject"],
+)
+```
+
+## Delete Entities
+If you'd like to purge data, you can delete entities specifying the primary key or delete all entities matching a particular filter expression.
+
+
+```python
+# Delete entities by primary key
+res = client.delete(collection_name="demo_collection", ids=[0, 2])
+
+print(res)
+
+# Delete entities by a filter expression
+res = client.delete(
+ collection_name="demo_collection",
+ filter="subject == 'biology'",
+)
+
+print(res)
+```
+
+ [0, 2]
+ [3, 4, 5]
+
+
+## Load Existing Data
+Since all data of Milvus Lite is stored in a local file, you can load all data into memory even after the program terminates, by creating a `MilvusClient` with the existing file. For example, this will recover the collections from "milvus_demo.db" file and continue to write data into it.
+
+
+```python
+from pymilvus import MilvusClient
+
+client = MilvusClient("milvus_demo.db")
+```
+
+## Drop the collection
+If you would like to delete all the data in a collection, you can drop the collection with
+
+
+```python
+# Drop collection
+client.drop_collection(collection_name="demo_collection")
+```
+
+## Learn More
+Milvus Lite is great for getting started with a local python program. If you have large scale data or would like to use Milvus in production, you can learn about deploying Milvus on [Docker](https://milvus.io/docs/install_standalone-docker.md) and [Kubernetes](https://milvus.io/docs/install_cluster-milvusoperator.md). All deployment modes of Milvus share the same API, so your client side code doesn't need to change much if moving to another deployment mode. Simply specify the [URI and Token](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) of a Milvus server deployed anywhere:
+
+
+```python
+client = MilvusClient(uri="http://localhost:19530", token="root:Milvus")
+```
+
+Milvus provides REST and gRPC API, with client libraries in languages such as [Python](https://milvus.io/docs/install-pymilvus.md), [Java](https://milvus.io/docs/install-java.md), [Go](https://milvus.io/docs/install-go.md), C# and [Node.js](https://milvus.io/docs/install-node.md).
diff --git a/preview/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md b/preview/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md
new file mode 100644
index 000000000..52084140f
--- /dev/null
+++ b/preview/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md
@@ -0,0 +1,97 @@
+---
+id: install_standalone-docker-compose.md
+label: Docker Compose
+related_key: Docker Compose
+summary: Learn how to install Milvus standalone with Docker Compose.
+title: Run Milvus with Docker Compose
+---
+
+# Run Milvus with Docker Compose
+
+This page illustrates how to launch a Milvus instance in Docker using Docker Compose.
+
+## Prerequisites
+
+- [Install Docker](https://docs.docker.com/get-docker/).
+- [Check the requirements for hardware and software](prerequisite-docker.md) prior to your installation.
+
+## Install Milvus
+
+Milvus provides a Docker Compose configuration file in the Milvus repository. To install Milvus using Docker Compose, just run
+
+```shell
+# Download the configuration file
+$ wget https://github.com/milvus-io/milvus/releases/download/v2.4.5/milvus-standalone-docker-compose.yml -O docker-compose.yml
+
+# Start Milvus
+$ sudo docker compose up -d
+
+Creating milvus-etcd ... done
+Creating milvus-minio ... done
+Creating milvus-standalone ... done
+```
+
+
+
+- If you failed to run the above command, please check whether your system has Docker Compose V1 installed. If this is the case, you are advised to migrate to Docker Compose V2 due to the notes on [this page](https://docs.docker.com/compose/).
+
+- If you encounter any issues pulling the image, contact us at community@zilliz.com with details about the problem, and we'll provide you with the necessary support.
+
+
+
+After starting up Milvus,
+
+- Containers named **milvus-standalone**, **milvus-minio**, and **milvus-etcd** are up.
+ - The **milvus-etcd** container does not expose any ports to the host and maps its data to **volumes/etcd** in the current folder.
+ - The **milvus-minio** container serves ports **9090** and **9091** locally with the default authentication credentials and maps its data to **volumes/minio** in the current folder.
+ - The **milvus-standalone** container serves ports **19530** locally with the default settings and maps its data to **volumes/milvus** in the current folder.
+
+You can check if the containers are up and running using the following command:
+
+```shell
+$ sudo docker compose ps
+
+ Name Command State Ports
+--------------------------------------------------------------------------------------------------------------------
+milvus-etcd etcd -advertise-client-url ... Up 2379/tcp, 2380/tcp
+milvus-minio /usr/bin/docker-entrypoint ... Up (healthy) 9000/tcp
+milvus-standalone /tini -- milvus run standalone Up 0.0.0.0:19530->19530/tcp, 0.0.0.0:9091->9091/tcp
+```
+
+You can stop and delete this container as follows
+
+```shell
+# Stop Milvus
+$ sudo docker compose down
+
+# Delete service data
+$ sudo rm -rf volumes
+```
+
+## What's next
+
+Having installed Milvus in Docker, you can:
+
+- Check [Quickstart](quickstart.md) to see what Milvus can do.
+
+- Learn the basic operations of Milvus:
+ - [Manage Databases](manage_databases.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
+
+- [Upgrade Milvus Using Helm Chart](upgrade_milvus_cluster-helm.md).
+- [Scale your Milvus cluster](scaleout.md).
+- Deploy your Milvu cluster on clouds:
+ - [Amazon EC2](aws.md)
+ - [Amazon EKS](eks.md)
+ - [Google Cloud](gcp.md)
+ - [Google Cloud Storage](gcs.md)
+ - [Microsoft Azure](azure.md)
+ - [Microsoft Azure Blob Storage](abs.md)
+- Explore [Milvus Backup](milvus_backup_overview.md), an open-source tool for Milvus data backups.
+- Explore [Birdwatcher](birdwatcher_overview.md), an open-source tool for debugging Milvus and dynamic configuration updates.
+- Explore [Attu](https://github.com/zilliztech/attu), an open-source GUI tool for intuitive Milvus management.
+- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/run-milvus-docker/install_standalone-docker.md b/preview/site/en/getstarted/run-milvus-docker/install_standalone-docker.md
new file mode 100644
index 000000000..05293ad58
--- /dev/null
+++ b/preview/site/en/getstarted/run-milvus-docker/install_standalone-docker.md
@@ -0,0 +1,81 @@
+---
+id: install_standalone-docker.md
+label: Docker
+related_key: Docker
+summary: Learn how to install Milvus standalone with Docker.
+title: Run Milvus in Docker
+---
+
+# Run Milvus in Docker
+
+This page illustrates how to launch a Milvus instance in Docker.
+
+
+## Prerequisites
+
+- [Install Docker](https://docs.docker.com/get-docker/).
+- [Check the requirements for hardware and software](prerequisite-docker.md) prior to your installation.
+
+
+## Install Milvus in Docker
+
+Milvus provides an installation script to install it as a docker container. The script is available in the [Milvus repository](https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh). To install Milvus in Docker, just run
+
+```shell
+# Download the installation script
+$ curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh
+
+# Start the Docker container
+$ bash standalone_embed.sh start
+```
+
+
+
+If you encounter any issues pulling the image, contact us at community@zilliz.com with details about the problem, and we'll provide you with the necessary support.
+
+
+
+After running the installation script:
+
+- A docker container named milvus has been started at port **19530**.
+- An embed etcd is installed along with Milvus in the same container and serves at port **2379**. Its configuration file is mapped to **embedEtcd.yaml** in the current folder.
+- To change the default Milvus configuration, add your settings to the **user.yaml** file in the current folder and then restart the service.
+- The Milvus data volume is mapped to **volumes/milvus** in the current folder.
+
+You can stop and delete this container as follows
+
+```shell
+# Stop Milvus
+$ bash standalone_embed.sh stop
+
+# Delete Milvus data
+$ bash standalone_embed.sh delete
+```
+
+## What's next
+
+Having installed Milvus in Docker, you can:
+
+- Check [Quickstart](quickstart.md) to see what Milvus can do.
+
+- Learn the basic operations of Milvus:
+ - [Manage Databases](manage_databases.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
+
+- [Upgrade Milvus Using Helm Chart](upgrade_milvus_cluster-helm.md).
+- [Scale your Milvus cluster](scaleout.md).
+- Deploy your Milvu cluster on clouds:
+ - [Amazon EC2](aws.md)
+ - [Amazon EKS](eks.md)
+ - [Google Cloud](gcp.md)
+ - [Google Cloud Storage](gcs.md)
+ - [Microsoft Azure](azure.md)
+ - [Microsoft Azure Blob Storage](abs.md)
+- Explore [Milvus Backup](milvus_backup_overview.md), an open-source tool for Milvus data backups.
+- Explore [Birdwatcher](birdwatcher_overview.md), an open-source tool for debugging Milvus and dynamic configuration updates.
+- Explore [Attu](https://milvus.io/docs/attu.md), an open-source GUI tool for intuitive Milvus management.
+- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/prerequisite-docker.md b/preview/site/en/getstarted/run-milvus-docker/prerequisite-docker.md
similarity index 75%
rename from preview/site/en/getstarted/prerequisite-docker.md
rename to preview/site/en/getstarted/run-milvus-docker/prerequisite-docker.md
index 3139f8658..960a32c50 100644
--- a/preview/site/en/getstarted/prerequisite-docker.md
+++ b/preview/site/en/getstarted/run-milvus-docker/prerequisite-docker.md
@@ -1,21 +1,22 @@
---
id: prerequisite-docker.md
-label: Install with Docker Compose
+label: Docker requirements
related_key: Docker
summary: Learn the necessary preparations before installing Milvus with Docker Compose.
+title: Requirements for Installing Milvus with Docker Compose
---
-# Environment Checklist for Milvus with Docker Compose
+# Requirements for Installing Milvus with Docker Compose
-Before you install Milvus, check your hardware and software to see if they meet the requirements.
+Before installing a Milvus instance, check your hardware and software to see if they meet the requirements.
## Hardware requirements
| Component | Requirement |Recommendation| Note |
| ------------------- | ------------------------------------------------------------ |--------------| ------------------------------------------------------------ |
-| CPU |
+
+If you encounter any issues pulling the image, contact us at community@zilliz.com with details about the problem, and we'll provide you with the necessary support.
+
+
+
+## Install Helm Chart for Milvus
+
+Helm is a K8s package manager that can help you deploy Milvus quickly.
+
+1. Add Milvus Helm repository.
+
+```
+$ helm repo add milvus https://zilliztech.github.io/milvus-helm/
+```
+
+
+
+The Milvus Helm Charts repo at `https://milvus-io.github.io/milvus-helm/` has been archived and you can get further updates from `https://zilliztech.github.io/milvus-helm/` as follows:
+
+```shell
+helm repo add zilliztech https://zilliztech.github.io/milvus-helm
+helm repo update
+# upgrade existing helm release
+helm upgrade my-release zilliztech/milvus
+```
+
+The archived repo is still available for the charts up to 4.0.31. For later releases, use the new repo instead.
+
+
+
+2. Update charts locally.
+
+```
+$ helm repo update
+```
+
+## Start Milvus
+
+Once you have installed the Helm chart, you can start Milvus on Kubernetes. In this section, we will guide you through the steps to start Milvus with GPU support.
+
+You should start Milvus with Helm by specifying the release name, the chart, and the parameters you expect to change. In this guide, we use my-release as the release name. To use a different release name, replace my-release in the following commands with the one you are using.
+
+Milvus allows you to assign one or more GPU devices to Milvus.
+
+### 1. Assign a single GPU device
+
+Milvus with GPU support allows you to assign one or more GPU devices.
+
+- Milvus cluster
+
+ ```bash
+ cat <
+
+
+- Milvus standalone
+
+ ```bash
+ cat <-
+
+
+If you encounter any issues pulling the image, contact us at community@zilliz.com with details about the problem, and we'll provide you with the necessary support.
+
+
+
+## Install Milvus
+
+To install Milvus with GPU support using Docker Compose, follow these steps.
+
+### 1. Download and configure the YAML file
+
+Download [`milvus-standalone-docker-compose-gpu.yml`](https://github.com/milvus-io/milvus/releases/download/v2.4.5/milvus-standalone-docker-compose-gpu.yml) and save it as docker-compose.yml manually, or with the following command.
+
+```shell
+$ wget https://github.com/milvus-io/milvus/releases/download/v2.4.5/milvus-standalone-docker-compose-gpu.yml -O docker-compose.yml
+```
+
+You need to make some changes to the environment variables of the standalone service in the YAML file as follows:
+
+- To assign a specific GPU device to Milvus, locate the `deploy.resources.reservations.devices[0].devices_ids` field in the definition of the `standalone` service and replace its value with the ID of the desired GPU. You can use the `nvidia-smi` tool, included with NVIDIA GPU display drivers, to determine the ID of a GPU device. Milvus supports multiple GPU devices.
+
+Assign a single GPU device to Milvus:
+
+```yaml
+...
+standalone:
+ ...
+ deploy:
+ resources:
+ reservations:
+ devices:
+ - driver: nvidia
+ capabilities: ["gpu"]
+ device_ids: ["0"]
+...
+```
+
+Assign multiple GPU devices to Milvus:
+
+```yaml
+...
+standalone:
+ ...
+ deploy:
+ resources:
+ reservations:
+ devices:
+ - driver: nvidia
+ capabilities: ["gpu"]
+ device_ids: ['0', '1']
+...
+```
+
+### 2. Start Milvus
+
+In the directory that holds docker-compose.yml, start Milvus by running:
+
+```shell
+$ sudo docker compose up -d
+
+Creating milvus-etcd ... done
+Creating milvus-minio ... done
+Creating milvus-standalone ... done
+```
+
+
+
+If you failed to run the above command, check whether your system has Docker Compose V1 installed. If this is the case, you are advised to migrate to Docker Compose V2 due to the notes on [this page](https://docs.docker.com/compose/).
+
+
+
+After starting up Milvus,
+
+- Containers named **milvus-standalone**, **milvus-minio**, and **milvus-etcd** are up.
+ - The **milvus-etcd** container does not expose any ports to the host and maps its data to **volumes/etcd** in the current folder.
+ - The **milvus-minio** container serves ports **9090** and **9091** locally with the default authentication credentials and maps its data to **volumes/minio** in the current folder.
+ - The **milvus-standalone** container serves ports **19530** locally with the default settings and maps its data to **volumes/milvus** in the current folder.
+
+You can check if the containers are up and running using the following command:
+
+```shell
+$ sudo docker compose ps
+
+ Name Command State Ports
+--------------------------------------------------------------------------------------------------------------------
+milvus-etcd etcd -advertise-client-url ... Up 2379/tcp, 2380/tcp
+milvus-minio /usr/bin/docker-entrypoint ... Up (healthy) 9000/tcp
+milvus-standalone /tini -- milvus run standalone Up 0.0.0.0:19530->19530/tcp, 0.0.0.0:9091->9091/tcp
+```
+
+If you have assigned multiple GPU devices to Milvus in docker-compose.yml, you can specify which GPU device is visible or available for use.
+
+Make GPU device `0` visible to Milvus:
+
+```shell
+$ CUDA_VISIBLE_DEVICES=0 ./milvus run standalone
+```
+
+Make GPU devices `0` and `1` visible to Milvus:
+
+```shell
+$ CUDA_VISIBLE_DEVICES=0,1 ./milvus run standalone
+```
+
+You can stop and delete this container as follows.
+
+```shell
+# Stop Milvus
+$ sudo docker compose down
+
+# Delete service data
+$ sudo rm -rf volumes
+```
+
+## Configure memory pool
+
+After Milvus is up and running, you can customize the memory pool by modifying the `initMemSize` and `maxMemSize` settings in the `milvus.yaml` file.
+
+
+
+The `milvus.yaml` file is located in the `/milvus/configs/` directory inside the Milvus container.
+
+
+
+To confgiure the memory pool, modify the `initMemSize` and `maxMemSize` settings in the `milvus.yaml` file as follows.
+
+1. Use the following command to copy `milvus.yaml` from the Milvus container to your local machine. Replace `
+
+Ensure that you can access the K8s cluster via `kubectl`. If you have not installed `kubectl` locally, see [Use kubectl inside minikube](https://minikube.sigs.k8s.io/docs/handbook/kubectl/).
+
+
+
+### How can I start a K8s cluster with GPU worker nodes?
+
+If you prefer to use GPU-enabled worker nodes, you can follow the steps below to create a K8s cluster with GPU worker nodes. We recommend installing Milvus on a K8s cluster with GPU worker nodes and using the default storage class provisioned.
+
+1. Prepare GPU worker nodes
+
+ To use GPU-enabled worker nodes, follow steps in [Prepare your GPU nodes](https://gitlab.com/nvidia/kubernetes/device-plugin/-/blob/main/README.md#preparing-your-gpu-nodes).
+
+2. Enable GPU support on K8s
+
+ Deploy the **nvidia-device-plugin** with Helm by following [these steps](https://gitlab.com/nvidia/kubernetes/device-plugin/-/blob/main/README.md#deployment-via-helm).
+
+ After setting up, view the GPU resources with the following command. Replace `
+
+If you encounter any issues pulling the image, contact us at community@zilliz.com with details about the problem, and we'll provide you with the necessary support.
+
+
+
+## Install Milvus Helm Chart
+
+Before installing Milvus Helm Charts, you need to add Milvus Helm repository.
+
+```
+$ helm repo add milvus https://zilliztech.github.io/milvus-helm/
+```
+
+
+
+The Milvus Helm Charts repo at `https://milvus-io.github.io/milvus-helm/` has been archived and you can get further updates from `https://zilliztech.github.io/milvus-helm/` as follows:
+
+```shell
+helm repo add zilliztech https://zilliztech.github.io/milvus-helm
+helm repo update
+# upgrade existing helm release
+helm upgrade my-release zilliztech/milvus
+```
+
+The archived repo is still available for the charts up to 4.0.31. For later releases, use the new repo instead.
+
+
+
+Then fetch Milvus charts from the repository as follows:
+
+```
+$ helm repo update
+```
+
+You can always run this command to fetch the latest Milvus Helm charts.
+
+## Online install
+
+### 1. Deploy a Milvus cluster
+
+Once you have installed the Helm chart, you can start Milvus on Kubernetes. This section will guide you through the steps to starting Milvus.
+
+```shell
+$ helm install my-release milvus/milvus
+```
+
+In the above command, `my-release` is the release name, and `milvus/milvus` is the locally installed chart repository. To use a different name, replace `my-release` with the one you see fit.
+
+The command above deploys a Milvus cluster with its components and dependencies using default configurations. To customize these settings, we recommend you use the [Milvus Sizing Tool](https://milvus.io/tools/sizing) to adjust the configurations based on your actual data size and then download the corresponding YAML file. To learn more about configuration parameters, refer to [Milvus System Configurations Checklist](https://milvus.io/docs/system_configuration.md).
+
+
+
+
+### 2. Check Milvus cluster status
+
+Run the following command to check the status of all pods in your Milvus cluster.
+
+```
+$ kubectl get pods
+```
+
+Once all pods are running, the output of the above command should be similar to the following:
+
+```
+NAME READY STATUS RESTARTS AGE
+my-release-etcd-0 1/1 Running 0 3m23s
+my-release-etcd-1 1/1 Running 0 3m23s
+my-release-etcd-2 1/1 Running 0 3m23s
+my-release-milvus-datacoord-6fd4bd885c-gkzwx 1/1 Running 0 3m23s
+my-release-milvus-datanode-68cb87dcbd-4khpm 1/1 Running 0 3m23s
+my-release-milvus-indexcoord-5bfcf6bdd8-nmh5l 1/1 Running 0 3m23s
+my-release-milvus-indexnode-5c5f7b5bd9-l8hjg 1/1 Running 0 3m24s
+my-release-milvus-proxy-6bd7f5587-ds2xv 1/1 Running 0 3m24s
+my-release-milvus-querycoord-579cd79455-xht5n 1/1 Running 0 3m24s
+my-release-milvus-querynode-5cd8fff495-k6gtg 1/1 Running 0 3m24s
+my-release-milvus-rootcoord-7fb9488465-dmbbj 1/1 Running 0 3m23s
+my-release-minio-0 1/1 Running 0 3m23s
+my-release-minio-1 1/1 Running 0 3m23s
+my-release-minio-2 1/1 Running 0 3m23s
+my-release-minio-3 1/1 Running 0 3m23s
+my-release-pulsar-autorecovery-86f5dbdf77-lchpc 1/1 Running 0 3m24s
+my-release-pulsar-bookkeeper-0 1/1 Running 0 3m23s
+my-release-pulsar-bookkeeper-1 1/1 Running 0 98s
+my-release-pulsar-broker-556ff89d4c-2m29m 1/1 Running 0 3m23s
+my-release-pulsar-proxy-6fbd75db75-nhg4v 1/1 Running 0 3m23s
+my-release-pulsar-zookeeper-0 1/1 Running 0 3m23s
+my-release-pulsar-zookeeper-metadata-98zbr 0/1 Completed 0 3m24s
+```
+
+### 3. Forward a local port to Milvus
+
+Run the following command to get the port at which your Milvus cluster serves.
+
+```bash
+$ kubectl get pod my-release-milvus-proxy-6bd7f5587-ds2xv --template
+='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
+19530
+```
+
+The output shows that the Milvus instance serves at the default port **19530**.
+
+-
+
+
+If you have deployed Milvus in standalone mode, change the pod name from `my-release-milvus-proxy-xxxxxxxxxx-xxxxx` to `my-release-milvus-xxxxxxxxxx-xxxxx`.
+
+
+
+Then, run the following command to forward a local port to the port at which Milvus serves.
+
+```bash
+$ kubectl port-forward service/my-release-milvus 27017:19530
+Forwarding from 127.0.0.1:27017 -> 19530
+```
+
+Optionally, you can use `:19530` instead of `27017:19530` in the above command to let `kubectl` allocate a local port for you so that you don't have to manage port conflicts.
+
+By default, kubectl's port-forwarding only listens on `localhost`. Use the `address` flag if you want Milvus to listen on the selected or all IP addresses. The following command makes port-forward listen on all IP addresses on the host machine.
+
+```bash
+$ kubectl port-forward --address 0.0.0.0 service/my-release-milvus 27017:19530
+Forwarding from 0.0.0.0:27017 -> 19530
+```
+
+## Offline install
+
+If you are in a network-restricted environment, follow the procedure in this section to start a Milvus cluster.
+
+### 1. Get Milvus manifest
+
+Run the following command to get the Milvus manifest.
+
+```shell
+$ helm template my-release milvus/milvus > milvus_manifest.yaml
+```
+
+The above command renders chart templates for a Milvus cluster and saves the output to a manifest file named `milvus_manifest.yaml`. Using this manifest, you can install a Milvus cluster with its components and dependencies in separate pods.
+
+
+
+- To install a Milvus instance in the standalone mode where all Milvus components are contained within a single pod, you should run `helm template my-release --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsar.enabled=false milvus/milvus > milvus_manifest.yaml` instead to render chart templates for a Milvus instance in a standalone mode.
+- To change Milvus configurations, download the [`value.yaml`](https://raw.githubusercontent.com/milvus-io/milvus-helm/master/charts/milvus/values.yaml) template, place your desired settings in it, and use `helm template -f values.yaml my-release milvus/milvus > milvus_manifest.yaml` to render the manifest accordingly.
+
+
+
+### 2. Download image-pulling script
+
+The image-pulling script is developed in Python. You should download the script along with its dependencies in the `requirement.txt` file.
+
+```shell
+$ wget https://raw.githubusercontent.com/milvus-io/milvus/master/deployments/offline/requirements.txt
+$ wget https://raw.githubusercontent.com/milvus-io/milvus/master/deployments/offline/save_image.py
+```
+
+### 3. Pull and save images
+
+Run the following command to pull and save the required images.
+
+```shell
+$ pip3 install -r requirements.txt
+$ python3 save_image.py --manifest milvus_manifest.yaml
+```
+
+The images are pulled into a sub-folder named `images` in the current directory.
+
+### 4. Load images
+
+You can now load the images to the hosts in the network-restricted environment as follows:
+
+```shell
+$ for image in $(find . -type f -name "*.tar.gz") ; do gunzip -c $image | docker load; done
+```
+
+### 5. Deploy Milvus
+
+```shell
+$ kubectl apply -f milvus_manifest.yaml
+```
+
+Till now, you can follow steps [2](#2-Check-Milvus-cluster-status) and [3](#3-Forward-a-local-port-to-Milvus) of the online install to check the cluster status and forward a local port to Milvus.
+
+## Upgrade running Milvus cluster
+
+Run the following command to upgrade your running Milvus cluster to the latest version:
+
+```shell
+$ helm repo update
+$ helm upgrade my-release zilliztech/milvus
+```
+
+## Uninstall Milvus
+
+Run the following command to uninstall Milvus.
+
+```bash
+$ helm uninstall my-release
+```
+
+## What's next
+
+Having installed Milvus in Docker, you can:
+
+- Check [Hello Milvus](quickstart.md) to see what Milvus can do.
+
+- Learn the basic operations of Milvus:
+ - [Manage Databases](manage_databases.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
+
+- [Upgrade Milvus Using Helm Chart](upgrade_milvus_cluster-helm.md).
+- [Scale your Milvus cluster](scaleout.md).
+- Deploy your Milvu cluster on clouds:
+ - [Amazon EC2](aws.md)
+ - [Amazon EKS](eks.md)
+ - [Google Cloud](gcp.md)
+ - [Google Cloud Storage](gcs.md)
+ - [Microsoft Azure](azure.md)
+ - [Microsoft Azure Blob Storage](abs.md)
+- Explore [Milvus Backup](milvus_backup_overview.md), an open-source tool for Milvus data backups.
+- Explore [Birdwatcher](birdwatcher_overview.md), an open-source tool for debugging Milvus and dynamic configuration updates.
+- Explore [Attu](https://milvus.io/docs/attu.md), an open-source GUI tool for intuitive Milvus management.
+- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md b/preview/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md
new file mode 100644
index 000000000..96e9f8b27
--- /dev/null
+++ b/preview/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md
@@ -0,0 +1,359 @@
+---
+id: install_cluster-milvusoperator.md
+label: Milvus Operator
+related_key: Kubernetes
+summary: Learn how to install Milvus cluster on Kubernetes using Milvus Operator
+title: Install Milvus Cluster with Milvus Operator
+---
+
+# Run Milvus in Kubernetes with Milvus Operator
+
+This page illustrates how to start a Milvus instance in Kubernetes using [Milvus Operator](https://github.com/zilliztech/milvus-operator).
+
+## Overview
+
+Milvus Operator is a solution that helps you deploy and manage a full Milvus service stack to target Kubernetes (K8s) clusters. The stack includes all Milvus components and relevant dependencies like etcd, Pulsar, and MinIO.
+
+## Prerequisites
+
+- [Create a K8s cluster](prerequisite-helm.md#How-can-I-start-a-K8s-cluster-locally-for-test-purposes).
+- Install a [StorageClass](https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/). You can check the installed StorageClass as follows.
+
+ ```bash
+ $ kubectl get sc
+
+ NAME PROVISIONER RECLAIMPOLICY VOLUMEBIINDINGMODE ALLOWVOLUMEEXPANSION AGE
+ standard (default) k8s.io/minikube-hostpath Delete Immediate false
+ ```
+
+- Check [the hardware and software requirements](prerequisite-helm.md) before installation.
+
+- Before installing Milvus, it is recommended to use the [Milvus Sizing Tool](https://milvus.io/tools/sizing) to estimate the hardware requirements based on your data size. This helps ensure optimal performance and resource allocation for your Milvus installation.
+
+
+
+If you encounter any issues pulling the image, contact us at community@zilliz.com with details about the problem, and we'll provide you with the necessary support.
+
+
+
+## Install Milvus Operator
+
+Milvus Operator defines a Milvus cluster custom resources on top of [Kubernetes Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). When custom resources are defined, you can use K8s APIs in a declarative way and manage the Milvus deployment stack to ensure its scalability and high availability.
+
+### 1. Install cert-manager
+
+Milvus Operator uses [cert-manager](https://cert-manager.io/docs/installation/supported-releases/) to provide a certificate for the webhook server.
+
+
+
+- You can safely skip this step if you choose to [deploy Milvus Operator using Helm](install_cluster-helm.md).
+- Milvus Operator requires cert-manager 1.1.3 or above.
+
+
+
+Run the following command to install cert-manager.
+
+```shell
+$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml
+```
+
+You will see the output similar to the following after the installation process ends.
+
+```shell
+customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
+customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
+customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
+customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
+customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
+customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
+namespace/cert-manager created
+serviceaccount/cert-manager-cainjector created
+...
+service/cert-manager created
+service/cert-manager-webhook created
+deployment.apps/cert-manager-cainjector created
+deployment.apps/cert-manager created
+deployment.apps/cert-manager-webhook created
+mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
+validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
+```
+
+You can check if cert-manager pods are running as follows:
+
+```shell
+$ kubectl get pods -n cert-manager
+
+NAME READY STATUS RESTARTS AGE
+cert-manager-848f547974-gccz8 1/1 Running 0 70s
+cert-manager-cainjector-54f4cc6b5-dpj84 1/1 Running 0 70s
+cert-manager-webhook-7c9588c76-tqncn 1/1 Running 0 70s
+```
+
+### 2. Install Milvus Operator
+
+You can install Milvus Operator in either of the following ways:
+
+- [With Helm](#Install-with-Helm)
+- [With kubectl](#Install-with-kubectl)
+
+#### Install with Helm
+
+Run the following command to install Milvus Operator with Helm.
+
+```shell
+$ helm install milvus-operator \
+ -n milvus-operator --create-namespace \
+ --wait --wait-for-jobs \
+ https://github.com/zilliztech/milvus-operator/releases/download/v0.9.17/milvus-operator-0.9.17.tgz
+```
+
+You will see the output similar to the following after the installation process ends.
+
+```shell
+NAME: milvus-operator
+LAST DEPLOYED: Thu Jul 7 13:18:40 2022
+NAMESPACE: milvus-operator
+STATUS: deployed
+REVISION: 1
+TEST SUITE: None
+NOTES:
+Milvus Operator Is Starting, use `kubectl get -n milvus-operator deploy/milvus-operator` to check if its successfully installed
+If Operator not started successfully, check the checker's log with `kubectl -n milvus-operator logs job/milvus-operator-checker`
+Full Installation doc can be found in https://github.com/zilliztech/milvus-operator/blob/main/docs/installation/installation.md
+Quick start with `kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_minimum.yaml`
+More samples can be found in https://github.com/zilliztech/milvus-operator/tree/main/config/samples
+CRD Documentation can be found in https://github.com/zilliztech/milvus-operator/tree/main/docs/CRD
+```
+
+#### Install with kubectl
+
+Run the following command to install Milvus Operator with `kubectl`.
+
+```shell
+$ kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/deploy/manifests/deployment.yaml
+```
+
+You will see the output similar to the following after the installation process ends.
+
+```shell
+namespace/milvus-operator created
+customresourcedefinition.apiextensions.k8s.io/milvusclusters.milvus.io created
+serviceaccount/milvus-operator-controller-manager created
+role.rbac.authorization.k8s.io/milvus-operator-leader-election-role created
+clusterrole.rbac.authorization.k8s.io/milvus-operator-manager-role created
+clusterrole.rbac.authorization.k8s.io/milvus-operator-metrics-reader created
+clusterrole.rbac.authorization.k8s.io/milvus-operator-proxy-role created
+rolebinding.rbac.authorization.k8s.io/milvus-operator-leader-election-rolebinding created
+clusterrolebinding.rbac.authorization.k8s.io/milvus-operator-manager-rolebinding created
+clusterrolebinding.rbac.authorization.k8s.io/milvus-operator-proxy-rolebinding created
+configmap/milvus-operator-manager-config created
+service/milvus-operator-controller-manager-metrics-service created
+service/milvus-operator-webhook-service created
+deployment.apps/milvus-operator-controller-manager created
+certificate.cert-manager.io/milvus-operator-serving-cert created
+issuer.cert-manager.io/milvus-operator-selfsigned-issuer created
+mutatingwebhookconfiguration.admissionregistration.k8s.io/milvus-operator-mutating-webhook-configuration created
+validatingwebhookconfiguration.admissionregistration.k8s.io/milvus-operator-validating-webhook-configuration created
+```
+
+You can check if the Milvus Operator pod is running as follows:
+
+```shell
+$ kubectl get pods -n milvus-operator
+
+NAME READY STATUS RESTARTS AGE
+milvus-operator-5fd77b87dc-msrk4 1/1 Running 0 46s
+```
+
+## Deploy Milvus
+
+### 1. Deploy a Milvus cluster
+
+Once the Milvus Operator pod is running, you can deploy a Milvus cluster as follows.
+
+```shell
+$ kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_cluster_default.yaml
+```
+
+The command above deploys a Milvus cluster with its components and dependencies in separate pods using default configurations. To customize these settings, we recommend you use the [Milvus Sizing Tool](https://milvus.io/tools/sizing) to adjust the configurations based on your actual data size and then download the corresponding YAML file. To learn more about configuration parameters, refer to [Milvus System Configurations Checklist](https://milvus.io/docs/system_configuration.md).
+
+
+
+- The release name should only contain letters, numbers and dashes. Dots are not allowed in the release name.
+- You can also deploy a Milvus instance in standalone mode, where all its components are contained within a single pod. To do so, change the configuration file URL in the above command to `https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_default.yaml`
+
+
+
+#### 2. Check Milvus cluster status
+
+Run the following command to check Milvus cluster status
+
+```shell
+$ kubectl get milvus my-release -o yaml
+```
+
+Once your Milvus cluster is ready, the output of the above command should be similar to the following. If the `status.status` field stays `Unhealthy`, your Milvus cluster is still under creation.
+
+```yaml
+apiVersion: milvus.io/v1alpha1
+kind: MilvusCluster
+metadata:
+...
+status:
+ conditions:
+ - lastTransitionTime: "2021-11-02T05:59:41Z"
+ reason: StorageReady
+ status: "True"
+ type: StorageReady
+ - lastTransitionTime: "2021-11-02T06:06:23Z"
+ message: Pulsar is ready
+ reason: PulsarReady
+ status: "True"
+ type: PulsarReady
+ - lastTransitionTime: "2021-11-02T05:59:41Z"
+ message: Etcd endpoints is healthy
+ reason: EtcdReady
+ status: "True"
+ type: EtcdReady
+ - lastTransitionTime: "2021-11-02T06:12:36Z"
+ message: All Milvus components are healthy
+ reason: MilvusClusterHealthy
+ status: "True"
+ type: MilvusReady
+ endpoint: my-release-milvus.default:19530
+ status: Healthy
+```
+
+Milvus Operator creates Milvus dependencies, such as etcd, Pulsar, and MinIO, and then Milvus components, such as proxy, coordinators, and nodes.
+
+Once your Milvus cluster is ready, the status of all pods in the Milvus cluster should be similar to the following.
+
+```shell
+$ kubectl get pods
+
+NAME READY STATUS RESTARTS AGE
+my-release-etcd-0 1/1 Running 0 14m
+my-release-etcd-1 1/1 Running 0 14m
+my-release-etcd-2 1/1 Running 0 14m
+my-release-milvus-datacoord-6c7bb4b488-k9htl 1/1 Running 0 6m
+my-release-milvus-datanode-5c686bd65-wxtmf 1/1 Running 0 6m
+my-release-milvus-indexcoord-586b9f4987-vb7m4 1/1 Running 0 6m
+my-release-milvus-indexnode-5b9787b54-xclbx 1/1 Running 0 6m
+my-release-milvus-proxy-84f67cdb7f-pg6wf 1/1 Running 0 6m
+my-release-milvus-querycoord-865cc56fb4-w2jmn 1/1 Running 0 6m
+my-release-milvus-querynode-5bcb59f6-nhqqw 1/1 Running 0 6m
+my-release-milvus-rootcoord-fdcccfc84-9964g 1/1 Running 0 6m
+my-release-minio-0 1/1 Running 0 14m
+my-release-minio-1 1/1 Running 0 14m
+my-release-minio-2 1/1 Running 0 14m
+my-release-minio-3 1/1 Running 0 14m
+my-release-pulsar-bookie-0 1/1 Running 0 14m
+my-release-pulsar-bookie-1 1/1 Running 0 14m
+my-release-pulsar-bookie-init-h6tfz 0/1 Completed 0 14m
+my-release-pulsar-broker-0 1/1 Running 0 14m
+my-release-pulsar-broker-1 1/1 Running 0 14m
+my-release-pulsar-proxy-0 1/1 Running 0 14m
+my-release-pulsar-proxy-1 1/1 Running 0 14m
+my-release-pulsar-pulsar-init-d2t56 0/1 Completed 0 14m
+my-release-pulsar-recovery-0 1/1 Running 0 14m
+my-release-pulsar-toolset-0 1/1 Running 0 14m
+my-release-pulsar-zookeeper-0 1/1 Running 0 14m
+my-release-pulsar-zookeeper-1 1/1 Running 0 13m
+my-release-pulsar-zookeeper-2 1/1 Running 0 13m
+```
+
+### 3. Forward a local port to Milvus
+
+Run the following command to get the port at which your Milvus cluster serves.
+
+```shell
+$ kubectl get pod my-release-milvus-proxy-84f67cdb7f-pg6wf --template
+='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
+19530
+```
+
+The output shows that the Milvus instance serves at the default port **19530**.
+
+
+
+If you have deployed Milvus in standalone mode, change the pod name from `my-release-milvus-proxy-xxxxxxxxxx-xxxxx` to `my-release-milvus-xxxxxxxxxx-xxxxx`.
+
+
+
+Then, run the following command to forward a local port to the port at which Milvus serves.
+
+```shell
+$ kubectl port-forward service/my-release-milvus 27017:19530
+Forwarding from 127.0.0.1:27017 -> 19530
+```
+
+Optionally, you can use `:19530` instead of `27017:19530` in the above command to let `kubectl` allocate a local port for you so that you don't have to manage port conflicts.
+
+By default, kubectl's port-forwarding only listens on `localhost`. Use the `address` flag if you want Milvus to listen on the selected or all IP addresses. The following command makes port-forward listen on all IP addresses on the host machine.
+
+```shell
+$ kubectl port-forward --address 0.0.0.0 service/my-release-milvus 27017:19530
+Forwarding from 0.0.0.0:27017 -> 19530
+```
+
+## Uninstall Milvus
+
+Run the following command to uninstall the Milvus cluster.
+
+```shell
+$ kubectl delete milvus my-release
+```
+
+
+
+- When you delete the Milvus cluster using the default configuration, dependencies like etcd, Pulsar, and MinIO are not deleted. Therefore, next time when you install the same Milvus cluster instance, these dependencies will be used again.
+- To delete the dependencies and private virtual clouds (PVCs) along with the Milvus cluster, see [configuration file](https://github.com/zilliztech/milvus-operator/blob/main/config/samples/milvus_deletion.yaml).
+
+
+
+## Uninstall Milvus Operator
+
+There are also two ways to uninstall Milvus Operator.
+
+- [Uninstall with Helm](#Uninstall-with-Helm)
+- [Uninstall with kubectl](#Uninstall-with-kubectl)
+
+#### Uninstall with Helm
+
+```shell
+$ helm -n milvus-operator uninstall milvus-operator
+```
+
+#### Uninstall with kubectl
+
+```shell
+$ kubectl delete -f https://raw.githubusercontent.com/zilliztech/milvus-operator/v0.9.17/deploy/manifests/deployment.yaml
+```
+
+## What's next
+
+Having installed Milvus in Docker, you can:
+
+- Check [Hello Milvus](quickstart.md) to see what Milvus can do.
+
+- Learn the basic operations of Milvus:
+ - [Manage Databases](manage_databases.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
+
+- [Upgrade Milvus Using Helm Chart](upgrade_milvus_cluster-helm.md).
+- [Scale your Milvus cluster](scaleout.md).
+- Deploy your Milvu cluster on clouds:
+ - [Amazon EC2](aws.md)
+ - [Amazon EKS](eks.md)
+ - [Google Cloud](gcp.md)
+ - [Google Cloud Storage](gcs.md)
+ - [Microsoft Azure](azure.md)
+ - [Microsoft Azure Blob Storage](abs.md)
+- Explore [Milvus Backup](milvus_backup_overview.md), an open-source tool for Milvus data backups.
+- Explore [Birdwatcher](birdwatcher_overview.md), an open-source tool for debugging Milvus and dynamic configuration updates.
+- Explore [Attu](https://milvus.io/docs/attu.md), an open-source GUI tool for intuitive Milvus management.
+- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/prerequisite-helm.md b/preview/site/en/getstarted/run-milvus-k8s/prerequisite-helm.md
similarity index 62%
rename from preview/site/en/getstarted/prerequisite-helm.md
rename to preview/site/en/getstarted/run-milvus-k8s/prerequisite-helm.md
index 046b3a401..849776c0b 100644
--- a/preview/site/en/getstarted/prerequisite-helm.md
+++ b/preview/site/en/getstarted/run-milvus-k8s/prerequisite-helm.md
@@ -3,20 +3,21 @@ id: prerequisite-helm.md
label: Install on Kubernetes
related_key: Kubernetes
summary: Learn the necessary preparations before installing Milvus with Helm.
+title: Requirements for running Milvus on Kubernetes
---
-# Environment Checklist for Milvus Milvus on Kubernetes
+# Requirements for running Milvus on Kubernetes
-Before you install Milvus, check your hardware and software to see if they meet the requirements.
+This page lists the hardware and software requirements to get Milvus up and running.
## Hardware requirements
| Component | Requirement |Recommendation| Note |
| ------------------- | ------------------------------------------------------------ |--------------| ------------------------------------------------------------ |
-| CPU |
+
+Ensure that you can access the K8s cluster via `kubectl`. If you have not installed `kubectl` locally, see [Use kubectl inside minikube](https://minikube.sigs.k8s.io/docs/handbook/kubectl/).
+
+
+
## What's next
- If your hardware and software meet the requirements, you can:
- - [Install Milvus standalone on Kubernetes](install_standalone-helm.md)
- - [Install Milvus cluster on Kubernetes](install_cluster-helm.md)
+ - [Run Milvus in Kubernets with Milvus Operator](install_cluster-milvusoperator.md)
+ - [Run Milvus in Kubernetes with Helm](install_cluster-helm.md)
- See [System Configuration](system_configuration.md) for parameters you can set while installing Milvus.
diff --git a/preview/site/en/getstarted/standalone/install_standalone-aptyum.md b/preview/site/en/getstarted/standalone/install_standalone-aptyum.md
new file mode 100644
index 000000000..c4d799b16
--- /dev/null
+++ b/preview/site/en/getstarted/standalone/install_standalone-aptyum.md
@@ -0,0 +1,83 @@
+---
+id: install_standalone-aptyum.md
+label: DEB/RPM
+related_key: Install
+order: 3
+group: install_standalone-docker.md
+summary: Learn how to install Milvus stanalone with dpkg/yum.
+title: Install Milvus Standalone with dpkg/yum
+deprecate: true
+---
+
+
+
+# Install Milvus Standalone with dpkg/yum
+
+This topic describes how to install Milvus standalone using package manager dpkg or yum on Linux systems.
+
+
+## Prerequisites
+
+Check [the requirements](prerequisite-docker.md) for hardware and software prior to your installation.
+
+## Install Milvus
+
+### Install Milvus with dpkg on Ubuntu
+
+```bash
+$ wget https://github.com/milvus-io/milvus/releases/download/v2.4.1/milvus_2.4.1-1_amd64.deb
+$ sudo apt-get update
+$ sudo dpkg -i milvus_2.4.1-1_amd64.deb
+$ sudo apt-get -f install
+```
+
+### Install Milvus with yum on RedHat9
+
+```bash
+$ sudo yum install -y https://github.com/milvus-io/milvus/releases/download/v2.4.1/milvus-2.4.1-1.el9.x86_64.rpm
+```
+
+## Check the status of Milvus
+
+```bash
+$ sudo systemctl restart milvus
+$ sudo systemctl status milvus
+```
+
+## Connect to Milvus
+
+Please refer to [Hello Milvus](https://milvus.io/docs/example_code.md), then run the example code.
+
+## Uninstall Milvus
+
+### Uninstall Milvus on Ubuntu
+
+```bash
+$ sudo dpkg -P milvus
+```
+
+### Uninstall Milvus on RedHat9
+
+```bash
+$ sudo yum remove -y milvus
+```
+
+## What's next
+
+Having installed Milvus, you can:
+
+- Check [Hello Milvus](quickstart.md) to run an example code with different SDKs to see what Milvus can do.
+- Check [In-memory Index](index.md) for more about CPU-compatible index types.
+
+- Learn the basic operations of Milvus:
+ - [Manage Databases](manage_databases.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
+
+- Explore [Milvus Backup](milvus_backup_overview.md), an open-source tool for Milvus data backups.
+- Explore [Birdwatcher](birdwatcher_overview.md), an open-source tool for debugging Milvus and dynamic configuration updates.
+- Explore [Attu](https://milvus.io/docs/attu.md), an open-source GUI tool for intuitive Milvus management.
+- [Monitor Milvus with Prometheus](monitor.md)
diff --git a/preview/site/en/getstarted/standalone/install_standalone-docker.md b/preview/site/en/getstarted/standalone/install_standalone-docker.md
deleted file mode 100644
index 34a3c1d23..000000000
--- a/preview/site/en/getstarted/standalone/install_standalone-docker.md
+++ /dev/null
@@ -1,88 +0,0 @@
----
-id: install_standalone-docker.md
-label: Docker Compose
-related_key: Docker
-order: 2
-group: install_standalone-helm.md
-summary: Learn how to install Milvus stanalone with Docker Compose.
----
-
-
-
-# Install Milvus Standalone with Docker Compose
-
-This topic describes how to install Milvus standalone using Docker Compose.
-
-## Prerequisites
-
-Check [the requirements](prerequisite-docker.md) for hardware and software prior to your installation.
-
-## Download the `YAML` file
-
-[Download](https://github.com/milvus-io/milvus/releases/download/v2.2.3/milvus-standalone-docker-compose.yml) `milvus-standalone-docker-compose.yml` and save it as `docker-compose.yml` manually, or with the following command.
-
-```
-$ wget https://github.com/milvus-io/milvus/releases/download/v2.2.3/milvus-standalone-docker-compose.yml -O docker-compose.yml
-```
-
-## Start Milvus
-
-In the same directory as the `docker-compose.yml` file, start up Milvus by running:
-
-```shell
-$ sudo docker-compose up -d
-```
-
-
-If your system has Docker Compose V2 installed instead of V1, use
-
-```text
-Creating milvus-etcd ... done
-Creating milvus-minio ... done
-Creating milvus-standalone ... done
-```
-
-Now check if the containers are up and running.
-
-```
-$ sudo docker-compose ps
-```
-
-After Milvus standalone starts, there will be three docker containers running, including the Milvus standalone service and its two dependencies.
-
-```
- Name Command State Ports
---------------------------------------------------------------------------------------------------------------------
-milvus-etcd etcd -advertise-client-url ... Up 2379/tcp, 2380/tcp
-milvus-minio /usr/bin/docker-entrypoint ... Up (healthy) 9000/tcp
-milvus-standalone /tini -- milvus run standalone Up 0.0.0.0:19530->19530/tcp, 0.0.0.0:9091->9091/tcp
-```
-
-## Stop Milvus
-
-To stop Milvus standalone, run:
-```
-sudo docker-compose down
-```
-
-To delete data after stopping Milvus, run:
-```
-sudo rm -rf volumes
-```
-
-## What's next
-
-Having installed Milvus, you can:
-
-- Check [Hello Milvus](example_code.md) to run an example code with different SDKs to see what Milvus can do.
-
-- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
-
-- Explore [MilvusDM](migrate_overview.md), an open-source tool designed for importing and exporting data in Milvus.
-- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/standalone/install_standalone-helm.md b/preview/site/en/getstarted/standalone/install_standalone-helm.md
index 830ff4d1f..278379588 100644
--- a/preview/site/en/getstarted/standalone/install_standalone-helm.md
+++ b/preview/site/en/getstarted/standalone/install_standalone-helm.md
@@ -1,16 +1,19 @@
---
id: install_standalone-helm.md
label: Helm
-order: 1
-group: install_standalone-helm.md
-summary: Learn how to install Milvus stanalone on Kubernetes.
+related_key: Helm
+order: 2
+group: install_standalone-docker.md
+summary: Learn how to install Milvus standalone on Kubernetes.
+title: Install Milvus Standalone with Kubernetes
+deprecate: true
---
-
+
# Install Milvus Standalone with Kubernetes
-This topic describes how to install Milvus standalone using Kubernetes (K8s).
+This topic describes how to install Milvus standalone using Kubernetes.
## Prerequisites
@@ -47,6 +50,21 @@ NAME PROVISIONER RECLAIMPOLICY VOLUMEBIINDI
standard (default) k8s.io/minikube-hostpath Delete Immediate false 3m36s
```
+### 4. Check the default storage class
+
+Milvus relies on the default storage class to automatically provision volumes for data persistence. Run the following command to check storage classes:
+
+```bash
+$ kubectl get sc
+```
+
+The command output should be similar to the following:
+
+```bash
+NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
+local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 461d
+```
+
## Install Helm Chart for Milvus
Helm is a K8s package manager that can help you deploy Milvus quickly.
@@ -54,9 +72,24 @@ Helm is a K8s package manager that can help you deploy Milvus quickly.
1. Add Milvus to Helm's repository.
```bash
-$ helm repo add milvus https://milvus-io.github.io/milvus-helm/
+$ helm repo add milvus https://zilliztech.github.io/milvus-helm/
```
+ docker compose instead of docker-compose . Check if this is the case with $ docker compose version . Read here for more information.
-
+
+The Milvus Helm Charts repo at `https://milvus-io.github.io/milvus-helm/` has been archived and you can get further updates from `https://zilliztech.github.io/milvus-helm/` as follows:
+
+```shell
+helm repo add zilliztech https://zilliztech.github.io/milvus-helm
+helm repo update
+# upgrade existing helm release
+helm upgrade my-release zilliztech/milvus
+```
+
+The archived repo is still available for the charts up to 4.0.31. For later releases, use the new repo instead.
+
+
+
2. Update your local chart repository.
```bash
@@ -65,7 +98,9 @@ $ helm repo update
## Start Milvus
-Start Milvus with Helm by specifying the release name, the chart, and parameters you expect to change. This topic uses my-release as the release name. To use a different release name, replace my-release in the command.
+Once you have installed the Helm chart, you can start Milvus on Kubernetes. In this section, we will guide you through the steps to start Milvus.
+
+You should start Milvus with Helm by specifying the release name, the chart, and the parameters you expect to change. In this guide, we use my-release as the release name. To use a different release name, replace my-release in the following commands with the one you are using.
```bash
$ helm install my-release milvus/milvus --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsar.enabled=false
@@ -112,6 +147,13 @@ $ kubectl port-forward service/my-release-milvus 27017:19530
Forwarding from 127.0.0.1:27017 -> 19530
```
+By default, ports forwarded by kubectl only listen on localhost. Use flag `address` if you want Milvus server to listen on selected IP or all addresses.
+
+```bash
+$ kubectl port-forward --address 0.0.0.0 service/my-release-milvus 27017:19530
+Forwarding from 0.0.0.0:27017 -> 19530
+```
+
## Uninstall Milvus
Run the following command to uninstall Milvus.
@@ -146,15 +188,18 @@ $ minikube delete
Having installed Milvus, you can:
-- Check [Hello Milvus](example_code.md) to run an example code with different SDKs to see what Milvus can do.
+- Check [Hello Milvus](quickstart.md) to run an example code with different SDKs to see what Milvus can do.
- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
-
-- [Upgrade Milvus Using Helm Chart](upgrade.md).
-- Explore [MilvusDM](migrate_overview.md), an open-source tool designed for importing and exporting data in Milvus.
+ - [Manage Databases](manage_databases.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
+
+- [Upgrade Milvus Using Helm Chart](upgrade_milvus_standalone-helm.md).
+- Explore [Milvus Backup](milvus_backup_overview.md), an open-source tool for Milvus data backups.
+- Explore [Birdwatcher](birdwatcher_overview.md), an open-source tool for debugging Milvus and dynamic configuration updates.
+- Explore [Attu](https://milvus.io/docs/attu.md), an open-source GUI tool for intuitive Milvus management.
- [Monitor Milvus with Prometheus](monitor.md).
diff --git a/preview/site/en/getstarted/standalone/install_standalone-operator.md b/preview/site/en/getstarted/standalone/install_standalone-operator.md
index 19cfdbae0..ac33846e1 100644
--- a/preview/site/en/getstarted/standalone/install_standalone-operator.md
+++ b/preview/site/en/getstarted/standalone/install_standalone-operator.md
@@ -1,12 +1,14 @@
---
id: install_standalone-operator.md
label: Milvus Operator
-order: 0
-group: install_standalone-helm.md
+order: 1
+group: install_standalone-docker.md
summary: Learn how to install Milvus stanalone with Milvus Operator.
+title: Install Milvus Standalone with Milvus Operator
+deprecate: true
---
-
+
# Install Milvus Standalone with Milvus Operator
@@ -146,7 +148,7 @@ There are two ways to install Milvus Operator on K8s:
helm install milvus-operator \
-n milvus-operator --create-namespace \
--wait --wait-for-jobs \
- https://github.com/milvus-io/milvus-operator/releases/download/v0.7.7/milvus-operator-0.7.7.tgz
+ https://github.com/zilliztech/milvus-operator/releases/download/v0.9.17/milvus-operator-0.9.17.tgz
```
If Milvus Operator is installed, you can see the following output.
@@ -160,16 +162,16 @@ TEST SUITE: None
NOTES:
Milvus Operator Is Starting, use `kubectl get -n milvus-operator deploy/milvus-operator` to check if its successfully installed
If Operator not started successfully, check the checker's log with `kubectl -n milvus-operator logs job/milvus-operator-checker`
-Full Installation doc can be found in https://github.com/milvus-io/milvus-operator/blob/main/docs/installation/installation.md
-Quick start with `kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/config/samples/milvus_minimum.yaml`
-More samples can be found in https://github.com/milvus-io/milvus-operator/tree/main/config/samples
-CRD Documentation can be found in https://github.com/milvus-io/milvus-operator/tree/main/docs/CRD
+Full Installation doc can be found in https://github.com/zilliztech/milvus-operator/blob/main/docs/installation/installation.md
+Quick start with `kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_minimum.yaml`
+More samples can be found in https://github.com/zilliztech/milvus-operator/tree/main/config/samples
+CRD Documentation can be found in https://github.com/zilliztech/milvus-operator/tree/main/docs/CRD
```
#### Install by `kubectl` command
```
-$ kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/deploy/manifests/deployment.yaml
+$ kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/deploy/manifests/deployment.yaml
```
If Milvus Operator is installed, you can see the following output.
@@ -209,7 +211,7 @@ milvus-operator-5fd77b87dc-msrk4 1/1 Running 0 46s
When Milvus Operator starts, run the following command to install Milvus.
```
-$ kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/config/samples/milvus_default.yaml
+$ kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_default.yaml
```
@@ -221,7 +223,31 @@ Run the following command to check the status of Milvus you just installed.
$ kubectl get milvus my-release -o yaml
```
-When the Milvus is successfully installed, you can learn how to [Connect to Milvus server](manage_connection.md).
+When the Milvus is successfully installed, you can learn how to [manage collections](manage-collections.md).
+
+## Connect to Milvus
+
+Verify which local port the Milvus server is listening on. Replace the pod name with your own.
+
+```bash
+$ kubectl get pod my-release-milvus-proxy-84f67cdb7f-pg6wf --template
+='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
+19530
+```
+
+Open a new terminal and run the following command to forward a local port to the port that Milvus uses. Optionally, omit the designated port and use `:19530` to let `kubectl` allocate a local port for you so that you don't have to manage port conflicts.
+
+```bash
+$ kubectl port-forward service/my-release-milvus 27017:19530
+Forwarding from 127.0.0.1:27017 -> 19530
+```
+
+By default, kubectl's port-forwarding only listens on localhost. Use flag `address` if you want Milvus server to listen on selected IP or all addresses.
+
+```bash
+$ kubectl port-forward --address 0.0.0.0 service/my-release-milvus 27017:19530
+Forwarding from 0.0.0.0:27017 -> 19530
+```
## Uninstall Milvus standalone
@@ -244,7 +270,7 @@ $ helm -n milvus-operator uninstall milvus-operator
### Uninstall Milvus Operator by `kubectl` command
```
-$ kubectl delete -f https://raw.githubusercontent.com/milvus-io/milvus-operator/v0.7.7/deploy/manifests/deployment.yaml
+$ kubectl delete -f https://raw.githubusercontent.com/zilliztech/milvus-operator/v0.9.17/deploy/manifests/deployment.yaml
```
## Delete the K8s cluster
@@ -254,15 +280,18 @@ When you no longer need the K8s cluster in the test environment, you can run `$
## What's next
Having installed Milvus, you can:
-- Check [Hello Milvus](example_code.md) to run an example code with different SDKs to see what Milvus can do.
+- Check [Hello Milvus](quickstart.md) to run an example code with different SDKs to see what Milvus can do.
- Learn the basic operations of Milvus:
- - [Connect to Milvus server](manage_connection.md)
- - [Create a collection](create_collection.md)
- - [Create a partition](create_partition.md)
- - [Insert data](insert_data.md)
- - [Conduct a vector search](search.md)
-- [Upgrade Milvus Using Helm Chart](upgrade.md)
-- Explore [MilvusDM](migrate_overview.md), an open-source tool designed for importing and exporting data in Milvus.
+ - [Manage Databases](manage_databases.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
+- [Upgrade Milvus Using Milvus Operator](upgrade_milvus_standalone-operator.md)
+- Explore [Milvus Backup](milvus_backup_overview.md), an open-source tool for Milvus data backups.
+- Explore [Birdwatcher](birdwatcher_overview.md), an open-source tool for debugging Milvus and dynamic configuration updates.
+- Explore [Attu](https://milvus.io/docs/attu.md), an open-source GUI tool for intuitive Milvus management.
- [Monitor Milvus with Prometheus](monitor.md)
diff --git a/preview/site/en/getstarted/with_gpu/install_standalone-helm-gpu.md b/preview/site/en/getstarted/with_gpu/install_standalone-helm-gpu.md
new file mode 100644
index 000000000..f24a79683
--- /dev/null
+++ b/preview/site/en/getstarted/with_gpu/install_standalone-helm-gpu.md
@@ -0,0 +1,326 @@
+---
+id: install_standalone-helm-gpu.md
+label: Standalone (Helm)
+order: 0
+group: install_standalone-helm-gpu.md
+related_key: Docker
+summary: Learn the necessary preparations before installing Milvus with Docker.
+title: Install Milvus Standalone with GPU Support
+deprecate: true
+---
+
+
+
+# Install Milvus Standalone with GPU Support
+
+Milvus now can use GPU devices to build indexes and perform ANN searches thanks to the contribution from NVIDIA. This guide will show you how to install Milvus with GPU support on your machine.
+
+## Prerequisites
+
+Before installing Milvus with GPU support, make sure you have the following prerequisites:
+
+- The compute capability of your GPU device is 6.0、7.0、7.5、8.0、8.6、9.0. To check whether your GPU device suffices the requirement, check [Your GPU Compute Capability](https://developer.nvidia.com/cuda-gpus) on the NVIDIA developer website.
+
+- You have installed the NVIDIA driver for your GPU device on one of [the supported Linux distributions](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#linux-distributions) and then the NVIDIA Container Toolkit following [this guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
+
+ For Ubuntu 22.04 users, you can install the driver and the container toolkit with the following commands:
+
+ ```shell
+ $ sudo apt install --no-install-recommends nvidia-headless-545 nvidia-utils-545
+ ```
+
+ For other OS users, please refer to the [official installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installing-on-ubuntu-and-debian).
+
+ You can check whether the driver has been installed correctly by running the following command:
+
+ ```shell
+ $ modinfo nvidia | grep "^version"
+ version: 545.29.06
+ ```
+
+ You are recommended to use the drivers of version 545 and above.
+
+- You have installed a Kubernetes cluster, and the `kubectl` command-line tool has been configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts.
+
+## Create a K8s cluster using minikube
+
+We recommend installing Milvus on K8s with [minikube](https://minikube.sigs.k8s.io/docs/), a tool that allows you to run K8s locally.
+
+### 1. Install minikube
+
+See [install minikube](https://minikube.sigs.k8s.io/docs/start/) for more information.
+
+### 2. Start a K8s cluster using minikube
+
+After installing minikube, run the following command to start a K8s cluster.
+
+```
+$ minikube start --gpus all
+```
+
+### 3. Check the K8s cluster status
+
+Run `$ kubectl cluster-info` to check the status of the K8s cluster you just created. Ensure that you can access the K8s cluster via `kubectl`. If you have not installed `kubectl` locally, see [Use kubectl inside minikube](https://minikube.sigs.k8s.io/docs/handbook/kubectl/).
+
+Minikube has a dependency on default StorageClass when installed. Check the dependency by running the following command. Other installation methods require manual configuration of the StorageClass. See [Change the Default StorageClass](https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/) for more information.
+
+```
+$ kubectl get sc
+```
+
+```
+NAME PROVISIONER RECLAIMPOLICY VOLUMEBIINDINGMODE ALLOWVOLUMEEXPANSION AGE
+standard (default) k8s.io/minikube-hostpath Delete Immediate false 3m36s
+```
+
+## Start a Kubernetes cluster with GPU worker nodes
+
+If you prefer to use GPU-enabled worker nodes, you can follow the steps below to create a K8s cluster with GPU worker nodes. We recommend installing Milvus on a Kubernetes cluster with GPU worker nodes and using the default storage class provisioned.
+
+### 1. Prepare GPU worker nodes
+
+See [Prepare GPU worker nodes](https://gitlab.com/nvidia/kubernetes/device-plugin/-/blob/main/README.md#preparing-your-gpu-nodes) for more information.
+
+### 2. Enable GPU support on Kubernetes
+
+See [install nvidia-device-plugin with helm](https://gitlab.com/nvidia/kubernetes/device-plugin/-/blob/main/README.md#deployment-via-helm) for more information.
+
+After setting up, run `kubectl describe node
+
+The Milvus Helm Charts repo at `https://milvus-io.github.io/milvus-helm/` has been archived and you can get further updates from `https://zilliztech.github.io/milvus-helm/` as follows:
+
+```shell
+helm repo add zilliztech https://zilliztech.github.io/milvus-helm
+helm repo update
+# upgrade existing helm release
+helm upgrade my-release zilliztech/milvus
+```
+
+The archived repo is still available for the charts up to 4.0.31. For later releases, use the new repo instead.
+
+
+
+2. Update your local chart repository.
+
+```bash
+$ helm repo update
+```
+
+## Start Milvus
+
+Once you have installed the Helm chart, you can start Milvus on Kubernetes. In this section, we will guide you through the steps to start Milvus with GPU support.
+
+You should start Milvus with Helm by specifying the release name, the chart, and the parameters you expect to change. In this guide, we use my-release as the release name. To use a different release name, replace my-release in the following commands with the one you are using.
+
+Milvus allows you to assign one or more GPU devices to Milvus.
+
+- Assign a single GPU device (recommended)
+
+ Run the following commands to assign a single GPU device to Milvus:
+
+ ```bash
+ cat <
+ See Milvus Helm Chart and Helm for more information.
+
+
+ Check the status of the running pods:
+
+ ```bash
+ $ kubectl get pods
+ ```
+
+After Milvus starts, the `READY` column displays `1/1` for all pods.
+
+```text
+NAME READY STATUS RESTARTS AGE
+my-release-etcd-0 1/1 Running 0 30s
+my-release-milvus-standalone-54c4f88cb9-f84pf 1/1 Running 0 30s
+my-release-minio-5564fbbddc-mz7f5 1/1 Running 0 30s
+```
+
+## Connect to Milvus
+
+Verify which local port the Milvus server is listening on. Replace the pod name with your own.
+
+```bash
+$ kubectl get pod my-release-milvus-standalone-54c4f88cb9-f84pf --template='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
+```
+
+```
+19530
+```
+
+Open a new terminal and run the following command to forward a local port to the port that Milvus uses. Optionally, omit the designated port and use `:19530` to let `kubectl` allocate a local port for you so that you don't have to manage port conflicts.
+
+```bash
+$ kubectl port-forward service/my-release-milvus 27017:19530
+```
+
+```
+Forwarding from 127.0.0.1:27017 -> 19530
+```
+
+By default, ports forwarded by kubectl only listen on localhost. Use flag `address` if you want Milvus server to listen on selected IP or all addresses.
+
+```bash
+$ kubectl port-forward --address 0.0.0.0 service/my-release-milvus 27017:19530
+Forwarding from 0.0.0.0:27017 -> 19530
+```
+
+## Uninstall Milvus
+
+Run the following command to uninstall Milvus.
+
+```bash
+$ helm uninstall my-release
+```
+
+## Stop the K8s cluster
+
+Stop the cluster and the minikube VM without deleting the resources you created.
+
+```bash
+$ minikube stop
+```
+
+Run `minikube start` to restart the cluster.
+
+## Delete the K8s cluster
+
+
+Run
+
+Delete the cluster, the minikube VM, and all resources you created including persistent volumes.
+
+```bash
+$ minikube delete
+```
+
+## What's next
+
+Having installed Milvus, you can:
+
+- Check [Hello Milvus](quickstart.md) to run an example code with different SDKs to see what Milvus can do.
+
+- Learn the basic operations of Milvus:
+ - [Manage Databases](manage_databases.md)
+ - [Manage Collections](manage-collections.md)
+ - [Manage Partitions](manage-partitions.md)
+ - [Insert, Upsert & Delete](insert-update-delete.md)
+ - [Single-Vector Search](single-vector-search.md)
+ - [Hybrid Search](multi-vector-search.md)
+
+- [Upgrade Milvus Using Helm Chart](upgrade_milvus_standalone-helm.md).
+- Explore [Milvus Backup](milvus_backup_overview.md), an open-source tool for Milvus data backups.
+- Explore [Birdwatcher](birdwatcher_overview.md), an open-source tool for debugging Milvus and dynamic configuration updates.
+- Explore [Attu](https://milvus.io/docs/attu.md), an open-source GUI tool for intuitive Milvus management.
+- [Monitor Milvus with Prometheus](monitor.md).
+
+
diff --git a/preview/site/en/home/home.json b/preview/site/en/home/home.json
index 58c168ab8..7c570843d 100644
--- a/preview/site/en/home/home.json
+++ b/preview/site/en/home/home.json
@@ -14,7 +14,7 @@
"title": "Quick Start",
"key": "Cluster",
"btnLabel": "Learn to use",
- "link": "example_code.md"
+ "link": "quickstart.md"
},
{
"title": "Milvus Bootcamp",
@@ -37,24 +37,24 @@
"items": [{
"label": "Use",
"list": [{
- "text": "Create a Collection",
- "link": "create_collection.md"
+ "text": "Manage Collections",
+ "link": "manage-collections.md"
},
{
- "text": "Insert Data",
- "link": "insert_data.md"
+ "text": "Insert, Upsert & Delete",
+ "link": "insert-update-delete.md"
},
{
- "text": "Build an Index",
- "link": "build_index.md"
+ "text": "Manage Indexes",
+ "link": "manage-indexes.md"
},
{
- "text": "Vector Similarity Search",
- "link": "search.md"
+ "text": "Single-Vector Search",
+ "link": "single-vector-search.md"
},
{
- "text": "Query",
- "link": "query.md"
+ "text": "Get & Scalar Query",
+ "link": "get-and-scalar-query.md"
}
]
},
@@ -116,16 +116,16 @@
},
"title": "Blog",
"items": [{
- "time": "June 29, 2021",
- "link": "https://medium.com/@xiaofan.luan/milvus-2-0-redefining-vector-database-438479a7dd6e",
- "title": "Milvus 2.0: Redefining Vector Database",
- "abstract": "Milvus 2.0 is available now. This refactored, cloud-native version powers image/video search, chatbots, and many more AI applications.",
+ "time": "Feb 19, 2024",
+ "link": "https://milvus.io/blog/what-milvus-version-to-start-with.md",
+ "title": "What Milvus version to start with",
+ "abstract": "Selecting the appropriate Milvus version is foremost to the success of any project leveraging vector search technology. With different Milvus versions tailored to varying requirements, understanding the importance of selecting the correct version is crucial for achieving the desired results.",
"imgSrc": "https://zilliz-cms.s3.us-west-2.amazonaws.com/2_0_cover_bbc582b347.jpg"
}]
},
"section5": {
"title": "What's new in docs",
- "date": "Oct 2021",
+ "date": "March 2024",
"list": [
"With the release of Milvus 2.0-RC7, we've made a number of compatibility and performance updates. Check out the Release Notes for details. Note that RC7 is not compatible with previous versions of Milvus 2.0.0 due to changes made to storage format.",
"Tutorials are updated with direct links to Jupyter notebook, GitHub repos, and online demos. Read the latest tutorial about how to use Milvus to build a DNA sequence classification model.",
diff --git a/preview/site/en/home/home.md b/preview/site/en/home/home.md
index 04aba4bd6..fe4ff885a 100644
--- a/preview/site/en/home/home.md
+++ b/preview/site/en/home/home.md
@@ -14,6 +14,18 @@ id: home.md
+$ kubectl logs `pod_name` to get the stderr log of the pod before deleting the cluster and all resources.
+
+
+
## Get Started
+
+
+
+Try Managed Milvus For Free!
+Try Zilliz Cloud for free! The easiest way to experience Milvus!
+
@@ -27,7 +39,7 @@ id: home.md
-
+
-
## Recommended articles
Quick Start
@@ -46,7 +58,6 @@ id: home.md
@@ -54,11 +65,11 @@ id: home.md
+
+This guide showcases how [VoyageAI's Embedding API](https://docs.voyageai.com/docs/embeddings) can be used with Milvus vector database to conduct semantic search on text.
+
+## Getting started
+Before you start, make sure you have the Voyage API key ready, or you get one from the [VoyageAI website](https://dash.voyageai.com/api-keys).
+
+The data used in this example are book titles. You can download the dataset [here](https://www.kaggle.com/datasets/jealousleopard/goodreadsbooks) and put it in the same directory where you run the following code.
+
+First, install the package for Milvus and Voyage AI:
+
+
+```python
+$ pip install --upgrade voyageai pymilvus
+```
+
+")
+
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+vectors = voyage_client.embed(texts=docs, model=MODEL_NAME, truncation=False).embeddings
+
+# Prepare data to be stored in Milvus vector database.
+# We can store the id, vector representation, raw text and labels such as "subject" in this case in Milvus.
+data = [
+ {"id": i, "vector": vectors[i], "text": docs[i], "subject": "history"}
+ for i in range(len(docs))
+]
+
+
+# Connect to Milvus, all data is stored in a local file named "milvus_voyage_demo.db"
+# in current directory. You can also connect to a remote Milvus server following this
+# instruction: https://milvus.io/docs/install_standalone-docker.md.
+milvus_client = MilvusClient(uri="milvus_voyage_demo.db")
+COLLECTION_NAME = "demo_collection" # Milvus collection name
+# Create a collection to store the vectors and text.
+if milvus_client.has_collection(collection_name=COLLECTION_NAME):
+ milvus_client.drop_collection(collection_name=COLLECTION_NAME)
+milvus_client.create_collection(collection_name=COLLECTION_NAME, dimension=DIMENSION)
+
+# Insert all data into Milvus vector database.
+res = milvus_client.insert(collection_name="demo_collection", data=data)
+
+print(res["insert_count"])
+```
+
+
- `local`: Store dumped files on local disks.
- `remote`: Store dumped files on object storage. | + | `target.remote.outputDir` | Output directory path in the cloud storage bucket. | + | `target.remote.cloud` | Cloud storage service provider. Example values: `aws`, `gcp`, `azure`. | + | `target.remote.region` | Cloud storage region. It can be any value if you use local MinIO. | + | `target.remote.bucket` | Bucket name for storing data. The value must be the same as the config in Milvus 2.x. For more information, refer to [System Configurations](https://milvus.io/docs/configure_minio.md#miniobucketName). | + | `target.remote.useIAM` | Whether to use an IAM Role for connection. | + | `target.remote.checkBucket` | Whether to check if the specified bucket exists in object storage. | + | `target.milvus2x` | Connection configs for the target Milvus 2.x server. | + | `target.milvus2x.endpoint` | Address of the target Milvus server. | + | `target.milvus2x.username` | Username for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + | `target.milvus2x.password` | Password for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + +## Start the migration task + +Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + +```bash +./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml +``` + +The following is an example of a successful migration log output: + +```bash +[task/load_base_task.go:94] ["[LoadTasker] Dec Task Processing-------------->"] [Count=0] [fileName=testfiles/output/zwh/migration/test_mul_field4/data_1_1.json] [taskId=442665677354739304] +[task/load_base_task.go:76] ["[LoadTasker] Progress Task --------------->"] [fileName=testfiles/output/zwh/migration/test_mul_field4/data_1_1.json] [taskId=442665677354739304] +[dbclient/cus_field_milvus2x.go:86] ["[Milvus2x] begin to ShowCollectionRows"] +[loader/cus_milvus2x_loader.go:66] ["[Loader] Static: "] [collection=test_mul_field4_rename1] [beforeCount=50000] [afterCount=100000] [increase=50000] +[loader/cus_milvus2x_loader.go:66] ["[Loader] Static Total"] ["Total Collections"=1] [beforeTotalCount=50000] [afterTotalCount=100000] [totalIncrease=50000] +[migration/es_starter.go:25] ["[Starter] migration ES to Milvus finish!!!"] [Cost=80.009174459] +[starter/starter.go:106] ["[Starter] Migration Success!"] [Cost=80.00928425] +[cleaner/remote_cleaner.go:27] ["[Remote Cleaner] Begin to clean files"] [bucket=a-bucket] [rootPath=testfiles/output/zwh/migration] +[cmd/start.go:32] ["[Cleaner] clean file success!"] +``` + +## Verify the result + +Once the migration task is executed, you can make API calls or use Attu to view the number of entities migrated. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). + +## Field mapping reference + +Review the table below to understand how field types in Elasticsearch indexes are mapped to field types in Milvus collections. + +For more information on supported data types in Milvus, refer to [Supported data types](https://milvus.io/docs/schema.md#Supported-data-types). + +| Elasticsearch Field Type | Milvus Field Type | Description | +| --- | --- | --- | +| dense_vector | FloatVector | Vector dimensions remain unchanged during migration. | +| keyword | VarChar | Set Max Length (1 to 65,535). Strings exceeding the limit can trigger migration errors. | +| text | VarChar | Set Max Length (1 to 65,535). Strings exceeding the limit can trigger migration errors. | +| long | Int64 | - | +| integer | Int32 | - | +| double | Double | - | +| float | Float | - | +| boolean | Bool | - | +| object | JSON | - | \ No newline at end of file diff --git a/preview/site/en/migrate/f2m.md b/preview/site/en/migrate/f2m.md index 573427f9d..637d7123c 100644 --- a/preview/site/en/migrate/f2m.md +++ b/preview/site/en/migrate/f2m.md @@ -1,76 +1,135 @@ --- id: f2m.md -title: Migrate from Faiss to Milvus +title: From Faiss related_key: Faiss, migrate, import summary: Learn how to migrate Faiss data to Milvus. --- -# Migrate Data from Faiss to Milvus +# From Faiss -This topic describes how to import data from Faiss to Milvus using [MilvusDM](migrate_overview.md), an open-source tool specifically designed for Milvus data migration. +This guide provides a comprehensive, step-by-step process for migrating data from Faiss to Milvus 2.x. By following this guide, you will be able to efficiently transfer your data, leveraging Milvus 2.x advanced features and improved performance. ## Prerequisites -You need to [install MilvusDM](milvusdm_install.md) before migrating Milvus data. - -## 1. Download YAML file - -Download the `F2M.yaml` file. - -``` -$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/F2M.yaml -``` - -## 2. Set the parameters - -Configuration parameters include: - -| Parameter | Description | Example | -| ------------------------- | ----------------------------------------- | ---------------------------- | -| `milvus_version` | Version of Milvus. | 2.0.0 | -| `data_path` | Path to the data in Faiss. | '/home/user/data/faiss.index' | -| `data_dir` | Directory of the HDF5 files. Set either `data_path` or `data_dir`. | '/Users/zilliz/Desktop/HDF5_data' | -| `dest_host` | Milvus server address. | '127.0.0.1' | -| `dest_port` | Milvus server port. | 19530 | -| `mode` | Mode of migration, including `skip`, `append`, and `overwrite`. This parameter works only when the specified collection name exists in the Milvus library.
index_file_size: 1024
metric_type: 'HAMMING'" | - -### Example - -The following example of configuration is for your reference. +- **Software versions**: + - Source Faiss + - Target Milvus: 2.x + - For installation details, see [Installing Faiss](https://github.com/facebookresearch/faiss/blob/main/INSTALL.md) and [Install Milvus](https://milvus.io/docs/install_standalone-docker.md). +- **Required tools**: + - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md). + +## Configure the migration + +Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory. + +```yaml +dumper: # configs for the migration job. + worker: + limit: 2 + workMode: faiss # operational mode of the migration job. + reader: + bufferSize: 1024 + writer: + bufferSize: 1024 +loader: + worker: + limit: 2 +source: # configs for the source Faiss index. + mode: local + local: + faissFile: ./testfiles/faiss/faiss_ivf_flat.index + +target: # configs for the target Milvus collection. + create: + collection: + name: test1w + shardsNums: 2 + dim: 256 + metricType: L2 + + mode: remote + remote: + outputDir: testfiles/output/ + cloud: aws + endpoint: 0.0.0.0:9000 + region: ap-southeast-1 + bucket: a-bucket + ak: minioadmin + sk: minioadmin + useIAM: false + useSSL: false + checkBucket: true + milvus2x: + endpoint: localhost:19530 + username: xxxxx + password: xxxxx ``` -F2M: - milvus_version: 2.0.0 - data_path: '/home/data/faiss1.index' - dest_host: '127.0.0.1' - dest_port: 19530 - mode: 'append' - dest_collection_name: 'test' - dest_partition_name: '' - collection_parameter: - dimension: 256 - index_file_size: 1024 - metric_type: 'L2' -``` - - -## 3. Migrate data from Faiss to Milvus - -Run MilvusDM to import data from Faiss to Milvus with the following command. - -``` -$ milvusdm --yaml F2M.yaml -``` - - -## What's next -- If you are interested in migrating data in other forms into Milvus, - - Learn how to [Migrate Data from HDF5 to Milvus](h2m.md). -- If you are looking for information about how to migrate data from Milvus 1.x to Milvus 2.0, - - Learn [version migration](m2m.md). -- If you are interested in learning more about the data migration tool, - - Read the overview of [MilvusDM](migrate_overview.md). +The following table describes the parameters in the example config file. For a full list of configs, refer to [Milvus Migration: Faiss to Milvus 2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_FAISS.md#migrationyaml-reference). + +- `dumper` + + | Parameter | Description | + | --- | --- | + | `dumper.worker.limit` | The concurrency of dumper threads. | + | `dumper.worker.workMode` | The operational mode of the migration job. Set to faiss when migrating from Faiss indexes. | + | `dumper.worker.reader.bufferSize` | Buffer size to read from Faiss in each batch. Unit: KB. | + | `dumper.worker.writer.bufferSize` | Buffer size to write to Milvus in each batch. Unit: KB. | + +- `loader` + + | Parameter | Description | + | --- | --- | + | `loader.worker.limit` | The concurrency of loader threads. | + +- `source` + + | Parameter | Description | + | --- | --- | + | `source.mode` | Specifies where the source files are read from. Valid values:
- `local`: reads files from a local disk.
- `remote`: reads files from remote storage. | + | `source.local.faissFile` | The directory path where the source files are located. For example, `/db/faiss.index`. | + +- `target` + + | Parameter | Description | + | --- | --- | + | `target.create.collection.name` | Name of the Milvus collection. | + | `target.create.collection.shardsNums` | Number of shards to be created in the collection. For more information on shards, refer to [Terminology](https://milvus.io/docs/glossary.md#Shard). | + | `target.create.collection.dim` | Dimension of the vector field. | + | `target.create.collection.metricType` | Metric type used to measure similarities between vectors. For more information, refer to [Terminology](https://milvus.io/docs/glossary.md#Metric-type). | + | `target.mode` | Storage location for dumped files. Valid values:
- `local`: Store dumped files on local disks.
- `remote`: Store dumped files on object storage. | + | `target.remote.outputDir` | Output directory path in the cloud storage bucket. | + | `target.remote.cloud` | Cloud storage service provider. Example values: `aws`, `gcp`, `azure`. | + | `target.remote.endpoint` | Endpoint of Milvus 2.x storage. | + | `target.remote.region` | Cloud storage region. It can be any value if you use local MinIO. | + | `target.remote.bucket` | Bucket name for storing data. The value must be the same as the config in Milvus 2.x. For more information, refer to [System Configurations](https://milvus.io/docs/configure_minio.md#miniobucketName). | + | `target.remote.ak` | Access key for Milvus 2.x storage. | + | `target.remote.sk` | Secret key for Milvus 2.x storage. | + | `target.remote.useIAM` | Whether to use an IAM Role for connection. | + | `target.remote.useSSL` | Whether to enable SSL when connecting to Milvus 2.x. For more information, refer to [Encryption in Transit](https://milvus.io/docs/tls.md#Encryption-in-Transit). | + | `target.remote.checkBucket` | Whether to check if the specified bucket exists in object storage. | + | `target.milvus2x.endpoint` | Address of the target Milvus server. | + | `target.milvus2x.username` | Username for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + | `target.milvus2x.password` | Password for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + +## Start the migration task + +1. Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + + ```bash + ./milvus-migration dump --config=/{YourConfigFilePath}/migration.yaml + ``` + + The command above converts the Faiss index data into NumPy files, and then uses the [bulkInsert](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md) operation to write the data to the target bucket. + +2. Once NumPy files are generated, import these files into Milvus 2.x with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + + ```bash + ./milvus-migration load --config=/{YourConfigFilePath}/migration.yaml + ``` + + +## Verify the result + +Once the migration task is executed, you can make API calls or use Attu to view the number of entities migrated. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). diff --git a/preview/site/en/migrate/from-m2x.md b/preview/site/en/migrate/from-m2x.md new file mode 100644 index 000000000..d9ac26bd7 --- /dev/null +++ b/preview/site/en/migrate/from-m2x.md @@ -0,0 +1,148 @@ +--- +id: from-m2x.md +summary: This guide provides a comprehensive, step-by-step process for migrating data from Milvus 2.3.x to Milvus 2.3.x or above. +title: From Milvus 2.3.x +--- + +# From Milvus 2.3.x + +This guide provides a comprehensive, step-by-step process for migrating data from Milvus 2.3.x to Milvus 2.3.x or above. + +## Prerequisites + +- **Software versions**: + - Source Milvus: 2.3.0+ (The tool uses the iterator to fetch source collection data, requiring source Milvus to be version 2.3.0 or above.) + - Target Milvus: 2.3.0+ +- **Required tools**: + - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md). +- **Data preparation**: + - Ensure that the source Milvus collection is loaded and ready for data export. + - If the target Milvus does not contain a collection corresponding to the source collection, the [milvus-migration](https://github.com/zilliztech/milvus-migration) tool will automatically create it. Note that after migration, the target collection will not be indexed, and you must manually index the collection afterward. + +## Configure the migration file + +Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory. + +```yaml +dumper: + worker: + workMode: milvus2x + reader: + bufferSize: 500 + +meta: + mode: config + version: 2.3.0 + collection: src_table_name + +source: + milvus2x: + endpoint: {milvus2x_domain}:{milvus2x_port} + username: xxxx + password: xxxxx + +target: + milvus2x: + endpoint: {milvus2x_domain}:{milvus2x_port} + username: xxxx + password: xxxxx +``` + +The following table describes the parameters in the example config file. For more information, refer to [Milvus Migration: Milvus2.x to Milvus2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_2X.md#milvus-migration-milvus2x-to-milvus2x). + +- `dumper` + + | Parameter | Description | + | --- | --- | + | `dumper.worker.workMode` | The operational mode of the migration job. Set to milvus2x when migrating from Milvus 2.x. | + | `dumper.worker.reader.bufferSize` | Buffer size to read from Milvus 2.x in each batch. | + +- `meta` + + | Parameter | Description | + | --- | --- | + | `meta.mode` | Specifies where the meta file is read from. Set to config, indicating that the meta config can be obtained from this migration.yaml file. | + | `meta.version` | Source Milvus version. Set to 2.3.0 or above. | + | `meta.collection` | Source collection name. | + +- `source` + + | Parameter | Description | + | --- | --- | + | `source.milvus2x.endpoint` | Address of the source Milvus server. | + | `source.milvus2x.username` | Username for the source Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | + | `source.milvus2x.password` | Password for the source Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | + +- `target` + + | Parameter | Description | + | --- | --- | + | `target.milvus2x.endpoint` | Address of the target Milvus server. | + | `target.milvus2x.username` | Username for the target Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | + | `target.milvus2x.password` | Password for the target Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | + +## Start the migration task + +You have two options to start the migration task - using CLI or making API requests. Choose the one that best fits your needs. + +### Option 1: Using CLI + +Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + +```bash +./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml +``` + +Monitor the logs for progress updates. Successful migration logs should include entries like: + +```bash +[INFO] [migration/milvus2x_starter.go:79] ["=================>JobProcess!"] [Percent=100] +[INFO] [migration/milvus2x_starter.go:27] ["[Starter] migration Milvus2x to Milvus2x finish!!!"] [Cost=94.877717375] +[INFO] [starter/starter.go:109] ["[Starter] Migration Success!"] [Cost=94.878243583] +``` + +### Option 2: Making API requests + +You can also use the Restful API to execute the migration. Start the API server with: + +```bash +./milvus-migration server run -p 8080 +``` + +Once the server starts successfully, place the `migration.yaml` file in the `configs/` directory of the project and start the migration using: + +```bash +curl -XPOST http://localhost:8080/api/v1/start +``` + +## Verify the result + +After the migration task is completed, use Attu to view the number of entities migrated. Additionally, you can create indexes and load collections in Attu. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). + +## Additional configuration options + +In addition to the basic configurations mentioned above, you can also add additional settings based on your specific requirements. + +- **Selective field migration**: If you need to migrate only specific fields in a collection rather than all fields, specify the fields to be migrated in the `meta` section of the `migration.yaml` file. + + ```yaml + meta: + fields: + - name: id + - name: title_vector + - name: reading_time + ``` + +- **Custom target collection**: To customize the properties of the target collection, add the related configurations in the `meta` section of the `migration.yaml` file. + + ```yaml + meta: + milvus: + collection: target_collection_name + shardNum: 2 + closeDynamicField: false + consistencyLevel: Customized + ``` + + +For detailed information, refer to [Milvus Migration: Milvus2.x to Milvus2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_2X.md#milvus-migration-milvus2x-to-milvus2x). diff --git a/preview/site/en/migrate/m2m.md b/preview/site/en/migrate/m2m.md new file mode 100644 index 000000000..c01a32535 --- /dev/null +++ b/preview/site/en/migrate/m2m.md @@ -0,0 +1,156 @@ +--- +id: m2m.md +summary: This guide provides a comprehensive, step-by-step process for migrating data from Milvus 1.x (including 0.9.x and above) to Milvus 2.x. +title: From Milvus 1.x +--- + +# From Milvus 1.x + +This guide provides a comprehensive, step-by-step process for migrating data from Milvus 1.x (including 0.9.x and above) to Milvus 2.x. By following this guide, you will be able to efficiently transfer your data, leveraging Milvus 2.x advanced features and improved performance. + +## Prerequisites + +- **Software versions**: + - Source Milvus: 0.9.x to 1.x + - Target Milvus: 2.x +- **Required tools**: + - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md). + +## Export metadata of the source Milvus installation + +To prepare migration data for Milvus 0.9.x through 1.x, stop the source Milvus or at least stop performing any DML operations in it. + +1. Export metadata of the source Milvus installation to `meta.json`. + + - For those installations using MySQL as the backend, run + + ```bash + ./milvus-migration export -m "user:password@tcp(adderss)/milvus?charset=utf8mb4&parseTime=True&loc=Local" -o outputDir + ``` + + - For those installations using SQLite as the backend, run + + ```bash + ./milvus-migration export -s /milvus/db/meta.sqlite -o outputDir + ``` + +2. Copy the `tables` folder of your Milvus installation, then move both `meta.json` and the `tables` folder to an empty folder. + + Once this step is done, the structure of the empty folder should look like this: + + ``` + migration_data + ├── meta.json + └── tables + ``` + +3. Upload the folder prepared in the preceding step to an S3 block storage bucket or directly use this local folder in the next section. + +## Configure the migration file + +Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory. + +```yaml +dumper: + worker: + limit: 2 + workMode: milvus1x + reader: + bufferSize: 1024 + writer: + bufferSize: 1024 +loader: + worker: + limit: 16 +meta: + mode: local + localFile: /outputDir/test/meta.json +source: + mode: local + local: + tablesDir: /db/tables/ +target: + mode: remote + remote: + outputDir: "migration/test/xx" + ak: xxxx + sk: xxxx + cloud: aws + region: us-west-2 + bucket: xxxxx + useIAM: true + checkBucket: false + milvus2x: + endpoint: "{yourMilvus2_xServerAddress}:{port}" + username: xxxx + password: xxxx +``` + +The following table describes the parameters in the example config file. For a full list of configs, refer to [Milvus Migration: Milvus1.x to Milvus 2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_1X.md#migrationyaml-reference). + +- `dumper` + + | Parameter | Description | + | --- | --- | + | `dumper.worker.limit` | The concurrency of dumper threads. | + | `dumper.worker.workMode` | The operational mode of the migration job. Set to `milvus1x` when migrating from Milvus 1.x. | + | `dumper.worker.reader.bufferSize` | Buffer size to read from Milvus 1.x in each batch. Unit: KB. | + | `dumper.worker.writer.bufferSize` | Buffer size to write to Milvus 2.x in each batch. Unit: KB. | + +- `loader` + + | Parameter | Description | + | --- | --- | + | `loader.worker.limit` | The concurrency of loader threads. | + +- `meta` + + | Parameter | Description | + | --- | --- | + | `meta.mode` | Specifies where the meta file meta.json is read from. Valid values: `local`, `remote`, `mysql`, `sqlite`. | + | `meta.localFile` | Local directory path where the `meta.json` file resides. This config is used only when `meta.mode` is set to `local`. For other meta configs, refer to [README_1X](https://github.com/zilliztech/milvus-migration/blob/main/README_1X.md#meta). | + +- `source` + + | Parameter | Description | + | --- | --- | + | `source.mode` | Specifies where the source files are read from. Valid values:
- `local`: reads files from a local disk.
- `remote`: reads files from remote storage. | + | `source.local.tablesDir` | The directory path where the source files are located. For example, `/db/tables/`. | + +- `target` + + | Parameter | Description | + | --- | --- | + | `target.mode` | Storage location for dumped files. Valid values:
- `local`: Store dumped files on local disks.
- `remote`: Store dumped files on object storage. | + | `target.remote.outputDir` | Output directory path in the cloud storage bucket. | + | `target.remote.ak` | Access key for Milvus 2.x storage. | + | `target.remote.sk` | Secret key for Milvus 2.x storage. | + | `target.remote.cloud` | Cloud storage service provider. Example values: `aws`, `gcp`, `azure`. | + | `target.remote.region` | Cloud storage region. It can be any value if you use local MinIO. | + | `target.remote.bucket` | Bucket name for storing data. The value must be the same as the config in Milvus 2.x. For more information, refer to [System Configurations](https://milvus.io/docs/configure_minio.md#miniobucketName). | + | `target.remote.useIAM` | Whether to use an IAM Role for connection. | + | `target.remote.checkBucket` | Whether to check if the specified bucket exists in object storage. | + | `target.milvus2x.endpoint` | Address of the target Milvus server. | + | `target.milvus2x.username` | Username for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + | `target.milvus2x.password` | Password for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + +## Start the migration task + +1. Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + + ```bash + ./milvus-migration dump --config=/{YourConfigFilePath}/migration.yaml + ``` + + The command above converts the source data in Milvus 1.x into NumPy files, and then uses the [bulkInsert](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md) operation to write the data to the target bucket. + +2. Once NumPy files are generated, import these files into Milvus 2.x with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + + ```bash + ./milvus-migration load --config=/{YourConfigFilePath}/migration.yaml + ``` + + +## Verify the result + +Once the migration task is executed, you can make API calls or use Attu to view the number of entities migrated. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). diff --git a/preview/site/en/migrate/migrate_overview.md b/preview/site/en/migrate/migrate_overview.md index 59a39eeb9..65a2d9be1 100644 --- a/preview/site/en/migrate/migrate_overview.md +++ b/preview/site/en/migrate/migrate_overview.md @@ -1,56 +1,51 @@ --- id: migrate_overview.md -summary: MilvusDM allows data migration between Milvus and many other sources of data. +summary: This article provides an overview of the Milvus-migration tool, including supported migrations, features, and architecture. +title: Milvus Migration Overview --- -# Overview -[MilvusDM](https://github.com/milvus-io/milvus-tools) (Milvus Data Migration) is an open-source tool designed specifically for importing and exporting data with Milvus. MilvusDM allows you to migrate data in a specific collection or partition. - - colors = Arrays.asList("green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey");
+List data = new ArrayList<>();
+
+for (int i=0; i<1000; i++) {
+ Random rand = new Random();
+ String current_color = colors.get(rand.nextInt(colors.size()-1));
+ Long current_tag = rand.nextLong(8999L) + 1000L;
+
+ // Generate an random-sized array
+ Long capacity = rand.nextLong(5L) + 1L;
+ List current_coord = new ArrayList<>();
+ current_coord.add(rand.nextLong(40L) + 1L);
+ current_coord.add(rand.nextLong(40L) + 1L);
+ for (int j=3; j Math.floor(Math.random() * 40))
+
+ data.push({
+ id: i,
+ vector: Array(5).fill(0).map(() => Math.random()),
+ color: current_color,
+ color_tag: current_tag,
+ color_coord: current_coord,
+ })
+}
+
+console.log(data[0])
+```
+
+You can view the structure of the generated data by checking its first entry.
+
+```
+{
+ id: 0,
+ vector: [
+ 0.0338537420906162,
+ 0.6844108238358322,
+ 0.28410588909961754,
+ 0.09752595400212116,
+ 0.22671013058761114
+ ],
+ color: 'orange',
+ color_tag: 5677,
+ color_coord: [ 3, 0, 18, 29 ]
+}
+```
+
+"
+# }
+```
+
+```java
+import io.milvus.v2.client.ConnectConfig;
+import io.milvus.v2.client.MilvusClientV2;
+import io.milvus.v2.common.DataType;
+import io.milvus.v2.common.IndexParam;
+import io.milvus.v2.service.collection.request.AddFieldReq;
+import io.milvus.v2.service.collection.request.CreateCollectionReq;
+import io.milvus.v2.service.collection.request.GetLoadStateReq;
+
+String CLUSTER_ENDPOINT = "http://localhost:19530";
+
+// 1. Connect to Milvus server
+ConnectConfig connectConfig = ConnectConfig.builder()
+ .uri(CLUSTER_ENDPOINT)
+ .build();
+
+MilvusClientV2 client = new MilvusClientV2(connectConfig);
+
+// 2. Create a collection in customized setup mode
+
+// 2.1 Create schema
+CreateCollectionReq.CollectionSchema schema = client.createSchema();
+
+// 2.2 Add fields to schema
+schema.addField(AddFieldReq.builder()
+ .fieldName("id")
+ .dataType(DataType.Int64)
+ .isPrimaryKey(true)
+ .autoID(false)
+ .build());
+
+schema.addField(AddFieldReq.builder()
+ .fieldName("vector")
+ .dataType(DataType.FloatVector)
+ .dimension(5)
+ .build());
+
+schema.addField(AddFieldReq.builder()
+ .fieldName("color")
+ .dataType(DataType.VarChar)
+ .maxLength(512)
+ .build());
+
+schema.addField(AddFieldReq.builder()
+ .fieldName("color_tag")
+ .dataType(DataType.Int64)
+ .build());
+
+// highlight-start
+schema.addField(AddFieldReq.builder()
+ .fieldName("color_coord")
+ .dataType(DataType.Array)
+ .elementType(DataType.Int64)
+ .maxCapacity(5)
+ .build());
+// highlight-end
+
+// 2.3 Prepare index parameters
+IndexParam indexParamForIdField = IndexParam.builder()
+ .fieldName("id")
+ .indexType(IndexParam.IndexType.STL_SORT)
+ .build();
+
+IndexParam indexParamForVectorField = IndexParam.builder()
+ .fieldName("vector")
+ .indexType(IndexParam.IndexType.IVF_FLAT)
+ .metricType(IndexParam.MetricType.IP)
+ .extraParams(Map.of("nlist", 1024))
+ .build();
+
+IndexParam indexParamForColorField = IndexParam.builder()
+ .fieldName("color")
+ .indexType(IndexParam.IndexType.TRIE)
+ .build();
+
+List indexParams = new ArrayList<>();
+indexParams.add(indexParamForIdField);
+indexParams.add(indexParamForVectorField);
+indexParams.add(indexParamForColorField);
+
+// 2.4 Create a collection with schema and index parameters
+CreateCollectionReq customizedSetupReq = CreateCollectionReq.builder()
+ .collectionName("test_collection")
+ .collectionSchema(schema)
+ .indexParams(indexParams)
+ .build();
+
+client.createCollection(customizedSetupReq);
+
+// 2.5 Check if the collection is loaded
+GetLoadStateReq getLoadStateReq = GetLoadStateReq.builder()
+ .collectionName("test_collection")
+ .build();
+
+Boolean isLoaded = client.getLoadState(getLoadStateReq);
+
+System.out.println(isLoaded);
+
+// Output:
+// true
+```
+
+```javascript
+const { MilvusClient, DataType, sleep } = require("@zilliz/milvus2-sdk-node")
+
+const address = "http://localhost:19530"
+
+// 1. Set up a Milvus Client
+client = new MilvusClient({address});
+
+// 2. Create a collection
+// 2.1 Define fields
+const fields = [
+ {
+ name: "id",
+ data_type: DataType.Int64,
+ is_primary_key: true,
+ auto_id: false
+ },
+ {
+ name: "vector",
+ data_type: DataType.FloatVector,
+ dim: 5
+ },
+ {
+ name: "color",
+ data_type: DataType.VarChar,
+ max_length: 512
+ },
+ {
+ name: "color_tag",
+ data_type: DataType.Int64,
+ },
+// highlight-start
+ {
+ name: "color_coord",
+ data_type: DataType.Array,
+ element_type: DataType.Int64,
+ max_capacity: 5
+ }
+// highlight-end
+]
+
+// 2.2 Prepare index parameters
+const index_params = [{
+ field_name: "vector",
+ index_type: "IVF_FLAT",
+ metric_type: "IP",
+ params: { nlist: 1024}
+}]
+
+// 2.3 Create a collection with fields and index parameters
+res = await client.createCollection({
+ collection_name: "test_collection",
+ fields: fields,
+ index_params: index_params
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+
+res = await client.getLoadState({
+ collection_name: "test_collection",
+})
+
+console.log(res.state)
+
+// Output
+//
+// LoadStateLoaded
+//
+```
+
+## Insert field values
+
+After creating a collection, you can insert arrays such as the one demonstrated in [Overview](#overview).
+
+
## Summary
-- Compared with Milvus 2.1, the QPS of Milvus 2.2.0 increases over 48% in cluster mode and over 75% in standalone mode.
+- Comparing with Milvus 2.1, the QPS of Milvus 2.2.0 increases over 48% in cluster mode and over 75% in standalone mode.
- Milvus 2.2.0 has an impressive capability to scale up and scale out:
- QPS increases linearly when expanding CPU cores from 8 to 32.
- QPS increases linearly when expanding Querynode replicas from 1 to 8.
diff --git a/preview/site/en/reference/bitset.md b/preview/site/en/reference/bitset.md
index 3cace29a9..0149c6605 100644
--- a/preview/site/en/reference/bitset.md
+++ b/preview/site/en/reference/bitset.md
@@ -1,11 +1,12 @@
---
id: bitset.md
summary: Learn about bitsets in Milvus.
+title: Bitset
---
# Bitset
-This topic introduces the bitset mechanism that helps enable key functionalities like [Time Travel](timetravel_ref.md), attribute filtering, and [delete operations](https://milvus.io/blog/2022-02-07-how-milvus-deletes-streaming-data-in-distributed-cluster.md) in Milvus.
+This topic introduces the bitset mechanism that helps enable key functionalities like attribute filtering and [delete operations](https://milvus.io/blog/2022-02-07-how-milvus-deletes-streaming-data-in-distributed-cluster.md) in Milvus.
## Overview
@@ -25,10 +26,6 @@ As bitsets contain only two possible values, they are perfect for storing result
Bitsets serve as a compact way to store information about whether a row in a segment is deleted. Deleted entities are marked with `1` in the corresponding bitset, which [will not be computed](https://milvus.io/blog/deleting-data-in-milvus.md) during a search or query.
-### Query with Time Travel
-
-When you search with Time Travel, Milvus uses bitsets to store information about whether data in a segment meet your timestamp requirement in `travel_timestamp`. Data are marked as `1` if their timestamp is larger than or equals to the requirement, meaning they are present at the given time. The exact process is more complicated for Time Travel to be as efficient as it can be. See [Bitset for timestamp](https://milvus.io/docs/v2.1.x/timetravel_ref.md#Bitset-for-timestamp) for more information.
-
## Examples
Here we present three examples that illustrate how bitsets are used in Milvus, with references to all three major implementations of bitsets discussed above. In all three cases, there is a segment with 8 entities and then a series of data manipulation language (DML) events takes place in the order shown below.
@@ -88,6 +85,5 @@ Now we have two bitsets after Time Travel and attribute filtering: `filter_bitse
Now that you know how bitsets work in Milvus, you might also want to:
-- Learn how to [use strings to filter](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md) your search results, or refer to [Hybrid Search](https://milvus.io/docs/v2.1.x/hybridsearch.md) on our docs.
-- Learn more about how to [search with Time Travel](https://milvus.io/docs/v2.1.x/timetravel.md).
+- Learn how to [use strings to filter](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md) your search results, or refer to [Hybrid Search](https://milvus.io/docs/hybridsearch.md) on our docs.
- Understand [how data are processed](https://milvus.io/docs/v2.1.x/data_processing.md) in Milvus.
diff --git a/preview/site/en/reference/boolean.md b/preview/site/en/reference/boolean.md
index 4e5aabba0..3c9807fdf 100644
--- a/preview/site/en/reference/boolean.md
+++ b/preview/site/en/reference/boolean.md
@@ -1,13 +1,14 @@
---
id: boolean.md
summary: Learn about boolean expression rules in Milvus.
+title: Scalar Filtering Rules
---
-# Boolean Expression Rules
+# Scalar Filtering Rules
## Overview
-A predicate expression outputs a boolean value. Milvus conducts scalar filtering by searching with predicates. A predicate expression, when evaluated, returns either TRUE or FALSE. View [Python SDK API Reference](/api-reference/pymilvus/v2.2.2/Collection/query().md) for instruction on using predicate expressions.
+A predicate expression outputs a boolean value. Milvus conducts scalar filtering by searching with predicates. A predicate expression, when evaluated, returns either TRUE or FALSE. View [Python SDK API Reference](/api-reference/pymilvus/v2.4.x/Collection/query().md) for instruction on using predicate expressions.
[EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) grammar rules describe boolean expressions rules:
@@ -36,6 +37,19 @@ CompareExpr = IDENTIFIER CmpOp IDENTIFIER
CmpOpRestricted = "<" | "<=";
CmpOp = ">" | ">=" | "<" | "<=" | "=="| "!=";
MatchOp = "like" | "LIKE";
+JsonArrayOps = JsonDefs "(" IDENTIFIER "," JsonExpr | JsonArray ")";
+JsonArrayDefs = "json_contains" | "JSON_CONTAINS"
+ | "json_contains_all" | "JSON_CONTAINS_ALL"
+ | "json_contains_any" | "JSON_CONTAINS_ANY";
+JsonExpr = Constant | ConstantArray | STRING | BOOLEAN;
+JsonArray = "[" JsonExpr { "," JsonExpr } "]";
+ArrayOps = ArrayDefs "(" IDENTIFIER "," ArrayExpr | Array ")";
+ArrayDefs = "array_contains" | "ARRAY_CONTAINS"
+ | "array_contains_all" | "ARRAY_CONTAINS_ALL"
+ | "array_contains_any" | "ARRAY_CONTAINS_ANY"
+ | "array_length" | "ARRAY_LENGTH";
+ArrayExpr = Constant | ConstantArray | STRING | BOOLEAN;
+Array = "[" ArrayExpr { "," ArrayExpr } "]";
```
The following table lists the description of each symbol mentioned in the above Boolean expression rules.
@@ -46,7 +60,7 @@ The following table lists the description of each symbol mentioned in the above
| = | Definition. |
| , | Concatenation. |
| ; | Termination. |
-| \| | Alternation. |
+| | | Alternation. |
| {...} | Repetition. |
| (...) | Grouping. |
| NIL | Empty. The expression can be an empty string. |
@@ -69,23 +83,22 @@ The following table lists the description of each symbol mentioned in the above
| SingleExpr | SingleExpr, namely single expression, can be either a TermExpr or a CompareExpr. |
| LogicalExpr | A LogicalExpr can be a BinaryLogicalOp on two LogicalExprs, or a UnaryLogicalOp on a single LogicalExpr, or a LogicalExpr grouped within parentheses, or a SingleExpr. The LogicalExpr is defined recursively. |
| Expr | Expr, an abbreviation meaning expression, can be LogicalExpr or NIL. |
-| MatchOp | A MatchOp, namely a match operator, compares a string to a string constant or a string prefix constant. |
+| MatchOp | A MatchOp, namely a match operator, compares a string to a string constant or a string prefix, infix, or suffix constant. |
+| JsonArrayOp | A JsonOp, namely a JSON operator, checks whether the specified identifier contains the specified elements. |
+| ArrayOp | An ArrayOp, namely an array operator, checks whether the specified identifier contains the specified elements. |
## Operators
-### Logical operators:
+### Logical operators
Logical operators perform a comparison between two expressions.
| Symbol| Operation | Example | Description |
| ----------| ------------- | ----------- | ------------------------- |
| 'and' && | and | expr1 && expr2 | True if both expr1 and expr2 are true. |
-| 'or' \|\| | or | expr1 \|\| expr2 | True if either expr1 or expr2 are true. |
+| 'or' || | or | expr1 || expr2 | True if either expr1 or expr2 are true. |
-
-
-
-### Binary arithmetic operators:
+### Binary arithmetic operators
Binary arithmetic operators contain two operands and can perform basic arithmetic operations and return the corresponding result.
@@ -98,8 +111,7 @@ Binary arithmetic operators contain two operands and can perform basic arithmeti
| ** | Power | a ** b | Raise the first operand to the power of the second operand. |
| % | Modulo | a % b | Divide the first operand by the second operand and yield the remainder portion. |
-
-### Relational operators:
+### Relational operators
Relational operators use symbols to check for equality, inequality, or relative order between two expressions.
@@ -112,43 +124,49 @@ Relational operators use symbols to check for equality, inequality, or relative
| <= | Less than or equal | a <= b | True if a is less than or equal to b. |
| >= | Greater than or equal | a >= b | True if a is greater than or equal to b. |
-
## Operator precedence and associativity
The following table lists the precedence and associativity of operators. Operators are listed top to bottom, in descending precedence.
-
-| Precedence | Operator | Description | Associativity |
-|------------|-----------|---------------|---------------|
-| 1 | + - | UnaryArithOp | Left-to-right |
-| 2 | not | UnaryLogicOp | Right-to-left |
-| 3 | ** | BinaryArithOp | Left-to-right |
-| 4 | * / % | BinaryArithOp | Left-to-right |
-| 5 | + - | BinaryArithOp | Left-to-right |
-| 6 | < <= > >= | CmpOp | Left-to-right |
-| 7 | == != | CmpOp | Left-to-right |
-| 8 | like LIKE | MatchOp | Left-to-right |
-| 9 | && and | BinaryLogicOp | Left-to-right |
-| 10 | \|\| or | BinaryLogicOp | Left-to-right |
-
-
-Expressions are normally evaluated from left to right. Complex expressions are evaluated one at a time. The order in which the expressions are evaluated is determined by the precedence of the operators used.
+| Precedence | Operator | Description | Associativity |
+|------------|---------------------------------------|---------------|---------------|
+| 1 | + - | UnaryArithOp | Left-to-right |
+| 2 | not | UnaryLogicOp | Right-to-left |
+| 3 | ** | BinaryArithOp | Left-to-right |
+| 4 | * / % | BinaryArithOp | Left-to-right |
+| 5 | + - | BinaryArithOp | Left-to-right |
+| 6 | < <= > >= | CmpOp | Left-to-right |
+| 7 | == != | CmpOp | Left-to-right |
+| 8 | like LIKE | MatchOp | Left-to-right |
+| 9 | json_contains JSON_CONTAINS | JsonArrayOp | Left-to-right |
+| 9 | array_contains ARRAY_CONTAINS | ArrayOp | Left-to-right |
+| 10 | json_contains_all JSON_CONTAINS_ALL | JsonArrayOp | Left-to-right |
+| 10 | array_contains_all ARRAY_CONTAINS_ALL | ArrayOp | Left-to-right |
+| 11 | json_contains_any JSON_CONTAINS_ANY | JsonArrayOp | Left-to-right |
+| 11 | array_contains_any ARRAY_CONTAINS_ANY | ArrayOp | Left-to-right |
+| 12 | array_length ARRAY_LENGTH | ArrayOp | Left-to-right |
+| 13 | && and | BinaryLogicOp | Left-to-right |
+| 14 | || or | BinaryLogicOp | Left-to-right |
+
+Expressions are normally evaluated from left to right. Complex expressions are evaluated one at a time. The order in which the expressions are evaluated is determined by the precedence of the operators used.
If an expression contains two or more operators with the same precedence, the operator to the left is evaluated first.
+Parentheses can be nested within expressions. Innermost parenthetical expressions are evaluated first.
## Usage
@@ -172,7 +190,6 @@ Samples of all available boolean expression usage in Milvus are listed as follow
VARCHAR > "str1"
```
-
2. BinaryLogicalOp and parentheses
```
@@ -182,7 +199,9 @@ VARCHAR > "str1"
3. TermExpr and UnaryLogicOp
@@ -85,14 +96,14 @@ id: home.md
+
+## Introduction
+This guide demonstrates how to use an open-source embedding model and large-language model on BentoCloud with Milvus vector database to build a RAG (Retrieval Augmented Generation) application.
+BentoCloud is an AI Inference Platform for fast-moving AI teams, offering fully-managed infrastructure tailored for model inference. It works in conjunction with BentoML, an open-source model serving framework, to facilitate the easy creation and deployment of high-performance model services. In this demo, we use Milvus Lite as vector database, which is the lightweight version of Milvus that can be embedded into your Python application.
+
+## Before you begin
+Milvus Lite is available on PyPI. You can install it via pip for Python 3.8+:
+
+
+```python
+$ pip install -U pymilvus bentoml
+```
+
+
+
+## What is DSPy
+DSPy, introduced by the Stanford NLP Group, stands as a groundbreaking programmatic framework designed to optimize prompts and weights within language models, particularly valuable in scenarios where large language models (LLMs) are integrated across multiple stages of a pipeline. Unlike conventional prompting engineering techniques reliant on manual crafting and tweaking, DSPy adopts a learning-based approach. By assimilating query-answer examples, DSPy generates optimized prompts dynamically, tailored to specific tasks. This innovative methodology enables the seamless reassembly of entire pipelines, eliminating the need for continuous manual prompt adjustments. DSPy's Pythonic syntax offers various composable and declarative modules, simplifying the instruction of LLMs.
+
+## Benefits of using DSPy
+- Programming Approach: DSPy provides a systematic programming approach for LM pipeline development by abstracting pipelines as text transformation graphs instead of just prompting the LLMs. Its declarative modules enable structured design and optimization, replacing the trial-and-error method of traditional prompt templates.
+- Performance Improvement: DSPy demonstrates significant performance gains over existing methods. Through case studies, it outperforms standard prompting and expert-created demonstrations, showcasing its versatility and effectiveness even when compiled to smaller LM models.
+- Modularized Abstraction: DSPy effectively abstracts intricate aspects of LM pipeline development, such as decomposition, fine-tuning, and model selection. With DSPy, a concise program can seamlessly translate into instructions for various models, such as GPT-4, Llama2-13b, or T5-base, streamlining development and enhancing performance.
+
+## Modules
+There are numerous components that contribute to constructing an LLM pipeline. Here, we'll describe some key components to provide a high-level understanding of how DSPy operates.
+
+
+
+Signature: Signatures in DSPy serve as declarative specifications, outlining the input/output behavior of modules, guiding the language model in task execution.
+Module: DSPy modules serve as fundamental components for programs leveraging language models (LMs). They abstract various prompting techniques, such as chain of thought or ReAct, and are adaptable to handle any DSPy Signature. With learnable parameters and the ability to process inputs and produce outputs, these modules can be combined to form larger programs, drawing inspiration from NN modules in PyTorch but tailored for LM applications.
+Optimizer: Optimizers in DSPy fine-tune the parameters of DSPy programs, such as prompts and LLM weights, to maximize specified metrics like accuracy, enhancing program efficiency.
+
+## Why Milvus in DSPy
+DSPy is a powerful programming framework that boosts RAG applications. Such application needs to retrieve useful information to enhance answer quality, which needs vector database. Milvus is a well-known open-source vector database to improve performance and scalability. With MilvusRM, a retriever module in DSPy, integrating Milvus becomes seamless. Now, developers can easily define and optimize RAG programs using DSPy, taking advantage of Milvus' strong vector search capabilities. This collaboration makes RAG applications more efficient and scalable, combining DSPy's programming capabilities with Milvus' search features.
+
+## Examples
+Now, let's walk through a quick example to demonstrate how to leverage Milvus in DSPy for optimizing a RAG application.
+
+### Prerequisites
+Before building the RAG app, install the DSPy and PyMilvus.
+
+
+```python
+$ pip install "dspy-ai[milvus]"
+$ pip install -U pymilvus
+```
+"
+MILVUS_URI = "example.db"
+MILVUS_TOKEN = ""
+
+from pymilvus import MilvusClient, DataType, Collection
+from dspy.retrieve.milvus_rm import openai_embedding_function
+
+client = MilvusClient(uri=MILVUS_URI, token=MILVUS_TOKEN)
+
+if "dspy_example" not in client.list_collections():
+ client.create_collection(
+ collection_name="dspy_example",
+ overwrite=True,
+ dimension=1536,
+ primary_field_name="id",
+ vector_field_name="embedding",
+ id_type="int",
+ metric_type="IP",
+ max_length=65535,
+ enable_dynamic=True,
+ )
+text = requests.get(
+ "https://raw.githubusercontent.com/wxywb/dspy_dataset_sample/master/sample_data.txt"
+).text
+
+for idx, passage in enumerate(text.split("\n")):
+ if len(passage) == 0:
+ continue
+ client.insert(
+ collection_name="dspy_example",
+ data=[
+ {
+ "id": idx,
+ "embedding": openai_embedding_function(passage)[0],
+ "text": passage,
+ }
+ ],
+ )
+```
+
+### Define MilvusRM.
+Now, you need to define the MilvusRM.
+
+
+```python
+from dspy.retrieve.milvus_rm import MilvusRM
+import dspy
+
+retriever_model = MilvusRM(
+ collection_name="dspy_example",
+ uri=MILVUS_URI,
+ token=MILVUS_TOKEN, # ignore this if no token is required for Milvus connection
+ embedding_function=openai_embedding_function,
+)
+turbo = dspy.OpenAI(model="gpt-3.5-turbo")
+dspy.settings.configure(lm=turbo)
+```
+
+### Building signatures
+Now that we have loaded the data, let's start defining the signatures for the sub-tasks of our pipeline. We can identify our simple input `question` and output `answer`, but since we are building a RAG pipeline, we’ll retrieve contextual information from Milvus. So let's define our signature as `context, question --> answer`.
+
+
+```python
+class GenerateAnswer(dspy.Signature):
+ """Answer questions with short factoid answers."""
+
+ context = dspy.InputField(desc="may contain relevant facts")
+ question = dspy.InputField()
+ answer = dspy.OutputField(desc="often between 1 and 5 words")
+```
+
+We include short descriptions for the `context` and `answer` fields to define clearer guidelines on what the model will receive and should generate.
+
+### Building the pipeline
+Now, let's define the RAG pipeline.
+
+
+```python
+class RAG(dspy.Module):
+ def __init__(self, rm):
+ super().__init__()
+ self.retrieve = rm
+
+ # This signature indicates the task imposed on the COT module.
+ self.generate_answer = dspy.ChainOfThought(GenerateAnswer)
+
+ def forward(self, question):
+ # Use milvus_rm to retrieve context for the question.
+ context = self.retrieve(question).passages
+ # COT module takes "context, query" and output "answer".
+ prediction = self.generate_answer(context=context, question=question)
+ return dspy.Prediction(
+ context=[item.long_text for item in context], answer=prediction.answer
+ )
+```
+
+### Executing the pipeline and getting the results
+Now, we’ve built this RAG pipeline. Let's try it out and get results.
+
+
+```python
+rag = RAG(retriever_model)
+print(rag("who write At My Window").answer)
+```
+
+ Townes Van Zandt
+
+
+We can evaluate the quantitative results on the dataset.
+
+
+```python
+from dspy.evaluate.evaluate import Evaluate
+from dspy.datasets import HotPotQA
+
+evaluate_on_hotpotqa = Evaluate(
+ devset=devset, num_threads=1, display_progress=False, display_table=5
+)
+
+metric = dspy.evaluate.answer_exact_match
+score = evaluate_on_hotpotqa(rag, metric=metric)
+print("rag:", score)
+```
+
+### Optimizing the pipeline
+After defining this program, the next step is compilation. This process updates the parameters within each module to enhance performance. The compilation process depends on three critical factors:
+- Training Set: We'll utilize the 20 question-answer examples from our training dataset for this demonstration.
+- Validation Metric: We will establish a simple `validate_context_and_answer` metric. This metric verifies the accuracy of the predicted answer and ensures that the retrieved context includes the answer.
+- Specific Optimizer (Teleprompter): DSPy's compiler incorporates multiple teleprompters designed to optimize your programs effectively.
+
+
+```python
+from dspy.teleprompt import BootstrapFewShot
+
+# Validation logic: check that the predicted answer is correct.# Also check that the retrieved context does contain that answer.
+
+
+def validate_context_and_answer(example, pred, trace=None):
+ answer_EM = dspy.evaluate.answer_exact_match(example, pred)
+ answer_PM = dspy.evaluate.answer_passage_match(example, pred)
+ return answer_EM and answer_PM
+
+
+# Set up a basic teleprompter, which will compile our RAG program.
+teleprompter = BootstrapFewShot(metric=validate_context_and_answer)
+
+# Compile!
+compiled_rag = teleprompter.compile(rag, trainset=trainset)
+
+# Now compiled_rag is optimized and ready to answer your new question!
+# Now, let’s evaluate the compiled RAG program.
+score = evaluate_on_hotpotqa(compiled_rag, metric=metric)
+print(score)
+print("compile_rag:", score)
+```
+
+The Ragas score has increased from its previous value of 50.0 to 52.0, indicating an enhancement in answer quality.
+
+## Summary
+DSPy marks a leap in language model interactions through its programmable interface, which facilitates algorithmic and automated optimization of model prompts and weights. By leveraging DSPy for RAG implementation, adaptability to varying language models or datasets becomes a breeze, drastically reducing the need for tedious manual interventions.
diff --git a/preview/site/en/integrations/integrate_with_fastgpt.md b/preview/site/en/integrations/integrate_with_fastgpt.md
new file mode 100644
index 000000000..84d25592f
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_fastgpt.md
@@ -0,0 +1,54 @@
+---
+id: integrate_with_fastgpt.md
+summary: This tutorial will guide you on how to swiftly deploy your own exclusive FastGPT application using [Milvus](https://milvus.io/).
+title: Deploying FastGPT with Milvus
+---
+
+# Deploying FastGPT with Milvus
+
+[FastGPT](https://fastgpt.in/) is a knowledge-based question and answer system built on the LLM large language model, offering ready-to-use capabilities for data processing and model invocation. Furthermore, it enables workflow orchestration through Flow visualization, thus facilitating complex question and answer scenarios. This tutorial will guide you on how to swiftly deploy your own exclusive FastGPT application using [Milvus](https://milvus.io/).
+
+## Download docker-compose.yml
+Ensure that you have already installed [Docker Compose](https://docs.docker.com/compose/).
+Execute the command below to download the docker-compose.yml file.
+```shell
+$ mkdir fastgpt
+$ cd fastgpt
+$ curl -O https://raw.githubusercontent.com/labring/FastGPT/main/projects/app/data/config.json
+
+# milvus version
+$ curl -o docker-compose.yml https://raw.githubusercontent.com/labring/FastGPT/main/files/docker/docker-compose-milvus.yml
+# zilliz version
+# curl -o docker-compose.yml https://raw.githubusercontent.com/labring/FastGPT/main/files/docker/docker-compose-zilliz.yml
+```
+> If you're using the Zilliz version, adjust the `MILVUS_ADDRESS` and `MILVUS_TOKEN` link parameters in the docker-compose.yml file, which corresponds to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in [Zilliz Cloud](https://zilliz.com/cloud).
+
+## Launch the Container
+Execute in the same directory as docker-compose.yml. Ensure that the docker-compose version is ideally above 2.17, as some automation commands may not function otherwise.
+```shell
+# Launch the container
+$ docker-compose up -d
+# Wait for 10s, OneAPI typically needs to restart a few times to initially connect to Mysql
+$ sleep 10
+# Restart oneapi (Due to certain issues with the default Key of OneAPI, it will display 'channel not found' if not restarted, this can be temporarily resolved by manually restarting once, while waiting for the author's fix)
+$ docker restart oneapi
+```
+
+## Access OneAPI to Add Models
+OneAPI can be accessed at `ip:3001`. The default username is root, and the password is 123456. You can alter the password after logging in.
+Using OpenAI's model as an example, click on the "Channel" tab, and select your chat model and embedding model under "Models".
+Input your [OpenAI API Key](https://platform.openai.com/docs/quickstart) in the "Secrets" section.
+For the use of models beyond OpenAI, and further information, please consult [One API](https://doc.fastgpt.in/docs/development/one-api/).
+
+## Setting Tokens
+Click on the "Tokens" tab. By default, there is a token `Initial Root Token`. You can also create a new token and set a quota on your own.
+Click "Copy" on your token, ensuring that the value of this token matches the `CHAT_API_KEY` value set in the docker-compose.yml file.
+
+## Accessing FastGPT
+At present, FastGPT can be directly accessed at `ip:3000` (please mind the firewall). The login username is root, with the password set to `DEFAULT_ROOT_PSW` within the docker-compose.yml environment variable. Should you require domain name access, you would need to install and configure [Nginx](https://nginx.org/en/) on your own.
+
+## Stop the Container
+Run the following command to stop the container.
+```shell
+$ docker-compose down
+```
\ No newline at end of file
diff --git a/preview/site/en/integrations/integrate_with_haystack.md b/preview/site/en/integrations/integrate_with_haystack.md
new file mode 100644
index 000000000..1e7082f6d
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_haystack.md
@@ -0,0 +1,217 @@
+---
+id: integrate_with_haystack.md
+summary: This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using Haystack and Milvus.
+title: Retrieval-Augmented Generation (RAG) with Milvus and Haystack
+---
+
+# Retrieval-Augmented Generation (RAG) with Milvus and Haystack
+
+
+
+This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using Haystack and Milvus.
+
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+
+[Haystack](https://haystack.deepset.ai/) is the open source Python framework by deepset for building custom apps with large language models (LLMs). [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
+
+
+
+## Prerequisites
+
+Before running this notebook, make sure you have the following dependencies installed:
+
+
+```python
+! pip install --upgrade --quiet pymilvus milvus-haystack markdown-it-py mdit_plain
+```
+
+
+
+A question answering system based on semantic search works by finding the most similar question from a dataset of question-answer pairs for a given query question. Once the most similar question is identified, the corresponding answer from the dataset is considered as the answer for the query. This approach relies on semantic similarity measures to determine the similarity between questions and retrieve relevant answers.
+
+This tutorial shows how to build a question answering system using [Hugging Face](https://huggingface.co) as the data loader & embedding generator for data processing and [Milvus](https://milvus.io) as the vector database for semantic search.
+
+## Before you begin
+
+You need to make sure all required dependencies are installed:
+
+- `pymilvus`: a python package works with the vector database service powered by Milvus or Zilliz Cloud.
+- `datasets`, `transformers`: Hugging Face packages manage data and utilize models.
+- `torch`: a powerful library provides efficient tensor computation and deep learning tools.
+
+
+```python
+$ pip install --upgrade pymilvus transformers datasets torch
+```
+
+
+
+This guide demonstrates how to use Jina AI embeddings and Milvus to conduct similarity search and retrieval tasks.
+
+## Who is Jina AI
+Jina AI, founded in 2020 in Berlin, is a pioneering AI company focused on revolutionizing the future of artificial intelligence through its search foundation. Specializing in multimodal AI, Jina AI aims to empower businesses and developers to harness the power of multimodal data for value creation and cost savings through its integrated suite of components, including embeddings, rerankers, prompt ops, and core infrastructure.
+Jina AI's cutting-edge embeddings boast top-tier performance, featuring an 8192 token-length model ideal for comprehensive data representation. Offering multilingual support and seamless integration with leading platforms like OpenAI, these embeddings facilitate cross-lingual applications.
+
+## Milvus and Jina AI's Embedding
+In order to store and search these embeddings efficiently for speed and scale, specific infrastructure designed for this purpose is required. Milvus is a widely known advanced open-source vector database capable of handling large-scale vector data. Milvus enables fast and accurate vector(embedding) search according plenty of metrics. Its scalability allows for seamless handling of massive volumes of image data, ensuring high-performance search operations even as datasets grow.
+
+## Examples
+Jina embeddings have been integrated into the PyMilvus model library. Now, we will demonstrate code examples to show how to use Jina embeddings in action.
+
+Before we start, we need to install model library for PyMilvus.
+
+
+```python
+$ pip install -U pymilvus
+$ pip install "pymilvus[model]"
+```
+
+"
+ef = JinaEmbeddingFunction("jina-embeddings-v2-base-en", jina_api_key)
+
+query = "what is information retrieval?"
+doc = "Information retrieval is the process of finding relevant information from a large collection of data or documents."
+
+qvecs = ef.encode_queries([query])
+dvecs = ef.encode_documents([doc])
+```
+
+## Bilingual Embeddings
+Jina AI's bilingual models enhance multilingual platforms, global support, and cross-lingual content discovery. Designed for German-English and Chinese-English translations, they foster understanding among diverse linguistic groups, simplifying interactions across languages.
+
+
+```python
+from pymilvus.model.dense import JinaEmbeddingFunction
+
+jina_api_key = ""
+ef = JinaEmbeddingFunction("jina-embeddings-v2-base-de", jina_api_key)
+
+query = "what is information retrieval?"
+doc = "Information Retrieval ist der Prozess, relevante Informationen aus einer großen Sammlung von Daten oder Dokumenten zu finden."
+
+qvecs = ef.encode_queries([query])
+dvecs = ef.encode_documents([doc])
+```
+
+## Code Embeddings
+Jina AI's code embedding model provides searching ability through code and documentation. It supports English and 30 popular programming languages that can be used for enhancing code navigation, streamlined code review and automated documentation assistance.
+
+
+```python
+from pymilvus.model.dense import JinaEmbeddingFunction
+
+jina_api_key = ""
+ef = JinaEmbeddingFunction("jina-embeddings-v2-base-code", jina_api_key)
+
+# Case1: Enhanced Code Navigation
+# query: text description of the functionality
+# document: relevant code snippet
+
+query = "function to calculate average in Python."
+doc = """
+def calculate_average(numbers):
+ total = sum(numbers)
+ count = len(numbers)
+ return total / count
+"""
+
+# Case2: Streamlined Code Review
+# query: text description of the programming concept
+# document: relevante code snippet or PR
+
+query = "pull quest related to Collection"
+doc = "fix:[restful v2] parameters of create collection ..."
+
+# Case3: Automatic Documentation Assistance
+# query: code snippet you need explanation
+# document: relevante document or DocsString
+
+query = "What is Collection in Milvus"
+doc = """
+In Milvus, you store your vector embeddings in collections. All vector embeddings within a collection share the same dimensionality and distance metric for measuring similarity.
+Milvus collections support dynamic fields (i.e., fields not pre-defined in the schema) and automatic incrementation of primary keys.
+"""
+
+qvecs = ef.encode_queries([query])
+dvecs = ef.encode_documents([doc])
+```
+
+## Semantic Search with Jina & Milvus
+With the powerful vector embedding function, we can combine the embeddings retrieved by utilizing Jina AI models with Milvus Lite vector database to perform semantic search.
+
+
+```python
+from pymilvus.model.dense import JinaEmbeddingFunction
+from pymilvus import MilvusClient
+
+jina_api_key = ""
+ef = JinaEmbeddingFunction("jina-embeddings-v2-base-en", jina_api_key)
+DIMENSION = 768 # size of jina-embeddings-v2-base-en
+
+doc = [
+ "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.",
+ "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.",
+ "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.",
+ "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.",
+]
+
+dvecs = ef.encode_documents(doc)
+
+data = [
+ {"id": i, "vector": dvecs[i], "text": doc[i], "subject": "history"}
+ for i in range(len(dvecs))
+]
+
+milvus_client = MilvusClient("./milvus_jina_demo.db")
+COLLECTION_NAME = "demo_collection" # Milvus collection name
+if milvus_client.has_collection(collection_name=COLLECTION_NAME):
+ milvus_client.drop_collection(collection_name=COLLECTION_NAME)
+milvus_client.create_collection(collection_name=COLLECTION_NAME, dimension=DIMENSION)
+
+res = milvus_client.insert(collection_name=COLLECTION_NAME, data=data)
+
+print(res["insert_count"])
+```
+
+"
+
+rf = JinaRerankFunction("jina-reranker-v1-base-en", jina_api_key)
+
+query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?"
+
+documents = [
+ "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.",
+ "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.",
+ "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.",
+ "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.",
+]
+
+rf(query, documents)
+```
+
+
+
+
+ [RerankResult(text="The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.", score=0.9370958209037781, index=1),
+ RerankResult(text='The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.', score=0.35420963168144226, index=3),
+ RerankResult(text="In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.", score=0.3498658835887909, index=0),
+ RerankResult(text='In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.', score=0.2728956639766693, index=2)]
diff --git a/preview/site/en/integrations/integrate_with_langchain.md b/preview/site/en/integrations/integrate_with_langchain.md
new file mode 100644
index 000000000..ad6547357
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_langchain.md
@@ -0,0 +1,311 @@
+---
+id: integrate_with_langchain.md
+summary: This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LangChain and Milvus.
+title: Retrieval-Augmented Generation (RAG) with Milvus and LangChain
+---
+
+# Retrieval-Augmented Generation (RAG) with Milvus and LangChain
+
+
+
+This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LangChain and Milvus.
+
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+
+[LangChain](https://www.langchain.com/) is a framework for developing applications powered by large language models (LLMs). [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
+
+
+
+## Prerequisites
+
+Before running this notebook, make sure you have the following dependencies installed:
+
+
+```python
+$ pip install --upgrade --quiet langchain langchain-core langchain-community langchain-text-splitters langchain-milvus langchain-openai bs4
+```
+
+ tags.
+If you don't know the answer, just say that you don't know, don't try to make up an answer.
+
+{context}
+
+
+
+{question}
+
+
+The response should be specific and use statistics or numbers when possible.
+
+Assistant:"""
+
+# Create a PromptTemplate instance with the defined template and input variables
+prompt = PromptTemplate(
+ template=PROMPT_TEMPLATE, input_variables=["context", "question"]
+)
+# Convert the vector store to a retriever
+retriever = vectorstore.as_retriever()
+
+
+# Define a function to format the retrieved documents
+def format_docs(docs):
+ return "\n\n".join(doc.page_content for doc in docs)
+```
+
+Use the LCEL(LangChain Expression Language) to build a RAG chain.
+
+
+```python
+# Define the RAG (Retrieval-Augmented Generation) chain for AI response generation
+rag_chain = (
+ {"context": retriever | format_docs, "question": RunnablePassthrough()}
+ | prompt
+ | llm
+ | StrOutputParser()
+)
+
+# rag_chain.get_graph().print_ascii()
+
+# Invoke the RAG chain with a specific question and retrieve the response
+res = rag_chain.invoke(query)
+res
+```
+
+
+
+
+ "Self-reflection of an AI agent involves the process of synthesizing memories into higher-level inferences over time to guide the agent's future behavior. It serves as a mechanism to create higher-level summaries of past events. One approach to self-reflection involves prompting the language model with the 100 most recent observations and asking it to generate the 3 most salient high-level questions based on those observations. This process helps the AI agent optimize believability in the current moment and over time."
+
+
+
+Congratulations! You have built a basic RAG chain powered by Milvus and LangChain.
+
+## Metadata filtering
+
+We can use the [Milvus Scalar Filtering Rules](https://milvus.io/docs/boolean.md) to filter the documents based on metadata. We have loaded the documents from two different sources, and we can filter the documents by the metadata `source`.
+
+
+```python
+vectorstore.similarity_search(
+ "What is CoT?",
+ k=1,
+ expr="source == 'https://lilianweng.github.io/posts/2023-06-23-agent/'",
+)
+
+# The same as:
+# vectorstore.as_retriever(search_kwargs=dict(
+# k=1,
+# expr="source == 'https://lilianweng.github.io/posts/2023-06-23-agent/'",
+# )).invoke("What is CoT?")
+```
+
+
+
+
+ [Document(page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\nSelf-Reflection#', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'pk': 449281835035555858})]
+
+
+
+If we want to dynamically change the search parameters without rebuilding the chain, we can [configure the runtime chain internals](https://python.langchain.com/v0.2/docs/how_to/configure/) . Let's define a new retriever with this dynamically configure and use it to build a new RAG chain.
+
+
+```python
+from langchain_core.runnables import ConfigurableField
+
+# Define a new retriever with a configurable field for search_kwargs
+retriever2 = vectorstore.as_retriever().configurable_fields(
+ search_kwargs=ConfigurableField(
+ id="retriever_search_kwargs",
+ )
+)
+
+# Invoke the retriever with a specific search_kwargs which filter the documents by source
+retriever2.with_config(
+ configurable={
+ "retriever_search_kwargs": dict(
+ expr="source == 'https://lilianweng.github.io/posts/2023-06-23-agent/'",
+ k=1,
+ )
+ }
+).invoke(query)
+```
+
+
+
+
+ [Document(page_content='Self-Reflection#\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\nThought: ...\nAction: ...\nObservation: ...\n... (Repeated many times)', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'pk': 449281835035555859})]
+
+
+
+
+```python
+# Define a new RAG chain with this dynamically configurable retriever
+rag_chain2 = (
+ {"context": retriever2 | format_docs, "question": RunnablePassthrough()}
+ | prompt
+ | llm
+ | StrOutputParser()
+)
+```
+
+Let's try this dynamically configurable RAG chain with different filter conditions.
+
+
+```python
+# Invoke this RAG chain with a specific question and config
+rag_chain2.with_config(
+ configurable={
+ "retriever_search_kwargs": dict(
+ expr="source == 'https://lilianweng.github.io/posts/2023-06-23-agent/'",
+ )
+ }
+).invoke(query)
+```
+
+
+
+
+ "Self-reflection of an AI agent involves the process of synthesizing memories into higher-level inferences over time to guide the agent's future behavior. It serves as a mechanism to create higher-level summaries of past events. One approach to self-reflection involves prompting the language model with the 100 most recent observations and asking it to generate the 3 most salient high-level questions based on those observations. This process helps the AI agent optimize believability in the current moment and over time."
+
+
+
+When we change the search condition to filter the documents by the second source, as the content of this blog source has nothing todo with the query question, we get an answer with no relevant information.
+
+
+```python
+rag_chain2.with_config(
+ configurable={
+ "retriever_search_kwargs": dict(
+ expr="source == 'https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/'",
+ )
+ }
+).invoke(query)
+```
+
+
+
+
+ "I'm sorry, but based on the provided context, there is no specific information or statistical data available regarding the self-reflection of an AI agent."
+
+
+
+----
+This tutorial focus the basic usage of Milvus LangChain integration and simple RAG approach. For more advanced RAG techniques, please refer to the [advanced rag bootcamp](https://github.com/milvus-io/bootcamp/tree/master/bootcamp/RAG/advanced_rag).
diff --git a/preview/site/en/integrations/integrate_with_langfuse.md b/preview/site/en/integrations/integrate_with_langfuse.md
new file mode 100644
index 000000000..1f91fce58
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_langfuse.md
@@ -0,0 +1,126 @@
+---
+id: integrate_with_langfuse.md
+summary: This is a simple cookbook that demonstrates how to use the LlamaIndex Langfuse integration. It uses Milvus Lite to store the documents and Query.
+title: Cookbook LlamaIndex & Milvus Integration
+---
+
+# Cookbook - LlamaIndex & Milvus Integration
+
+
+
+
+
+This is a simple cookbook that demonstrates how to use the [LlamaIndex Langfuse integration](https://langfuse.com/docs/integrations/llama-index/get-started). It uses Milvus Lite to store the documents and Query.
+
+[Milvus Lite](https://github.com/milvus-io/milvus-lite/) is the lightweight version of Milvus, an open-source vector database that powers AI applications with vector embeddings and similarity search.
+
+## Setup
+
+Make sure you have both `llama-index` and `langfuse` installed.
+
+
+```python
+$ pip install llama-index langfuse llama-index-vector-stores-milvus --upgrade
+```
+
+Initialize the integration. Get your API keys from the [Langfuse project settings](https://cloud.langfuse.com), and replace public_key secret_key with your key values. This example uses OpenAI for embeddings and chat completions, so you also need to specify your OpenAI key in environment variable.
+
+
+```python
+import os
+
+# Get keys for your project from the project settings page
+# https://cloud.langfuse.com
+os.environ["LANGFUSE_PUBLIC_KEY"] = ""
+os.environ["LANGFUSE_SECRET_KEY"] = ""
+os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com" # 🇪🇺 EU region
+# os.environ["LANGFUSE_HOST"] = "https://us.cloud.langfuse.com" # 🇺🇸 US region
+
+# Your openai key
+os.environ["OPENAI_API_KEY"] = ""
+```
+
+
+```python
+from llama_index.core import Settings
+from llama_index.core.callbacks import CallbackManager
+from langfuse.llama_index import LlamaIndexCallbackHandler
+
+langfuse_callback_handler = LlamaIndexCallbackHandler()
+Settings.callback_manager = CallbackManager([langfuse_callback_handler])
+```
+
+## Index using Milvus Lite
+
+
+```python
+from llama_index.core import Document
+
+doc1 = Document(text="""
+Maxwell "Max" Silverstein, a lauded movie director, screenwriter, and producer, was born on October 25, 1978, in Boston, Massachusetts. A film enthusiast from a young age, his journey began with home movies shot on a Super 8 camera. His passion led him to the University of Southern California (USC), majoring in Film Production. Eventually, he started his career as an assistant director at Paramount Pictures. Silverstein's directorial debut, “Doors Unseen,” a psychological thriller, earned him recognition at the Sundance Film Festival and marked the beginning of a successful directing career.
+""")
+doc2 = Document(text="""
+Throughout his career, Silverstein has been celebrated for his diverse range of filmography and unique narrative technique. He masterfully blends suspense, human emotion, and subtle humor in his storylines. Among his notable works are "Fleeting Echoes," "Halcyon Dusk," and the Academy Award-winning sci-fi epic, "Event Horizon's Brink." His contribution to cinema revolves around examining human nature, the complexity of relationships, and probing reality and perception. Off-camera, he is a dedicated philanthropist living in Los Angeles with his wife and two children.
+""")
+```
+
+
+```python
+# Example index construction + LLM query
+
+from llama_index.core import VectorStoreIndex
+from llama_index.core import StorageContext
+from llama_index.vector_stores.milvus import MilvusVectorStore
+
+
+vector_store = MilvusVectorStore(
+ uri="tmp/milvus_demo.db", dim=1536, overwrite=False
+)
+storage_context = StorageContext.from_defaults(vector_store=vector_store)
+
+index = VectorStoreIndex.from_documents(
+ [doc1,doc2], storage_context=storage_context
+)
+```
+
+## Query
+
+
+```python
+# Query
+response = index.as_query_engine().query("What did he do growing up?")
+print(response)
+```
+
+
+```python
+# Chat
+response = index.as_chat_engine().chat("What did he do growing up?")
+print(response)
+```
+
+## Explore traces in Langfuse
+
+
+```python
+# As we want to immediately see result in Langfuse, we need to flush the callback handler
+langfuse_callback_handler.flush()
+```
+
+Done! ✨ You see traces of your index and query in your Langfuse project.
+
+Example traces (public links):
+1. [Query](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/2b26fc72-044f-4b0b-a3c3-485328975161)
+2. [Query (chat)](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/72503163-2b25-4693-9cc9-56190b8e32b9)
+
+Trace in Langfuse:
+
+
+
+
+## Interested in more advanced features?
+
+See the full [integration docs](https://langfuse.com/docs/integrations/llama-index/get-started) to learn more about advanced features and how to use them:
+
+- Interoperability with Langfuse Python SDK and other integrations
+- Add custom metadata and attributes to the traces
diff --git a/preview/site/en/integrations/integrate_with_llamaindex.md b/preview/site/en/integrations/integrate_with_llamaindex.md
new file mode 100644
index 000000000..dcc8aa07f
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_llamaindex.md
@@ -0,0 +1,229 @@
+---
+id: integrate_with_llamaindex.md
+summary: This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LlamaIndex and Milvus.
+title: Retrieval-Augmented Generation (RAG) with Milvus and LlamaIndex
+---
+
+# Retrieval-Augmented Generation (RAG) with Milvus and LlamaIndex
+
+
+
+This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LlamaIndex and Milvus.
+
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+
+[LlamaIndex](https://www.llamaindex.ai/) is a simple, flexible data framework for connecting custom data sources to large language models (LLMs). [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
+
+In this notebook we are going to show a quick demo of using the MilvusVectorStore.
+
+## Before you begin
+
+### Install dependencies
+Code snippets on this page require pymilvus and llamaindex dependencies. You can install them using the following commands:
+
+
+```python
+$ pip install pymilvus>=2.4.2
+```
+
+
+```python
+$ pip install llama-index-vector-stores-milvus
+```
+
+
+```python
+$ pip install llama-index
+```
+
+
+
+This guide showcases how [OpenAI's Embedding API](https://platform.openai.com/docs/guides/embeddings) can be used with Milvus vector database to conduct semantic search on text.
+
+## Getting started
+Before you start, make sure you have the OpenAI API key ready, or you get one from the [OpenAI website](https://openai.com/index/openai-api/).
+
+The data used in this example are book titles. You can download the dataset [here](https://www.kaggle.com/datasets/jealousleopard/goodreadsbooks) and put it in the same directory where you run the following code.
+
+First, install the package for Milvus and OpenAI:
+
+```shell
+pip install --upgrade openai pymilvus
+```
+
+")
+
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+
+vectors = [
+ vec.embedding
+ for vec in openai_client.embeddings.create(input=docs, model=MODEL_NAME).data
+]
+
+# Prepare data to be stored in Milvus vector database.
+# We can store the id, vector representation, raw text and labels such as "subject" in this case in Milvus.
+data = [
+ {"id": i, "vector": vectors[i], "text": docs[i], "subject": "history"}
+ for i in range(len(docs))
+]
+
+
+# Connect to Milvus, all data is stored in a local file named "milvus_openai_demo.db"
+# in current directory. You can also connect to a remote Milvus server following this
+# instruction: https://milvus.io/docs/install_standalone-docker.md.
+milvus_client = MilvusClient(uri="milvus_openai_demo.db")
+COLLECTION_NAME = "demo_collection" # Milvus collection name
+# Create a collection to store the vectors and text.
+if milvus_client.has_collection(collection_name=COLLECTION_NAME):
+ milvus_client.drop_collection(collection_name=COLLECTION_NAME)
+milvus_client.create_collection(collection_name=COLLECTION_NAME, dimension=DIMENSION)
+
+# Insert all data into Milvus vector database.
+res = milvus_client.insert(collection_name="demo_collection", data=data)
+
+print(res["insert_count"])
+```
+
+
+
+This guide demonstrates how to use Ragas to evaluate a Retrieval-Augmented Generation (RAG) pipeline built upon [Milvus](https://milvus.io/).
+
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+
+[Ragas](https://docs.ragas.io/en/latest/index.html#) is a framework that helps you evaluate your RAG pipelines. There are existing tools and frameworks that help you build these pipelines but evaluating it and quantifying your pipeline performance can be hard. This is where Ragas (RAG Assessment) comes in.
+
+
+
+
+## Prerequisites
+
+Before running this notebook, make sure you have the following dependencies installed:
+
+
+```python
+$ pip install --upgrade pymilvus openai requests tqdm pandas ragas
+```
+
+ tags to provide an answer to the question enclosed in tags.
+
+{context}
+
+
+{question}
+
+"""
+
+ def _prepare_milvus(
+ self, milvus_client: MilvusClient, collection_name: str = "rag_collection"
+ ):
+ self.milvus_client = milvus_client
+ self.collection_name = collection_name
+ if self.milvus_client.has_collection(self.collection_name):
+ self.milvus_client.drop_collection(self.collection_name)
+ embedding_dim = len(self._emb_text("foo"))
+ self.milvus_client.create_collection(
+ collection_name=self.collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+ )
+
+ def load(self, texts: List[str]):
+ """
+ Load the text data into Milvus.
+ """
+ data = []
+ for i, line in enumerate(tqdm(texts, desc="Creating embeddings")):
+ data.append({"id": i, "vector": self._emb_text(line), "text": line})
+
+ self.milvus_client.insert(collection_name=self.collection_name, data=data)
+
+ def retrieve(self, question: str, top_k: int = 3) -> List[str]:
+ """
+ Retrieve the most similar text data to the given question.
+ """
+ search_res = self.milvus_client.search(
+ collection_name=self.collection_name,
+ data=[self._emb_text(question)],
+ limit=top_k,
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+ )
+ retrieved_texts = [res["entity"]["text"] for res in search_res[0]]
+ return retrieved_texts[:top_k]
+
+ def answer(
+ self,
+ question: str,
+ retrieval_top_k: int = 3,
+ return_retrieved_text: bool = False,
+ ):
+ """
+ Answer the given question with the retrieved knowledge.
+ """
+ retrieved_texts = self.retrieve(question, top_k=retrieval_top_k)
+ user_prompt = self.USER_PROMPT.format(
+ context="\n".join(retrieved_texts), question=question
+ )
+ response = self.openai_client.chat.completions.create(
+ model=self.llm_model,
+ messages=[
+ {"role": "system", "content": self.SYSTEM_PROMPT},
+ {"role": "user", "content": user_prompt},
+ ],
+ )
+ if not return_retrieved_text:
+ return response.choices[0].message.content
+ else:
+ return response.choices[0].message.content, retrieved_texts
+```
+
+Let's initialize the RAG class with OpenAI and Milvus clients.
+
+
+```python
+openai_client = OpenAI()
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+my_rag = RAG(openai_client=openai_client, milvus_client=milvus_client)
+```
+
+
+
+## Performing the search
+
+With all the data inserted into Milvus, we can start performing our searches. In this example, we are going to search for movies based on the plot. Because we are doing a batch search, the search time is shared across the movie searches.
+
+```python
+# Search for titles that closest match these phrases.
+search_terms = ['A movie about cars', 'A movie about monsters']
+
+# Search the database based on input text
+def embed_search(data):
+ embeds = transformer.encode(data)
+ return [x for x in embeds]
+
+search_data = embed_search(search_terms)
+
+start = time.time()
+res = collection.search(
+ data=search_data, # Embeded search value
+ anns_field="embedding", # Search across embeddings
+ param={},
+ limit = TOP_K, # Limit to top_k results per search
+ output_fields=['title'] # Include title field in result
+)
+end = time.time()
+
+for hits_i, hits in enumerate(res):
+ print('Title:', search_terms[hits_i])
+ print('Search Time:', end-start)
+ print('Results:')
+ for hit in hits:
+ print( hit.entity.get('title'), '----', hit.distance)
+ print()
+```
+
+The output should be similar to the following:
+
+```shell
+Title: A movie about cars
+Search Time: 0.08636689186096191
+Results:
+Youth's Endearing Charm ---- 1.0954499244689941
+From Leadville to Aspen: A Hold-Up in the Rockies ---- 1.1019384860992432
+Gentlemen of Nerve ---- 1.1331942081451416
+
+Title: A movie about monsters
+Search Time: 0.08636689186096191
+Results:
+The Suburbanite ---- 1.0666425228118896
+Youth's Endearing Charm ---- 1.1072258949279785
+The Godless Girl ---- 1.1511223316192627
+```
+
diff --git a/preview/site/en/integrations/integrate_with_snowpark.md b/preview/site/en/integrations/integrate_with_snowpark.md
new file mode 100644
index 000000000..9deefc8d3
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_snowpark.md
@@ -0,0 +1,310 @@
+---
+id: integrate_with_snowpark.md
+summary: This guide demonstrates how to start a Milvus demo on Snowpark container services.
+title: Milvus on Snowpark Container Services
+---
+
+# Milvus on Snowpark Container Services
+
+This guide demonstrates how to start a Milvus demo on Snowpark container services.
+
+## About Snowpark Container Services
+
+Snowpark Container Services is a fully managed container offering designed to facilitate the deployment, management, and scaling of containerized applications within the Snowflake ecosystem. This service enables users to run containerized workloads directly within Snowflake, ensuring that data doesn’t need to be moved out of the Snowflake environment for processing. For more information, please refer to the official introduction: [Snowpark Container Services](https://docs.snowflake.com/en/developer-guide/snowpark-container-services/overview).
+
+## Configure Milvus demo
+
+The following will let users understand the capabilities of Milvus and how to use Milvus in SPCS through configuration and code.
+
+### 1. Obtain account information
+
+Download the SPCS client: [SnowSQL](https://docs.snowflake.com/en/user-guide/snowsql-install-config), then log in to your account.
+
+```shell
+snowsql -a ${instance_name} -u ${user_name}
+```
+
+The rule of `${instance_name}` is `${org_name}-${acct_name}`. The relevant information can be obtained by logging in to [app.snowflake.com](http://app.snowflake.com/sn) and checking the personal account information.
+
+
+
+### 2. Configure Role and privileges
+
+Configure OAUTH integration.
+
+```sql
+USE ROLE ACCOUNTADMIN;
+CREATE SECURITY INTEGRATION SNOWSERVICES_INGRESS_OAUTH
+ TYPE=oauth
+ OAUTH_CLIENT=snowservices_ingress
+ ENABLED=true;
+
+USE ROLE ACCOUNTADMIN;
+GRANT BIND SERVICE ENDPOINT ON ACCOUNT TO ROLE SYSADMIN;
+```
+
+Create a role for the service, note that the `${PASSWORD}` part here needs to be replaced by the user when the demo is
+
+```sql
+USE ROLE SECURITYADMIN;
+CREATE ROLE MILVUS_ROLE;
+
+USE ROLE USERADMIN;
+CREATE USER milvus_user
+ PASSWORD='milvususerok'
+ DEFAULT_ROLE = MILVUS_ROLE
+ DEFAULT_SECONDARY_ROLES = ('ALL')
+ MUST_CHANGE_PASSWORD = FALSE;
+
+USE ROLE SECURITYADMIN;
+GRANT ROLE MILVUS_ROLE TO USER milvus_user;
+```
+
+### 3. Create data storage configuration
+
+- Create warehouse and database
+
+ ```sql
+ USE ROLE SYSADMIN;
+ CREATE OR REPLACE WAREHOUSE MILVUS_WAREHOUSE WITH
+ WAREHOUSE_SIZE='X-SMALL'
+ AUTO_SUSPEND = 180
+ AUTO_RESUME = true
+ INITIALLY_SUSPENDED=false;
+
+ USE ROLE SYSADMIN;
+ CREATE DATABASE IF NOT EXISTS MILVUS_DEMO;
+ USE DATABASE MILVUS_DEMO;
+ CREATE IMAGE REPOSITORY MILVUS_DEMO.PUBLIC.MILVUS_REPO;
+ CREATE OR REPLACE STAGE YAML_STAGE;
+ CREATE OR REPLACE STAGE DATA ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');
+ CREATE OR REPLACE STAGE FILES ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');
+ ```
+
+- Grant role privileges
+
+ ```sql
+ USE ROLE SECURITYADMIN;
+ GRANT ALL PRIVILEGES ON DATABASE MILVUS_DEMO TO MILVUS_ROLE;
+ GRANT ALL PRIVILEGES ON SCHEMA MILVUS_DEMO.PUBLIC TO MILVUS_ROLE;
+ GRANT ALL PRIVILEGES ON WAREHOUSE MILVUS_WAREHOUSE TO MILVUS_ROLE;
+ GRANT ALL PRIVILEGES ON STAGE MILVUS_DEMO.PUBLIC.FILES TO MILVUS_ROLE;
+ ```
+
+- Configure ACL
+
+ ```sql
+ USE ROLE ACCOUNTADMIN;
+ USE DATABASE MILVUS_DEMO;
+ USE SCHEMA PUBLIC;
+ CREATE NETWORK RULE allow_all_rule
+ TYPE = 'HOST_PORT'
+ MODE= 'EGRESS'
+ VALUE_LIST = ('0.0.0.0:443','0.0.0.0:80');
+
+ CREATE EXTERNAL ACCESS INTEGRATION allow_all_eai
+ ALLOWED_NETWORK_RULES=(allow_all_rule)
+ ENABLED=TRUE;
+
+ GRANT USAGE ON INTEGRATION allow_all_eai TO ROLE SYSADMIN;
+ ```
+
+### 4. Create images
+
+The image used by Milvus needs to be built locally and then uploaded by user. For the relevant configuration of the image, please refer to [this repo](https://github.com/dald001/milvus_on_spcs). After cloning the code, go to the root directory of the project and prepare to build the image.
+
+- Build images locally
+
+ Open your local shell and begin to build images.
+
+ ```shell
+ cd ${repo_git_root_path}
+ docker build --rm --no-cache --platform linux/amd64 -t milvus ./images/milvus
+ docker build --rm --no-cache --platform linux/amd64 -t jupyter ./images/jupyter
+ ```
+
+ There are two images here, the first one is running the Milvus database, and the second one is the notebook used for display.
+
+ After the local images are built, prepare to tag and upload them.
+
+- Tag built images
+
+ Log in to the docker hub of SPCS.
+
+ ```shell
+ docker login ${instance_name}.registry.snowflakecomputing.com -u ${user_name}
+ ```
+
+ And you can tag images for spcs now.
+
+ ```shell
+ docker tag milvus ${instance_name}.registry.snowflakecomputing.com/milvus_demo/public/milvus_repo/milvus
+ docker tag jupyter ${instance_name}.registry.snowflakecomputing.com/milvus_demo/public/milvus_repo/jupyter
+ ```
+
+ Then use `docker images | grep milvus` in the local shell to check whether the image has been packaged and tagged successfully.
+
+ ```shell
+ docker images | grep milvus
+
+ ${instance_name}.registry.snowflakecomputing.com/milvus_demo/public/milvus_repo/milvus latest 3721bbb8f62b 2 days ago 2.95GB
+ ${instance_name}.registry.snowflakecomputing.com/milvus_demo/public/milvus_repo/jupyter latest 20633f5bcadf 2 days ago 2GB
+ ```
+
+- Push images to SPCS
+
+ ```shell
+ docker push ${instance_name}.registry.snowflakecomputing.com/milvus_demo/public/milvus_repo/milvus
+ docker push ${instance_name}.registry.snowflakecomputing.com/milvus_demo/public/milvus_repo/jupyter
+ ```
+
+### 5. Create and start services
+
+Let us go back to the SnowSQL shell.
+
+- Create Compute pools
+
+```sql
+USE ROLE SYSADMIN;
+CREATE COMPUTE POOL IF NOT EXISTS MILVUS_COMPUTE_POOL
+ MIN_NODES = 1
+ MAX_NODES = 1
+ INSTANCE_FAMILY = CPU_X64_S
+ AUTO_RESUME = true;
+CREATE COMPUTE POOL IF NOT EXISTS JUPYTER_COMPUTE_POOL
+ MIN_NODES = 1
+ MAX_NODES = 1
+ INSTANCE_FAMILY = CPU_X64_S
+ AUTO_RESUME = true;
+```
+
+Check the compute pools through `DESCRIBE` until the status is `ACTIVE` or `IDLE`.
+
+```sql
+DESCRIBE COMPUTE POOL MILVUS_COMPUTE_POOL;
+DESCRIBE COMPUTE POOL JUPYTER_COMPUTE_POOL;
+```
+
+
+
+- Upload spec files
+
+After creating the compute pool, start preparing the spce file for the service. The files are also in [this repo](https://github.com/dald001/milvus_on_spcs). Please refer to the specs directory.
+
+Open the spec files of these two services, find `${org_name}-${acct_name}` in the spec file, and replace them with ${instance_name} of your own account. After modification, use SnowSQL to complete the upload.
+
+```sql
+PUT file://${path/to/jupyter.yaml} @yaml_stage overwrite=true auto_compress=false;
+PUT file://${path/to/milvus.yaml} @yaml_stage overwrite=true auto_compress=false;
+```
+
+- Create service
+
+When the upload is complete, you are ready to create the service, Continue to complete the process of creating the service.
+
+```sql
+USE ROLE SYSADMIN;
+USE DATABASE MILVUS_DEMO;
+USE SCHEMA PUBLIC;
+
+CREATE SERVICE MILVUS
+ IN COMPUTE POOL MILVUS_COMPUTE_POOL
+ FROM @YAML_STAGE
+ SPEC='milvus.yaml'
+ MIN_INSTANCES=1
+ MAX_INSTANCES=1;
+
+CREATE SERVICE JUPYTER
+ IN COMPUTE POOL JUPYTER_COMPUTE_POOL
+ FROM @YAML_STAGE
+ SPEC='jupyter.yaml'
+ MIN_INSTANCES=1
+ MAX_INSTANCES=1;
+```
+
+The services can also be viewed through `SHOW SERVICES;`.
+
+```sql
+SHOW SERVICES;
+
++---------+---------------+-------------+----------+----------------------+--------------------------------------------------------+-----------------
+| name | database_name | schema_name | owner | compute_pool | dns_name | ......
+|---------+---------------+-------------+----------+----------------------+--------------------------------------------------------+-----------------
+| JUPYTER | MILVUS_DEMO | PUBLIC | SYSADMIN | JUPYTER_COMPUTE_POOL | jupyter.public.milvus-demo.snowflakecomputing.internal | ......
+| MILVUS | MILVUS_DEMO | PUBLIC | SYSADMIN | MILVUS_COMPUTE_POOL | milvus.public.milvus-demo.snowflakecomputing.internal | ......
++---------+---------------+-------------+----------+----------------------+--------------------------------------------------------+-----------------
+```
+
+If you encounter problems starting the service, you can view service information through `CALL SYSTEM$GET_SERVICE_STATUS('milvus');`.
+
+
+
+More information can be obtained through `CALL SYSTEM$GET_SERVICE_LOGS('milvus', '0', 'milvus', 10);`.
+
+## Use Notebook
+
+Use **SnowSQL** to grant permissions.
+
+```sql
+USE ROLE SECURITYADMIN;
+GRANT USAGE ON SERVICE MILVUS_DEMO.PUBLIC.JUPYTER TO ROLE MILVUS_ROLE;
+```
+
+Then view and record the endpoint of the Jupyter nootbook.
+
+```sql
+USE ROLE SYSADMIN;
+SHOW ENDPOINTS IN SERVICE MILVUS_DEMO.PUBLIC.JUPYTER;
+```
+
+Record the `ingress_url` part of the information, then open the browser and enter the `ingress_url`, use the milvus_user account to log in to the website.
+
+
+
+Opening the notebook through the `ingress_url`, double-click the `TestMilvus.ipynb` file on the page to try out Milvus. Select the first part of the code block, and click the **Run** button to start establishing the connection and initializing the embedding function.
+
+
+
+After establishing the connection, continue to click **RUN**. The code will turn a piece of text into vector data after embedding processing, and then insert it into Milvus.
+
+```python
+docs = [
+ "Artificial intelligence was founded as an academic discipline in 1956.",
+ "Alan Turing was the first person to conduct substantial research in AI.",
+ "Born in Maida Vale, London, Turing was raised in southern England.",
+]
+```
+
+Then use a text as a query: "Who started AI research?", perform the query after embedding processing, and finally obtain and display the most relevant results.
+
+
+
+For more information on the usage of the Milvus client, you can refer to the [Milvus Doc](quickstart.md) section.
+
+## 7. Clean up
+
+After verification, you can use SnowSQL to cleanup the services, roles, and data resources.
+
+```sql
+USE ROLE ACCOUNTADMIN;
+DROP USER milvus_user;
+
+USE ROLE SYSADMIN;
+DROP SERVICE MILVUS;
+DROP SERVICE JUPYTER;
+
+DROP COMPUTE POOL MILVUS_COMPUTE_POOL;
+DROP COMPUTE POOL JUPYTER_COMPUTE_POOL;
+
+DROP IMAGE REPOSITORY MILVUS_DEMO.PUBLIC.MILVUS_REPO;
+DROP DATABASE MILVUS_DEMO;
+DROP WAREHOUSE MILVUS_WAREHOUSE;
+
+USE ROLE ACCOUNTADMIN;
+DROP ROLE MILVUS_ROLE;
+DROP SECURITY INTEGRATION SNOWSERVICES_INGRESS_OAUTH;
+```
+
+## About Milvus
+
+For more information about Milvus, you can start with the [Milvus introduction](overview.md) and [Quick start](quickstart.md). Of course, there is a more detailed introduction to the API, refer to the [Python](https://milvus.io/api-reference/pymilvus/v2.4.x/About.md) and [Java](https://milvus.io/api-reference/java/v2.3.x/About.md) versions, and there is also information about [Embeddings](https://milvus.io/docs/embeddings.md) and [Integrations](https://milvus.io/docs/integrate_with_openai.md) for reference.
diff --git a/preview/site/en/integrations/integrate_with_spark.md b/preview/site/en/integrations/integrate_with_spark.md
new file mode 100644
index 000000000..4c379e9ed
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_spark.md
@@ -0,0 +1,491 @@
+---
+id: integrate_with_spark.md
+summary: This page discusses the Spark-Milvus connector.
+title: Spark-Milvus Connector User Guide
+---
+
+# Spark-Milvus Connector User Guide
+
+The Spark-Milvus Connector (https://github.com/zilliztech/spark-milvus) provides seamless integration between Apache Spark and Milvus, combining the data processing and ML features of Apache Spark with the vector data storage and search capabilities of Milvus. This integration enables various interesting applications, including:
+
+- Efficiently load vector data into Milvus in large batches,
+- Move data between Milvus and other storage systems or databases,
+- Analyzing the data in Milvus by leveraging Spark MLlib and other AI tools.
+
+## Quick start
+
+### Preparation
+
+The Spark-Milvus Connector supports Scala and Python programming languages. Users can use it with **Pyspark** or **Spark-shell**. To run this demo, set up a Spark environment containing Spark-Milvus Connector dependency in the following steps:
+
+1. Install Apache Spark (version >= 3.3.0)
+
+ You can install Apache Spark by referring to the [official documentation](https://spark.apache.org/docs/latest/).
+
+2. Download the **spark-milvus** jar file.
+
+ ```
+ wget https://github.com/zilliztech/spark-milvus/raw/1.0.0-SNAPSHOT/output/spark-milvus-1.0.0-SNAPSHOT.jar
+ ```
+
+3. Start the Spark runtime with **spark-milvus** jar as one of the dependencies.
+
+ To start the Spark runtime with the Spark-Milvus Connector, add the downloaded **spark-milvus** as the dependency to the command.
+
+ - **pyspark**
+
+ ```
+ ./bin/pyspark --jars spark-milvus-1.0.0-SNAPSHOT.jar
+ ```
+
+ - **spark-shell**
+
+ ```
+ ./bin/spark-shell --jars spark-milvus-1.0.0-SNAPSHOT.jar
+ ```
+
+### Demo
+
+In this demo, we create a sample Spark DataFrame with vector data and write it to Milvus through the Spark-Milvus Connector. A collection will be created in Milvus automatically based on the schema and specified options.
+
+
+
+```python
+from pyspark.sql import SparkSession
+
+columns = ["id", "text", "vec"]
+data = [(1, "a", [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]),
+ (2, "b", [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]),
+ (3, "c", [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]),
+ (4, "d", [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0])]
+sample_df = spark.sparkContext.parallelize(data).toDF(columns)
+sample_df.write \
+ .mode("append") \
+ .option("milvus.host", "localhost") \
+ .option("milvus.port", "19530") \
+ .option("milvus.collection.name", "hello_spark_milvus") \
+ .option("milvus.collection.vectorField", "vec") \
+ .option("milvus.collection.vectorDim", "8") \
+ .option("milvus.collection.primaryKeyField", "id") \
+ .format("milvus") \
+ .save()
+```
+
+```scala
+import org.apache.spark.sql.{SaveMode, SparkSession}
+
+object Hello extends App {
+
+ val spark = SparkSession.builder().master("local[*]")
+ .appName("HelloSparkMilvus")
+ .getOrCreate()
+
+ import spark.implicits._
+
+ // Create DataFrame
+ val sampleDF = Seq(
+ (1, "a", Seq(1.0,2.0,3.0,4.0,5.0)),
+ (2, "b", Seq(1.0,2.0,3.0,4.0,5.0)),
+ (3, "c", Seq(1.0,2.0,3.0,4.0,5.0)),
+ (4, "d", Seq(1.0,2.0,3.0,4.0,5.0))
+ ).toDF("id", "text", "vec")
+
+ // set milvus options
+ val milvusOptions = Map(
+ "milvus.host" -> "localhost" -> uri,
+ "milvus.port" -> "19530",
+ "milvus.collection.name" -> "hello_spark_milvus",
+ "milvus.collection.vectorField" -> "vec",
+ "milvus.collection.vectorDim" -> "5",
+ "milvus.collection.primaryKeyField", "id"
+ )
+
+ sampleDF.write.format("milvus")
+ .options(milvusOptions)
+ .mode(SaveMode.Append)
+ .save()
+}
+```
+
+After executing the above code, you can view the inserted data in Milvus using SDK or Attu (A Milvus Dashboard). You can find a collection named `hello_spark_milvus` created with 4 entities already inserted into it.
+
+## Features & concepts
+
+### Milvus options
+
+In the [Quick Start](#Quick-start) section, we have shown setting options during operations with Milvus. These options are abstracted as Milvus Options. They are used to create connections to Milvus and control other Milvus behaviors. Not all of the options are mandatory.
+
+| Option Key | Default Value | Description |
+| ---------- | ------------- | ----------- |
+| `milvus.host` | `localhost` | Milvus server host. See [Manage Milvus Connections](https://milvus.io/docs/manage_connection.md) for detail. |
+| `milvus.port` | `19530` | Milvus server port. See [Manage Milvus Connections](https://milvus.io/docs/manage_connection.md) for detail. |
+| `milvus.username` | `root` | Username for Milvus server. See [Manage Milvus Connections](https://milvus.io/docs/manage_connection.md) for detail. |
+| `milvus.password` | `Milvus` | Password for Milvus server. See [Manage Milvus Connections](https://milvus.io/docs/manage_connection.md) for detail. |
+| `milvus.uri` | `--` | Milvus server URI. See [Manage Milvus Connections](https://milvus.io/docs/manage_connection.md) for detail. |
+| `milvus.token` | `--` | Milvus server token. See [Manage Milvus Connections](https://milvus.io/docs/manage_connection.md) for detail. |
+| `milvus.database.name` | `default` | Name of the Milvus database to read or write. |
+| `milvus.collection.name` | `hello_milvus` | Name of the Milvus collection to read or write. |
+| `milvus.collection.primaryKeyField` | `None` | Name of the primary key field in the collection. Required if the collection does not exist. |
+| `milvus.collection.vectorField` | `None` | Name of the vector field in the collection. Required if the collection does not exist. |
+| `milvus.collection.vectorDim` | `None` | Dimension of the vector field in the collection. Required if the collection does not exist. |
+| `milvus.collection.autoID` | `false` | If the collection does not exist, this option specifies whether to automatically generate IDs for the entities. For more information, see [create_collection](https://milvus.io/docs/create_collection.md) |
+| `milvus.bucket` | `a-bucket` | Bucket name in the Milvus storage. This should be the same as `minio.bucketName` in [milvus.yaml](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml). |
+| `milvus.rootpath` | `files` | Root path of the Milvus storage. This should be the same as `minio.rootpath` in [milvus.yaml](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml). |
+| `milvus.fs` | `s3a://` | File system of the Milvus storage. The value `s3a://` applies to open-source Spark. Use `s3://` for Databricks. |
+| `milvus.storage.endpoint` | `localhost:9000` | Endpoint of the Milvus storage. This should be the same as `minio.address`:`minio.port` in [milvus.yaml](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml). |
+| `milvus.storage.user` | `minioadmin` | User of the Milvus storage. This should be the same as `minio.accessKeyID` in [milvus.yaml](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml). |
+| `milvus.storage.password` | `minioadmin` | Password of the Milvus storage. This should be the same as `minio.secretAccessKey` in [milvus.yaml](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml). |
+| `milvus.storage.useSSL` | `false` | Whether to use SSL for the Milvus storage. This should be the same as `minio.useSSL` in [milvus.yaml](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml). |
+
+## Milvus data format
+
+The Spark-Milvus Connector supports reading and writing data in the following Milvus data formats:
+
+- `milvus`: Milvus data format for seamless conversion from Spark DataFrame to Milvus entities.
+- `milvusbinlog`: Milvus data format for reading Milvus build-in binlog data.
+- `mjson`: Milvus JSON format for bulk-inserting data into Milvus.
+
+### milvus
+
+In [Quick start](#Quick-start), we use the **milvus** format to write sample data into a Milvus cluster. The **milvus** format is a new data format that supports seamlessly writing Spark DataFrame data into Milvus Collections. This is achieved by batch calls to the Insert API of Milvus SDK. If a collection doesn't exist in Milvus, a new collection will be created based on the schema of the Dataframe. However, the automatically created collection may not support all features of the collection schema. Therefore, it is recommended to create a collection via SDK first and then use spark-milvus for writing. For further information, please refer to [the demo](https://github.com/zilliztech/spark-milvus/blob/main/examples/src/main/scala/InsertDemo.scala).
+
+### milvusbinlog
+
+The new data format **milvusbinlog** is for reading Milvus built-in binlog data. Binlog is Milvus's internal data storage format based on parquet. Unfortunately, it can't be read by a regular parquet library, so we implemented this new data format to help Spark job read it.
+It is not recommended to use **milvusbinlog** directly unless you are familiar with the milvus internal storage details. We suggest using the [MilvusUtils](#MilvusUtils) function that will be introduced in the next section.
+
+```scalar
+val df = spark.read
+ .format("milvusbinlog")
+ .load(path)
+ .withColumnRenamed("val", "embedding")
+```
+
+### mjson
+
+Milvus provides [Bulkinsert](https://milvus.io/docs/bulk_insert.md) functionality for better writing performance when operating with large datasets. However, the JSON format used by Milvus is slightly different than Spark's default JSON output format.
+To resolve this, we introduce **mjson** data format to generate data that meets Milvus requirements. Here is an example that shows the difference between JSON-lines and **mjson**:
+
+- JSON-lines:
+
+ ```json
+ {"book_id": 101, "word_count": 13, "book_intro": [1.1, 1.2]}
+ {"book_id": 102, "word_count": 25, "book_intro": [2.1, 2.2]}
+ {"book_id": 103, "word_count": 7, "book_intro": [3.1, 3.2]}
+ {"book_id": 104, "word_count": 12, "book_intro": [4.1, 4.2]}
+ {"book_id": 105, "word_count": 34, "book_intro": [5.1, 5.2]}
+ ```
+
+- mjson (Required for Milvus Bulkinsert):
+
+ ```json
+ {
+ "rows":[
+ {"book_id": 101, "word_count": 13, "book_intro": [1.1, 1.2]},
+ {"book_id": 102, "word_count": 25, "book_intro": [2.1, 2.2]},
+ {"book_id": 103, "word_count": 7, "book_intro": [3.1, 3.2]},
+ {"book_id": 104, "word_count": 12, "book_intro": [4.1, 4.2]},
+ {"book_id": 105, "word_count": 34, "book_intro": [5.1, 5.2]}
+ ]
+ }
+ ```
+
+This will be improved in the future. We recommend using parquet format in spark-milvus intergration if your Milvus version is v2.3.7+ which supports bulkinsert with Parquet format. See [Demo](https://github.com/zilliztech/spark-milvus/blob/main/examples/src/main/scala/BulkInsertDemo.scala) on Github.
+
+## MilvusUtils
+
+MilvusUtils contains several useful util functions. Currently it is only supported in Scala. More usage examples are in [Advanced Usage](#Advanced-Usage) section.
+
+### MilvusUtils.readMilvusCollection
+
+**MilvusUtils.readMilvusCollection** is a simple interface for loading a whole Milvus collection into a Spark Dataframe. It wraps various operations, including calling Milvus SDK, reading **milvusbinlog** and common union/join operations.
+
+```scala
+val collectionDF = MilvusUtils.readMilvusCollection(spark, milvusOptions)
+```
+
+### MilvusUtils.bulkInsertFromSpark
+
+**MilvusUtils.bulkInsertFromSpark** provides a convenient way to import Spark output files into Milvus in a large batch. It wraps the **Bullkinsert** API of the Milvus SDK.
+
+```scala
+df.write.format("parquet").save(outputPath)
+MilvusUtils.bulkInsertFromSpark(spark, milvusOptions, outputPath, "parquet")
+```
+
+## Advanced Usage
+
+In this section, you will find advanced usage examples of the Spark-Milvus Connector for data analysis and migration. For more demos, see [examples](https://github.com/zilliztech/spark-milvus/tree/main/examples/src/main/scala).
+
+### MySQL -> embedding -> Milvus
+
+In this demo, we will
+
+1. Read data from MySQL through the Spark-MySQL Connector,
+2. Generate embedding (using Word2Vec as an example), and
+3. Write embedded data into Milvus.
+
+To enable the Spark-MySQL Connector, you need to add the following dependency to your Spark environment:
+
+```
+spark-shell --jars spark-milvus-1.0.0-SNAPSHOT.jar,mysql-connector-j-x.x.x.jar
+```
+
+```scala
+import org.apache.spark.ml.feature.{Tokenizer, Word2Vec}
+import org.apache.spark.sql.functions.udf
+import org.apache.spark.sql.{SaveMode, SparkSession}
+import zilliztech.spark.milvus.MilvusOptions._
+
+import org.apache.spark.ml.linalg.Vector
+
+object Mysql2MilvusDemo extends App {
+
+ val spark = SparkSession.builder().master("local[*]")
+ .appName("Mysql2MilvusDemo")
+ .getOrCreate()
+
+ import spark.implicits._
+
+ // Create DataFrame
+ val sampleDF = Seq(
+ (1, "Milvus was created in 2019 with a singular goal: store, index, and manage massive embedding vectors generated by deep neural networks and other machine learning (ML) models."),
+ (2, "As a database specifically designed to handle queries over input vectors, it is capable of indexing vectors on a trillion scale. "),
+ (3, "Unlike existing relational databases which mainly deal with structured data following a pre-defined pattern, Milvus is designed from the bottom-up to handle embedding vectors converted from unstructured data."),
+ (4, "As the Internet grew and evolved, unstructured data became more and more common, including emails, papers, IoT sensor data, Facebook photos, protein structures, and much more.")
+ ).toDF("id", "text")
+
+ // Write to MySQL Table
+ sampleDF.write
+ .mode(SaveMode.Append)
+ .format("jdbc")
+ .option("driver","com.mysql.cj.jdbc.Driver")
+ .option("url", "jdbc:mysql://localhost:3306/test")
+ .option("dbtable", "demo")
+ .option("user", "root")
+ .option("password", "123456")
+ .save()
+
+ // Read from MySQL Table
+ val dfMysql = spark.read
+ .format("jdbc")
+ .option("driver","com.mysql.cj.jdbc.Driver")
+ .option("url", "jdbc:mysql://localhost:3306/test")
+ .option("dbtable", "demo")
+ .option("user", "root")
+ .option("password", "123456")
+ .load()
+
+ val tokenizer = new Tokenizer().setInputCol("text").setOutputCol("tokens")
+ val tokenizedDf = tokenizer.transform(dfMysql)
+
+ // Learn a mapping from words to Vectors.
+ val word2Vec = new Word2Vec()
+ .setInputCol("tokens")
+ .setOutputCol("vectors")
+ .setVectorSize(128)
+ .setMinCount(0)
+ val model = word2Vec.fit(tokenizedDf)
+
+ val result = model.transform(tokenizedDf)
+
+ val vectorToArrayUDF = udf((v: Vector) => v.toArray)
+ // Apply the UDF to the DataFrame
+ val resultDF = result.withColumn("embedding", vectorToArrayUDF($"vectors"))
+ val milvusDf = resultDF.drop("tokens").drop("vectors")
+
+ milvusDf.write.format("milvus")
+ .option(MILVUS_HOST, "localhost")
+ .option(MILVUS_PORT, "19530")
+ .option(MILVUS_COLLECTION_NAME, "text_embedding")
+ .option(MILVUS_COLLECTION_VECTOR_FIELD, "embedding")
+ .option(MILVUS_COLLECTION_VECTOR_DIM, "128")
+ .option(MILVUS_COLLECTION_PRIMARY_KEY, "id")
+ .mode(SaveMode.Append)
+ .save()
+}
+```
+
+### Milvus -> Transform -> Milvus
+
+In this demo, we will
+
+1. Read data from a Milvus collection,
+2. Apply a transformation (using PCA as an example), and
+3. Write the transformed data into another Milvus via the Bulkinsert API.
+
+
-## What's new in docs
+## What's new in docs
-_Nov 2022 - Milvus 2.2.0 release_
+_Mar 2024 - Milvus 2.4.0 release_
-- Added guidance on how to [bulk insert entities from files](bulk_insert.md).
-- Added description of [disk-based ANN index](disk_index.md).
-- Added guidance on how to configure [quotas_and_limits](configure_quota_limits.md).
+- Added guidance on how to [conduct hybrid search](multi-vector-search.md).
+- Added description of [GPU index](gpu_index.md).
+- Added guidance on how to [embed your data using PyMilvus](embeddings.md).
- Added guidance on how to [enable RBAC](rbac.md) with Milvus Operator.
-- Added guidance on how to [migrate metadata](upgrade_milvus_cluster-helm.md) from 2.1.x to 2.2.0
+- Added descriptions of [Milvus CDC](milvus-cdc-overview.md).
diff --git a/preview/site/en/integrations/integrate_with_bentoml.md b/preview/site/en/integrations/integrate_with_bentoml.md
new file mode 100644
index 000000000..e576caa71
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_bentoml.md
@@ -0,0 +1,295 @@
+---
+id: integrate_with_bentoml.md
+summary: This guide demonstrates how to use an open-source embedding model and large-language model on BentoCloud with Milvus vector database to build a Retrieval Augmented Generation (RAG) application.
+title: Retrieval-Augmented Generation (RAG) with Milvus and BentoML
+---
+
+# Retrieval-Augmented Generation (RAG) with Milvus and BentoML
+
+
+
+## Introduction
+This guide demonstrates how to use an open-source embedding model and large-language model on BentoCloud with Milvus vector database to build a RAG (Retrieval Augmented Generation) application.
+BentoCloud is an AI Inference Platform for fast-moving AI teams, offering fully-managed infrastructure tailored for model inference. It works in conjunction with BentoML, an open-source model serving framework, to facilitate the easy creation and deployment of high-performance model services. In this demo, we use Milvus Lite as vector database, which is the lightweight version of Milvus that can be embedded into your Python application.
+
+## Before you begin
+Milvus Lite is available on PyPI. You can install it via pip for Python 3.8+:
+
+
+```python
+$ pip install -U pymilvus bentoml
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime** (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+After sign in the BentoCloud, we can interact with deployed BentoCloud Services in Deployments, and the corresponding END_POINT and API are located in Playground -> Python.
+You can download the city data [here](https://github.com/ytang07/bento_octo_milvus_RAG/tree/main/data).
+
+## Serving Embeddings with BentoML/BentoCloud
+To use this endpoint, import `bentoml` and set up an HTTP client using the `SyncHTTPClient` by specifying the endpoint and optionally the token (if you turn on `Endpoint Authorization` on BentoCloud). Alternatively, you can use the same model served through BentoML using its [Sentence Transformers Embeddings](https://github.com/bentoml/BentoSentenceTransformers) repository.
+
+
+```python
+import bentoml
+
+BENTO_EMBEDDING_MODEL_END_POINT = "BENTO_EMBEDDING_MODEL_END_POINT"
+BENTO_API_TOKEN = "BENTO_API_TOKEN"
+
+embedding_client = bentoml.SyncHTTPClient(
+ BENTO_EMBEDDING_MODEL_END_POINT, token=BENTO_API_TOKEN
+)
+```
+
+Once we connect to the embedding_client, we need to process our data. We provided several functions to perform data splitting and embedding.
+
+Read files and preprocess the text into a list of strings.
+
+
+```python
+# naively chunk on newlines
+def chunk_text(filename: str) -> list:
+ with open(filename, "r") as f:
+ text = f.read()
+ sentences = text.split("\n")
+ return sentences
+```
+
+First we need to download the city data.
+
+
+```python
+import os
+import requests
+import urllib.request
+
+# set up the data source
+repo = "ytang07/bento_octo_milvus_RAG"
+directory = "data"
+save_dir = "./city_data"
+api_url = f"https://api.github.com/repos/{repo}/contents/{directory}"
+
+
+response = requests.get(api_url)
+data = response.json()
+
+if not os.path.exists(save_dir):
+ os.makedirs(save_dir)
+
+for item in data:
+ if item["type"] == "file":
+ file_url = item["download_url"]
+ file_path = os.path.join(save_dir, item["name"])
+ urllib.request.urlretrieve(file_url, file_path)
+```
+
+Next, we process each of the files we have.
+
+
+```python
+# please upload your data directory under this file's folder
+cities = os.listdir("city_data")
+# store chunked text for each of the cities in a list of dicts
+city_chunks = []
+for city in cities:
+ chunked = chunk_text(f"city_data/{city}")
+ cleaned = []
+ for chunk in chunked:
+ if len(chunk) > 7:
+ cleaned.append(chunk)
+ mapped = {"city_name": city.split(".")[0], "chunks": cleaned}
+ city_chunks.append(mapped)
+```
+
+Splits a list of strings into a list of embeddings, each grouped 25 text strings.
+
+
+```python
+def get_embeddings(texts: list) -> list:
+ if len(texts) > 25:
+ splits = [texts[x : x + 25] for x in range(0, len(texts), 25)]
+ embeddings = []
+ for split in splits:
+ embedding_split = embedding_client.encode(sentences=split)
+ embeddings += embedding_split
+ return embeddings
+ return embedding_client.encode(
+ sentences=texts,
+ )
+```
+
+Now, we need to match up embeddings and text chunks. Since the list embeddings and the list of sentences should match by index, we can `enumerate` through either list to match them up.
+
+
+```python
+entries = []
+for city_dict in city_chunks:
+ # No need for the embeddings list if get_embeddings already returns a list of lists
+ embedding_list = get_embeddings(city_dict["chunks"]) # returns a list of lists
+ # Now match texts with embeddings and city name
+ for i, embedding in enumerate(embedding_list):
+ entry = {
+ "embedding": embedding,
+ "sentence": city_dict["chunks"][
+ i
+ ], # Assume "chunks" has the corresponding texts for the embeddings
+ "city": city_dict["city_name"],
+ }
+ entries.append(entry)
+ print(entries)
+```
+
+## Inserting Data into a Vector Database for Retrieval
+With our embeddings and data prepared, we can insert the vectors together with metadata into Milvus Lite for vector search later. The first step in this section is to start a client by connecting to Milvus Lite.
+We simply import the `MilvusClient` module and initialize a Milvus Lite client that connects to your Milvus Lite vector database. The dimension size comes from the size of the embedding model, e.g. the Sentence Transformer model `all-MiniLM-L6-v2` produces vectors of 384 dimension.
+
+
+```python
+from pymilvus import MilvusClient
+
+COLLECTION_NAME = "Bento_Milvus_RAG" # random name for your collection
+DIMENSION = 384
+
+# Initialize a Milvus Lite client
+milvus_client = MilvusClient("milvus_demo.db")
+```
+
+
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+Or with old connections.connect API (not recommended):
+
+
+```python
+from pymilvus import connections
+
+connections.connect(uri="milvus_demo.db")
+```
+
+## Creating Your Milvus Lite Collection
+Creating a collection using Milvus Lite involves two steps: first, defining the schema, and second, defining the index. For this section, we need one module: DataType tells us what type of data will be in a field. We also need to use two functions to create schema and add fields. create_schema(): creates a collection schema, add_field(): adds a field to the schema of a collection.
+
+
+```python
+from pymilvus import MilvusClient, DataType, Collection
+
+# Create schema
+schema = MilvusClient.create_schema(
+ auto_id=True,
+ enable_dynamic_field=True,
+)
+
+# 3.2. Add fields to schema
+schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
+schema.add_field(field_name="embedding", datatype=DataType.FLOAT_VECTOR, dim=DIMENSION)
+```
+
+Now that we have created our schema and successfully defined data field, we need to define the index. In terms of search, an "index" defines how we are going to map our data out for retrieval. We use the default choice [AUTOINDEX](https://docs.zilliz.com/docs/autoindex-explained) to index our data for this project.
+
+Next, we create the collection with the previously given name, schema and index. Finally, we insert the previously processed data.
+
+
+```python
+# prepare index parameters
+index_params = milvus_client.prepare_index_params()
+
+# add index
+index_params.add_index(
+ field_name="embedding",
+ index_type="AUTOINDEX", # use autoindex instead of other complex indexing method
+ metric_type="COSINE", # L2, COSINE, or IP
+)
+
+# create collection
+if milvus_client.has_collection(collection_name=COLLECTION_NAME):
+ milvus_client.drop_collection(collection_name=COLLECTION_NAME)
+milvus_client.create_collection(
+ collection_name=COLLECTION_NAME, schema=schema, index_params=index_params
+)
+
+# Outside the loop, now you upsert all the entries at once
+milvus_client.insert(collection_name=COLLECTION_NAME, data=entries)
+```
+
+## Set up Your LLM for RAG
+To build a RAG app, we need to deploy an LLM on BentoCloud. Let’s use the latest Llama3 LLM. Once it is up and running, simply copy the endpoint and token of this model service and set up a client for it.
+
+
+```python
+BENTO_LLM_END_POINT = "BENTO_LLM_END_POINT"
+
+llm_client = bentoml.SyncHTTPClient(BENTO_LLM_END_POINT, token=BENTO_API_TOKEN)
+```
+
+## LLM Instructions
+Now, we set up the LLM instructions with the prompt, context, and the question. Here is the function that behaves as an LLM and it then returns the output from the client in a string format.
+
+
+```python
+def dorag(question: str, context: str):
+
+ prompt = (
+ f"You are a helpful assistant. The user has a question. Answer the user question based only on the context: {context}. \n"
+ f"The user question is {question}"
+ )
+
+ results = llm_client.generate(
+ max_tokens=1024,
+ prompt=prompt,
+ )
+
+ res = ""
+ for result in results:
+ res += result
+
+ return res
+```
+
+## A RAG Example
+Now we’re ready to ask a question. This function simply takes a question and then does RAG to generate the relevant context from the background information. Then, we pass the context and the question to dorag() and get the result.
+
+
+```python
+question = "What state is Cambridge in?"
+
+
+def ask_a_question(question):
+ embeddings = get_embeddings([question])
+ res = milvus_client.search(
+ collection_name=COLLECTION_NAME,
+ data=embeddings, # search for the one (1) embedding returned as a list of lists
+ anns_field="embedding", # Search across embeddings
+ limit=5, # get me the top 5 results
+ output_fields=["sentence"], # get the sentence/chunk and city
+ )
+
+ sentences = []
+ for hits in res:
+ for hit in hits:
+ print(hit)
+ sentences.append(hit["entity"]["sentence"])
+ context = ". ".join(sentences)
+ return context
+
+
+context = ask_a_question(question=question)
+print(context)
+```
+
+Implement RAG
+
+
+```python
+print(dorag(question=question, context=context))
+```
+
+For the example question asking which state Cambridge is in, we can print the entire response from BentoML. However, if we take the time to parse through it, it just looks nicer, and it should tell us that Cambridge is located in Massachusetts.
diff --git a/preview/site/en/integrations/integrate_with_cohere.md b/preview/site/en/integrations/integrate_with_cohere.md
new file mode 100644
index 000000000..52ba6d49c
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_cohere.md
@@ -0,0 +1,258 @@
+---
+id: integrate_with_cohere.md
+summary: This page goes over how to search for the best answer to questions using Milvus as the Vector Database and Hugging Face as the embedding system.
+title: Question Answering Using Milvus and Cohere
+---
+
+# Question Answering Using Milvus and Cohere
+
+This page illustrates how to create a question-answering system based on the SQuAD dataset using Milvus as the vector database and Cohere as the embedding system.
+
+## Before you begin
+
+Code snippets on this page require **pymilvus**, **cohere**, **pandas**, **numpy**, and **tqdm** installed. Among these packages, **pymilvus** is the client for Milvus. If not present on your system, run the following commands to install them:
+
+```shell
+pip install pymilvus cohere pandas numpy tqdm
+```
+
+Then you need to load the modules to be used in this guide.
+
+```python
+import cohere
+import pandas
+import numpy as np
+from tqdm import tqdm
+from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection, utility
+```
+
+## Parameters
+
+Here we can find the parameters used in the following snippets. Some of them need to be changed to fit your environment. Beside each is a description of what it is.
+
+```python
+FILE = 'https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json' # The SQuAD dataset url
+COLLECTION_NAME = 'question_answering_db' # Collection name
+DIMENSION = 1024 # Embeddings size, cohere embeddings default to 4096 with the large model
+COUNT = 5000 # How many questions to embed and insert into Milvus
+BATCH_SIZE = 96 # How large of batches to use for embedding and insertion
+MILVUS_HOST = 'localhost' # Milvus server URI
+MILVUS_PORT = '19530'
+COHERE_API_KEY = 'replace-this-with-the-cohere-api-key' # API key obtained from Cohere
+```
+
+To know more about the model and dataset used on this page, refer to [co:here](https://cohere.ai/) and [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/).
+
+## Prepare the dataset
+
+In this example, we are going to use the Stanford Question Answering Dataset (SQuAD) as our truth source for answering questions. This dataset comes in the form of a JSON file and we are going to use **pandas** to load it in.
+
+```python
+# Download the dataset
+dataset = pandas.read_json(FILE)
+
+# Clean up the dataset by grabbing all the question answer pairs
+simplified_records = []
+for x in dataset['data']:
+ for y in x['paragraphs']:
+ for z in y['qas']:
+ if len(z['answers']) != 0:
+ simplified_records.append({'question': z['question'], 'answer': z['answers'][0]['text']})
+
+# Grab the amount of records based on COUNT
+simplified_records = pandas.DataFrame.from_records(simplified_records)
+simplified_records = simplified_records.sample(n=min(COUNT, len(simplified_records)), random_state = 42)
+
+# Check the length of the cleaned dataset matches count
+print(len(simplified_records))
+```
+
+The output should be the number of records in the dataset
+
+```shell
+5000
+```
+
+## Create a collection
+
+This section deals with Milvus and setting up the database for this use case. Within Milvus, we need to set up a collection and index it.
+
+```python
+# Connect to Milvus Database
+connections.connect(host=MILVUS_HOST, port=MILVUS_PORT)
+
+# Remove collection if it already exists
+if utility.has_collection(COLLECTION_NAME):
+ utility.drop_collection(COLLECTION_NAME)
+
+# Create collection which includes the id, title, and embedding.
+fields = [
+ FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),
+ FieldSchema(name='original_question', dtype=DataType.VARCHAR, max_length=1000),
+ FieldSchema(name='answer', dtype=DataType.VARCHAR, max_length=1000),
+ FieldSchema(name='original_question_embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)
+]
+schema = CollectionSchema(fields=fields)
+collection = Collection(name=COLLECTION_NAME, schema=schema)
+
+# Create an IVF_FLAT index for collection.
+index_params = {
+ 'metric_type':'IP',
+ 'index_type':"IVF_FLAT",
+ 'params':{"nlist": 1024}
+}
+collection.create_index(field_name="original_question_embedding", index_params=index_params)
+collection.load()
+```
+
+## Insert data
+
+Once we have the collection set up we need to start inserting our data. This is done in three steps
+
+- reading the data,
+- embedding the original questions, and
+- inserting the data into the collection we've just created on Milvus.
+
+In this example, the data includes the original question, the original question's embedding, and the answer to the original question.
+
+```python
+# Set up a co:here client.
+cohere_client = cohere.Client(COHERE_API_KEY)
+
+# Extract embeddings from questions using Cohere
+def embed(texts, input_type):
+ res = cohere_client.embed(texts, model='embed-multilingual-v3.0', input_type=input_type)
+ return res.embeddings
+
+# Insert each question, answer, and qustion embedding
+total = pandas.DataFrame()
+for batch in tqdm(np.array_split(simplified_records, (COUNT/BATCH_SIZE) + 1)):
+ questions = batch['question'].tolist()
+ embeddings = embed(questions, "search_document")
+
+ data = [
+ {
+ 'original_question': x,
+ 'answer': batch['answer'].tolist()[i],
+ 'original_question_embedding': embeddings[i]
+ } for i, x in enumerate(questions)
+ ]
+
+ collection.insert(data=data)
+
+time.sleep(10)
+```
+
+## Ask questions
+
+Once all the data is inserted into the Milvus collection, we can ask the system questions by taking our question phrase, embedding it with Cohere, and searching with the collection.
+
+
+
+Searches performed on data right after insertion might be a little slower as searching unindexed data is done in a brute-force manner. Once the new data is automatically indexed, the searches will speed up.
+
+
+
+```python
+# Search the cluster for an answer to a question text
+def search(text, top_k = 5):
+
+ # AUTOINDEX does not require any search params
+ search_params = {}
+
+ results = collection.search(
+ data = embed([text], "search_query"), # Embeded the question
+ anns_field='original_question_embedding',
+ param=search_params,
+ limit = top_k, # Limit to top_k results per search
+ output_fields=['original_question', 'answer'] # Include the original question and answer in the result
+ )
+
+ distances = results[0].distances
+ entities = [ x.entity.to_dict()['entity'] for x in results[0] ]
+
+ ret = [ {
+ "answer": x[1]["answer"],
+ "distance": x[0],
+ "original_question": x[1]['original_question']
+ } for x in zip(distances, entities)]
+
+ return ret
+
+# Ask these questions
+search_questions = ['What kills bacteria?', 'What\'s the biggest dog?']
+
+# Print out the results in order of [answer, similarity score, original question]
+
+ret = [ { "question": x, "candidates": search(x) } for x in search_questions ]
+```
+
+The output should be similar to the following:
+
+```shell
+# Output
+#
+# [
+# {
+# "question": "What kills bacteria?",
+# "candidates": [
+# {
+# "answer": "farming",
+# "distance": 0.6261022090911865,
+# "original_question": "What makes bacteria resistant to antibiotic treatment?"
+# },
+# {
+# "answer": "Phage therapy",
+# "distance": 0.6093736886978149,
+# "original_question": "What has been talked about to treat resistant bacteria?"
+# },
+# {
+# "answer": "oral contraceptives",
+# "distance": 0.5902313590049744,
+# "original_question": "In therapy, what does the antibacterial interact with?"
+# },
+# {
+# "answer": "slowing down the multiplication of bacteria or killing the bacteria",
+# "distance": 0.5874154567718506,
+# "original_question": "How do antibiotics work?"
+# },
+# {
+# "answer": "in intensive farming to promote animal growth",
+# "distance": 0.5667208433151245,
+# "original_question": "Besides in treating human disease where else are antibiotics used?"
+# }
+# ]
+# },
+# {
+# "question": "What's the biggest dog?",
+# "candidates": [
+# {
+# "answer": "English Mastiff",
+# "distance": 0.7875324487686157,
+# "original_question": "What breed was the largest dog known to have lived?"
+# },
+# {
+# "answer": "forest elephants",
+# "distance": 0.5886962413787842,
+# "original_question": "What large animals reside in the national park?"
+# },
+# {
+# "answer": "Rico",
+# "distance": 0.5634892582893372,
+# "original_question": "What is the name of the dog that could ID over 200 things?"
+# },
+# {
+# "answer": "Iditarod Trail Sled Dog Race",
+# "distance": 0.546872615814209,
+# "original_question": "Which dog-sled race in Alaska is the most famous?"
+# },
+# {
+# "answer": "part of the family",
+# "distance": 0.5387814044952393,
+# "original_question": "Most people today describe their dogs as what?"
+# }
+# ]
+# }
+# ]
+
+```
diff --git a/preview/site/en/integrations/integrate_with_dspy.md b/preview/site/en/integrations/integrate_with_dspy.md
new file mode 100644
index 000000000..4616f52cc
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_dspy.md
@@ -0,0 +1,231 @@
+---
+id: integrate_with_dspy.md
+summary: This guide demonstrates how to use MilvusRM, one of DSPy's retriever modules, to optimize RAG programs.
+title: Integrate Milvus with DSPy
+---
+
+# Integrate Milvus with DSPy
+
+
+
+## What is DSPy
+DSPy, introduced by the Stanford NLP Group, stands as a groundbreaking programmatic framework designed to optimize prompts and weights within language models, particularly valuable in scenarios where large language models (LLMs) are integrated across multiple stages of a pipeline. Unlike conventional prompting engineering techniques reliant on manual crafting and tweaking, DSPy adopts a learning-based approach. By assimilating query-answer examples, DSPy generates optimized prompts dynamically, tailored to specific tasks. This innovative methodology enables the seamless reassembly of entire pipelines, eliminating the need for continuous manual prompt adjustments. DSPy's Pythonic syntax offers various composable and declarative modules, simplifying the instruction of LLMs.
+
+## Benefits of using DSPy
+- Programming Approach: DSPy provides a systematic programming approach for LM pipeline development by abstracting pipelines as text transformation graphs instead of just prompting the LLMs. Its declarative modules enable structured design and optimization, replacing the trial-and-error method of traditional prompt templates.
+- Performance Improvement: DSPy demonstrates significant performance gains over existing methods. Through case studies, it outperforms standard prompting and expert-created demonstrations, showcasing its versatility and effectiveness even when compiled to smaller LM models.
+- Modularized Abstraction: DSPy effectively abstracts intricate aspects of LM pipeline development, such as decomposition, fine-tuning, and model selection. With DSPy, a concise program can seamlessly translate into instructions for various models, such as GPT-4, Llama2-13b, or T5-base, streamlining development and enhancing performance.
+
+## Modules
+There are numerous components that contribute to constructing an LLM pipeline. Here, we'll describe some key components to provide a high-level understanding of how DSPy operates.
+
+
+
+Signature: Signatures in DSPy serve as declarative specifications, outlining the input/output behavior of modules, guiding the language model in task execution.
+Module: DSPy modules serve as fundamental components for programs leveraging language models (LMs). They abstract various prompting techniques, such as chain of thought or ReAct, and are adaptable to handle any DSPy Signature. With learnable parameters and the ability to process inputs and produce outputs, these modules can be combined to form larger programs, drawing inspiration from NN modules in PyTorch but tailored for LM applications.
+Optimizer: Optimizers in DSPy fine-tune the parameters of DSPy programs, such as prompts and LLM weights, to maximize specified metrics like accuracy, enhancing program efficiency.
+
+## Why Milvus in DSPy
+DSPy is a powerful programming framework that boosts RAG applications. Such application needs to retrieve useful information to enhance answer quality, which needs vector database. Milvus is a well-known open-source vector database to improve performance and scalability. With MilvusRM, a retriever module in DSPy, integrating Milvus becomes seamless. Now, developers can easily define and optimize RAG programs using DSPy, taking advantage of Milvus' strong vector search capabilities. This collaboration makes RAG applications more efficient and scalable, combining DSPy's programming capabilities with Milvus' search features.
+
+## Examples
+Now, let's walk through a quick example to demonstrate how to leverage Milvus in DSPy for optimizing a RAG application.
+
+### Prerequisites
+Before building the RAG app, install the DSPy and PyMilvus.
+
+
+```python
+$ pip install "dspy-ai[milvus]"
+$ pip install -U pymilvus
+```
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime** (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+### Loading the dataset
+In this example, we use the HotPotQA, a collection of complex question-answer pairs, as our training dataset. We can load them through the HotPotQA class.
+
+
+```python
+from dspy.datasets import HotPotQA
+
+# Load the dataset.
+dataset = HotPotQA(
+ train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0
+)
+
+# Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.
+trainset = [x.with_inputs("question") for x in dataset.train]
+devset = [x.with_inputs("question") for x in dataset.dev]
+```
+
+### Ingest data into the Milvus vector database
+Ingest the context information into the Milvus collection for vector retrieval. This collection should have an `embedding` field and a `text` field. We use OpenAI's `text-embedding-3-small` model as the default query embedding function in this case.
+
+
+```python
+import requests
+import os
+
+os.environ["OPENAI_API_KEY"] = "
+
+This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using Haystack and Milvus.
+
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+
+[Haystack](https://haystack.deepset.ai/) is the open source Python framework by deepset for building custom apps with large language models (LLMs). [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
+
+
+
+## Prerequisites
+
+Before running this notebook, make sure you have the following dependencies installed:
+
+
+```python
+! pip install --upgrade --quiet pymilvus milvus-haystack markdown-it-py mdit_plain
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime** (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+We will use the models from OpenAI. You should prepare the [api key](https://platform.openai.com/docs/quickstart) `OPENAI_API_KEY` as an environment variable.
+
+
+```python
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+```
+
+## Prepare the data
+
+We use an online content about [Leonardo Da Vinci](https://www.gutenberg.org/cache/epub/7785/pg7785.txt) as a store of private knowledge for our RAG pipeline, which is a good data source for a simple RAG pipeline.
+
+Download it and save it as a local text file.
+
+
+```python
+import os
+import urllib.request
+
+url = "https://www.gutenberg.org/cache/epub/7785/pg7785.txt"
+file_path = "./davinci.txt"
+
+if not os.path.exists(file_path):
+ urllib.request.urlretrieve(url, file_path)
+```
+
+## Create the indexing Pipeline
+
+Create an indexing pipeline that converts the text into documents, splits them into sentences, and embeds them. The documents are then written to the Milvus document store.
+
+
+```python
+from haystack import Pipeline
+from haystack.components.converters import MarkdownToDocument
+from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
+from haystack.components.preprocessors import DocumentSplitter
+from haystack.components.writers import DocumentWriter
+from haystack.utils import Secret
+
+from milvus_haystack import MilvusDocumentStore
+from milvus_haystack.milvus_embedding_retriever import MilvusEmbeddingRetriever
+
+
+document_store = MilvusDocumentStore(
+ connection_args={"uri": "./milvus.db"},
+ # connection_args={"uri": "http://localhost:19530"},
+ # connection_args={"uri": YOUR_ZILLIZ_CLOUD_URI, "token": Secret.from_env_var("ZILLIZ_CLOUD_API_KEY")},
+ drop_old=True,
+)
+```
+
+
+
+For the connection_args:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+
+```python
+indexing_pipeline = Pipeline()
+indexing_pipeline.add_component("converter", MarkdownToDocument())
+indexing_pipeline.add_component(
+ "splitter", DocumentSplitter(split_by="sentence", split_length=2)
+)
+indexing_pipeline.add_component("embedder", OpenAIDocumentEmbedder())
+indexing_pipeline.add_component("writer", DocumentWriter(document_store))
+indexing_pipeline.connect("converter", "splitter")
+indexing_pipeline.connect("splitter", "embedder")
+indexing_pipeline.connect("embedder", "writer")
+indexing_pipeline.run({"converter": {"sources": [file_path]}})
+
+print("Number of documents:", document_store.count_documents())
+```
+
+ Converting markdown files to Documents: 100%|█| 1/
+ Calculating embeddings: 100%|█| 9/9 [00:05<00:00,
+ E20240516 10:40:32.945937 5309095 milvus_local.cpp:189] [SERVER][GetCollection][] Collecton HaystackCollection not existed
+ E20240516 10:40:32.946677 5309095 milvus_local.cpp:189] [SERVER][GetCollection][] Collecton HaystackCollection not existed
+ E20240516 10:40:32.946704 5309095 milvus_local.cpp:189] [SERVER][GetCollection][] Collecton HaystackCollection not existed
+ E20240516 10:40:32.946725 5309095 milvus_local.cpp:189] [SERVER][GetCollection][] Collecton HaystackCollection not existed
+
+
+ Number of documents: 277
+
+
+## Create the retrieval pipeline
+
+Create a retrieval pipeline that retrieves documents from the Milvus document store using a vector similarity search engine.
+
+
+```python
+question = 'Where is the painting "Warrior" currently stored?'
+
+retrieval_pipeline = Pipeline()
+retrieval_pipeline.add_component("embedder", OpenAITextEmbedder())
+retrieval_pipeline.add_component(
+ "retriever", MilvusEmbeddingRetriever(document_store=document_store, top_k=3)
+)
+retrieval_pipeline.connect("embedder", "retriever")
+
+retrieval_results = retrieval_pipeline.run({"embedder": {"text": question}})
+
+for doc in retrieval_results["retriever"]["documents"]:
+ print(doc.content)
+ print("-" * 10)
+```
+
+ ). The
+ composition of this oil-painting seems to have been built up on the
+ second cartoon, which he had made some eight years earlier, and which
+ was apparently taken to France in 1516 and ultimately lost.
+ ----------
+
+ This "Baptism of Christ," which is now in the Accademia in Florence
+ and is in a bad state of preservation, appears to have been a
+ comparatively early work by Verrocchio, and to have been painted
+ in 1480-1482, when Leonardo would be about thirty years of age.
+
+ To about this period belongs the superb drawing of the "Warrior," now
+ in the Malcolm Collection in the British Museum.
+ ----------
+ " Although he
+ completed the cartoon, the only part of the composition which he
+ eventually executed in colour was an incident in the foreground
+ which dealt with the "Battle of the Standard." One of the many
+ supposed copies of a study of this mural painting now hangs on the
+ south-east staircase in the Victoria and Albert Museum.
+ ----------
+
+
+## Create the RAG pipeline
+
+Create a RAG pipeline that combines the MilvusEmbeddingRetriever and the OpenAIGenerator to answer the question using the retrieved documents.
+
+
+```python
+from haystack.utils import Secret
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+
+prompt_template = """Answer the following query based on the provided context. If the context does
+ not include an answer, reply with 'I don't know'.\n
+ Query: {{query}}
+ Documents:
+ {% for doc in documents %}
+ {{ doc.content }}
+ {% endfor %}
+ Answer:
+ """
+
+rag_pipeline = Pipeline()
+rag_pipeline.add_component("text_embedder", OpenAITextEmbedder())
+rag_pipeline.add_component(
+ "retriever", MilvusEmbeddingRetriever(document_store=document_store, top_k=3)
+)
+rag_pipeline.add_component("prompt_builder", PromptBuilder(template=prompt_template))
+rag_pipeline.add_component(
+ "generator",
+ OpenAIGenerator(
+ api_key=Secret.from_token(os.getenv("OPENAI_API_KEY")),
+ generation_kwargs={"temperature": 0},
+ ),
+)
+rag_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
+rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
+rag_pipeline.connect("prompt_builder", "generator")
+
+results = rag_pipeline.run(
+ {
+ "text_embedder": {"text": question},
+ "prompt_builder": {"query": question},
+ }
+)
+print("RAG answer:", results["generator"]["replies"][0])
+```
+
+ RAG answer: The painting "Warrior" is currently stored in the Malcolm Collection in the British Museum.
+
+
+For more information about how to use milvus-haystack, please refer to the [milvus-haystack Readme](https://github.com/milvus-io/milvus-haystack).
diff --git a/preview/site/en/integrations/integrate_with_hugging-face.md b/preview/site/en/integrations/integrate_with_hugging-face.md
new file mode 100644
index 000000000..c5c956904
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_hugging-face.md
@@ -0,0 +1,213 @@
+---
+id: integrate_with_hugging-face.md
+summary: This tutorial shows how to build a question answering system using Hugging Face as the data loader & embedding generator for data processing and Milvus as the vector database for semantic search.
+title: Question Answering Using Milvus and Hugging Face
+---
+
+# Question Answering Using Milvus and Hugging Face
+
+
+
+A question answering system based on semantic search works by finding the most similar question from a dataset of question-answer pairs for a given query question. Once the most similar question is identified, the corresponding answer from the dataset is considered as the answer for the query. This approach relies on semantic similarity measures to determine the similarity between questions and retrieve relevant answers.
+
+This tutorial shows how to build a question answering system using [Hugging Face](https://huggingface.co) as the data loader & embedding generator for data processing and [Milvus](https://milvus.io) as the vector database for semantic search.
+
+## Before you begin
+
+You need to make sure all required dependencies are installed:
+
+- `pymilvus`: a python package works with the vector database service powered by Milvus or Zilliz Cloud.
+- `datasets`, `transformers`: Hugging Face packages manage data and utilize models.
+- `torch`: a powerful library provides efficient tensor computation and deep learning tools.
+
+
+```python
+$ pip install --upgrade pymilvus transformers datasets torch
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**. (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+## Prepare data
+
+In this section, we will load example question-answer pairs from the Hugging Face Datasets. As a demo, we only take partial data from the validation split of [SQuAD](https://huggingface.co/datasets/rajpurkar/squad).
+
+
+```python
+from datasets import load_dataset
+
+
+DATASET = "squad" # Name of dataset from HuggingFace Datasets
+INSERT_RATIO = 0.001 # Ratio of example dataset to be inserted
+
+data = load_dataset(DATASET, split="validation")
+# Generates a fixed subset. To generate a random subset, remove the seed.
+data = data.train_test_split(test_size=INSERT_RATIO, seed=42)["test"]
+# Clean up the data structure in the dataset.
+data = data.map(
+ lambda val: {"answer": val["answers"]["text"][0]},
+ remove_columns=["id", "answers", "context"],
+)
+
+# View summary of example data
+print(data)
+```
+
+ Dataset({
+ features: ['title', 'question', 'answer'],
+ num_rows: 11
+ })
+
+
+To generate embeddings for questions, you are able to select a text embedding model from Hugging Face Models. In this tutorial, we will use a small sentencce embedding model [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) as example.
+
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+MODEL = (
+ "sentence-transformers/all-MiniLM-L6-v2" # Name of model from HuggingFace Models
+)
+INFERENCE_BATCH_SIZE = 64 # Batch size of model inference
+
+# Load tokenizer & model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained(MODEL)
+model = AutoModel.from_pretrained(MODEL)
+
+
+def encode_text(batch):
+ # Tokenize sentences
+ encoded_input = tokenizer(
+ batch["question"], padding=True, truncation=True, return_tensors="pt"
+ )
+
+ # Compute token embeddings
+ with torch.no_grad():
+ model_output = model(**encoded_input)
+
+ # Perform pooling
+ token_embeddings = model_output[0]
+ attention_mask = encoded_input["attention_mask"]
+ input_mask_expanded = (
+ attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+ )
+ sentence_embeddings = torch.sum(
+ token_embeddings * input_mask_expanded, 1
+ ) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+ # Normalize embeddings
+ batch["question_embedding"] = torch.nn.functional.normalize(
+ sentence_embeddings, p=2, dim=1
+ )
+ return batch
+
+
+data = data.map(encode_text, batched=True, batch_size=INFERENCE_BATCH_SIZE)
+data_list = data.to_list()
+```
+
+## Insert data
+
+Now we have question-answer pairs ready with question embeddings. The next step is to insert them into the vector database.
+
+We will first need to connect to Milvus service and create a Milvus collection.
+
+
+```python
+from pymilvus import MilvusClient
+
+
+MILVUS_URI = "./huggingface_milvus_test.db" # Connection URI
+COLLECTION_NAME = "huggingface_test" # Collection name
+DIMENSION = 384 # Embedding dimension depending on model
+
+milvus_client = MilvusClient(MILVUS_URI)
+if milvus_client.has_collection(collection_name=COLLECTION_NAME):
+ milvus_client.drop_collection(collection_name=COLLECTION_NAME)
+milvus_client.create_collection(
+ collection_name=COLLECTION_NAME,
+ dimension=DIMENSION,
+ auto_id=True, # Enable auto id
+ enable_dynamic_field=True, # Enable dynamic fields
+ vector_field_name="question_embedding", # Map vector field name and embedding column in dataset
+ consistency_level="Strong", # To enable search with latest data
+)
+```
+
+
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+Insert all data into the collection:
+
+
+```python
+milvus_client.insert(collection_name=COLLECTION_NAME, data=data_list)
+```
+
+
+
+
+ {'insert_count': 11,
+ 'ids': [450072488481390592, 450072488481390593, 450072488481390594, 450072488481390595, 450072488481390596, 450072488481390597, 450072488481390598, 450072488481390599, 450072488481390600, 450072488481390601, 450072488481390602],
+ 'cost': 0}
+
+
+
+## Ask questions
+
+Once all the data is inserted into Milvus, we can ask questions and see what the closest answers are.
+
+
+```python
+questions = {
+ "question": [
+ "What is LGM?",
+ "When did Massachusetts first mandate that children be educated in schools?",
+ ]
+}
+
+# Generate question embeddings
+question_embeddings = [v.tolist() for v in encode_text(questions)["question_embedding"]]
+
+# Search across Milvus
+search_results = milvus_client.search(
+ collection_name=COLLECTION_NAME,
+ data=question_embeddings,
+ limit=3, # How many search results to output
+ output_fields=["answer", "question"], # Include these fields in search results
+)
+
+# Print out results
+for q, res in zip(questions["question"], search_results):
+ print("Question:", q)
+ for r in res:
+ print(
+ {
+ "answer": r["entity"]["answer"],
+ "score": r["distance"],
+ "original question": r["entity"]["question"],
+ }
+ )
+ print("\n")
+```
+
+ Question: What is LGM?
+ {'answer': 'Last Glacial Maximum', 'score': 0.956273078918457, 'original question': 'What does LGM stands for?'}
+ {'answer': 'coordinate the response to the embargo', 'score': 0.2120140939950943, 'original question': 'Why was this short termed organization created?'}
+ {'answer': '"Reducibility Among Combinatorial Problems"', 'score': 0.1945795714855194, 'original question': 'What is the paper written by Richard Karp in 1972 that ushered in a new era of understanding between intractability and NP-complete problems?'}
+
+
+ Question: When did Massachusetts first mandate that children be educated in schools?
+ {'answer': '1852', 'score': 0.9709997177124023, 'original question': 'In what year did Massachusetts first require children to be educated in schools?'}
+ {'answer': 'several regional colleges and universities', 'score': 0.34164726734161377, 'original question': 'In 1890, who did the university decide to team up with?'}
+ {'answer': '1962', 'score': 0.1931006908416748, 'original question': 'When were stromules discovered?'}
diff --git a/preview/site/en/integrations/integrate_with_jina.md b/preview/site/en/integrations/integrate_with_jina.md
new file mode 100644
index 000000000..8d9b67ed7
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_jina.md
@@ -0,0 +1,210 @@
+---
+id: integrate_with_jina.md
+summary: This guide demonstrates how to use Jina embeddings and Milvus to conduct similarity search and retrieval tasks.
+title: Integrate Milvus with Jina
+---
+
+# Integrate Milvus with Jina AI
+
+
+
+This guide demonstrates how to use Jina AI embeddings and Milvus to conduct similarity search and retrieval tasks.
+
+## Who is Jina AI
+Jina AI, founded in 2020 in Berlin, is a pioneering AI company focused on revolutionizing the future of artificial intelligence through its search foundation. Specializing in multimodal AI, Jina AI aims to empower businesses and developers to harness the power of multimodal data for value creation and cost savings through its integrated suite of components, including embeddings, rerankers, prompt ops, and core infrastructure.
+Jina AI's cutting-edge embeddings boast top-tier performance, featuring an 8192 token-length model ideal for comprehensive data representation. Offering multilingual support and seamless integration with leading platforms like OpenAI, these embeddings facilitate cross-lingual applications.
+
+## Milvus and Jina AI's Embedding
+In order to store and search these embeddings efficiently for speed and scale, specific infrastructure designed for this purpose is required. Milvus is a widely known advanced open-source vector database capable of handling large-scale vector data. Milvus enables fast and accurate vector(embedding) search according plenty of metrics. Its scalability allows for seamless handling of massive volumes of image data, ensuring high-performance search operations even as datasets grow.
+
+## Examples
+Jina embeddings have been integrated into the PyMilvus model library. Now, we will demonstrate code examples to show how to use Jina embeddings in action.
+
+Before we start, we need to install model library for PyMilvus.
+
+
+```python
+$ pip install -U pymilvus
+$ pip install "pymilvus[model]"
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**. (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+## General-Purpose Embedding
+Jina AI's core embedding model, excels in understanding detailed text, making it ideal for semantic search, content classification thus supports advanced sentiment analysis, text summarization, and personalized recommendation systems.
+
+
+```python
+from pymilvus.model.dense import JinaEmbeddingFunction
+
+jina_api_key = "
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+With all data in Milvus vector database, we can now perform semantic search by generating vector embedding for the query and conduct vector search.
+
+
+```python
+queries = "What event in 1956 marked the official birth of artificial intelligence as a discipline?"
+qvecs = ef.encode_queries([queries])
+
+res = milvus_client.search(
+ collection_name=COLLECTION_NAME, # target collection
+ data=[qvecs[0]], # query vectors
+ limit=3, # number of returned entities
+ output_fields=["text", "subject"], # specifies fields to be returned
+)[0]
+
+for result in res:
+ print(result)
+```
+
+ {'id': 1, 'distance': 0.8802614808082581, 'entity': {'text': "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.", 'subject': 'history'}}
+
+
+## Jina Reranker
+Jina Ai also provides rerankers to further enhance retrieval quality after searching using embeddings.
+
+
+```python
+from pymilvus.model.reranker import JinaRerankFunction
+
+jina_api_key = "
+
+This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LangChain and Milvus.
+
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+
+[LangChain](https://www.langchain.com/) is a framework for developing applications powered by large language models (LLMs). [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
+
+
+
+## Prerequisites
+
+Before running this notebook, make sure you have the following dependencies installed:
+
+
+```python
+$ pip install --upgrade --quiet langchain langchain-core langchain-community langchain-text-splitters langchain-milvus langchain-openai bs4
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**. (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+We will use the models from OpenAI. You should prepare the [api key](https://platform.openai.com/docs/quickstart) `OPENAI_API_KEY` as an environment variable.
+
+
+```python
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+```
+
+
+## Prepare the data
+
+We use the Langchain WebBaseLoader to load documents from web sources and split them into chunks using the RecursiveCharacterTextSplitter.
+
+
+
+```python
+import bs4
+from langchain_community.document_loaders import WebBaseLoader
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+
+# Create a WebBaseLoader instance to load documents from web sources
+loader = WebBaseLoader(
+ web_paths=(
+ "https://lilianweng.github.io/posts/2023-06-23-agent/",
+ "https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/",
+ ),
+ bs_kwargs=dict(
+ parse_only=bs4.SoupStrainer(
+ class_=("post-content", "post-title", "post-header")
+ )
+ ),
+)
+# Load documents from web sources using the loader
+documents = loader.load()
+# Initialize a RecursiveCharacterTextSplitter for splitting text into chunks
+text_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
+
+# Split the documents into chunks using the text_splitter
+docs = text_splitter.split_documents(documents)
+
+# Let's take a look at the first document
+docs[1]
+```
+
+
+
+
+ Document(page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\nSelf-Reflection#', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'})
+
+
+
+As we can see, the document is already split into chunks. And the content of the data is about the AI agent.
+
+## Build RAG chain with Milvus Vector Store
+
+We will initialize a Milvus vector store with the documents, which load the documents into the Milvus vector store and build an index under the hood.
+
+
+```python
+from langchain_milvus import Milvus, Zilliz
+from langchain_openai import OpenAIEmbeddings
+
+embeddings = OpenAIEmbeddings()
+
+vectorstore = Milvus.from_documents( # or Zilliz.from_documents
+ documents=docs,
+ embedding=embeddings,
+ connection_args={
+ "uri": "./milvus_demo.db",
+ },
+ drop_old=True, # Drop the old Milvus collection if it exists
+)
+```
+
+
+
+For the `connection_args`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, replace `Milvus.from_documents` with `Zilliz.from_documents`, and adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+Search the documents in the Milvus vector store using a test query question. Let's take a look at the top 1 document.
+
+
+```python
+query = "What is self-reflection of an AI Agent?"
+vectorstore.similarity_search(query, k=1)
+```
+
+
+
+
+ [Document(page_content='Self-Reflection#\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\nThought: ...\nAction: ...\nObservation: ...\n... (Repeated many times)', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'pk': 449281835035555859})]
+
+
+
+
+```python
+from langchain_core.runnables import RunnablePassthrough
+from langchain_core.prompts import PromptTemplate
+from langchain_core.output_parsers import StrOutputParser
+from langchain_openai import ChatOpenAI
+
+# Initialize the OpenAI language model for response generation
+llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)
+
+# Define the prompt template for generating AI responses
+PROMPT_TEMPLATE = """
+Human: You are an AI assistant, and provides answers to questions by using fact based and statistical information when possible.
+Use the following pieces of information to provide a concise answer to the question enclosed in
+
+
+This is a simple cookbook that demonstrates how to use the [LlamaIndex Langfuse integration](https://langfuse.com/docs/integrations/llama-index/get-started). It uses Milvus Lite to store the documents and Query.
+
+[Milvus Lite](https://github.com/milvus-io/milvus-lite/) is the lightweight version of Milvus, an open-source vector database that powers AI applications with vector embeddings and similarity search.
+
+## Setup
+
+Make sure you have both `llama-index` and `langfuse` installed.
+
+
+```python
+$ pip install llama-index langfuse llama-index-vector-stores-milvus --upgrade
+```
+
+Initialize the integration. Get your API keys from the [Langfuse project settings](https://cloud.langfuse.com), and replace public_key secret_key with your key values. This example uses OpenAI for embeddings and chat completions, so you also need to specify your OpenAI key in environment variable.
+
+
+```python
+import os
+
+# Get keys for your project from the project settings page
+# https://cloud.langfuse.com
+os.environ["LANGFUSE_PUBLIC_KEY"] = ""
+os.environ["LANGFUSE_SECRET_KEY"] = ""
+os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com" # 🇪🇺 EU region
+# os.environ["LANGFUSE_HOST"] = "https://us.cloud.langfuse.com" # 🇺🇸 US region
+
+# Your openai key
+os.environ["OPENAI_API_KEY"] = ""
+```
+
+
+```python
+from llama_index.core import Settings
+from llama_index.core.callbacks import CallbackManager
+from langfuse.llama_index import LlamaIndexCallbackHandler
+
+langfuse_callback_handler = LlamaIndexCallbackHandler()
+Settings.callback_manager = CallbackManager([langfuse_callback_handler])
+```
+
+## Index using Milvus Lite
+
+
+```python
+from llama_index.core import Document
+
+doc1 = Document(text="""
+Maxwell "Max" Silverstein, a lauded movie director, screenwriter, and producer, was born on October 25, 1978, in Boston, Massachusetts. A film enthusiast from a young age, his journey began with home movies shot on a Super 8 camera. His passion led him to the University of Southern California (USC), majoring in Film Production. Eventually, he started his career as an assistant director at Paramount Pictures. Silverstein's directorial debut, “Doors Unseen,” a psychological thriller, earned him recognition at the Sundance Film Festival and marked the beginning of a successful directing career.
+""")
+doc2 = Document(text="""
+Throughout his career, Silverstein has been celebrated for his diverse range of filmography and unique narrative technique. He masterfully blends suspense, human emotion, and subtle humor in his storylines. Among his notable works are "Fleeting Echoes," "Halcyon Dusk," and the Academy Award-winning sci-fi epic, "Event Horizon's Brink." His contribution to cinema revolves around examining human nature, the complexity of relationships, and probing reality and perception. Off-camera, he is a dedicated philanthropist living in Los Angeles with his wife and two children.
+""")
+```
+
+
+```python
+# Example index construction + LLM query
+
+from llama_index.core import VectorStoreIndex
+from llama_index.core import StorageContext
+from llama_index.vector_stores.milvus import MilvusVectorStore
+
+
+vector_store = MilvusVectorStore(
+ uri="tmp/milvus_demo.db", dim=1536, overwrite=False
+)
+storage_context = StorageContext.from_defaults(vector_store=vector_store)
+
+index = VectorStoreIndex.from_documents(
+ [doc1,doc2], storage_context=storage_context
+)
+```
+
+## Query
+
+
+```python
+# Query
+response = index.as_query_engine().query("What did he do growing up?")
+print(response)
+```
+
+
+```python
+# Chat
+response = index.as_chat_engine().chat("What did he do growing up?")
+print(response)
+```
+
+## Explore traces in Langfuse
+
+
+```python
+# As we want to immediately see result in Langfuse, we need to flush the callback handler
+langfuse_callback_handler.flush()
+```
+
+Done! ✨ You see traces of your index and query in your Langfuse project.
+
+Example traces (public links):
+1. [Query](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/2b26fc72-044f-4b0b-a3c3-485328975161)
+2. [Query (chat)](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/72503163-2b25-4693-9cc9-56190b8e32b9)
+
+Trace in Langfuse:
+
+
+
+
+## Interested in more advanced features?
+
+See the full [integration docs](https://langfuse.com/docs/integrations/llama-index/get-started) to learn more about advanced features and how to use them:
+
+- Interoperability with Langfuse Python SDK and other integrations
+- Add custom metadata and attributes to the traces
diff --git a/preview/site/en/integrations/integrate_with_llamaindex.md b/preview/site/en/integrations/integrate_with_llamaindex.md
new file mode 100644
index 000000000..dcc8aa07f
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_llamaindex.md
@@ -0,0 +1,229 @@
+---
+id: integrate_with_llamaindex.md
+summary: This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LlamaIndex and Milvus.
+title: Retrieval-Augmented Generation (RAG) with Milvus and LlamaIndex
+---
+
+# Retrieval-Augmented Generation (RAG) with Milvus and LlamaIndex
+
+
+
+This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LlamaIndex and Milvus.
+
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+
+[LlamaIndex](https://www.llamaindex.ai/) is a simple, flexible data framework for connecting custom data sources to large language models (LLMs). [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
+
+In this notebook we are going to show a quick demo of using the MilvusVectorStore.
+
+## Before you begin
+
+### Install dependencies
+Code snippets on this page require pymilvus and llamaindex dependencies. You can install them using the following commands:
+
+
+```python
+$ pip install pymilvus>=2.4.2
+```
+
+
+```python
+$ pip install llama-index-vector-stores-milvus
+```
+
+
+```python
+$ pip install llama-index
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**. (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+### Setup OpenAI
+
+Lets first begin by adding the openai api key. This will allow us to access chatgpt.
+
+
+```python
+import openai
+
+openai.api_key = "sk-***********"
+```
+
+### Prepare data
+
+You can download sample data with the following commands:
+
+
+```python
+! mkdir -p 'data/'
+! wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham_essay.txt'
+! wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O 'data/uber_2021.pdf'
+```
+
+## Getting Started
+
+### Generate our data
+As a first example, lets generate a document from the file `paul_graham_essay.txt`. It is a single essay from Paul Graham titled `What I Worked On`. To generate the documents we will use the SimpleDirectoryReader.
+
+
+```python
+from llama_index.core import SimpleDirectoryReader
+
+# load documents
+documents = SimpleDirectoryReader(
+ input_files=["./data/paul_graham_essay.txt"]
+).load_data()
+
+print("Document ID:", documents[0].doc_id)
+```
+
+ Document ID: 95f25e4d-f270-4650-87ce-006d69d82033
+
+
+### Create an index across the data
+
+Now that we have a document, we can can create an index and insert the document.
+
+> Please note that **Milvus Lite** requires `pymilvus>=2.4.2`.
+
+
+```python
+# Create an index over the documents
+from llama_index.core import VectorStoreIndex, StorageContext
+from llama_index.vector_stores.milvus import MilvusVectorStore
+
+
+vector_store = MilvusVectorStore(uri="./milvus_demo.db", dim=1536, overwrite=True)
+storage_context = StorageContext.from_defaults(vector_store=vector_store)
+index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)
+```
+
+
+
+For the parameters of `MilvusVectorStore`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+### Query the data
+Now that we have our document stored in the index, we can ask questions against the index. The index will use the data stored in itself as the knowledge base for chatgpt.
+
+
+```python
+query_engine = index.as_query_engine()
+res = query_engine.query("What did the author learn?")
+print(res)
+```
+
+ The author learned that philosophy courses in college were boring to him, leading him to switch his focus to studying AI.
+
+
+
+```python
+res = query_engine.query("What challenges did the disease pose for the author?")
+print(res)
+```
+
+ The disease posed challenges for the author as it affected his mother's health, leading to a stroke caused by colon cancer. This resulted in her losing her balance and needing to be placed in a nursing home. The author and his sister were determined to help their mother get out of the nursing home and back to her house.
+
+
+This next test shows that overwriting removes the previous data.
+
+
+```python
+from llama_index.core import Document
+
+
+vector_store = MilvusVectorStore(uri="./milvus_demo.db", dim=1536, overwrite=True)
+storage_context = StorageContext.from_defaults(vector_store=vector_store)
+index = VectorStoreIndex.from_documents(
+ [Document(text="The number that is being searched for is ten.")],
+ storage_context,
+)
+query_engine = index.as_query_engine()
+res = query_engine.query("Who is the author?")
+print(res)
+```
+
+ The author is the individual who created the context information.
+
+
+The next test shows adding additional data to an already existing index.
+
+
+```python
+del index, vector_store, storage_context, query_engine
+
+vector_store = MilvusVectorStore(uri="./milvus_demo.db", overwrite=False)
+storage_context = StorageContext.from_defaults(vector_store=vector_store)
+index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)
+query_engine = index.as_query_engine()
+res = query_engine.query("What is the number?")
+print(res)
+```
+
+ The number is ten.
+
+
+
+```python
+res = query_engine.query("Who is the author?")
+print(res)
+```
+
+ Paul Graham
+
+
+## Metadata filtering
+
+We can generate results by filtering specific sources. The following example illustrates loading all documents from the directory and subsequently filtering them based on metadata.
+
+
+```python
+from llama_index.core.vector_stores import ExactMatchFilter, MetadataFilters
+
+# Load all the two documents loaded before
+documents_all = SimpleDirectoryReader("./data/").load_data()
+
+vector_store = MilvusVectorStore(uri="./milvus_demo.db", dim=1536, overwrite=True)
+storage_context = StorageContext.from_defaults(vector_store=vector_store)
+index = VectorStoreIndex.from_documents(documents_all, storage_context)
+```
+
+We want to only retrieve documents from the file `uber_2021.pdf`.
+
+
+```python
+filters = MetadataFilters(
+ filters=[ExactMatchFilter(key="file_name", value="uber_2021.pdf")]
+)
+query_engine = index.as_query_engine(filters=filters)
+res = query_engine.query("What challenges did the disease pose for the author?")
+
+print(res)
+```
+
+ The disease posed challenges related to the adverse impact on the business and operations, including reduced demand for Mobility offerings globally, affecting travel behavior and demand. Additionally, the pandemic led to driver supply constraints, impacted by concerns regarding COVID-19, with uncertainties about when supply levels would return to normal. The rise of the Omicron variant further affected travel, resulting in advisories and restrictions that could adversely impact both driver supply and consumer demand for Mobility offerings.
+
+
+We get a different result this time when retrieve from the file `paul_graham_essay.txt`.
+
+
+```python
+filters = MetadataFilters(
+ filters=[ExactMatchFilter(key="file_name", value="paul_graham_essay.txt")]
+)
+query_engine = index.as_query_engine(filters=filters)
+res = query_engine.query("What challenges did the disease pose for the author?")
+
+print(res)
+```
+
+ The disease posed challenges for the author as it affected his mother's health, leading to a stroke caused by colon cancer. This resulted in his mother losing her balance and needing to be placed in a nursing home. The author and his sister were determined to help their mother get out of the nursing home and back to her house.
+
diff --git a/preview/site/en/integrations/integrate_with_memgpt.md b/preview/site/en/integrations/integrate_with_memgpt.md
new file mode 100644
index 000000000..ea5f54c1d
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_memgpt.md
@@ -0,0 +1,109 @@
+---
+id: integrate_with_memgpt.md
+summary: MemGPT makes it easy to build and deploy stateful LLM agents. With Milvus integration, you can build agents with connections to external data sources (RAG).
+title: MemGPT with Milvus Integration
+---
+
+# MemGPT with Milvus Integration
+
+[MemGPT](https://memgpt.readme.io/docs/index) makes it easy to build and deploy stateful LLM agents. With Milvus integration, you can build agents with connections to external data sources (RAG).
+
+In this example, we're going to use MemGPT to chat with a custom data source which is stored in Milvus.
+
+## Configuration
+
+To run MemGPT, you should make sure the Python version >= 3.10.
+
+To enable the Milvus backend, make sure to install the required dependencies with:
+
+```shell
+$ pip install 'pymemgpt[milvus]'
+```
+
+You can configure Milvus connection via command
+
+```shell
+$ memgpt configure
+```
+
+```shell
+...
+? Select storage backend for archival data: milvus
+? Enter the Milvus connection URI (Default: ~/.memgpt/milvus.db): ~/.memgpt/milvus.db
+```
+You just set the URI to the local file path, e.g. `~/.memgpt/milvus.db`, which will automatically invoke the local Milvus service instance through Milvus Lite.
+
+If you have large scale of data such as more than a million docs, we recommend setting up a more performant Milvus server on [docker or kubenetes](https://milvus.io/docs/quickstart.md).
+And in this case, your URI should be the server URI, e.g. `http://localhost:19530`.
+
+## Creating an external data source
+
+To feed external data into a MemGPT chatbot, we first need to create a data source.
+
+To download the MemGPT research paper we'll use `curl` (you can also just download the PDF from your browser):
+
+```shell
+# we're saving the file as "memgpt_research_paper.pdf"
+$ curl -L -o memgpt_research_paper.pdf https://arxiv.org/pdf/2310.08560.pdf
+```
+
+Now that we have the paper downloaded, we can create a MemGPT data source using `memgpt load`:
+
+```shell
+$ memgpt load directory --name memgpt_research_paper --input-files=memgpt_research_paper.pdf
+```
+
+```text
+Loading files: 100%|███████████████████████████████████| 1/1 [00:00<00:00, 3.94file/s]
+Loaded 74 passages and 13 documents from memgpt_research_paper
+```
+
+## Attaching the data source to a MemGPT agent
+
+Now that we've created this data source, we can attach it to a MemGPT chatbot at any time.
+
+Let's create a new chatbot using the `memgpt_doc` persona (but you can use any persona you want):
+
+```shell
+# reminder: `memgpt run --persona memgpt_doc` will create a new MemGPT agent using the `memgpt_doc` persona
+$ memgpt run --persona memgpt_doc
+```
+
+Once we're chatting with the agent, we can "attach" the data source to the agent's archival memory:
+
+```text
+? Would you like to select an existing agent? No
+
+🧬 Creating new agent...
+-> 🤖 Using persona profile: 'sam_pov'
+-> 🧑 Using human profile: 'basic'
+🎉 Created new agent 'PoliteButterfly' (id=d26e1981-ff36-4095-97a0-61a1601dfb5d)
+
+Hit enter to begin (will request first MemGPT message)
+
+💭 Interesting, I've got a first-time user. Time to present myself and get to understand the user's needs. I wonder what brings Chad here today.
+🤖 Greetings Chad! I'm MemGPT. How may I assist you today?
+
+> Enter your message: /attach
+? Select data source memgpt_research_paper
+100%|███████████████████████████████████| 1/1 [00:00<00:00, 4.81it/s]
+```
+
+## Testing out our new chatbot
+
+Now that the data has been loaded into the chatbot's memory, we can start to ask questions about it:
+
+```text
+> Enter your message: The paper I loaded into your archival memory is called MemGPT. Can you tell me more about it?
+
+💭 It seems Chad would like to know more about the paper titled 'MemGPT' that's stored in my archival memory. It's time to search the archival memory for details and background data to provide a comprehensive
+understanding of the paper. Let's remember to page through results until I've gathered enough information to answer Chad's enquiry.
+⚡🧠 [function] searching memory with archival_memory_search
+💭 Based on the found results, MemGPT is an AI system designed for tasks like extended conversations and document analysis. It efficiently manages different tiers of storage to provide extended context within
+the LLM’s limited cap. Also, noteworthy from my findings is that MemGPT performs impressively in deep memory retrieval (DMR) and document QA tasks. Let's prepare a comprehensive response for Chad.
+🤖 Certainly, Chad. MemGPT, or MemoryGPT, is a system specially designed for tasks like extended conversations and document analysis which are traditionally hindered by the limited context windows of modern
+Large Language Models (LLMs). It uses a technique inspired by hierarchical memory systems in traditional operating systems, called virtual context management. This technique allows for an effective provision
+of extended context within an LLM’s limited context window by intelligently managing different storage tiers, much like paging between physical memory and disk. Notably, MemGPT outperforms the fixed-context
+baselines in deep memory retrieval and it's able to analyze large documents with precision.
+
+```
\ No newline at end of file
diff --git a/preview/site/en/integrations/integrate_with_openai.md b/preview/site/en/integrations/integrate_with_openai.md
new file mode 100644
index 000000000..246992162
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_openai.md
@@ -0,0 +1,138 @@
+---
+id: integrate_with_openai.md
+title: Semantic Search with Milvus and OpenAI
+summary: This page discusses vector database integration with OpenAI's embedding API.
+---
+
+# Semantic Search with Milvus and OpenAI
+
+
+
+This guide showcases how [OpenAI's Embedding API](https://platform.openai.com/docs/guides/embeddings) can be used with Milvus vector database to conduct semantic search on text.
+
+## Getting started
+Before you start, make sure you have the OpenAI API key ready, or you get one from the [OpenAI website](https://openai.com/index/openai-api/).
+
+The data used in this example are book titles. You can download the dataset [here](https://www.kaggle.com/datasets/jealousleopard/goodreadsbooks) and put it in the same directory where you run the following code.
+
+First, install the package for Milvus and OpenAI:
+
+```shell
+pip install --upgrade openai pymilvus
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**. (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+With this, we're ready to generate embeddings and use vector database to conduct semantic search.
+
+## Searching book titles with OpenAI & Milvus
+
+In the following example, we load book title data from the downloaded CSV file, use OpenAI embedding model to generate vector representations, and store them in Milvus vector database for semantic search.
+
+
+```python
+from openai import OpenAI
+from pymilvus import MilvusClient
+
+MODEL_NAME = "text-embedding-3-small" # Which model to use, please check https://platform.openai.com/docs/guides/embeddings for available models
+DIMENSION = 1536 # Dimension of vector embedding
+
+# Connect to OpenAI with API Key.
+openai_client = OpenAI(api_key="
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+With all data in Milvus vector database, we can now perform semantic search by generating vector embedding for the query and conduct vector search.
+
+
+```python
+queries = ["When was artificial intelligence founded?"]
+
+query_vectors = [
+ vec.embedding
+ for vec in openai_client.embeddings.create(input=queries, model=MODEL_NAME).data
+]
+
+res = milvus_client.search(
+ collection_name=COLLECTION_NAME, # target collection
+ data=query_vectors, # query vectors
+ limit=2, # number of returned entities
+ output_fields=["text", "subject"], # specifies fields to be returned
+)
+
+for q in queries:
+ print("Query:", q)
+ for result in res:
+ print(result)
+ print("\n")
+```
+
+You should see the following as the output:
+
+
+```python
+[
+ {
+ "id": 0,
+ "distance": -0.772376537322998,
+ "entity": {
+ "text": "Artificial intelligence was founded as an academic discipline in 1956.",
+ "subject": "history",
+ },
+ },
+ {
+ "id": 1,
+ "distance": -0.58596271276474,
+ "entity": {
+ "text": "Alan Turing was the first person to conduct substantial research in AI.",
+ "subject": "history",
+ },
+ },
+]
+```
diff --git a/preview/site/en/integrations/integrate_with_pytorch.md b/preview/site/en/integrations/integrate_with_pytorch.md
new file mode 100644
index 000000000..515b17fb1
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_pytorch.md
@@ -0,0 +1,250 @@
+---
+id: integrate_with_pytorch.md
+summary: This page discusses image search using Milvus
+title: Image Search with Milvus - Integration
+---
+
+# Image Search with Milvus
+
+On this page, we are going to be going over a simple image search example using Milvus. The dataset we are searching through is the Impressionist-Classifier Dataset found on [Kaggle](https://www.kaggle.com/datasets/delayedkarma/impressionist-classifier-data). For this example, we have rehosted the data in a public google drive.
+
+For this example, we are just using the Torchvision pre-trained Resnet50 model for embeddings. Let's get started!
+
+## Installing the requirements
+
+For this example, we are going to be using `pymilvus` to connect to use Milvus, `torch` for running the embedding model, `torchvision` for the actual model and preprocessing, `gdown` to download the example dataset and `tqdm` for loading bars.
+
+```shell
+pip install pymilvus torch gdown torchvision tqdm
+```
+
+## Grabbing the data
+
+We are going to use `gdown` to grab the zip from Google Drive and then decompress it with the built-in `zipfile` library.
+
+```python
+import gdown
+import zipfile
+
+url = 'https://drive.google.com/uc?id=1OYDHLEy992qu5C4C8HV5uDIkOWRTAR1_'
+output = './paintings.zip'
+gdown.download(url, output)
+
+with zipfile.ZipFile("./paintings.zip","r") as zip_ref:
+ zip_ref.extractall("./paintings")
+```
+
+
+
+The size of the dataset is 2.35 GB, and the time spent downloading it depends on your network condition.
+
+
+
+## Global Arguments
+
+These are some of the main global arguments that we will be using for easier tracking and updating.
+
+```python
+# Milvus Setup Arguments
+COLLECTION_NAME = 'image_search' # Collection name
+DIMENSION = 2048 # Embedding vector size in this example
+MILVUS_HOST = "localhost"
+MILVUS_PORT = "19530"
+
+# Inference Arguments
+BATCH_SIZE = 128
+TOP_K = 3
+```
+
+## Setting up Milvus
+
+At this point, we are going to begin setting up Milvus. The steps are as follows:
+
+1. Connect to the Milvus instance using the provided URI.
+
+ ```python
+ from pymilvus import connections
+
+ # Connect to the instance
+ connections.connect(host=MILVUS_HOST, port=MILVUS_PORT)
+ ```
+
+2. If the collection already exists, drop it.
+
+ ```python
+ from pymilvus import utility
+
+ # Remove any previous collections with the same name
+ if utility.has_collection(COLLECTION_NAME):
+ utility.drop_collection(COLLECTION_NAME)
+ ```
+
+3. Create the collection that holds the ID, the file path of the image, and its embedding.
+
+ ```python
+ from pymilvus import FieldSchema, CollectionSchema, DataType, Collection
+
+ # Create collection which includes the id, filepath of the image, and image embedding
+ fields = [
+ FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),
+ FieldSchema(name='filepath', dtype=DataType.VARCHAR, max_length=200), # VARCHARS need a maximum length, so for this example they are set to 200 characters
+ FieldSchema(name='image_embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)
+ ]
+ schema = CollectionSchema(fields=fields)
+ collection = Collection(name=COLLECTION_NAME, schema=schema)
+ ```
+
+4. Create an index on the newly created collection and load it into memory.
+
+ ```python
+ # Create an AutoIndex index for collection
+ index_params = {
+ 'metric_type':'L2',
+ 'index_type':"IVF_FLAT",
+ 'params':{'nlist': 16384}
+ }
+ collection.create_index(field_name="image_embedding", index_params=index_params)
+ collection.load()
+ ```
+
+Once these steps are done, the collection is ready to be inserted into and searched. Any added data will be indexed automatically and be available to search immediately. If the data is very fresh, the search might be slower as brute force searching will be used on data that is still in process of being indexed.
+
+## Inserting the data
+
+For this example, we are going to use the ResNet50 model provided by `torch` and its model hub. To get the embeddings, we are taking off the final classification layer, which results in the model giving us embeddings of 2048 dimensions. All the vision models found on `torch` use the same preprocessing that we have included here.
+
+In these next few steps we will be:
+
+1. Loading the data.
+
+ ```python
+ import glob
+
+ # Get the filepaths of the images
+ paths = glob.glob('./paintings/paintings/**/*.jpg', recursive=True)
+ len(paths)
+ ```
+
+2. Preprocessing the data into batches.
+
+ ```python
+ import torch
+
+ # Load the embedding model with the last layer removed
+ model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True)
+ model = torch.nn.Sequential(*(list(model.children())[:-1]))
+ model.eval()
+ ```
+
+3. Embedding the data.
+
+ ```python
+ from torchvision import transforms
+
+ # Preprocessing for images
+ preprocess = transforms.Compose([
+ transforms.Resize(256),
+ transforms.CenterCrop(224),
+ transforms.ToTensor(),
+ transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+ ])
+ ```
+
+4. Inserting the data.
+
+ ```python
+ from PIL import Image
+ from tqdm import tqdm
+
+ # Embed function that embeds the batch and inserts it
+ def embed(data):
+ with torch.no_grad():
+ output = model(torch.stack(data[0])).squeeze()
+ collection.insert([data[1], output.tolist()])
+
+ data_batch = [[],[]]
+
+ # Read the images into batches for embedding and insertion
+ for path in tqdm(paths):
+ im = Image.open(path).convert('RGB')
+ data_batch[0].append(preprocess(im))
+ data_batch[1].append(path)
+ if len(data_batch[0]) % BATCH_SIZE == 0:
+ embed(data_batch)
+ data_batch = [[],[]]
+
+ # Embed and insert the remainder
+ if len(data_batch[0]) != 0:
+ embed(data_batch)
+
+ # Call a flush to index any unsealed segments.
+ collection.flush()
+ ```
+
+
+
+ - This step is relatively time-consuming because embedding takes time. Take a sip of coffee and relax.
+ - PyTorch may not work well with Python 3.9 and earlier versions. Consider using Python 3.10 and later versions instead.
+
+
+
+## Performing the search
+
+With all the data inserted into Milvus, we can start performing our searches. In this example, we are going to search for two example images. Because we are doing a batch search, the search time is shared across the images of the batch.
+
+```python
+import glob
+
+# Get the filepaths of the search images
+search_paths = glob.glob('./paintings/test_paintings/**/*.jpg', recursive=True)
+len(search_paths)
+```
+
+```python
+import time
+from matplotlib import pyplot as plt
+
+# Embed the search images
+def embed(data):
+ with torch.no_grad():
+ ret = model(torch.stack(data))
+ # If more than one image, use squeeze
+ if len(ret) > 1:
+ return ret.squeeze().tolist()
+ # Squeeze would remove batch for single image, so using flatten
+ else:
+ return torch.flatten(ret, start_dim=1).tolist()
+
+data_batch = [[],[]]
+
+for path in search_paths:
+ im = Image.open(path).convert('RGB')
+ data_batch[0].append(preprocess(im))
+ data_batch[1].append(path)
+
+embeds = embed(data_batch[0])
+start = time.time()
+res = collection.search(embeds, anns_field='image_embedding', param={'nprobe': 128}, limit=TOP_K, output_fields=['filepath'])
+finish = time.time()
+```
+
+```python
+# Show the image results
+f, axarr = plt.subplots(len(data_batch[1]), TOP_K + 1, figsize=(20, 10), squeeze=False)
+
+for hits_i, hits in enumerate(res):
+ axarr[hits_i][0].imshow(Image.open(data_batch[1][hits_i]))
+ axarr[hits_i][0].set_axis_off()
+ axarr[hits_i][0].set_title('Search Time: ' + str(finish - start))
+ for hit_i, hit in enumerate(hits):
+ axarr[hits_i][hit_i + 1].imshow(Image.open(hit.entity.get('filepath')))
+ axarr[hits_i][hit_i + 1].set_axis_off()
+ axarr[hits_i][hit_i + 1].set_title('Distance: ' + str(hit.distance))
+
+# Save the search result in a separate image file alongside your script.
+plt.savefig('search_result.png')
+```
+
+The search result image should be similar to the following:
+
+
\ No newline at end of file
diff --git a/preview/site/en/integrations/integrate_with_ragas.md b/preview/site/en/integrations/integrate_with_ragas.md
new file mode 100644
index 000000000..e7f911ad3
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_ragas.md
@@ -0,0 +1,356 @@
+---
+id: integrate_with_ragas.md
+summary: This guide demonstrates how to use Ragas to evaluate a Retrieval-Augmented Generation (RAG) pipeline built upon Milvus.
+title: Evaluation with Ragas
+---
+
+# Evaluation with Ragas
+
+
+
+This guide demonstrates how to use Ragas to evaluate a Retrieval-Augmented Generation (RAG) pipeline built upon [Milvus](https://milvus.io/).
+
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+
+[Ragas](https://docs.ragas.io/en/latest/index.html#) is a framework that helps you evaluate your RAG pipelines. There are existing tools and frameworks that help you build these pipelines but evaluating it and quantifying your pipeline performance can be hard. This is where Ragas (RAG Assessment) comes in.
+
+
+
+
+## Prerequisites
+
+Before running this notebook, make sure you have the following dependencies installed:
+
+
+```python
+$ pip install --upgrade pymilvus openai requests tqdm pandas ragas
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime** (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+We will use OpenAI as the LLM in this example. You should prepare the [api key](https://platform.openai.com/docs/quickstart) `OPENAI_API_KEY` as an environment variable.
+
+
+```python
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+```
+
+## Define the RAG pipeline
+
+We will define the RAG class that use Milvus as the vector store, and OpenAI as the LLM.
+The class contains the `load` method, which loads the text data into Milvus, the `retrieve` method, which retrieves the most similar text data to the given question, and the `answer` method, which answers the given question with the retrieved knowledge.
+
+
+```python
+from typing import List
+from tqdm import tqdm
+from openai import OpenAI
+from pymilvus import MilvusClient
+
+
+class RAG:
+ """
+ RAG (Retrieval-Augmented Generation) class built upon OpenAI and Milvus.
+ """
+
+ def __init__(self, openai_client: OpenAI, milvus_client: MilvusClient):
+ self._prepare_openai(openai_client)
+ self._prepare_milvus(milvus_client)
+
+ def _emb_text(self, text: str) -> List[float]:
+ return (
+ self.openai_client.embeddings.create(input=text, model=self.embedding_model)
+ .data[0]
+ .embedding
+ )
+
+ def _prepare_openai(
+ self,
+ openai_client: OpenAI,
+ embedding_model: str = "text-embedding-3-small",
+ llm_model: str = "gpt-3.5-turbo",
+ ):
+ self.openai_client = openai_client
+ self.embedding_model = embedding_model
+ self.llm_model = llm_model
+ self.SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+ self.USER_PROMPT = """
+Use the following pieces of information enclosed in
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+## Run the RAG pipeline and get results
+
+We use the [Milvus development guide](https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md) to be as the private knowledge in our RAG, which is a good data source for a simple RAG pipeline.
+
+Download it and load it into the rag pipeline.
+
+
+```python
+import os
+import urllib.request
+
+url = "https://raw.githubusercontent.com/milvus-io/milvus/master/DEVELOPMENT.md"
+file_path = "./Milvus_DEVELOPMENT.md"
+
+if not os.path.exists(file_path):
+ urllib.request.urlretrieve(url, file_path)
+with open(file_path, "r") as file:
+ file_text = file.read()
+
+# We simply use "# " to separate the content in the file, which can roughly separate the content of each main part of the markdown file.
+text_lines = file_text.split("# ")
+my_rag.load(text_lines) # Load the text data into RAG pipeline
+```
+
+ Creating embeddings: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 47/47 [00:16<00:00, 2.80it/s]
+
+
+Let's define a query question about the content of the development guide documentation. And then use the `answer` method to get the answer and the retrieved context texts.
+
+
+```python
+question = "what is the hardware requirements specification if I want to build Milvus and run from source code?"
+my_rag.answer(question, return_retrieved_text=True)
+```
+
+
+
+
+ ('The hardware requirements specification to build and run Milvus from source code is 8GB of RAM and 50GB of free disk space.',
+ ['Hardware Requirements\n\nThe following specification (either physical or virtual machine resources) is recommended for Milvus to build and run from source code.\n\n```\n- 8GB of RAM\n- 50GB of free disk space\n```\n\n##',
+ 'Building Milvus on a local OS/shell environment\n\nThe details below outline the hardware and software requirements for building on Linux and MacOS.\n\n##',
+ "Software Requirements\n\nAll Linux distributions are available for Milvus development. However a majority of our contributor worked with Ubuntu or CentOS systems, with a small portion of Mac (both x86_64 and Apple Silicon) contributors. If you would like Milvus to build and run on other distributions, you are more than welcome to file an issue and contribute!\n\nHere's a list of verified OS types where Milvus can successfully build and run:\n\n- Debian/Ubuntu\n- Amazon Linux\n- MacOS (x86_64)\n- MacOS (Apple Silicon)\n\n##"])
+
+
+
+Now let's prepare some questions with its corresponding ground truth answers. We get answers and contexts from our RAG pipeline.
+
+
+```python
+from datasets import Dataset
+import pandas as pd
+
+question_list = [
+ "what is the hardware requirements specification if I want to build Milvus and run from source code?",
+ "What is the programming language used to write Knowhere?",
+ "What should be ensured before running code coverage?",
+]
+ground_truth_list = [
+ "If you want to build Milvus and run from source code, the recommended hardware requirements specification is:\n\n- 8GB of RAM\n- 50GB of free disk space.",
+ "The programming language used to write Knowhere is C++.",
+ "Before running code coverage, you should make sure that your code changes are covered by unit tests.",
+]
+contexts_list = []
+answer_list = []
+for question in tqdm(question_list, desc="Answering questions"):
+ answer, contexts = my_rag.answer(question, return_retrieved_text=True)
+ contexts_list.append(contexts)
+ answer_list.append(answer)
+
+df = pd.DataFrame(
+ {
+ "question": question_list,
+ "contexts": contexts_list,
+ "answer": answer_list,
+ "ground_truth": ground_truth_list,
+ }
+)
+rag_results = Dataset.from_pandas(df)
+df
+```
+
+ Answering questions: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:03<00:00, 1.29s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+## Evaluation with Ragas
+
+We use Ragas to evaluate the performance of our RAG pipeline results.
+
+Ragas provides a set of metrics that is easy to use. We take `Answer relevancy`, `Faithfulness`, `Context recall`, and `Context precision` as the metrics to evaluate our RAG pipeline. For more information about the metrics, please refer to the [Ragas Metrics](https://docs.ragas.io/en/latest/concepts/metrics/index.html).
+
+
+```python
+from ragas import evaluate
+from ragas.metrics import (
+ answer_relevancy,
+ faithfulness,
+ context_recall,
+ context_precision,
+)
+
+result = evaluate(
+ rag_results,
+ metrics=[
+ answer_relevancy,
+ faithfulness,
+ context_recall,
+ context_precision,
+ ],
+)
+
+result
+```
+
+
+ Evaluating: 0%| | 0/12 [00:00
+
+The above operation is relatively time-consuming because embedding takes time. To keep the time consumed at an acceptable level, try setting `COUNT` in [Global parameters](#Global-parameters) to an appropriate value. Take a break and enjoy a cup of coffee!
+
+| + | question | +contexts | +answer | +ground_truth | +
|---|---|---|---|---|
| 0 | +what is the hardware requirements specificatio... | +[Hardware Requirements\n\nThe following specif... | +The hardware requirements specification for bu... | +If you want to build Milvus and run from sourc... | +
| 1 | +What is the programming language used to write... | +[CMake & Conan\n\nThe algorithm library of Mil... | +The programming language used to write the Kno... | +The programming language used to write Knowher... | +
| 2 | +What should be ensured before running code cov... | +[Code coverage\n\nBefore submitting your pull ... | +Before running code coverage, you should ensur... | +Before running code coverage, you should make ... | +
+
+The PCA model is a tranformation model that reduces the dimensionality of embedding vectors, which is a common operation in machine learning.
+You can add any other processing operations, such as filtering, joining, or normalizing, to the transformation step.
+
+
+
+```scala
+import org.apache.spark.ml.feature.PCA
+import org.apache.spark.ml.linalg.{Vector, Vectors}
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.functions.udf
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+import zilliztech.spark.milvus.{MilvusOptions, MilvusUtils}
+
+import scala.collection.JavaConverters._
+
+object TransformDemo extends App {
+ val sparkConf = new SparkConf().setMaster("local")
+ val spark = SparkSession.builder().config(sparkConf).getOrCreate()
+
+ import spark.implicits._
+
+ val host = "localhost"
+ val port = 19530
+ val user = "root"
+ val password = "Milvus"
+ val fs = "s3a://"
+ val bucketName = "a-bucket"
+ val rootPath = "files"
+ val minioAK = "minioadmin"
+ val minioSK = "minioadmin"
+ val minioEndpoint = "localhost:9000"
+ val collectionName = "hello_spark_milvus1"
+ val targetCollectionName = "hello_spark_milvus2"
+
+ val properties = Map(
+ MilvusOptions.MILVUS_HOST -> host,
+ MilvusOptions.MILVUS_PORT -> port.toString,
+ MilvusOptions.MILVUS_COLLECTION_NAME -> collectionName,
+ MilvusOptions.MILVUS_BUCKET -> bucketName,
+ MilvusOptions.MILVUS_ROOTPATH -> rootPath,
+ MilvusOptions.MILVUS_FS -> fs,
+ MilvusOptions.MILVUS_STORAGE_ENDPOINT -> minioEndpoint,
+ MilvusOptions.MILVUS_STORAGE_USER -> minioAK,
+ MilvusOptions.MILVUS_STORAGE_PASSWORD -> minioSK,
+ )
+
+ // 1, configurations
+ val milvusOptions = new MilvusOptions(new CaseInsensitiveStringMap(properties.asJava))
+
+ // 2, batch read milvus collection data to dataframe
+ // Schema: dim of `embeddings` is 8
+ // +-+------------+------------+------------------+
+ // | | field name | field type | other attributes |
+ // +-+------------+------------+------------------+
+ // |1| "pk" | Int64 | is_primary=True |
+ // | | | | auto_id=False |
+ // +-+------------+------------+------------------+
+ // |2| "random" | Double | |
+ // +-+------------+------------+------------------+
+ // |3|"embeddings"| FloatVector| dim=8 |
+ // +-+------------+------------+------------------+
+ val arrayToVectorUDF = udf((arr: Seq[Double]) => Vectors.dense(arr.toArray[Double]))
+ val collectionDF = MilvusUtils.readMilvusCollection(spark, milvusOptions)
+ .withColumn("embeddings_vec", arrayToVectorUDF($"embeddings"))
+ .drop("embeddings")
+
+ // 3. Use PCA to reduce dim of vector
+ val dim = 4
+ val pca = new PCA()
+ .setInputCol("embeddings_vec")
+ .setOutputCol("pca_vec")
+ .setK(dim)
+ .fit(collectionDF)
+ val vectorToArrayUDF = udf((v: Vector) => v.toArray)
+ // embeddings dim number reduce to 4
+ // +-+------------+------------+------------------+
+ // | | field name | field type | other attributes |
+ // +-+------------+------------+------------------+
+ // |1| "pk" | Int64 | is_primary=True |
+ // | | | | auto_id=False |
+ // +-+------------+------------+------------------+
+ // |2| "random" | Double | |
+ // +-+------------+------------+------------------+
+ // |3|"embeddings"| FloatVector| dim=4 |
+ // +-+------------+------------+------------------+
+ val pcaDf = pca.transform(collectionDF)
+ .withColumn("embeddings", vectorToArrayUDF($"pca_vec"))
+ .select("pk", "random", "embeddings")
+
+ // 4. Write PCAed data to S3
+ val outputPath = "s3a://a-bucket/result"
+ pcaDf.write
+ .mode("overwrite")
+ .format("parquet")
+ .save(outputPath)
+
+ // 5. Config MilvusOptions of target table
+ val targetProperties = Map(
+ MilvusOptions.MILVUS_HOST -> host,
+ MilvusOptions.MILVUS_PORT -> port.toString,
+ MilvusOptions.MILVUS_COLLECTION_NAME -> targetCollectionName,
+ MilvusOptions.MILVUS_BUCKET -> bucketName,
+ MilvusOptions.MILVUS_ROOTPATH -> rootPath,
+ MilvusOptions.MILVUS_FS -> fs,
+ MilvusOptions.MILVUS_STORAGE_ENDPOINT -> minioEndpoint,
+ MilvusOptions.MILVUS_STORAGE_USER -> minioAK,
+ MilvusOptions.MILVUS_STORAGE_PASSWORD -> minioSK,
+ )
+ val targetMilvusOptions = new MilvusOptions(new CaseInsensitiveStringMap(targetProperties.asJava))
+
+ // 6. Bulkinsert Spark output files into milvus
+ MilvusUtils.bulkInsertFromSpark(spark, targetMilvusOptions, outputPath, "parquet")
+}
+```
+
+### Databricks -> Zilliz Cloud
+
+If you are using Zilliz Cloud (the managed Milvus service), you can leverage its convenient Data Import API. Zilliz Cloud provides comprehensive tools and documentation to help you efficiently move your data from various data sources, including Spark and Databricks. Simply set up an S3 bucket as an intermediary and open its access to your Zilliz Cloud account. The Zilliz Cloud's Data Import API will automatically load the full batch of data from the S3 bucket to your Zilliz Cloud cluster.
+
+**Preparations**
+
+1. Load the Spark runtime by adding a jar file to your Databricks Cluster.
+
+ You can install a library in different ways. This screenshot shows uploading a jar from local to the cluster. For more information, see [Cluster Libraries](https://docs.databricks.com/en/libraries/cluster-libraries.html) in the Databricks documentation.
+
+ 
+
+2. Create an S3 bucket and configure it as an external storage location for you Databricks cluster.
+
+ Bulkinsert required data to be stored in a temporary bucket so that Zilliz Cloud can import the data in a batch. You can create an S3 bucket and configure it as an external location of databricks. Please refer to [External locations](https://docs.databricks.com/en/sql/language-manual/sql-ref-external-locations.html) for details.
+
+3. Secure you Databricks credentials.
+
+ For more details, refer to the instructions on the blog [Securely Managing Credentials in Databricks](https://www.databricks.com/blog/2018/06/04/securely-managing-credentials-in-databricks.html).
+
+**Demo**
+
+Here's a code snippet showcasing the batch data migration process. Similar to the above Milvus example, you just need to replace the credential and S3 bucket address.
+
+```scala
+// Write the data in batch into the Milvus bucket storage.
+val outputPath = "s3://my-temp-bucket/result"
+df.write
+ .mode("overwrite")
+ .format("mjson")
+ .save(outputPath)
+// Specify Milvus options.
+val targetProperties = Map(
+ MilvusOptions.MILVUS_URI -> zilliz_uri,
+ MilvusOptions.MILVUS_TOKEN -> zilliz_token,
+ MilvusOptions.MILVUS_COLLECTION_NAME -> targetCollectionName,
+ MilvusOptions.MILVUS_BUCKET -> bucketName,
+ MilvusOptions.MILVUS_ROOTPATH -> rootPath,
+ MilvusOptions.MILVUS_FS -> fs,
+ MilvusOptions.MILVUS_STORAGE_ENDPOINT -> minioEndpoint,
+ MilvusOptions.MILVUS_STORAGE_USER -> minioAK,
+ MilvusOptions.MILVUS_STORAGE_PASSWORD -> minioSK,
+)
+val targetMilvusOptions = new MilvusOptions(new CaseInsensitiveStringMap(targetProperties.asJava))
+
+// Bulk insert Spark output files into Milvus
+MilvusUtils.bulkInsertFromSpark(spark, targetMilvusOptions, outputPath, "mjson")
+```
+
+## Hands-on
+
+To help you quickly get started with the Spark-Milvus Connector, we have prepared a notebook that walks you through both the streaming and batch data transfer processes, with Milvus and Zilliz Cloud.
+
+- [Spark-Milvus Connector Hands-on](https://zilliz.com/databricks_zilliz_demos)
diff --git a/preview/site/en/integrations/integrate_with_vanna.md b/preview/site/en/integrations/integrate_with_vanna.md
new file mode 100644
index 000000000..e08a757f7
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_vanna.md
@@ -0,0 +1,376 @@
+---
+id: integrate_with_vanna.md
+summary: This guide demonstrates how to use Vanna to generate and execute SQL queries based on your data stored in a database.
+title: Write SQL with Vanna and Milvus
+---
+
+# Write SQL with Vanna and Milvus
+
+[Vanna](https://vanna.ai/) is an open-source Python RAG (Retrieval-Augmented Generation) framework for SQL generation and related functionality. [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
+
+Vanna works in two easy steps - train a RAG "model" on your data, and then ask questions which will return SQL queries that can be set up to run on your database. This guide demonstrates how to use Vanna to generate and execute SQL queries based on your data stored in a database.
+
+
+
+## Prerequisites
+
+Before running this notebook, make sure you have the following dependencies installed:
+
+
+```python
+$ pip install "vanna[milvus,openai]"
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime** (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+And you need set the `OPENAI_API_KEY` in your environment variables. You can get the API key from [OpenAI](https://platform.openai.com/docs/guides/production-best-practices/api-keys).
+
+
+```python
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+```
+
+## Data preparation
+
+First, we need to inherit from the `Milvus_VectorStore` and `OpenAI_Chat` classes from Vanna and define a new class `VannaMilvus` that combines capabilities from both.
+
+
+```python
+from pymilvus import MilvusClient, model
+from vanna.milvus import Milvus_VectorStore
+from vanna.openai import OpenAI_Chat
+
+
+class VannaMilvus(Milvus_VectorStore, OpenAI_Chat):
+ def __init__(self, config=None):
+ Milvus_VectorStore.__init__(self, config=config)
+ OpenAI_Chat.__init__(self, config=config)
+```
+
+We initialize the `VannaMilvus` class with the necessary configuration parameters. We use a `milvus_client` instance to store embeddings and the `model.DefaultEmbeddingFunction()` initialized from [milvus_model](https://milvus.io/docs/embeddings.md) to generate embeddings.C
+
+
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+
+```python
+milvus_uri = "./milvus_vanna.db"
+
+milvus_client = MilvusClient(uri=milvus_uri)
+
+vn_milvus = VannaMilvus(
+ config={
+ "api_key": os.getenv("OPENAI_API_KEY"),
+ "model": "gpt-3.5-turbo",
+ "milvus_client": milvus_client,
+ "embedding_function": model.DefaultEmbeddingFunction(),
+ "n_results": 2, # The number of results to return from Milvus semantic search.
+ }
+)
+```
+
+This is a simple example with only a few sample of data, so we set `n_results` to 2 to make sure we search for the top 2 most similar results.
+In practice, you should set `n_results` to a higher value when dealing with larger training dataset.
+
+We will use a sample SQLite database with few tables containing some sample data.
+
+
+```python
+import sqlite3
+
+sqlite_path = "./my-database.sqlite"
+sql_connect = sqlite3.connect(sqlite_path)
+c = sql_connect.cursor()
+
+init_sqls = """
+CREATE TABLE IF NOT EXISTS Customer (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Name TEXT NOT NULL,
+ Company TEXT NOT NULL,
+ City TEXT NOT NULL,
+ Phone TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS Company (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Name TEXT NOT NULL,
+ Industry TEXT NOT NULL,
+ Location TEXT NOT NULL,
+ EmployeeCount INTEGER NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS User (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Username TEXT NOT NULL UNIQUE,
+ Email TEXT NOT NULL UNIQUE
+);
+
+INSERT INTO Customer (Name, Company, City, Phone)
+VALUES ('John Doe', 'ABC Corp', 'New York', '123-456-7890');
+
+INSERT INTO Customer (Name, Company, City, Phone)
+VALUES ('Jane Smith', 'XYZ Inc', 'Los Angeles', '098-765-4321');
+
+INSERT INTO Company (Name, Industry, Location, EmployeeCount)
+VALUES ('ABC Corp', 'cutting-edge technology', 'New York', 100);
+
+INSERT INTO User (Username, Email)
+VALUES ('johndoe123', 'johndoe123@example.com');
+"""
+
+for sql in init_sqls.split(";"):
+ c.execute(sql)
+
+sql_connect.commit()
+
+# Connect to the SQLite database
+vn_milvus.connect_to_sqlite(sqlite_path)
+```
+
+## Train with data
+We can train the model on the DDL data of the SQLite database. We get the DDL data and feed it to the `train` function.
+
+
+```python
+# If there exists training data, we should remove it before training.
+existing_training_data = vn_milvus.get_training_data()
+if len(existing_training_data) > 0:
+ for _, training_data in existing_training_data.iterrows():
+ vn_milvus.remove_training_data(training_data["id"])
+
+# Get the DDL of the SQLite database
+df_ddl = vn_milvus.run_sql("SELECT type, sql FROM sqlite_master WHERE sql is not null")
+
+# Train the model on the DDL data
+for ddl in df_ddl["sql"].to_list():
+ vn_milvus.train(ddl=ddl)
+```
+
+ Adding ddl: CREATE TABLE Customer (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Name TEXT NOT NULL,
+ Company TEXT NOT NULL,
+ City TEXT NOT NULL,
+ Phone TEXT NOT NULL
+ )
+ Adding ddl: CREATE TABLE sqlite_sequence(name,seq)
+ Adding ddl: CREATE TABLE Company (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Name TEXT NOT NULL,
+ Industry TEXT NOT NULL,
+ Location TEXT NOT NULL,
+ EmployeeCount INTEGER NOT NULL
+ )
+ Adding ddl: CREATE TABLE User (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Username TEXT NOT NULL UNIQUE,
+ Email TEXT NOT NULL UNIQUE
+ )
+
+
+Besides training on the DDL data, we can also train on the documentation and SQL queries of the database.
+
+
+```python
+# Add documentation about your business terminology or definitions.
+vn_milvus.train(
+ documentation="ABC Corp specializes in cutting-edge technology solutions and innovation."
+)
+vn_milvus.train(
+ documentation="XYZ Inc is a global leader in manufacturing and supply chain management."
+)
+
+# You can also add SQL queries to your training data.
+vn_milvus.train(sql="SELECT * FROM Customer WHERE Name = 'John Doe'")
+```
+
+ Adding documentation....
+ Adding documentation....
+ Using model gpt-3.5-turbo for 65.0 tokens (approx)
+ Question generated with sql: What are the details of the customer named John Doe?
+ Adding SQL...
+
+
+
+
+
+ '595b185c-e6ad-47b0-98fd-0e93ef9b6a0a-sql'
+
+
+
+Let's take a look at the training data.
+
+
+```python
+training_data = vn_milvus.get_training_data()
+training_data
+```
+
+
+
+
+
+
+
+
+
+
+
+
+## Generate SQLs and execute them
+As we have trained with the DDL data, the table structure is now available for generating SQL queries.
+
+Let's try a simple question.
+
+
+```python
+sql = vn_milvus.generate_sql("what is the phone number of John Doe?")
+vn_milvus.run_sql(sql)
+```
+
+ SQL Prompt: [{'role': 'system', 'content': "You are a SQLite expert. Please help to generate a SQL query to answer the question. Your response should ONLY be based on the given context and follow the response guidelines and format instructions. \n===Tables \nCREATE TABLE Customer (\n ID INTEGER PRIMARY KEY AUTOINCREMENT,\n Name TEXT NOT NULL,\n Company TEXT NOT NULL,\n City TEXT NOT NULL,\n Phone TEXT NOT NULL\n)\n\nCREATE TABLE User (\n ID INTEGER PRIMARY KEY AUTOINCREMENT,\n Username TEXT NOT NULL UNIQUE,\n Email TEXT NOT NULL UNIQUE\n)\n\n\n===Additional Context \n\nABC Corp specializes in cutting-edge technology solutions and innovation.\n\nXYZ Inc is a global leader in manufacturing and supply chain management.\n\n===Response Guidelines \n1. If the provided context is sufficient, please generate a valid SQL query without any explanations for the question. \n2. If the provided context is almost sufficient but requires knowledge of a specific string in a particular column, please generate an intermediate SQL query to find the distinct strings in that column. Prepend the query with a comment saying intermediate_sql \n3. If the provided context is insufficient, please explain why it can't be generated. \n4. Please use the most relevant table(s). \n5. If the question has been asked and answered before, please repeat the answer exactly as it was given before. \n"}, {'role': 'user', 'content': 'What are the details of the customer named John Doe?'}, {'role': 'assistant', 'content': "SELECT * FROM Customer WHERE Name = 'John Doe'"}, {'role': 'user', 'content': 'what is the phone number of John Doe?'}]
+ Using model gpt-3.5-turbo for 367.25 tokens (approx)
+ LLM Response: SELECT Phone FROM Customer WHERE Name = 'John Doe'
+
+
+
+
+
+| # | +id | +question | +content | +
|---|---|---|---|
| 0 | +595b185c-e6ad-47b0-98fd-0e93ef9b6a0a-sql | +What are the details of the customer named Joh... | +SELECT * FROM Customer WHERE Name = 'John Doe' | +
| 0 | +25f4956c-e370-4097-994f-996f22d145fa-ddl | +None | +CREATE TABLE Company (\n ID INTEGER PRIMARY... | +
| 1 | +b95ecc66-f65b-49dc-a9f1-c1842ad230ff-ddl | +None | +CREATE TABLE Customer (\n ID INTEGER PRIMAR... | +
| 2 | +fcc73d15-30a5-4421-9d73-b8c3b0ed5305-ddl | +None | +CREATE TABLE sqlite_sequence(name,seq) | +
| 3 | +feae618c-5910-4f6f-8b4b-6cc3e03aec06-ddl | +None | +CREATE TABLE User (\n ID INTEGER PRIMARY KE... | +
| 0 | +79a48db1-ba1f-4fd5-be99-74f2ca2eaeeb-doc | +None | +XYZ Inc is a global leader in manufacturing an... | +
| 1 | +9f9df1b8-ae62-4823-ad28-d7e0f2d1f4c0-doc | +None | +ABC Corp specializes in cutting-edge technolog... | +
+
+
+
+
+
+
+
+Here is a more complex question. The manufacturing corporation name information is in the document data, which is background information. The generated SQL query will retrieve the customer information based on the specific manufacturing corporation name.
+
+
+```python
+sql = vn_milvus.generate_sql("which customer works for a manufacturing corporation?")
+vn_milvus.run_sql(sql)
+```
+
+ SQL Prompt: [{'role': 'system', 'content': "You are a SQLite expert. Please help to generate a SQL query to answer the question. Your response should ONLY be based on the given context and follow the response guidelines and format instructions. \n===Tables \nCREATE TABLE Company (\n ID INTEGER PRIMARY KEY AUTOINCREMENT,\n Name TEXT NOT NULL,\n Industry TEXT NOT NULL,\n Location TEXT NOT NULL,\n EmployeeCount INTEGER NOT NULL\n)\n\nCREATE TABLE Customer (\n ID INTEGER PRIMARY KEY AUTOINCREMENT,\n Name TEXT NOT NULL,\n Company TEXT NOT NULL,\n City TEXT NOT NULL,\n Phone TEXT NOT NULL\n)\n\n\n===Additional Context \n\nXYZ Inc is a global leader in manufacturing and supply chain management.\n\nABC Corp specializes in cutting-edge technology solutions and innovation.\n\n===Response Guidelines \n1. If the provided context is sufficient, please generate a valid SQL query without any explanations for the question. \n2. If the provided context is almost sufficient but requires knowledge of a specific string in a particular column, please generate an intermediate SQL query to find the distinct strings in that column. Prepend the query with a comment saying intermediate_sql \n3. If the provided context is insufficient, please explain why it can't be generated. \n4. Please use the most relevant table(s). \n5. If the question has been asked and answered before, please repeat the answer exactly as it was given before. \n"}, {'role': 'user', 'content': 'What are the details of the customer named John Doe?'}, {'role': 'assistant', 'content': "SELECT * FROM Customer WHERE Name = 'John Doe'"}, {'role': 'user', 'content': 'which customer works for a manufacturing corporation?'}]
+ Using model gpt-3.5-turbo for 384.25 tokens (approx)
+ LLM Response: SELECT *
+ FROM Customer
+ WHERE Company = 'XYZ Inc'
+
+
+
+
+
+| # | +Phone | +
|---|---|
| 0 | +123-456-7890 | +
+
+
+
+
+
+
+
+Disconnect from the SQLite and Milvus and remove them to free up resources.
+
+
+```python
+sql_connect.close()
+milvus_client.close()
+
+os.remove(sqlite_path)
+if os.path.exists(milvus_uri):
+ os.remove(milvus_uri)
+```
diff --git a/preview/site/en/integrations/integrate_with_voxel51.md b/preview/site/en/integrations/integrate_with_voxel51.md
new file mode 100644
index 000000000..f06b92f5d
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_voxel51.md
@@ -0,0 +1,315 @@
+---
+id: integrate_with_voxel51.md
+summary: This page discusses the integration with voxel51
+title: Conduct Vision Searches with Milvus and FiftyOne
+---
+
+# Conduct Vision Searches with Milvus and FiftyOne
+
+[FiftyOne](https://docs.voxel51.com/) is an open-source tool for building high-quality datasets and computer vision models. This guide helps you integrate the similarity search capabilities of Milvus into FiftyOne, enabling you to conduct vision searches on your own datasets.
+
+FiftyOne provides an API to create Milvus collections, upload vectors, and run similarity queries, both [programmatically](https://docs.voxel51.com/integrations/milvus.html#milvus-query) in Python and via point-and-click in the App. The demonstration on this page focuses on the programmatic integration.
+
+## Prerequisites
+
+Before starting, make sure you have the following:
+
+- A running [Milvus server](install_standalone-docker.md).
+- A Python environment with `pymilvus` and `fiftyone` installed.
+- A [dataset](https://docs.voxel51.com/user_guide/dataset_creation/index.html#loading-datasets) of images to search.
+
+## Installing Requirements
+
+For this example, we are going to use `pymilvus` and `fiftyone`. You can install them by running the following commands:
+
+```shell
+python3 -m pip install pymilvus fiftyone torch torchvision
+```
+
+## Basic recipe
+
+The basic workflow to use Milvus to create a similarity index on your FiftyOne datasets and use this to query your data is as follows:
+
+1. Load a [dataset](https://docs.voxel51.com/user_guide/dataset_creation/index.html#loading-datasets) into FiftyOne
+2. Compute vector embeddings for samples or patches in your dataset, or select a model to use the generate embeddings.
+3. Use the [`compute_similarity()`](https://docs.voxel51.com/api/fiftyone.brain.html#fiftyone.brain.compute_similarity) method to generate a Milvus similarity index for the samples or object patches in a dataset by setting the parameter `backend="milvus"` and specifying a `brain_key` of your choice.
+4. Use this Milvus similarity index to query your data with [`sort_by_similarity()`](https://docs.voxel51.com/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.sort_by_similarity).
+5. If desired, delete the index.
+
+## Procedures
+
+The example below demonstrates the above workflow.
+
+### 1. Load a dataset into FiftyOne and compute embeddings for the samples
+
+The following code uses the sample image set provided by FiftyOne to demonstrate the integration. You can prepare your own image set by referring to [this article](https://docs.voxel51.com/user_guide/dataset_creation/index.html#loading-datasets).
+
+```python
+import fiftyone as fo
+import fiftyone.brain as fob
+import fiftyone.zoo as foz
+
+# Step 1: Load your data into FiftyOne
+dataset = foz.load_zoo_dataset("quickstart")
+
+# Steps 2 and 3: Compute embeddings and create a similarity index
+milvus_index = fob.compute_similarity(
+ dataset,
+ brain_key="milvus_index",
+ backend="milvus",
+)
+```
+
+### 2. Conduct vision similarity searches
+
+You can now use the Milvus similarity index to conduct vision similarity searches on your dataset.
+
+```python
+# Step 4: Query your data
+query = dataset.first().id # query by sample ID
+view = dataset.sort_by_similarity(
+ query,
+ brain_key="milvus_index",
+ k=10, # limit to 10 most similar samples
+)
+
+# Step 5 (optional): Cleanup
+
+# Delete the Milvus collection
+milvus_index.cleanup()
+
+# Delete run record from FiftyOne
+dataset.delete_brain_run("milvus_index")
+```
+
+### 3. Delete the index
+
+If you no longer need the Milvus similarity index, you can delete it using the following code:
+
+```python
+# Step 5: Delete the index
+milvus_index.delete()
+```
+
+## Use the Milvus backend
+
+By default, calling [`compute_similarity()`](https://docs.voxel51.com/api/fiftyone.brain.html#fiftyone.brain.compute_similarity) or `sort_by_similarity()` will use a sklearn backend.
+
+To use the Milvus backend, simply set the optional backend parameter of [`compute_similarity()`](https://docs.voxel51.com/api/fiftyone.brain.html#fiftyone.brain.compute_similarity) to `"milvus"`:
+
+```python
+import fiftyone.brain as fob
+
+fob.compute_similarity(..., backend="milvus", ...)
+```
+
+Alternatively, you can permanently configure FiftyOne to use the Milvus backend by setting the following environment variable:
+
+```shell
+export FIFTYONE_BRAIN_DEFAULT_SIMILARITY_BACKEND=milvus
+```
+
+or by setting the `default_similarity_backend` parameter of your [brain config](https://docs.voxel51.com/user_guide/brain.html#brain-config) located at `~/.fiftyone/brain_config.json`:
+
+```json
+{
+ "default_similarity_backend": "milvus"
+}
+```
+
+## Authentication
+
+If you are using a custom Milvus server, you can provide your credentials in a variety of ways.
+
+### Environment variables (recommended)
+
+The recommended way to configure your Milvus credentials is to store them in the environment variables shown below, which are automatically accessed by FiftyOne whenever a connection to Milvus is made.
+
+```python
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_URI=XXXXXX
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_USER=XXXXXX
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_PASSWORD=XXXXXX
+
+# also available if necessary
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_SECURE=true
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_TOKEN=XXXXXX
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_DB_NAME=XXXXXX
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_CLIENT_KEY_PATH=XXXXXX
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_CLIENT_PEM_PATH=XXXXXX
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_CA_PEM_PATH=XXXXXX
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_SERVER_PEM_PATH=XXXXXX
+export FIFTYONE_BRAIN_SIMILARITY_MILVUS_SERVER_NAME=XXXXXX
+```
+
+### FiftyOne Brain config
+
+You can also store your credentials in your [brain config](https://docs.voxel51.com/user_guide/brain.html#brain-config) located at `~/.fiftyone/brain_config.json`:
+
+```python
+{
+ "similarity_backends": {
+ "milvus": {
+ "uri": "XXXXXX",
+ "user": "XXXXXX",
+ "password": "XXXXXX",
+
+ # also available if necessary
+ "secure": true,
+ "token": "XXXXXX",
+ "db_name": "XXXXXX",
+ "client_key_path": "XXXXXX",
+ "client_pem_path": "XXXXXX",
+ "ca_pem_path": "XXXXXX",
+ "server_pem_path": "XXXXXX",
+ "server_name": "XXXXXX"
+ }
+ }
+}
+```
+
+Note that this file will not exist until you create it.
+
+### Keyword arguments
+
+You can manually provide your Milvus credentials as keyword arguments each time you call methods like [`compute_similarity()`](https://docs.voxel51.com/api/fiftyone.brain.html#fiftyone.brain.compute_similarity) that require connections to Milvus:
+
+```python
+import fiftyone.brain as fob
+
+milvus_index = fob.compute_similarity(
+ ...
+ backend="milvus",
+ brain_key="milvus_index",
+ uri="XXXXXX",
+ user="XXXXXX",
+ password="XXXXXX",
+
+ # also available if necessary
+ secure=True,
+ token="XXXXXX",
+ db_name="XXXXXX",
+ client_key_path="XXXXXX",
+ client_pem_path="XXXXXX",
+ ca_pem_path="XXXXXX",
+ server_pem_path="XXXXXX",
+ server_name="XXXXXX",
+)
+```
+
+Note that, when using this strategy, you must manually provide the credentials when loading an index later via [`load_brain_results()`](https://docs.voxel51.com/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.load_brain_results):
+
+```python
+milvus_index = dataset.load_brain_results(
+ "milvus_index",
+ uri="XXXXXX",
+ user="XXXXXX",
+ password="XXXXXX",
+
+ # also available if necessary
+ secure=True,
+ token="XXXXXX",
+ db_name="XXXXXX",
+ client_key_path="XXXXXX",
+ client_pem_path="XXXXXX",
+ ca_pem_path="XXXXXX",
+ server_pem_path="XXXXXX",
+ server_name="XXXXXX",
+)
+```
+
+### Milvus config parameters
+
+The Milvus backend supports a variety of query parameters that can be used to customize your similarity queries. These parameters include:
+
+- **collection_name** (*None*): the name of the Milvus collection to use or create. If none is provided, a new collection will be created
+
+- **metric** (*"dotproduct"*): the embedding distance metric to use when creating a new index. The supported values are (`"dotproduct"`, `"euclidean"`)
+
+- **consistency_level** (*"Session"*): the consistency level to use. Supported values are (`"Strong"`, `"Session"`, `"Bounded"`, `"Eventually"`)
+
+For detailed information on these parameters, see the [Milvus authentication documentation](authenticate.md) and [Milvus consistency levels documentation](consistency.md).
+
+You can specify these parameters via any of the strategies described in the previous section. Here’s an example of a [brain config](https://docs.voxel51.com/user_guide/brain.html#brain-config) that includes all of the available parameters:
+
+```json
+{
+ "similarity_backends": {
+ "milvus": {
+ "collection_name": "your_collection",
+ "metric": "dotproduct",
+ "consistency_level": "Strong"
+ }
+ }
+}
+```
+
+However, typically these parameters are directly passed to [`compute_similarity()`](https://docs.voxel51.com/api/fiftyone.brain.html#fiftyone.brain.compute_similarity) to configure a specific new index:
+
+```python
+milvus_index = fob.compute_similarity(
+ ...
+ backend="milvus",
+ brain_key="milvus_index",
+ collection_name="your_collection",
+ metric="dotproduct",
+ consistency_level="Strong",
+)
+```
+
+## Manage brain runs
+
+FiftyOne provides a variety of methods that you can use to manage brain runs.
+
+For example, you can call [`list_brain_runs()`](https://docs.voxel51.com/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.list_brain_runs) to see the available brain keys on a dataset:
+
+```python
+import fiftyone.brain as fob
+
+# List all brain runs
+dataset.list_brain_runs()
+
+# Only list similarity runs
+dataset.list_brain_runs(type=fob.Similarity)
+
+# Only list specific similarity runs
+dataset.list_brain_runs(
+ type=fob.Similarity,
+ patches_field="ground_truth",
+ supports_prompts=True,
+)
+```
+
+Or, you can use [`get_brain_info()`](https://docs.voxel51.com/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.get_brain_info) to retrieve information about the configuration of a brain run:
+
+```python
+info = dataset.get_brain_info(brain_key)
+print(info)
+```
+
+Use [`load_brain_results()`](https://docs.voxel51.com/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.load_brain_results) to load the [`SimilarityIndex`](https://docs.voxel51.com/api/fiftyone.brain.similarity.html#fiftyone.brain.similarity.SimilarityIndex) instance for a brain run.
+
+You can use [`rename_brain_run()`](https://docs.voxel51.com/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.rename_brain_run) to rename the brain key associated with an existing similarity results run:
+
+```python
+dataset.rename_brain_run(brain_key, new_brain_key)
+```
+
+Finally, you can use [`delete_brain_run()`](https://docs.voxel51.com/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.delete_brain_run) to delete a brain run:
+
+```python
+dataset.delete_brain_run(brain_key)
+```
+
+| # | +ID | +Name | +Company | +City | +Phone | +
|---|---|---|---|---|---|
| 0 | +2 | +Jane Smith | +XYZ Inc | +Los Angeles | +098-765-4321 | +
+
+Calling [`delete_brain_run()`](https://docs.voxel51.com/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.delete_brain_run) only deletes the record of the brain run from your FiftyOne dataset; it will not delete any associated Milvus collection, which you can do as follows:
+
+```python
+# Delete the Milvus collection
+milvus_index = dataset.load_brain_results(brain_key)
+milvus_index.cleanup()
+```
+
+
+
+For common vector search workflow on a FiftyOne dataset using the Milvus backend, see [Examples here](https://docs.voxel51.com/integrations/milvus.html#examples).
diff --git a/preview/site/en/integrations/integrate_with_voyageai.md b/preview/site/en/integrations/integrate_with_voyageai.md
new file mode 100644
index 000000000..80a026cc0
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_voyageai.md
@@ -0,0 +1,118 @@
+---
+id: integrate_with_voyageai.md
+title: Semantic Search with Milvus and VoyageAI
+summary: This page discusses vector database integration with VoyageAI's embedding API.
+---
+
+# Semantic Search with Milvus and VoyageAI
+
+
+
+This guide showcases how [VoyageAI's Embedding API](https://docs.voyageai.com/docs/embeddings) can be used with Milvus vector database to conduct semantic search on text.
+
+## Getting started
+Before you start, make sure you have the Voyage API key ready, or you get one from the [VoyageAI website](https://dash.voyageai.com/api-keys).
+
+The data used in this example are book titles. You can download the dataset [here](https://www.kaggle.com/datasets/jealousleopard/goodreadsbooks) and put it in the same directory where you run the following code.
+
+First, install the package for Milvus and Voyage AI:
+
+
+```python
+$ pip install --upgrade voyageai pymilvus
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**. (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+With this, we're ready to generate embeddings and use vector database to conduct semantic search.
+
+## Searching book titles with VoyageAI & Milvus
+
+In the following example, we load book title data from the downloaded CSV file, use Voyage AI embedding model to generate vector representations, and store them in Milvus vector database for semantic search.
+
+
+```python
+import voyageai
+from pymilvus import MilvusClient
+
+MODEL_NAME = "voyage-law-2" # Which model to use, please check https://docs.voyageai.com/docs/embeddings for available models
+DIMENSION = 1024 # Dimension of vector embedding
+
+# Connect to VoyageAI with API Key.
+voyage_client = voyageai.Client(api_key="
+
+As for the argument of `MilvusClient`:
+
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+With all data in Milvus vector database, we can now perform semantic search by generating vector embedding for the query and conduct vector search.
+
+
+```python
+queries = ["When was artificial intelligence founded?"]
+
+query_vectors = voyage_client.embed(
+ texts=queries, model=MODEL_NAME, truncation=False
+).embeddings
+
+res = milvus_client.search(
+ collection_name=COLLECTION_NAME, # target collection
+ data=query_vectors, # query vectors
+ limit=2, # number of returned entities
+ output_fields=["text", "subject"], # specifies fields to be returned
+)
+
+for q in queries:
+ print("Query:", q)
+ for result in res:
+ print(result)
+ print("\n")
+```
+
+ Query: When was artificial intelligence founded?
+ [{'id': 0, 'distance': 0.7196218371391296, 'entity': {'text': 'Artificial intelligence was founded as an academic discipline in 1956.', 'subject': 'history'}}, {'id': 1, 'distance': 0.6297335028648376, 'entity': {'text': 'Alan Turing was the first person to conduct substantial research in AI.', 'subject': 'history'}}]
+
+
+
diff --git a/preview/site/en/integrations/integrate_with_whyhow.md b/preview/site/en/integrations/integrate_with_whyhow.md
new file mode 100644
index 000000000..8c1cf8211
--- /dev/null
+++ b/preview/site/en/integrations/integrate_with_whyhow.md
@@ -0,0 +1,186 @@
+---
+id: integrate_with_whyhow.md
+summary: This guide demonstrates how to use whyhow.ai and Milvus Lite to conduct Rule-based Retrieval.
+title: Integrate Milvus with WhyHow
+---
+
+# Integrate Milvus with WhyHow
+
+This guide demonstrates how to use whyhow.ai and Milvus Lite to conduct Rule-based Retrieval.
+
+## Overview
+
+WhyHow is a platform which provides developers the building blocks they need to organize, contextualize, and reliably retrieve unstructured data to perform complex RAG. The Rule-based Retrieval package is a Python package developed by WhyHow that enables people to create and manage Retrieval Augmented Generation (RAG) applications with advanced filtering capabilities.
+
+## Installation
+
+Before you start, please install all the necessary python packages for later usage.
+
+```shell
+pip install --upgrade pymilvus, whyhow_rbr
+```
+
+Next, we need to initialize the Milvus client to implement the Rule-based Retrieval by using Milvus Lite.
+
+```python
+from pymilvus import MilvusClient
+
+# Milvus Lite local path
+path="./milvus_demo.db" # random name for local milvus lite db path
+
+# Initialize the ClientMilvus
+milvus_client = ClientMilvus(path)
+```
+
+You can also initialize the Milvus client through Milvus Cloud
+
+```python
+from pymilvus import MilvusClient
+
+# Milvus Cloud credentials
+YOUR_MILVUS_CLOUD_END_POINT = "YOUR_MILVUS_CLOUD_END_POINT"
+YOUR_MILVUS_CLOUD_TOKEN = "YOUR_MILVUS_CLOUD_TOKEN"
+
+# Initialize the ClientMilvus
+milvus_client = ClientMilvus(
+ milvus_uri=YOUR_MILVUS_CLOUD_END_POINT,
+ milvus_token=YOUR_MILVUS_CLOUD_TOKEN,
+)
+```
+
+## Create Collection
+
+### Defining necessary variables
+
+```python
+# Define collection name
+COLLECTION_NAME="YOUR_COLLECTION_NAME" # take your own collection name
+
+# Define vector dimension size
+DIMENSION=1536 # decide by the model you use
+```
+
+### Add schema
+
+Before inserting any data into Milvus Lite database, we need to first define the data field, which is called schema in here. Through create object `CollectionSchema` and add data field through `add_field()`, we can control our data type and their characteristics. This step is mandatory before inserting any data into Milvus.
+
+```python
+schema = milvus_client.create_schema(auto_id=True) # Enable id matching
+
+schema = milvus_client.add_field(schema=schema, field_name="id", datatype=DataType.INT64, is_primary=True)
+schema = milvus_client.add_field(schema=schema, field_name="embedding", datatype=DataType.FLOAT_VECTOR, dim=DIMENSION)
+```
+
+### Create index
+
+For each schema, it is better to have an index so that the querying will be much more efficient. To create an index, we first need an `index_params` and later add more index data on this `IndexParams` object.
+
+```python
+# Start to indexing data field
+index_params = milvus_client.prepare_index_params()
+index_params = milvus_client.add_index(
+ index_params=index_params, # pass in index_params object
+ field_name="embedding",
+ index_type="AUTOINDEX", # use autoindex instead of other complex indexing method
+ metric_type="COSINE", # L2, COSINE, or IP
+)
+```
+
+This method is a thin wrapper around the official Milvus implementation ([official docs](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/add_index.md)).
+
+### Create collection
+
+After defining all the data field and indexing them, we now need to create our database collection so that we can access our data quick and precise. What needs to be mentioned is that we initialized the `enable_dynamic_field` to be true so that you can upload any data freely. The cost is that data querying might be inefficient.
+
+```python
+# Create Collection
+milvus_client.create_collection(
+ collection_name=COLLECTION_NAME,
+ schema=schema,
+ index_params=index_params
+)
+```
+
+## Upload documents
+
+After creating a collection, we are ready to populate it with documents. In `whyhow_rbr` this is done using the `upload_documents` method of the `MilvusClient`. It performs the following steps under the hood:
+
+- **Preprocessing**: Reading and splitting the provided PDF files into chunks
+- **Embedding**: Embedding all the chunks using an OpenAI model
+- **Inserting**: Uploading both the embeddings and the metadata to Milvus Lite
+
+```python
+# get pdfs
+pdfs = ["harry-potter.pdf", "game-of-thrones.pdf"] # replace to your pdfs path
+
+# Uploading the PDF document
+milvus_client.upload_documents(
+ collection_name=COLLECTION_NAME,
+ documents=pdfs
+)
+```
+
+## Question answering
+
+Now we can finally move to retrieval augmented generation.
+
+```python
+# Search data and implement RAG!
+res = milvus_client.search(
+ question='What food does Harry Potter like to eat?',
+ collection_name=COLLECTION_NAME,
+ anns_field='embedding',
+ output_fields='text'
+)
+print(res['answer'])
+print(res['matches'])
+```
+
+### Rules
+
+In the previous example, every single document in our index was considered. However, sometimes it might be beneficial to only retrieve documents satisfying some predefined conditions (e.g. `filename=harry-potter.pdf`). In `whyhow_rbr` through Milvus Lite, this can be done via adjusting searching parameters.
+
+A rule can control the following metadata attributes
+
+- `filename` name of the file
+- `page_numbers` list of integers corresponding to page numbers (0 indexing)
+- `id` unique identifier of a chunk (this is the most "extreme" filter)
+- Other rules base on [Boolean Expressions](https://milvus.io/docs/boolean.md)
+
+```python
+# RULES(search on book harry-potter on page 8):
+PARTITION_NAME='harry-potter' # search on books
+page_number='page_number == 8'
+
+# first create a partitions to store the book and later search on this specific partition:
+milvus_client.crate_partition(
+ collection_name=COLLECTION_NAME,
+ partition_name=PARTITION_NAME # separate base on your pdfs type
+)
+
+# search with rules
+res = milvus_client.search(
+ question='Tell me about the greedy method',
+ collection_name=COLLECTION_NAME,
+ partition_names=PARTITION_NAME,
+ filter=page_number, # append any rules follow the Boolean Expression Rule
+ anns_field='embedding',
+ output_fields='text'
+)
+print(res['answer'])
+print(res['matches'])
+```
+
+In this example, we first create a partition that store harry-potter related pdfs, and through searching within this partition, we can get the most direct information. Also, we apply page numbers as a filter to specify the exact page we wish to search on. Remember, the filer parameter needs to follow the [boolean rule](https://milvus.io/docs/boolean.md).
+
+### Clean up
+
+At last, after implementing all the instructions, you can clean up the database by calling `drop_collection()`.
+
+```python
+# Clean up
+milvus_client.drop_collection(
+ collection_name=COLLECTION_NAME
+)
+```
+
diff --git a/preview/site/en/menuStructure/en.json b/preview/site/en/menuStructure/en.json
index c9160fbd5..936a05219 100644
--- a/preview/site/en/menuStructure/en.json
+++ b/preview/site/en/menuStructure/en.json
@@ -1,1315 +1,1371 @@
-{
- "menuList": [
+[
{
- "id": "about",
- "title": "About Milvus",
- "label1": "",
- "label2": "",
- "label3": "",
- "order": 0,
- "isMenu": true
- },
- {
- "id": "overview.md",
- "title": "What is Milvus",
- "label1": "about",
- "label2": "",
- "label3": "",
- "order": 0
- },
- {
- "id": "milvus_adopters.md",
- "title": "Milvus Adopters",
- "label1": "about",
- "label2": "",
- "label3": "",
- "order": 1
- },
- {
- "id": "roadmap",
- "title": "Milvus Roadmap",
- "label1": "about",
- "label2": "",
- "label3": "",
- "order": 2,
- "outLink": "https://wiki.lfaidata.foundation/display/MIL/Milvus+2.X+Roadmap+and+Time+schedule"
- },
- {
- "id": "limitations.md",
- "title": "Milvus Limits",
- "label1": "about",
- "label2": "",
- "label3": "",
- "order": 3
- },
- {
- "id": "release_notes.md",
- "title": "Releases",
- "label1": "about",
- "label2": "",
- "label3": "",
- "order": 4
- },
- {
- "id": "enhancementproposals",
- "title": "Enhancement Proposals",
- "label1": "about",
- "label2": "",
- "label3": "",
- "order": 5,
- "outLink": "https://wiki.lfaidata.foundation/pages/viewpage.action?pageId=43287103"
- },
- {
- "id": "bootcamp_about",
- "title": "Bootcamp",
- "label1": "about",
- "label2": "",
- "label3": "",
- "order": 6,
- "outLink": "https://milvus.io/bootcamp"
- },
- {
- "id": "getstarted",
- "title": "Get Started",
- "label1": "",
- "label2": "",
- "label3": "",
- "order": 1,
- "isMenu": true
- },
- {
- "id": "prerequisites",
- "title": "Prerequisites",
- "label1": "getstarted",
- "label2": "",
- "label3": "",
- "order": 0
- },
- {
- "id": "prerequisite-helm.md",
- "title": "On Kubernetes",
- "label1": "getstarted",
- "label2": "prerequisites",
- "label3": "",
- "order": 0
- },
- {
- "id": "prerequisite-docker.md",
- "title": "With Docker Compose",
- "label1": "getstarted",
- "label2": "prerequisites",
- "label3": "",
- "order": 1
- },
- {
- "id": "installmilvus",
- "title": "Install Milvus",
- "label1": "getstarted",
- "label2": "",
- "label3": "",
- "order": 1,
- "isMenu": true
-
- },
- {
- "id": "install_standalone-operator.md",
- "title": "Milvus Standalone",
- "label1": "getstarted",
- "label2": "installmilvus",
- "label3": "",
- "order": 0
- },
- {
- "id": "install_cluster-milvusoperator.md",
- "title": "Milvus Cluster",
- "label1": "getstarted",
- "label2": "installmilvus",
- "label3": "",
- "order": 1
- },
- {
- "id": "install_offline-helm.md",
- "title": "Install Offline",
- "label1": "getstarted",
- "label2": "installmilvus",
- "label3": "",
- "order": 2
- },
- {
- "id": "install_embedded_milvus.md",
- "title": "Install Embedded Milvus",
- "label1": "getstarted",
- "label2": "installmilvus",
- "label3": "",
- "order": 3
- },
- {
- "id": "install_sdks",
- "title": "Install SDKs",
- "label1": "getstarted",
- "label2": "",
- "label3": "",
- "order": 2
- },
- {
- "id": "install-pymilvus.md",
- "title": "Pymilvus",
- "label1": "getstarted",
- "label2": "install_sdks",
- "label3": "",
- "order": 0
- },
- {
- "id": "install-java.md",
- "title": "Java SDK",
- "label1": "getstarted",
- "label2": "install_sdks",
- "label3": "",
- "order": 1
- },
- {
- "id": "install-go.md",
- "title": "Go SDK",
- "label1": "getstarted",
- "label2": "install_sdks",
- "label3": "",
- "order": 2
- },
- {
- "id": "install-node.md",
- "title": "Node.js SDK",
- "label1": "getstarted",
- "label2": "install_sdks",
- "label3": "",
- "order": 3
- },
- {
- "id": "example_code.md",
- "title": "Hello Milvus",
- "label1": "getstarted",
- "label2": "",
- "label3": "",
- "order": 3
- },
- {
- "id": "userguide",
- "title": "User Guide",
- "label1": "",
- "label2": "",
- "order": 2,
- "isMenu": true
- },
- {
- "id": "manage_connection.md",
- "title": "Manage Milvus Connections",
- "label1": "userguide",
- "label2": "",
- "label3": "",
- "order": 0
- },
- {
- "id": "manage_collection",
- "title": "Manage Collections",
- "label1": "userguide",
- "label2": "",
- "label3": "",
- "order": 1,
- "isMenu": true
- },
- {
- "id": "create_collection.md",
- "title": "Create a Collection",
- "label1": "userguide",
- "label2": "manage_collection",
- "label3": "",
- "order": 0
- },
- {
- "id": "modify_collection.md",
- "title": "Modify a Collection",
- "label1": "userguide",
- "label2": "manage_collection",
- "label3": "",
- "order": 1
- },
- {
- "id": "check_collection.md",
- "title": "Check Collection Information",
- "label1": "userguide",
- "label2": "manage_collection",
- "label3": "",
- "order": 2
- },
- {
- "id": "drop_collection.md",
- "title": "Drop a Collection",
- "label1": "userguide",
- "label2": "manage_collection",
- "label3": "",
- "order": 3
- },
- {
- "id": "collection_alias.md",
- "title": "Manage Collection Alias",
- "label1": "userguide",
- "label2": "manage_collection",
- "label3": "",
- "order": 4
- },
- {
- "id": "load_collection.md",
- "title": "Load Collection",
- "label1": "userguide",
- "label2": "manage_collection",
- "label3": "",
- "order": 5
- },
- {
- "id": "release_collection.md",
- "title": "Release Collection",
- "label1": "userguide",
- "label2": "manage_collection",
- "label3": "",
- "order": 6
- },
- {
- "id": "manage_partition",
- "title": "Manage Partitions",
- "label1": "userguide",
- "label2": "",
- "label3": "",
- "order": 2,
- "isMenu": true
- },
- {
- "id": "create_partition.md",
- "title": "Create a Partition",
- "label1": "userguide",
- "label2": "manage_partition",
- "label3": "",
- "order": 0
- },
- {
- "id": "check_partition.md",
- "title": "Check Partition Information",
- "label1": "userguide",
- "label2": "manage_partition",
- "label3": "",
- "order": 1
- },
- {
- "id": "drop_partition.md",
- "title": "Drop a Partition",
- "label1": "userguide",
- "label2": "manage_partition",
- "label3": "",
- "order": 2
- },
- {
- "id": "load_partition.md",
- "title": "Load a Partition",
- "label1": "userguide",
- "label2": "manage_partition",
- "label3": "",
- "order": 3
- },
- {
- "id": "release_partition.md",
- "title": "Release a Partition",
- "label1": "userguide",
- "label2": "manage_partition",
- "label3": "",
- "order": 4
- },
- {
- "id": "manage_data",
- "title": "Manage Data",
- "label1": "userguide",
- "label2": "",
- "label3": "",
- "order": 3,
- "isMenu": true
- },
- {
- "id": "insert_data.md",
- "title": "Insert Entities",
- "label1": "userguide",
- "label2": "manage_data",
- "label3": "",
- "order": 0
- },
- {
- "id": "bulk_insert.md",
- "title": "Insert Entities from Files",
- "label1": "userguide",
- "label2": "manage_data",
- "label3": "",
- "order": 1
- },
- {
- "id": "delete_data.md",
- "title": "Delete Entities",
- "label1": "userguide",
- "label2": "manage_data",
- "label3": "",
- "order": 2
- },
- {
- "id": "compact_data.md",
- "title": "Compact Data",
- "label1": "userguide",
- "label2": "manage_data",
- "label3": "",
- "order": 3
- },
- {
- "id": "manage_index",
- "title": "Manage Indexes",
- "label1": "userguide",
- "label2": "",
- "label3": "",
- "order": 4,
- "isMenu": true
- },
- {
- "id": "build_index.md",
- "title": "Build an Index on Vectors",
- "label1": "userguide",
- "label2": "manage_index",
- "label3": "",
- "order": 0
- },
- {
- "id": "build_scalar_index.md",
- "title": "Build an Index on Scalars",
- "label1": "userguide",
- "label2": "manage_index",
- "label3": "",
- "order": 1
- },
- {
- "id": "drop_index.md",
- "title": "Drop an Index",
- "label1": "userguide",
- "label2": "manage_index",
- "label3": "",
- "order": 2
- },
- {
- "id": "search",
- "title": "Search and Query",
- "label1": "userguide",
- "label2": "",
- "label3": "",
- "order": 5,
- "isMenu": true
- },
- {
- "id": "search.md",
- "title": "Search",
- "label1": "userguide",
- "label2": "search",
- "label3": "",
- "order": 0
- },
- {
- "id": "hybridsearch.md",
- "title": "Hybrid Search",
- "label1": "userguide",
- "label2": "search",
- "label3": "",
- "order": 2
- },
- {
- "id": "query.md",
- "title": "Query",
- "label1": "userguide",
- "label2": "search",
- "label3": "",
- "order": 1
- },
- {
- "id": "timetravel.md",
- "title": "Search with Time Travel",
- "label1": "userguide",
- "label2": "",
- "label3": "",
- "order": 6
- },
- {
- "id": "calculate_distance.md",
- "title": "Calculate Distance",
- "label1": "userguide",
- "label2": "",
- "label3": "",
- "order": 7
- },
- {
- "id": "bootcamp_userguide",
- "title": "Bootcamp",
- "label1": "userguide",
- "label2": "",
- "label3": "",
- "order": 8,
- "outLink": "https://milvus.io/bootcamp"
- },
- {
- "id": "admin_guide",
- "title": "Administration Guide",
- "label1": "",
- "label2": "",
- "order": 3,
- "isMenu": true
- },
- {
- "id": "configure_milvus",
- "title": "Configure Milvus",
- "label1": "admin_guide",
- "label2": "",
- "order": 0,
- "isMenu": true
- },
- {
- "id": "configure-helm.md",
- "title": "With Helm",
- "label1": "admin_guide",
- "label2": "configure_milvus",
- "label3": "",
- "order": 1
- },
- {
- "id": "configure-docker.md",
- "title": "With Docker",
- "label1": "admin_guide",
- "label2": "configure_milvus",
- "label3": "",
- "order": 2
- },
- {
- "id": "configure_operator.md",
- "title": "With Milvus Operator",
- "label1": "admin_guide",
- "label2": "configure_milvus",
- "label3": "",
- "order": 0
- },
- {
- "id": "manage_dependencies",
- "title": "Manage Dependencies",
- "label1": "admin_guide",
- "label2": "",
- "order": 1,
- "isMenu": true
- },
- {
- "id": "docker_helm_dependencies",
- "title": "With Docker or Helm",
- "label1": "admin_guide",
- "label2": "manage_dependencies",
- "label3": "",
- "order": 0,
- "isMenu": true
- },
- {
- "id": "deploy_s3.md",
- "title": "Object Storage",
- "label1": "admin_guide",
- "label2": "manage_dependencies",
- "label3": "docker_helm_dependencies",
- "order": 0
- },
- {
- "id": "deploy_etcd.md",
- "title": "Meta Storage",
- "label1": "admin_guide",
- "label2": "manage_dependencies",
- "label3": "docker_helm_dependencies",
- "order": 1
- },
- {
- "id": "deploy_pulsar.md",
- "title": "Message Storage",
- "label1": "admin_guide",
- "label2": "manage_dependencies",
- "label3": "docker_helm_dependencies",
- "order": 2
- },
- {
- "id": "milvus_operator_dependencies",
- "title": "With Milvus Operator",
- "label1": "admin_guide",
- "label2": "manage_dependencies",
- "label3": "",
- "order": 1,
- "isMenu": true
- },
- {
- "id": "object_storage_operator.md",
- "title": "Object Storage",
- "label1": "admin_guide",
- "label2": "manage_dependencies",
- "label3": "milvus_operator_dependencies",
- "order": 0
- },
- {
- "id": "meta_storage_operator.md",
- "title": "Meta Storage",
- "label1": "admin_guide",
- "label2": "manage_dependencies",
- "label3": "milvus_operator_dependencies",
- "order": 1
- },
- {
- "id": "message_storage_operator.md",
- "title": "Message Storage",
- "label1": "admin_guide",
- "label2": "manage_dependencies",
- "label3": "milvus_operator_dependencies",
- "order": 2
- },
- {
- "id": "allocate.md",
- "title": "Allocate Resources",
- "label1": "admin_guide",
- "label2": "",
- "label3": "",
- "order": 2
- },
- {
- "id": "cloud",
- "title": "Deploy on Clouds",
- "label1": "admin_guide",
- "label2": "",
- "label3": "",
- "order": 3,
- "isMenu": true
- },
- {
- "id": "aws",
- "title": "AWS",
- "label1": "admin_guide",
- "label2": "cloud",
- "label3": "",
- "order": 0,
- "isMenu": true
- },
- {
- "id": "aws.md",
- "title": "Amazon EC2",
- "label1": "admin_guide",
- "label2": "cloud",
- "label3": "aws",
- "order": 0
- },
- {
- "id": "eks.md",
- "title": "Amazon EKS",
- "label1": "admin_guide",
- "label2": "cloud",
- "label3": "aws",
- "order": 1
- },
- {
- "id": "gcp",
- "title": "GCP",
- "label1": "admin_guide",
- "label2": "cloud",
- "label3": "",
- "order": 2
- },
- {
- "id": "gcp.md",
- "title": "GCP",
- "label1": "admin_guide",
- "label2": "cloud",
- "label3": "gcp",
- "order": 0
- },
- {
- "id": "gcp_layer7.md",
- "title": "Layer-7 Load Balancing",
- "label1": "admin_guide",
- "label2": "cloud",
- "label3": "gcp",
- "order": 1
- },
- {
- "id": "azure.md",
- "title": "Azure",
- "label1": "admin_guide",
- "label2": "cloud",
- "label3": "",
- "order": 2
- },
- {
- "id": "scaleout.md",
- "title": "Scale a Milvus Cluster",
- "label1": "admin_guide",
- "label2": "",
- "label3": "",
- "order": 3
- },
- {
- "id": "resource_group.md",
- "title": "Manage Resource Groups",
- "label1": "admin_guide",
- "label2": "",
- "label3": "",
- "order": 5
- },
- {
- "id": "upgrade_milvus",
- "title": "Upgrade",
- "label1": "admin_guide",
- "label2": "",
- "label3": "",
- "order": 6
- },
- {
- "id": "upgrade_milvus_cluster-operator.md",
- "title": "Upgrade Milvus Cluster",
- "label1": "admin_guide",
- "label2": "upgrade_milvus",
- "label3": "",
- "order": 0
- },
- {
- "id": "upgrade_milvus_standalone-operator.md",
- "title": "Upgrade Milvus Standalone",
- "label1": "admin_guide",
- "label2": "upgrade_milvus",
- "label3": "",
- "order": 1
- },
- {
- "id": "monitor",
- "title": "Monitor and Alert",
- "label1": "admin_guide",
- "label2": "",
- "label3": "",
- "order": 5,
- "isMenu": true
- },
- {
- "id": "monitor_overview.md",
- "title": "Monitoring Architecture",
- "label1": "admin_guide",
- "label2": "monitor",
- "label3": "",
- "order": 0
- },
- {
- "id": "monitor.md",
- "title": "Deploy Monitoring Services",
- "label1": "admin_guide",
- "label2": "monitor",
- "label3": "",
- "order": 1
- },
- {
- "id": "visualize.md",
- "title": "Visualize Milvus Metrics",
- "label1": "admin_guide",
- "label2": "monitor",
- "label3": "",
- "order": 2
- },
- {
- "id": "metrics_dashboard.md",
- "title": "Milvus Metrics Dashboard",
- "label1": "admin_guide",
- "label2": "monitor",
- "label3": "",
- "order": 3
- },
- {
- "id": "alert.md",
- "title": "Create an Alert",
- "label1": "admin_guide",
- "label2": "monitor",
- "label3": "",
- "order": 4
- },
- {
- "id": "rbac.md",
- "title": "Enable RBAC",
- "label1": "admin_guide",
- "label2": "",
- "label3": "",
- "order": 7,
- "isMenu": true
- },
- {
- "id": "security",
- "title": "Security",
- "label1": "admin_guide",
- "label2": "",
- "label3": "",
- "order": 8,
- "isMenu": true
- },
- {
- "id": "authenticate.md",
- "title": "Enable Authentication",
- "label1": "admin_guide",
- "label2": "security",
- "label3": "",
- "order": 0
- },
- {
- "id": "tls.md",
- "title": "Encryption in Transit",
- "label1": "admin_guide",
- "label2": "security",
- "label3": "",
- "order": 1
- },
- {
- "id": "benchmark.md",
- "title": "Benchmarks",
- "label1": "",
- "label2": "",
- "order": 4,
- "isMenu": true
- },
- {
- "id": "tools",
- "title": "Tools",
- "label1": "",
- "label2": "",
- "order": 5,
- "isMenu": true
- },
- {
- "id": "cli",
- "title": "Milvus CLI",
- "label1": "tools",
- "label2": "",
- "label3": "",
- "order":0,
- "isMenu": true
- },
- {
- "id": "cli_overview.md",
- "title":"Overview",
- "label1":"tools",
- "label2":"cli",
- "label3": "",
- "order":0
- },
- {
- "id": "install_cli.md",
- "title":"Installation",
- "label1":"tools",
- "label2":"cli",
- "label3": "",
- "order":1
- },
- {
- "id": "cli_commands.md",
- "title":"Commands",
- "label1":"tools",
- "label2":"cli",
- "label3": "",
- "order":2
- },
- {
- "id": "milvus_migration",
- "title": "Milvus Migration",
- "label1": "tools",
- "label2": "",
- "label3": "",
- "order":2,
- "isMenu": true
- },
- {
- "id": "milvus_migration_overview.md",
- "title": "Overview",
- "label1": "tools",
- "label2": "milvus_migration",
- "label3": "",
- "order":0,
- "isMenu": true
- },
- {
- "id": "milvus_migration-sqlite.md",
- "title": "User Guide",
- "label1": "tools",
- "label2": "milvus_migration",
- "label3": "",
- "order":1,
- "isMenu": true
- },
- {
- "id": "milvus_migration_conf.md",
- "title": "Configuration Guide",
- "label1": "tools",
- "label2": "milvus_migration",
- "label3": "",
- "order":2,
- "isMenu": true
- },
- {
- "id": "milvus_backup",
- "title": "Milvus Backup",
- "label1": "tools",
- "label2": "",
- "label3": "",
- "order":3,
- "isMenu": true
- },
- {
- "id": "milvus_backup_overview.md",
- "title": "Overview",
- "label1": "tools",
- "label2": "milvus_backup",
- "label3": "",
- "order":0,
- "isMenu": true
- },
- {
- "id": "milvus_backup-cli.md",
- "title": "Commands",
- "label1": "tools",
- "label2": "milvus_backup",
- "label3": "",
- "order":1,
- "isMenu": true
- },
- {
- "id": "milvus_backup-api.md",
- "title": "API",
- "label1": "tools",
- "label2": "milvus_backup",
- "label3": "",
- "order":2,
- "isMenu": true
- },
- {
- "id": "milvus_attu",
- "title": "Attu",
- "label1": "tools",
- "label2": "",
- "label3": "",
- "order": 3,
- "isMenu": true
- },
- {
- "id": "attu.md",
- "title": "Overview",
- "label1": "tools",
- "label2": "milvus_attu",
- "label3": "",
- "order": 0
- },
- {
- "id": "attu_install-docker.md",
- "title": "Installation",
- "label1": "tools",
- "label2": "milvus_attu",
- "label3": "",
- "order": 1
- },
- {
- "id": "attu_collection.md",
- "title": "Manage Collections",
- "label1": "tools",
- "label2": "milvus_attu",
- "label3": "",
- "order": 2
- },
- {
- "id": "attu_partition.md",
- "title": "Manage Partitions",
- "label1": "tools",
- "label2": "milvus_attu",
- "label3": "",
- "order": 3
- },
- {
- "id": "attu_data.md",
- "title": "Manage Data",
- "label1": "tools",
- "label2": "milvus_attu",
- "label3": "",
- "order": 4
- },
- {
- "id": "attu_index.md",
- "title": "Manage Index",
- "label1": "tools",
- "label2": "milvus_attu",
- "label3": "",
- "order": 5
- },
- {
- "id": "attu_query.md",
- "title": "Query Data",
- "label1": "tools",
- "label2": "milvus_attu",
- "label3": "",
- "order": 6
- },
- {
- "id": "attu_search.md",
- "title": "Search Data",
- "label1": "tools",
- "label2": "milvus_attu",
- "label3": "",
- "order": 7
- },
- {
- "id": "attu_system.md",
- "title": "System Status",
- "label1": "tools",
- "label2": "milvus_attu",
- "label3": "",
- "order": 8
- },
- {
- "id": "attu_faq.md",
- "title": "Attu FAQ",
- "label1": "tools",
- "label2": "milvus_attu",
- "label3": "",
- "order": 9
- },
- {
- "id": "milvusdm",
- "title": "MilvusDM",
- "label1": "tools",
- "label2": "",
- "label3": "",
- "order": 4,
- "isMenu": true
- },
- {
- "id": "migrate_overview.md",
- "title": "Overview",
- "label1": "tools",
- "label2": "milvusdm",
- "label3": "",
- "order": 0
- },
- {
- "id": "milvusdm_install.md",
- "title": "Installation",
- "label1": "tools",
- "label2": "milvusdm",
- "label3": "",
- "order": 1
- },
- {
- "id": "sizing",
- "title": "Milvus Sizing Tool",
- "label1": "tools",
- "label2": "",
- "label3": "",
- "order": 5,
- "outLink": "https://milvus.io/tools/sizing/"
- },
- {
- "id": "reference",
- "title": "Reference",
- "label1": "",
- "label2": "",
- "order": 5,
- "isMenu": true
- },
- {
- "id": "architecture",
- "title": "Architecture",
- "label1": "reference",
- "label2": "",
- "label3": "",
- "order": 0,
- "isMenu": true
- },
- {
- "id": "architecture_overview.md",
- "title": "Overview",
- "label1": "reference",
- "label2": "architecture",
- "label3": "",
- "order": 0
- },
- {
- "id": "four_layers.md",
- "title": "Storage/Computing",
- "label1": "reference",
- "label2": "architecture",
- "label3": "",
- "order": 1
- },
- {
- "id": "main_components.md",
- "title": "Main Components",
- "label1": "reference",
- "label2": "architecture",
- "label3": "",
- "order": 2
- },
- {
- "id": "data_processing.md",
- "title": "Data Processing",
- "label1": "reference",
- "label2": "architecture",
- "label3": "",
- "order": 3
- },
- {
- "id": "bitset.md",
- "title": "Bitset",
- "label1": "reference",
- "label2": "",
- "order": 1
- },
- {
- "id": "boolean.md",
- "title": "Boolean Expression Rules",
- "label1": "reference",
- "label2": "",
- "order": 2
- },
- {
- "id": "consistency.md",
- "title": "Consistency",
- "label1": "reference",
- "label2": "",
- "order": 3
- },
- {
- "id": "coordinator_ha.md",
- "title": "Coordinator HA",
- "label1": "reference",
- "label2": "",
- "label3": "",
- "order": 4
- },
- {
- "id": "glossary.md",
- "title": "Glossary",
- "label1": "reference",
- "label2": "",
- "order": 5
- },
- {
- "id": "replica.md",
- "title": "In-memory Replica",
- "label1": "reference",
- "label2": "",
- "order": 6
- },
- {
- "id": "knowhere.md",
- "title": "Knowhere",
- "label1": "reference",
- "label2": "",
- "order": 7
- },
- {
- "id": "schema.md",
- "title": "Schema",
- "label1": "reference",
- "label2": "",
- "order": 8
- },
- {
- "id": "metric.md",
- "title": "Similarity Metrics",
- "label1": "reference",
- "label2": "",
- "order": 9
- },
- {
- "id": "system_configuration.md",
- "title": "System Configurations",
- "label1": "reference",
- "label2": "",
- "order": 10
- },
- {
- "id": "timestamp.md",
- "title": "Timestamp",
- "label1": "reference",
- "label2": "",
- "order": 11
- },
- {
- "id": "time_sync.md",
- "title": "Time Synchronization",
- "label1": "reference",
- "label2": "",
- "order": 12
- },
- {
- "id": "timetravel_ref.md",
- "title": "Time Travel",
- "label1": "reference",
- "label2": "",
- "order": 13
- },
- {
- "id": "users_and_roles.md",
- "title": "Users and Roles",
- "label1": "reference",
- "label2": "",
- "order": 14
- },
- {
- "id": "vector_index",
- "title": "Vector Index",
- "label1": "reference",
- "label2": "",
- "order": 15
- },
- {
- "id": "index.md",
- "title": "In-memory Index",
- "label1": "reference",
- "label2": "vector_index",
- "order": 0
- },
- {
- "id": "disk_index.md",
- "title": "On-disk Index",
- "label1": "reference",
- "label2": "vector_index",
- "order": 1
- },
- {
- "id": "scalar_index.md",
- "title": "Scalar Index",
- "label1": "reference",
- "label2": "",
- "order": 16
- },
- {
- "id": "tutorials",
- "title": "Example Applications",
- "label1": "",
- "label2": "",
- "order": 6,
- "isMenu": true
- },
- {
- "id": "image_similarity_search.md",
- "title": "Image Similarity Search",
- "label1": "tutorials",
- "label2": "",
- "label3": "",
- "order": 0
- },
- {
- "id": "question_answering_system.md",
- "title": "Question Answering System",
- "label1": "tutorials",
- "label2": "",
- "label3": "",
- "order": 1
- },
- {
- "id": "recommendation_system.md",
- "title": "Recommender System",
- "label1": "tutorials",
- "label2": "",
- "label3": "",
- "order": 2
- },
- {
- "id": "video_similarity_search.md",
- "title": "Video Similarity Search",
- "label1": "tutorials",
- "label2": "",
- "label3": "",
- "order": 3
- },
- {
- "id": "audio_similarity_search.md",
- "title": "Audio Similarity Search",
- "label1": "tutorials",
- "label2": "",
- "label3": "",
- "order": 4
- },
- {
- "id": "molecular_similarity_search.md",
- "title": "Molecular Similarity Search",
- "label1": "tutorials",
- "label2": "",
- "label3": "",
- "order": 5
- },
- {
- "id": "dna_sequence_classification.md",
- "title": "DNA Sequence Classification",
- "label1": "tutorials",
- "label2": "",
- "label3": "",
- "order": 6
- },
- {
- "id": "text_search_engine.md",
- "title": "Text Search Engine",
- "label1": "tutorials",
- "label2": "",
- "label3": "",
- "order": 7
- },
- {
- "id": "text_image_search.md",
- "title": "Text to Image Search Engine",
- "label1": "tutorials",
- "label2": "",
- "label3": "",
- "order": 8
- },
- {
- "id": "image_deduplication_system.md",
- "title": "Image Deduplication System",
- "label1": "tutorials",
- "label2": "",
- "label3": "",
- "order": 9
- },
- {
- "id": "faq",
- "title": "FAQs",
- "label1": "",
- "label2": "",
- "order": 7,
- "isMenu": true
- },
-
- {
- "id": "performance_faq.md",
- "title": "Performance FAQs",
- "label1": "faq",
- "label2": "",
- "label3": "",
- "order": 0
- },
- {
- "id": "product_faq.md",
- "title": "Product FAQs",
- "label1": "faq",
- "label2": "",
- "label3": "",
- "order": 1
- },
- {
- "id": "operational_faq.md",
- "title": "Operational FAQs",
- "label1": "faq",
- "label2": "",
- "order": 2
- },
- {
- "id": "troubleshooting.md",
- "title": "Troubleshooting",
- "label1": "faq",
- "label2": "",
- "order": 3
- }
- ]
-}
+ "label": "About Milvus",
+ "id": "about",
+ "isMenu": true,
+ "order": 0,
+ "children": [
+ {
+ "label": "What is Milvus",
+ "id": "overview.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Milvus Adopters",
+ "id": "milvus_adopters.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Milvus Roadmap",
+ "id": "roadmap.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Releases",
+ "id": "release_notes.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "Benchmarks",
+ "id": "benchmark.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "Comparison",
+ "id": "comparison.md",
+ "order": 5,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Get Started",
+ "id": "getstarted",
+ "isMenu": true,
+ "order": 1,
+ "children": [
+ {
+ "label": "Quickstart",
+ "id": "quickstart.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Install Milvus",
+ "id": "install_milvus",
+ "order": 1,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Run Milvus Lite",
+ "id": "milvus_lite.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Run Milvus in Docker",
+ "id": "run_milvus_docker",
+ "order": 1,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Requirements",
+ "id": "prerequisite-docker.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Docker",
+ "id": "install_standalone-docker.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Docker Compose",
+ "id": "install_standalone-docker-compose.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Run Milvus in Kubernetes",
+ "id": "run_milvus_k8s",
+ "order": 2,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Requirements",
+ "id": "prerequisite-helm.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Milvus Operator",
+ "id": "install_cluster-milvusoperator.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Helm Chart",
+ "id": "install_cluster-helm.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Run Milvus with GPU",
+ "id": "run_milvus_gpu",
+ "order": 3,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Requirements",
+ "id": "prerequisite-gpu.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Helm Chart",
+ "id": "install_cluster-helm-gpu.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Docker Compose",
+ "id": "install_standalone-docker-compose-gpu.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "label": "Install SDKs",
+ "id": "install_sdks",
+ "order": 2,
+ "children": [
+ {
+ "label": "PyMilvus",
+ "id": "install-pymilvus.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Java SDK",
+ "id": "install-java.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Go SDK",
+ "id": "install-go.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Node.js SDK",
+ "id": "install-node.md",
+ "order": 3,
+ "children": []
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "label": "Concepts",
+ "id": "concepts",
+ "isMenu": true,
+ "order": 2,
+ "children": [
+ {
+ "label": "Architecture",
+ "id": "architecture",
+ "order": 0,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Overview",
+ "id": "architecture_overview.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Storage/Computing",
+ "id": "four_layers.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Main Components",
+ "id": "main_components.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Data Processing",
+ "id": "data_processing.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "Knowhere",
+ "id": "knowhere.md",
+ "order": 5,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Bitset",
+ "id": "bitset.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Consistency",
+ "id": "consistency.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Multi-tenancy",
+ "id": "multi_tenancy.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "Timestamp",
+ "id": "timestamp.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "Similarity Metrics",
+ "id": "metric.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "Time Synchronization",
+ "id": "time_sync.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "Vector Index",
+ "id": "vector_index",
+ "order": 7,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "In-memory Index",
+ "id": "index.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "On-disk Index",
+ "id": "disk_index.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "GPU Index",
+ "id": "gpu_index.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Scalar Index",
+ "id": "scalar_index.md",
+ "order": 8,
+ "children": []
+ },
+ {
+ "label": "Reranking",
+ "id": "reranking.md",
+ "order": 9,
+ "children": []
+ },
+ {
+ "label": "Users and Roles",
+ "id": "users_and_roles.md",
+ "order": 10,
+ "children": []
+ },
+ {
+ "label": "In-memory Replica",
+ "id": "replica.md",
+ "order": 11,
+ "children": []
+ },
+ {
+ "label": "Terminology",
+ "id": "glossary.md",
+ "order": 12,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "User Guide",
+ "id": "userguide",
+ "isMenu": true,
+ "order": 3,
+ "children": [
+ {
+ "label": "Manage Databases",
+ "id": "manage_databases.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Manage Schema",
+ "id": "schema.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Manage Collections",
+ "id": "manage-collections.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Manage Indexes",
+ "id": "manage_indexes",
+ "order": 3,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Index Vector Fields",
+ "id": "index-vector-fields.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Index Scalar Fields",
+ "id": "index-scalar-fields.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Index with GPU",
+ "id": "index-with-gpu.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Manage Partitions",
+ "id": "manage-partitions.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "Insert, Upsert & Delete",
+ "id": "insert-update-delete.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "Data Import",
+ "id": "data_import",
+ "order": 6,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Prepare Source Data",
+ "id": "prepare-source-data.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Import Data",
+ "id": "import-data.md",
+ "order": 1,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Search, Query & Get",
+ "id": "search_query_get",
+ "order": 7,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Single-Vector Search",
+ "id": "single-vector-search.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Hybrid Search",
+ "id": "multi-vector-search.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Get & Scalar Query",
+ "id": "get-and-scalar-query.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "With Iterators",
+ "id": "with-iterators.md",
+ "order": 3,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Advanced Features",
+ "id": "advanced_features",
+ "order": 8,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Dynamic Field",
+ "id": "enable-dynamic-field.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Partition Key",
+ "id": "use-partition-key.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "JSON Field",
+ "id": "use-json-fields.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Array Field",
+ "id": "array_data_type.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "Sparse Vector",
+ "id": "sparse_vector.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "Scalar Filtering Rules",
+ "id": "boolean.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "MMap Storage",
+ "id": "mmap.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "Manage Resource Groups",
+ "id": "resource_group.md",
+ "order": 7,
+ "children": []
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "label": "Models",
+ "id": "models",
+ "isMenu": true,
+ "order": 4,
+ "children": [
+ {
+ "label": "Embeddings",
+ "id": "embeddings",
+ "order": 0,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Overview",
+ "id": "embeddings.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "OpenAI",
+ "id": "embed-with-openai.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Sentence Transformers",
+ "id": "embed-with-sentence-transform.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "BGE M3",
+ "id": "embed-with-bgm-m3.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "BM25",
+ "id": "embed-with-bm25.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "SPLADE",
+ "id": "embed-with-splade.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "Voyage",
+ "id": "embed-with-voyage.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "Jina AI",
+ "id": "embed-with-jina.md",
+ "order": 7,
+ "children": []
+ },
+ {
+ "label": "Cohere",
+ "id": "embed-with-cohere.md",
+ "order": 8,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Rerankers",
+ "id": "rerankers",
+ "order": 1,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Overview",
+ "id": "rerankers-overview.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "BGE",
+ "id": "rerankers-bge.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Cohere",
+ "id": "rerankers-cohere.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Cross Encoder",
+ "id": "rerankers-cross-encoder.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "Voyage",
+ "id": "rerankers-voyage.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "Jina AI",
+ "id": "rerankers-jina.md",
+ "order": 5,
+ "children": []
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "label": "Milvus Migration",
+ "id": "milvus_migration",
+ "isMenu": true,
+ "order": 5,
+ "children": [
+ {
+ "label": "Overview",
+ "id": "migrate_overview.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Install Migration Tool",
+ "id": "milvusdm_install.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "From Elasticsearch",
+ "id": "es2m.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "From Faiss",
+ "id": "f2m.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "From Milvus 1.x",
+ "id": "m2m.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "From Milvus 2.3.x",
+ "id": "from-m2x.md",
+ "order": 5,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Administration Guide",
+ "id": "admin_guide",
+ "isMenu": true,
+ "order": 9,
+ "children": [
+ {
+ "label": "Deployment",
+ "id": "deployment",
+ "order": 0,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "On Premises",
+ "id": "on_premises",
+ "order": 0,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "With Docker",
+ "id": "configure-docker.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "With Helm",
+ "id": "configure-helm.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "With Milvus Operator",
+ "id": "configure_operator.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Allocate Resources",
+ "id": "allocate.md",
+ "order": 3,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "On Clouds",
+ "id": "on_cloud",
+ "order": 1,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Amazon EC2",
+ "id": "aws.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Amazon EKS",
+ "id": "eks.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Google Cloud",
+ "id": "gcp.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Google Cloud Storage",
+ "id": "gcs.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "GCP Layer-7 Load Balancing",
+ "id": "gcp_layer7.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "Microsoft Azure",
+ "id": "azure.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "Microsoft Azure Blob Storage",
+ "id": "abs.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "Ingress Nginx",
+ "id": "ingress.md",
+ "order": 7,
+ "children": []
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "label": "Configuration",
+ "id": "configuration",
+ "order": 1,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "System Configurations",
+ "id": "system_configuration.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "On the Fly",
+ "id": "dynamic_config.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Limit Collection Counts",
+ "id": "limit_collection_counts.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Configure Chunk Cache",
+ "id": "chunk_cache.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "Coordinator HA",
+ "id": "coordinator_ha.md",
+ "order": 4,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Manage Dependencies",
+ "id": "manage_dependencies",
+ "order": 2,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "With Docker or Helm",
+ "id": "docker_helm_dependencies",
+ "order": 0,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Object Storage",
+ "id": "deploy_s3.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Meta Storage",
+ "id": "deploy_etcd.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Message Storage",
+ "id": "deploy_pulsar.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "With Milvus Operator",
+ "id": "milvus_operator_dependencies",
+ "order": 1,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Object Storage",
+ "id": "object_storage_operator.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Meta Storage",
+ "id": "meta_storage_operator.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Message Storage",
+ "id": "message_storage_operator.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "label": "Scaling",
+ "id": "scaling",
+ "order": 4,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Scale Cluster",
+ "id": "scaleout.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Scale Dependencies",
+ "id": "scale-dependencies.md",
+ "order": 1,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Upgrade",
+ "id": "upgrade_milvus",
+ "order": 7,
+ "children": [
+ {
+ "label": "Upgrade Milvus Cluster",
+ "id": "upgrade_milvus_cluster-operator.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Upgrade Milvus Standalone",
+ "id": "upgrade_milvus_standalone-operator.md",
+ "order": 1,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Monitoring, Alerts & Logs",
+ "id": "monitor_alert_log",
+ "order": 8,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Monitoring",
+ "id": "monitoring",
+ "order": 0,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Architecture",
+ "id": "monitor_overview.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Deploy Monitoring Services",
+ "id": "monitor.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Visualize Metrics",
+ "id": "visualize.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Milvus Metrics Dashboard",
+ "id": "metrics_dashboard.md",
+ "order": 3,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Alerts",
+ "id": "alerts",
+ "order": 1,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Create an Alert",
+ "id": "alert.md",
+ "order": 0,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Logs",
+ "id": "logs",
+ "order": 2,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Configure Grafana Loki",
+ "id": "configure_grafana_loki.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Configure Access Logs",
+ "id": "configure_access_logs.md",
+ "order": 1,
+ "children": []
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "label": "Security",
+ "id": "security",
+ "order": 9,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Enable Authentication",
+ "id": "authenticate.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Enable RBAC",
+ "id": "rbac.md",
+ "order": 1,
+ "isMenu": true,
+ "children": []
+ },
+ {
+ "label": "Encryption in Transit",
+ "id": "tls.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "label": "Tools",
+ "id": "tools",
+ "isMenu": true,
+ "order": 6,
+ "children": [
+ {
+ "label": "Milvus CLI",
+ "id": "cli",
+ "order": 0,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Overview",
+ "id": "cli_overview.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Installation",
+ "id": "install_cli.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Commands",
+ "id": "cli_commands.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Milvus Backup",
+ "id": "milvus_backup",
+ "order": 1,
+ "children": [
+ {
+ "label": "Overview",
+ "id": "milvus_backup_overview.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Commands",
+ "id": "milvus_backup_cli.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "RESTful API",
+ "id": "milvus_backup_api.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Milvus CDC",
+ "id": "milvus_cdc",
+ "order": 3,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Overview",
+ "id": "milvus-cdc-overview.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Deploy CDC Server",
+ "id": "deploy-cdc-server.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Manage CDC Tasks",
+ "id": "manage-cdc-tasks.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Monitoring",
+ "id": "cdc-monitoring.md",
+ "order": 3,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Attu(Milvus GUI)",
+ "id": "milvus_attu",
+ "order": 4,
+ "externalLink": "https://github.com/zilliztech/attu",
+ "children": []
+ },
+ {
+ "label": "Birdwatcher",
+ "id": "birdwatcher",
+ "order": 5,
+ "children": [
+ {
+ "label": "Overview",
+ "id": "birdwatcher_overview.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Install Guides",
+ "id": "birdwatcher_install_guides.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Usage Guides",
+ "id": "birdwatcher_usage_guides.md",
+ "order": 2,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Milvus Sizing Tool",
+ "id": "sizing",
+ "order": 6,
+ "externalLink": "https://milvus.io/tools/sizing/",
+ "children": []
+ },
+ {
+ "label": "Milvus Connectors",
+ "id": "milvus_connectors",
+ "order": 7,
+ "isMenu": true,
+ "children": [
+ {
+ "label": "Spark",
+ "id": "integrate_with_spark.md",
+ "order": 0,
+ "children": []
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "label": "Integrations",
+ "id": "integrations",
+ "isMenu": true,
+ "order": 8,
+ "children": [
+ {
+ "label": "OpenAI",
+ "id": "integrate_with_openai.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Cohere",
+ "id": "integrate_with_cohere.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "HuggingFace",
+ "id": "integrate_with_hugging-face.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "PyTorch",
+ "id": "integrate_with_pytorch.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "SentenceTransformers",
+ "id": "integrate_with_sentencetransformers.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "LlamaIndex",
+ "id": "integrate_with_llamaindex.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "LangChain",
+ "id": "integrate_with_langchain.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "HayStack",
+ "id": "integrate_with_haystack.md",
+ "order": 7,
+ "children": []
+ },
+ {
+ "label": "FiftyOne",
+ "id": "integrate_with_voxel51.md",
+ "order": 7,
+ "children": []
+ },
+ {
+ "label": "VoyageAI",
+ "id": "integrate_with_voyageai.md",
+ "order": 8,
+ "children": []
+ },
+ {
+ "label": "BentoML",
+ "id": "integrate_with_bentoml.md",
+ "order": 9,
+ "children": []
+ },
+ {
+ "label": "DSPy",
+ "id": "integrate_with_dspy.md",
+ "order": 10,
+ "children": []
+ },
+ {
+ "label": "Jina AI",
+ "id": "integrate_with_jina.md",
+ "order": 11,
+ "children": []
+ },
+ {
+ "label": "Snowflake",
+ "id": "integrate_with_snowpark.md",
+ "order": 12,
+ "children": []
+ },
+ {
+ "label": "WhyHow",
+ "id": "integrate_with_whyhow.md",
+ "order": 13,
+ "children": []
+ },
+ {
+ "label": "LangFuse",
+ "id": "integrate_with_langfuse.md",
+ "order": 14,
+ "children": []
+ },
+ {
+ "label": "Ragas",
+ "id": "integrate_with_ragas.md",
+ "order": 15,
+ "children": []
+ },
+ {
+ "label": "MemGPT",
+ "id": "integrate_with_memgpt.md",
+ "order": 16,
+ "children": []
+ },
+ {
+ "label": "FastGPT",
+ "id": "integrate_with_fastgpt.md",
+ "order": 17,
+ "children": []
+ },
+ {
+ "label": "Vanna",
+ "id": "integrate_with_vanna.md",
+ "order": 18,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "Tutorials",
+ "id": "tutorials",
+ "isMenu": true,
+ "order": 7,
+ "children": [
+ {
+ "label": "Build RAG with Milvus",
+ "id": "build-rag-with-milvus.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Image Search with Milvus",
+ "id": "image_similarity_search.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Question Answering System",
+ "id": "question_answering_system.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Recommender System",
+ "id": "recommendation_system.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "Video Similarity Search",
+ "id": "video_similarity_search.md",
+ "order": 4,
+ "children": []
+ },
+ {
+ "label": "Audio Similarity Search",
+ "id": "audio_similarity_search.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "DNA Sequence Classification",
+ "id": "dna_sequence_classification.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "Text Search Engine",
+ "id": "text_search_engine.md",
+ "order": 7,
+ "children": []
+ },
+ {
+ "label": "Text to Image Search Engine",
+ "id": "text_image_search.md",
+ "order": 8,
+ "children": []
+ },
+ {
+ "label": "Image Deduplication System",
+ "id": "image_deduplication_system.md",
+ "order": 9,
+ "children": []
+ }
+ ]
+ },
+ {
+ "label": "FAQs",
+ "id": "faq",
+ "isMenu": true,
+ "order": 9,
+ "children": [
+ {
+ "label": "Performance FAQs",
+ "id": "performance_faq.md",
+ "order": 0,
+ "children": []
+ },
+ {
+ "label": "Product FAQs",
+ "id": "product_faq.md",
+ "order": 1,
+ "children": []
+ },
+ {
+ "label": "Operational FAQs",
+ "id": "operational_faq.md",
+ "order": 2,
+ "children": []
+ },
+ {
+ "label": "Milvus Limits",
+ "id": "limitations.md",
+ "order": 3,
+ "children": []
+ },
+ {
+ "label": "Troubleshooting",
+ "id": "troubleshooting.md",
+ "order": 4,
+ "children": []
+ }
+ ]
+ }
+]
diff --git a/preview/site/en/migrate/es2m.md b/preview/site/en/migrate/es2m.md
new file mode 100644
index 000000000..88317c25e
--- /dev/null
+++ b/preview/site/en/migrate/es2m.md
@@ -0,0 +1,175 @@
+---
+id: es2m.md
+summary: This guide provides a comprehensive, step-by-step process for migrating data from Elasticsearch to Milvus 2.x.
+title: From Elasticsearch
+---
+
+# From Elasticsearch
+
+This guide provides a comprehensive, step-by-step process for migrating data from Elasticsearch to Milvus 2.x. By following this guide, you will be able to efficiently transfer your data, leveraging Milvus 2.x advanced features and improved performance.
+
+## Prerequisites
+
+- **Software versions**:
+ - Source Elasticsearch: 7.x or 8.x
+ - Target Milvus: 2.x
+ - For installation details, refer to [Installing Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) and [Install Milvus](https://milvus.io/docs/install_standalone-docker.md).
+- **Required tools**:
+ - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md).
+- **Supported data types for migration**: The fields to migrate from the source Elasticsearch index are of the following types - [dense_vector](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/dense-vector.html#dense-vector), [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/keyword.html#keyword-field-type), [text](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/text.html#text-field-type), [long](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [integer](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [double](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [float](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [boolean](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/boolean.html), [object](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/object.html). Data types not listed here are currently not supported for migration. Refer to [Field mapping reference](#field-mapping-reference) for detailed information on data mappings between Milvus collections and Elasticsearch indexes.
+- **Elasticsearch index requirements**:
+ - The source Elasticsearch index must contain a vector field of the `dense_vector` type. Migration cannot start without a vector field.
+
+## Configure the migration file
+
+Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory.
+
+```yaml
+dumper: # configs for the migration job.
+ worker:
+ workMode: "elasticsearch" # operational mode of the migration job.
+ reader:
+ bufferSize: 2500 # buffer size to read from Elasticsearch in each batch. A value ranging from 2000 to 4000 is recommended.
+meta: # meta configs for the source Elasticsearch index and target Milvus 2.x collection.
+ mode: "config" # specifies the source for meta configs. currently, onlly `config` is supported.
+ version: "8.9.1"
+ index: "qatest_index" # identifies the Elasticsearch index to migrate data from.
+ fields: # fields within the Elasticsearch index to be migrated.
+ - name: "my_vector" # name of the Elasticsearch field.
+ type: "dense_vector" # data type of the Elasticsearch field.
+ dims: 128 # dimension of the vector field. required only when `type` is `dense_vector`.
+ - name: "id"
+ pk: true # specifies if the field serves as a primary key.
+ type: "long"
+ - name: "num"
+ type: "integer"
+ - name: "double1"
+ type: "double"
+ - name: "text1"
+ maxLen: 1000 # max. length of data fields. required only for `keyword` and `text` data types.
+ type: "text"
+ - name: "bl1"
+ type: "boolean"
+ - name: "float1"
+ type: "float"
+ milvus: # configs specific to creating the collection in Milvus 2.x
+ collection: "Collection_01" # name of the Milvus collection. defaults to the Elasticsearch index name if not specified.
+ closeDynamicField: false # specifies whether to disable the dynamic field in the collection. defaults to `false`.
+ shardNum: 2 # number of shards to be created in the collection.
+ consistencyLevel: Strong # consistency level for Milvus collection.
+source: # connection configs for the source Elasticsearch server
+ es:
+ urls:
+ - "http://10.15.1.***:9200" # address of the source Elasticsearch server.
+ username: "" # username for the Elasticsearch server.
+ password: "" # password for the Elasticsearch server.
+target:
+ mode: "remote" # storage location for dumped files. valid values: `remote` and `local`.
+ remote: # configs for remote storage
+ outputDir: "migration/milvus/test" # output directory path in the cloud storage bucket.
+ cloud: "aws" # cloud storage service provider. Examples: `aws`, `gcp`, `azure`, etc.
+ region: "us-west-2" # region of the cloud storage; can be any value if using local Minio.
+ bucket: "zilliz-aws-us-****-*-********" # bucket name for storing data; must align with configs in milvus.yaml for Milvus 2.x.
+ useIAM: true # whether to use an IAM Role for connection.
+ checkBucket: false # checks if the specified bucket exists in the storage.
+ milvus2x: # connection configs for the target Milvus 2.x server
+ endpoint: "http://10.102.*.**:19530" # address of the target Milvus server.
+ username: "****" # username for the Milvus 2.x server.
+ password: "******" # password for the Milvus 2.x server.
+```
+
+The following table describes the parameters in the example config file. For a full list of configs, refer to [Milvus Migration: Elasticsearch to Milvus 2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_ES.md#migrationyaml-reference).
+
+- `dumper`
+
+ | Parameter | Description |
+ | --- | --- |
+ | `dumper.worker.workMode` | The operational mode of the migration job. Set to `elasticsearch` when migrating from Elasticsearch indexes. |
+ | `dumper.worker.reader.bufferSize` | Buffer size to read from Elasticsearch in each batch. Unit: KB. |
+
+- `meta`
+
+ | Parameter | Description |
+ | --- | --- |
+ | `meta.mode` | Specifies the source for meta configs. Currently, only `config` is supported. |
+ | `meta.index` | Identifies the Elasticsearch index to migrate data from. |
+ | `meta.fields` | Fields within the Elasticsearch index to be migrated. |
+ | `meta.fields.name` | Name of the Elasticsearch field. |
+ | `meta.fields.maxLen` | Maximum length of the field. This parameter is required only when `meta.fields.type` is `keyword` or `text`. |
+ | `meta.fields.pk` | Specifies if the field serves as the primary key. |
+ | `meta.fields.type` | Data type of the Elasticsearch field. Currently, the following data types in Elasticsearch are supported: [dense_vector](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/dense-vector.html#dense-vector), [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/keyword.html#keyword-field-type), [text](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/text.html#text-field-type), [long](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [integer](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [double](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [float](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [boolean](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/boolean.html), [object](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/object.html). |
+ | `meta.fields.dims` | Dimension of the vector field. This parameter is required only when `meta.fields.type` is `dense_vector`. |
+ | `meta.milvus` | Configs specific to creating the collection in Milvus 2.x. |
+ | `meta.milvus.collection` | Name of the Milvus collection. Defaults to the Elasticsearch index name if not specified. |
+ | `meta.milvus.closeDynamicField` | Specifies whether to disable the dynamic field in the collection. Defaults to `false`. For more information on dynamic fields, refer to [Enable Dynamic Field](https://milvus.io/docs/enable-dynamic-field.md#Enable-Dynamic-Field). |
+ | `meta.milvus.shardNum` | Number of shards to be created in the collection. For more information on shards, refer to [Terminology](https://milvus.io/docs/glossary.md#Shard). |
+ | `meta.milvus.consistencyLevel` | Consistency level for the collection in Milvus. For more information, refer to [Consistency](https://milvus.io/docs/consistency.md). |
+
+- `source`
+
+ | Parameter | Description |
+ | --- | --- |
+ | `source.es` | Connection configs for the source Elasticsearch server. |
+ | `source.es.urls` | Address of the source Elasticsearch server. |
+ | `source.es.username` | Username for the Elasticsearch server. |
+ | `source.es.password` | Password for the Elasticsearch server. |
+
+- `target`
+
+ | Parameter | Description |
+ | --- | --- |
+ | `target.mode` | Storage location for dumped files. Valid values:- `local`: Store dumped files on local disks.
- `remote`: Store dumped files on object storage. | + | `target.remote.outputDir` | Output directory path in the cloud storage bucket. | + | `target.remote.cloud` | Cloud storage service provider. Example values: `aws`, `gcp`, `azure`. | + | `target.remote.region` | Cloud storage region. It can be any value if you use local MinIO. | + | `target.remote.bucket` | Bucket name for storing data. The value must be the same as the config in Milvus 2.x. For more information, refer to [System Configurations](https://milvus.io/docs/configure_minio.md#miniobucketName). | + | `target.remote.useIAM` | Whether to use an IAM Role for connection. | + | `target.remote.checkBucket` | Whether to check if the specified bucket exists in object storage. | + | `target.milvus2x` | Connection configs for the target Milvus 2.x server. | + | `target.milvus2x.endpoint` | Address of the target Milvus server. | + | `target.milvus2x.username` | Username for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + | `target.milvus2x.password` | Password for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + +## Start the migration task + +Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + +```bash +./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml +``` + +The following is an example of a successful migration log output: + +```bash +[task/load_base_task.go:94] ["[LoadTasker] Dec Task Processing-------------->"] [Count=0] [fileName=testfiles/output/zwh/migration/test_mul_field4/data_1_1.json] [taskId=442665677354739304] +[task/load_base_task.go:76] ["[LoadTasker] Progress Task --------------->"] [fileName=testfiles/output/zwh/migration/test_mul_field4/data_1_1.json] [taskId=442665677354739304] +[dbclient/cus_field_milvus2x.go:86] ["[Milvus2x] begin to ShowCollectionRows"] +[loader/cus_milvus2x_loader.go:66] ["[Loader] Static: "] [collection=test_mul_field4_rename1] [beforeCount=50000] [afterCount=100000] [increase=50000] +[loader/cus_milvus2x_loader.go:66] ["[Loader] Static Total"] ["Total Collections"=1] [beforeTotalCount=50000] [afterTotalCount=100000] [totalIncrease=50000] +[migration/es_starter.go:25] ["[Starter] migration ES to Milvus finish!!!"] [Cost=80.009174459] +[starter/starter.go:106] ["[Starter] Migration Success!"] [Cost=80.00928425] +[cleaner/remote_cleaner.go:27] ["[Remote Cleaner] Begin to clean files"] [bucket=a-bucket] [rootPath=testfiles/output/zwh/migration] +[cmd/start.go:32] ["[Cleaner] clean file success!"] +``` + +## Verify the result + +Once the migration task is executed, you can make API calls or use Attu to view the number of entities migrated. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). + +## Field mapping reference + +Review the table below to understand how field types in Elasticsearch indexes are mapped to field types in Milvus collections. + +For more information on supported data types in Milvus, refer to [Supported data types](https://milvus.io/docs/schema.md#Supported-data-types). + +| Elasticsearch Field Type | Milvus Field Type | Description | +| --- | --- | --- | +| dense_vector | FloatVector | Vector dimensions remain unchanged during migration. | +| keyword | VarChar | Set Max Length (1 to 65,535). Strings exceeding the limit can trigger migration errors. | +| text | VarChar | Set Max Length (1 to 65,535). Strings exceeding the limit can trigger migration errors. | +| long | Int64 | - | +| integer | Int32 | - | +| double | Double | - | +| float | Float | - | +| boolean | Bool | - | +| object | JSON | - | \ No newline at end of file diff --git a/preview/site/en/migrate/f2m.md b/preview/site/en/migrate/f2m.md index 573427f9d..637d7123c 100644 --- a/preview/site/en/migrate/f2m.md +++ b/preview/site/en/migrate/f2m.md @@ -1,76 +1,135 @@ --- id: f2m.md -title: Migrate from Faiss to Milvus +title: From Faiss related_key: Faiss, migrate, import summary: Learn how to migrate Faiss data to Milvus. --- -# Migrate Data from Faiss to Milvus +# From Faiss -This topic describes how to import data from Faiss to Milvus using [MilvusDM](migrate_overview.md), an open-source tool specifically designed for Milvus data migration. +This guide provides a comprehensive, step-by-step process for migrating data from Faiss to Milvus 2.x. By following this guide, you will be able to efficiently transfer your data, leveraging Milvus 2.x advanced features and improved performance. ## Prerequisites -You need to [install MilvusDM](milvusdm_install.md) before migrating Milvus data. - -## 1. Download YAML file - -Download the `F2M.yaml` file. - -``` -$ wget https://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/F2M.yaml -``` - -## 2. Set the parameters - -Configuration parameters include: - -| Parameter | Description | Example | -| ------------------------- | ----------------------------------------- | ---------------------------- | -| `milvus_version` | Version of Milvus. | 2.0.0 | -| `data_path` | Path to the data in Faiss. | '/home/user/data/faiss.index' | -| `data_dir` | Directory of the HDF5 files. Set either `data_path` or `data_dir`. | '/Users/zilliz/Desktop/HDF5_data' | -| `dest_host` | Milvus server address. | '127.0.0.1' | -| `dest_port` | Milvus server port. | 19530 | -| `mode` | Mode of migration, including `skip`, `append`, and `overwrite`. This parameter works only when the specified collection name exists in the Milvus library.
index_file_size: 1024
metric_type: 'HAMMING'" | - -### Example - -The following example of configuration is for your reference. +- **Software versions**: + - Source Faiss + - Target Milvus: 2.x + - For installation details, see [Installing Faiss](https://github.com/facebookresearch/faiss/blob/main/INSTALL.md) and [Install Milvus](https://milvus.io/docs/install_standalone-docker.md). +- **Required tools**: + - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md). + +## Configure the migration + +Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory. + +```yaml +dumper: # configs for the migration job. + worker: + limit: 2 + workMode: faiss # operational mode of the migration job. + reader: + bufferSize: 1024 + writer: + bufferSize: 1024 +loader: + worker: + limit: 2 +source: # configs for the source Faiss index. + mode: local + local: + faissFile: ./testfiles/faiss/faiss_ivf_flat.index + +target: # configs for the target Milvus collection. + create: + collection: + name: test1w + shardsNums: 2 + dim: 256 + metricType: L2 + + mode: remote + remote: + outputDir: testfiles/output/ + cloud: aws + endpoint: 0.0.0.0:9000 + region: ap-southeast-1 + bucket: a-bucket + ak: minioadmin + sk: minioadmin + useIAM: false + useSSL: false + checkBucket: true + milvus2x: + endpoint: localhost:19530 + username: xxxxx + password: xxxxx ``` -F2M: - milvus_version: 2.0.0 - data_path: '/home/data/faiss1.index' - dest_host: '127.0.0.1' - dest_port: 19530 - mode: 'append' - dest_collection_name: 'test' - dest_partition_name: '' - collection_parameter: - dimension: 256 - index_file_size: 1024 - metric_type: 'L2' -``` - - -## 3. Migrate data from Faiss to Milvus - -Run MilvusDM to import data from Faiss to Milvus with the following command. - -``` -$ milvusdm --yaml F2M.yaml -``` - - -## What's next -- If you are interested in migrating data in other forms into Milvus, - - Learn how to [Migrate Data from HDF5 to Milvus](h2m.md). -- If you are looking for information about how to migrate data from Milvus 1.x to Milvus 2.0, - - Learn [version migration](m2m.md). -- If you are interested in learning more about the data migration tool, - - Read the overview of [MilvusDM](migrate_overview.md). +The following table describes the parameters in the example config file. For a full list of configs, refer to [Milvus Migration: Faiss to Milvus 2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_FAISS.md#migrationyaml-reference). + +- `dumper` + + | Parameter | Description | + | --- | --- | + | `dumper.worker.limit` | The concurrency of dumper threads. | + | `dumper.worker.workMode` | The operational mode of the migration job. Set to faiss when migrating from Faiss indexes. | + | `dumper.worker.reader.bufferSize` | Buffer size to read from Faiss in each batch. Unit: KB. | + | `dumper.worker.writer.bufferSize` | Buffer size to write to Milvus in each batch. Unit: KB. | + +- `loader` + + | Parameter | Description | + | --- | --- | + | `loader.worker.limit` | The concurrency of loader threads. | + +- `source` + + | Parameter | Description | + | --- | --- | + | `source.mode` | Specifies where the source files are read from. Valid values:
- `local`: reads files from a local disk.
- `remote`: reads files from remote storage. | + | `source.local.faissFile` | The directory path where the source files are located. For example, `/db/faiss.index`. | + +- `target` + + | Parameter | Description | + | --- | --- | + | `target.create.collection.name` | Name of the Milvus collection. | + | `target.create.collection.shardsNums` | Number of shards to be created in the collection. For more information on shards, refer to [Terminology](https://milvus.io/docs/glossary.md#Shard). | + | `target.create.collection.dim` | Dimension of the vector field. | + | `target.create.collection.metricType` | Metric type used to measure similarities between vectors. For more information, refer to [Terminology](https://milvus.io/docs/glossary.md#Metric-type). | + | `target.mode` | Storage location for dumped files. Valid values:
- `local`: Store dumped files on local disks.
- `remote`: Store dumped files on object storage. | + | `target.remote.outputDir` | Output directory path in the cloud storage bucket. | + | `target.remote.cloud` | Cloud storage service provider. Example values: `aws`, `gcp`, `azure`. | + | `target.remote.endpoint` | Endpoint of Milvus 2.x storage. | + | `target.remote.region` | Cloud storage region. It can be any value if you use local MinIO. | + | `target.remote.bucket` | Bucket name for storing data. The value must be the same as the config in Milvus 2.x. For more information, refer to [System Configurations](https://milvus.io/docs/configure_minio.md#miniobucketName). | + | `target.remote.ak` | Access key for Milvus 2.x storage. | + | `target.remote.sk` | Secret key for Milvus 2.x storage. | + | `target.remote.useIAM` | Whether to use an IAM Role for connection. | + | `target.remote.useSSL` | Whether to enable SSL when connecting to Milvus 2.x. For more information, refer to [Encryption in Transit](https://milvus.io/docs/tls.md#Encryption-in-Transit). | + | `target.remote.checkBucket` | Whether to check if the specified bucket exists in object storage. | + | `target.milvus2x.endpoint` | Address of the target Milvus server. | + | `target.milvus2x.username` | Username for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + | `target.milvus2x.password` | Password for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + +## Start the migration task + +1. Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + + ```bash + ./milvus-migration dump --config=/{YourConfigFilePath}/migration.yaml + ``` + + The command above converts the Faiss index data into NumPy files, and then uses the [bulkInsert](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md) operation to write the data to the target bucket. + +2. Once NumPy files are generated, import these files into Milvus 2.x with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + + ```bash + ./milvus-migration load --config=/{YourConfigFilePath}/migration.yaml + ``` + + +## Verify the result + +Once the migration task is executed, you can make API calls or use Attu to view the number of entities migrated. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). diff --git a/preview/site/en/migrate/from-m2x.md b/preview/site/en/migrate/from-m2x.md new file mode 100644 index 000000000..d9ac26bd7 --- /dev/null +++ b/preview/site/en/migrate/from-m2x.md @@ -0,0 +1,148 @@ +--- +id: from-m2x.md +summary: This guide provides a comprehensive, step-by-step process for migrating data from Milvus 2.3.x to Milvus 2.3.x or above. +title: From Milvus 2.3.x +--- + +# From Milvus 2.3.x + +This guide provides a comprehensive, step-by-step process for migrating data from Milvus 2.3.x to Milvus 2.3.x or above. + +## Prerequisites + +- **Software versions**: + - Source Milvus: 2.3.0+ (The tool uses the iterator to fetch source collection data, requiring source Milvus to be version 2.3.0 or above.) + - Target Milvus: 2.3.0+ +- **Required tools**: + - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md). +- **Data preparation**: + - Ensure that the source Milvus collection is loaded and ready for data export. + - If the target Milvus does not contain a collection corresponding to the source collection, the [milvus-migration](https://github.com/zilliztech/milvus-migration) tool will automatically create it. Note that after migration, the target collection will not be indexed, and you must manually index the collection afterward. + +## Configure the migration file + +Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory. + +```yaml +dumper: + worker: + workMode: milvus2x + reader: + bufferSize: 500 + +meta: + mode: config + version: 2.3.0 + collection: src_table_name + +source: + milvus2x: + endpoint: {milvus2x_domain}:{milvus2x_port} + username: xxxx + password: xxxxx + +target: + milvus2x: + endpoint: {milvus2x_domain}:{milvus2x_port} + username: xxxx + password: xxxxx +``` + +The following table describes the parameters in the example config file. For more information, refer to [Milvus Migration: Milvus2.x to Milvus2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_2X.md#milvus-migration-milvus2x-to-milvus2x). + +- `dumper` + + | Parameter | Description | + | --- | --- | + | `dumper.worker.workMode` | The operational mode of the migration job. Set to milvus2x when migrating from Milvus 2.x. | + | `dumper.worker.reader.bufferSize` | Buffer size to read from Milvus 2.x in each batch. | + +- `meta` + + | Parameter | Description | + | --- | --- | + | `meta.mode` | Specifies where the meta file is read from. Set to config, indicating that the meta config can be obtained from this migration.yaml file. | + | `meta.version` | Source Milvus version. Set to 2.3.0 or above. | + | `meta.collection` | Source collection name. | + +- `source` + + | Parameter | Description | + | --- | --- | + | `source.milvus2x.endpoint` | Address of the source Milvus server. | + | `source.milvus2x.username` | Username for the source Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | + | `source.milvus2x.password` | Password for the source Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | + +- `target` + + | Parameter | Description | + | --- | --- | + | `target.milvus2x.endpoint` | Address of the target Milvus server. | + | `target.milvus2x.username` | Username for the target Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | + | `target.milvus2x.password` | Password for the target Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | + +## Start the migration task + +You have two options to start the migration task - using CLI or making API requests. Choose the one that best fits your needs. + +### Option 1: Using CLI + +Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + +```bash +./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml +``` + +Monitor the logs for progress updates. Successful migration logs should include entries like: + +```bash +[INFO] [migration/milvus2x_starter.go:79] ["=================>JobProcess!"] [Percent=100] +[INFO] [migration/milvus2x_starter.go:27] ["[Starter] migration Milvus2x to Milvus2x finish!!!"] [Cost=94.877717375] +[INFO] [starter/starter.go:109] ["[Starter] Migration Success!"] [Cost=94.878243583] +``` + +### Option 2: Making API requests + +You can also use the Restful API to execute the migration. Start the API server with: + +```bash +./milvus-migration server run -p 8080 +``` + +Once the server starts successfully, place the `migration.yaml` file in the `configs/` directory of the project and start the migration using: + +```bash +curl -XPOST http://localhost:8080/api/v1/start +``` + +## Verify the result + +After the migration task is completed, use Attu to view the number of entities migrated. Additionally, you can create indexes and load collections in Attu. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). + +## Additional configuration options + +In addition to the basic configurations mentioned above, you can also add additional settings based on your specific requirements. + +- **Selective field migration**: If you need to migrate only specific fields in a collection rather than all fields, specify the fields to be migrated in the `meta` section of the `migration.yaml` file. + + ```yaml + meta: + fields: + - name: id + - name: title_vector + - name: reading_time + ``` + +- **Custom target collection**: To customize the properties of the target collection, add the related configurations in the `meta` section of the `migration.yaml` file. + + ```yaml + meta: + milvus: + collection: target_collection_name + shardNum: 2 + closeDynamicField: false + consistencyLevel: Customized + ``` + + +For detailed information, refer to [Milvus Migration: Milvus2.x to Milvus2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_2X.md#milvus-migration-milvus2x-to-milvus2x). diff --git a/preview/site/en/migrate/m2m.md b/preview/site/en/migrate/m2m.md new file mode 100644 index 000000000..c01a32535 --- /dev/null +++ b/preview/site/en/migrate/m2m.md @@ -0,0 +1,156 @@ +--- +id: m2m.md +summary: This guide provides a comprehensive, step-by-step process for migrating data from Milvus 1.x (including 0.9.x and above) to Milvus 2.x. +title: From Milvus 1.x +--- + +# From Milvus 1.x + +This guide provides a comprehensive, step-by-step process for migrating data from Milvus 1.x (including 0.9.x and above) to Milvus 2.x. By following this guide, you will be able to efficiently transfer your data, leveraging Milvus 2.x advanced features and improved performance. + +## Prerequisites + +- **Software versions**: + - Source Milvus: 0.9.x to 1.x + - Target Milvus: 2.x +- **Required tools**: + - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md). + +## Export metadata of the source Milvus installation + +To prepare migration data for Milvus 0.9.x through 1.x, stop the source Milvus or at least stop performing any DML operations in it. + +1. Export metadata of the source Milvus installation to `meta.json`. + + - For those installations using MySQL as the backend, run + + ```bash + ./milvus-migration export -m "user:password@tcp(adderss)/milvus?charset=utf8mb4&parseTime=True&loc=Local" -o outputDir + ``` + + - For those installations using SQLite as the backend, run + + ```bash + ./milvus-migration export -s /milvus/db/meta.sqlite -o outputDir + ``` + +2. Copy the `tables` folder of your Milvus installation, then move both `meta.json` and the `tables` folder to an empty folder. + + Once this step is done, the structure of the empty folder should look like this: + + ``` + migration_data + ├── meta.json + └── tables + ``` + +3. Upload the folder prepared in the preceding step to an S3 block storage bucket or directly use this local folder in the next section. + +## Configure the migration file + +Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory. + +```yaml +dumper: + worker: + limit: 2 + workMode: milvus1x + reader: + bufferSize: 1024 + writer: + bufferSize: 1024 +loader: + worker: + limit: 16 +meta: + mode: local + localFile: /outputDir/test/meta.json +source: + mode: local + local: + tablesDir: /db/tables/ +target: + mode: remote + remote: + outputDir: "migration/test/xx" + ak: xxxx + sk: xxxx + cloud: aws + region: us-west-2 + bucket: xxxxx + useIAM: true + checkBucket: false + milvus2x: + endpoint: "{yourMilvus2_xServerAddress}:{port}" + username: xxxx + password: xxxx +``` + +The following table describes the parameters in the example config file. For a full list of configs, refer to [Milvus Migration: Milvus1.x to Milvus 2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_1X.md#migrationyaml-reference). + +- `dumper` + + | Parameter | Description | + | --- | --- | + | `dumper.worker.limit` | The concurrency of dumper threads. | + | `dumper.worker.workMode` | The operational mode of the migration job. Set to `milvus1x` when migrating from Milvus 1.x. | + | `dumper.worker.reader.bufferSize` | Buffer size to read from Milvus 1.x in each batch. Unit: KB. | + | `dumper.worker.writer.bufferSize` | Buffer size to write to Milvus 2.x in each batch. Unit: KB. | + +- `loader` + + | Parameter | Description | + | --- | --- | + | `loader.worker.limit` | The concurrency of loader threads. | + +- `meta` + + | Parameter | Description | + | --- | --- | + | `meta.mode` | Specifies where the meta file meta.json is read from. Valid values: `local`, `remote`, `mysql`, `sqlite`. | + | `meta.localFile` | Local directory path where the `meta.json` file resides. This config is used only when `meta.mode` is set to `local`. For other meta configs, refer to [README_1X](https://github.com/zilliztech/milvus-migration/blob/main/README_1X.md#meta). | + +- `source` + + | Parameter | Description | + | --- | --- | + | `source.mode` | Specifies where the source files are read from. Valid values:
- `local`: reads files from a local disk.
- `remote`: reads files from remote storage. | + | `source.local.tablesDir` | The directory path where the source files are located. For example, `/db/tables/`. | + +- `target` + + | Parameter | Description | + | --- | --- | + | `target.mode` | Storage location for dumped files. Valid values:
- `local`: Store dumped files on local disks.
- `remote`: Store dumped files on object storage. | + | `target.remote.outputDir` | Output directory path in the cloud storage bucket. | + | `target.remote.ak` | Access key for Milvus 2.x storage. | + | `target.remote.sk` | Secret key for Milvus 2.x storage. | + | `target.remote.cloud` | Cloud storage service provider. Example values: `aws`, `gcp`, `azure`. | + | `target.remote.region` | Cloud storage region. It can be any value if you use local MinIO. | + | `target.remote.bucket` | Bucket name for storing data. The value must be the same as the config in Milvus 2.x. For more information, refer to [System Configurations](https://milvus.io/docs/configure_minio.md#miniobucketName). | + | `target.remote.useIAM` | Whether to use an IAM Role for connection. | + | `target.remote.checkBucket` | Whether to check if the specified bucket exists in object storage. | + | `target.milvus2x.endpoint` | Address of the target Milvus server. | + | `target.milvus2x.username` | Username for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + | `target.milvus2x.password` | Password for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | + +## Start the migration task + +1. Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + + ```bash + ./milvus-migration dump --config=/{YourConfigFilePath}/migration.yaml + ``` + + The command above converts the source data in Milvus 1.x into NumPy files, and then uses the [bulkInsert](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md) operation to write the data to the target bucket. + +2. Once NumPy files are generated, import these files into Milvus 2.x with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. + + ```bash + ./milvus-migration load --config=/{YourConfigFilePath}/migration.yaml + ``` + + +## Verify the result + +Once the migration task is executed, you can make API calls or use Attu to view the number of entities migrated. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). diff --git a/preview/site/en/migrate/migrate_overview.md b/preview/site/en/migrate/migrate_overview.md index 59a39eeb9..65a2d9be1 100644 --- a/preview/site/en/migrate/migrate_overview.md +++ b/preview/site/en/migrate/migrate_overview.md @@ -1,56 +1,51 @@ --- id: migrate_overview.md -summary: MilvusDM allows data migration between Milvus and many other sources of data. +summary: This article provides an overview of the Milvus-migration tool, including supported migrations, features, and architecture. +title: Milvus Migration Overview --- -# Overview -[MilvusDM](https://github.com/milvus-io/milvus-tools) (Milvus Data Migration) is an open-source tool designed specifically for importing and exporting data with Milvus. MilvusDM allows you to migrate data in a specific collection or partition. - -
-Currently, MilvusDM is only supported in Milvus 1.x version.
-
-
-To substantially improve data management efficiency and reduce DevOps costs, MilvusDM supports the following migration channels:
-- [Milvus to Milvus](m2m.md): Migrates data between instances of Milvus.
-- [Faiss to Milvus](f2m.md): Imports unzipped data from Faiss to Milvus.
-- [HDF5 to Milvus](h2m.md): Imports HDF5 files into Milvus.
-- [Milvus to HDF5](m2h.md): Saves the data in Milvus as HDF5 files.
-
-
-
-MilvusDM is hosted on GitHub. To install MilvusDM, run:
-```
-pip3 install pymilvusdm
-```
-
-## MilvusDM File Structure
-The flow chart below shows how MilvusDM performs different tasks according to the .yaml file it receives:
-
-
-
-MilvusDM file structure:
-
-- pymilvusdm
- - core
- - **milvus_client.py**: Performs client operations in Milvus.
- - **read_data.py**: Reads the HDF5 files on your local drive. (Add your code here to support reading data files in other formats.)
- - **read_faiss_data.py**: Reads Faiss data files.
- - **read_milvus_data.py**: Reads Milvus data files.
- - **read_milvus_meta.py**: Reads Milvus metadata.
- - **data_to_milvus.py**: Creates collections or partitions as specified in .yaml files and imports vectors and the corresponding IDs into Milvus.
- - **save_data.py**: Saves data as HDF5 files.
- - **write_logs.py**: Writes `debug`/`info`/`error` logs during runtime.
- - **faiss_to_milvus.py**: Imports Faiss data into Milvus.
- - **hdf5_to_milvus.py**: Imports HDF5 files into Milvus.
- - **milvus_to_milvus.py**: Migrates data from a source Milvus to a target Milvus.
- - **milvus_to_hdf5.py**: Saves Milvus data as HDF5 files.
- - **main.py**: Executes tasks as specified by the received .yaml file.
- - **setting.py**: Stores configurations for MilvusDM operation.
-- **setup.py**: Creates and uploads pymilvusdm file packages to PyPI (Python Package Index).
-## Enhancement Plan
-In future releases, MilvusDM will provide more new features including MilvusDump and MilvusRestore to support exporting all Milvus data, restoring data in specified collections and partitions, resuming interrupted download and more.
-
-
-
-The MilvusDM project is open sourced on GitHub. Any contribution to the project is welcome. Give it a star 🌟, and feel free to file an issue or submit your own code!
-
+# Milvus Migration Overview
+
+Recognizing the diverse needs of user base, Milvus has expanded its migration tools to not only facilitate upgrades from earlier Milvus 1.x versions but also to enable seamless integration of data from other systems like [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html) and [Faiss](https://github.com/facebookresearch/faiss). The [Milvus-migration](https://github.com/zilliztech/milvus-migration) project is designed to bridge the gap between these varied data environments and the latest advancements in Milvus technology, ensuring you can harness improved features and performance seamlessly.
+
+## Supported migrations
+
+The [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool supports a variety of migration paths to accommodate different user needs:
+
+- [Elasticsearch to Milvus 2.x](es2m.md): Enabling users to migrate data from Elasticsearch environments to take advantage of Milvus's optimized vector search capabilities.
+- [Faiss to Milvus 2.x](f2m.md): Providing experimental support for transferring data from Faiss, a popular library for efficient similarity search.
+- [Milvus 1.x to Milvus 2.x](m2m.md): Ensuring data from earlier versions is transitioned smoothly to the latest framework.
+- [Milvus 2.3.x to Milvus 2.3.x or above](from-m2x.md): Providing a one-time migration path for users who have already migrated to 2.3.x.
+
+## Features
+
+Milvus-migration is designed with robust features to handle diverse migration scenarios:
+
+- Multiple interaction methods: You can perform migrations via a command line interface or through a Restful API, with flexibility in how migrations are executed.
+- Support for various file formats and cloud storage: The [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool can handle data stored in local files as well as in cloud storage solutions such as S3, OSS, and GCP, ensuring broad compatibility.
+- Data type handling: [Milvus-migration](https://github.com/zilliztech/milvus-migration) is capable of dealing with both vector data and scalar fields, making it a versatile choice for different data migration needs.
+
+## Architecture
+
+The architecture of [Milvus-migration](https://github.com/zilliztech/milvus-migration) is strategically designed to facilitate efficient data streaming, parsing, and writing processes, enabling robust migration capabilities across various data sources.
+
+
+
+In the preceding figure:
+
+- **Data source**: [Milvus-migration](https://github.com/zilliztech/milvus-migration) supports multiple data sources including Elasticsearch via the [scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html), local or cloud storage data files, and Milvus 1.x databases. These are accessed and read in a streamlined manner to initiate the migration process.
+- **Stream pipeline**:
+ - **Parse process**: Data from the sources is parsed according to its format. For example, for a data source from Elasticsearch, an Elasticsearch format parser is employed, while other formats use respective parsers. This step is crucial for transforming raw data into a structured format that can be further processed.
+ - **Convert process**: Following parsing, data undergoes conversion where fields are filtered, data types are converted, and table names are adjusted according to the target Milvus 2.x schema. This ensures that the data conforms to the expected structure and types in Milvus.
+- **Data writing and loading**:
+ - **Write data**: The processed data is written into intermediate JSON or NumPy files, ready to be loaded into Milvus 2.x.
+ - **Load data**: Data is finally loaded into Milvus 2.x using the [BulkInsert](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md) operation, which efficiently writes large volumes of data into Milvus storage systems, either cloud-based or filestore.
+
+## Future plans
+
+The development team is committed to enhancing [Milvus-migration](https://github.com/zilliztech/milvus-migration) with features such as:
+
+- **Support for more data sources**: Plans to extend support to additional databases and file systems, such as Pinecone, Chroma, Qdrant. If you need support for a specific data source, please submit your request through this [GitHub issue link](https://github.com/zilliztech/milvus-migration/issues).
+- **Command simplification**: Efforts to streamline the command process for easier execution.
+- **SPI parser** / **convert**: The architecture expects to include Service Provider Interface (SPI) tools for both parsing and converting. These tools allow for custom implementations that users can plug into the migration process to handle specific data formats or conversion rules.
+- **Checkpoint resumption**: Enabling migrations to resume from the last checkpoint to enhance reliability and efficiency in case of interruptions. Save points will be created to ensure data integrity and are stored in databases such as SQLite or MySQL to track the progress of the migration process.
diff --git a/preview/site/en/migrate/milvusdm_install.md b/preview/site/en/migrate/milvusdm_install.md
index f8c51433f..d3f16d4e4 100644
--- a/preview/site/en/migrate/milvusdm_install.md
+++ b/preview/site/en/migrate/milvusdm_install.md
@@ -1,56 +1,49 @@
---
id: milvusdm_install.md
-summary: Learn how to install MilvusDM to migrate your data.
+summary: Learn how to install Milvus-Migration to migrate your data.
+title: Install Migration Tool
---
-# Install MilvusDM
-
-MilvusDM is an open-source tool designed specifically for importing and exporting data with Milvus. This page shows you how to install MilvusDM.
-
-
- The pymilvusdm2.0 is used for migrating data from Milvus(0.10.x or 1.x) to Milvus2.x.
-
-
-## Before you begin
-
-Ensure you meet the requirements for operating system and software before installing MilvusDM.
-
-
-| Operating system | Supported versions |
-| --------------- | ----------------- |
-| CentOS | 7.5 or higher |
-| Ubuntu LTS | 18.04 or higher |
-
-
-| Software | Version |
-| --------------------------- | ----------------------------- |
-| [Milvus](https://milvus.io/) | 0.10.x or 1.x or 2.x |
-| Python3 | 3.7 or higher |
-| pip3 | Should be in correspondence to the Python version. |
-
-## Install MilvusDM
-
-1. Add the following two lines to the **~/.bashrc** file:
-
-```bash
-export MILVUSDM_PATH='/home/$user/milvusdm'
-export LOGS_NUM=0
-```
-
-- `MILVUSDM_PATH`: This parameter defines the working path of MilvusDM. Logs and data generated by MilvusDM will be stored in this path. The default value is `/home/$user/milvusdm`.
-
-- `LOGS_NUM`： MilvusDM generates one log file each day. This parameter defines the number of log files you would like to save. The default value is 0, which means all log files are saved.
-
-2. Configure environment variables：
-
-```shell
-$ source ~/.bashrc
-```
-
-3. Install MilvusDM with `pip`:
-
-```shell
-$ pip3 install pymilvusdm==2.0
-```
-
-
+# Install Migration Tool
+
+We support downloading the executable binary file or compiling the Milvus-migration tool from source.
+
+## Download the executable binary
+
+1. Download the latest release from the [Milvus-Migration GitHub repository](https://github.com/zilliztech/milvus-migration/tags).
+2. Extract the downloaded file to obtain the `milvus-migration` executable binary.
+
+## Compile from source
+
+Alternatively, download and compile the source to obtain an executable binary file.
+
+1. Clone the Milvus-Migration repository:
+
+ ```bash
+ # clone the source project
+ git clone https://github.com/zilliztech/milvus-migration.git
+ ```
+
+2. Navigate to the project directory:
+
+ ```bash
+ cd milvus-migration
+ ```
+
+3. Compile the project to obtain the executable file:
+
+ ```bash
+ # compile the project to obtain an executable file
+ go get & go build
+ ```
+
+ This will generate the `milvus-migration` executable in the project directory.
+
+
+## What's next
+
+Having the Milvus-migration tool installed, you can migrate data from various sources:
+
+- [From Elasticsearch](es2m.md)
+- [From Faiss](f2m.md)
+- [From Milvus 1.x](m2m.md)
\ No newline at end of file
diff --git a/preview/site/en/reference/architecture/architecture.md b/preview/site/en/reference/architecture/architecture.md
index e72ed0c48..bfb3039ca 100644
--- a/preview/site/en/reference/architecture/architecture.md
+++ b/preview/site/en/reference/architecture/architecture.md
@@ -1,6 +1,7 @@
---
id: architecture.md
title: Architecture
+deprecate: true
---
# Architecture
diff --git a/preview/site/en/reference/architecture/architecture_overview.md b/preview/site/en/reference/architecture/architecture_overview.md
index 12026bba5..d0151c701 100644
--- a/preview/site/en/reference/architecture/architecture_overview.md
+++ b/preview/site/en/reference/architecture/architecture_overview.md
@@ -1,17 +1,18 @@
---
id: architecture_overview.md
-summary: Milvus provides a fast, reliable, and stable vector database built specifically for similarity search and artificial intelligence.
+summary: Milvus provides a fast, reliable, and stable vector database built specifically for similarity search and artificial intelligence.
+title: Milvus Architecture Overview
---
# Milvus Architecture Overview
-Built on top of popular vector search libraries including Faiss, Annoy, HNSW, and more, Milvus was designed for similarity search on dense vector datasets containing millions, billions, or even trillions of vectors. Before proceeding, familiarize yourself with the [basic principles](glossary.md) of embedding retrieval.
+Built on top of popular vector search libraries including Faiss, HNSW, DiskANN, SCANN and more, Milvus was designed for similarity search on dense vector datasets containing millions, billions, or even trillions of vectors. Before proceeding, familiarize yourself with the [basic principles](glossary.md) of embedding retrieval.
-Milvus also supports data sharding, data persistence, streaming data ingestion, hybrid search between vector and scalar data, time travel, and many other advanced functions. The platform offers performance on demand and can be optimized to suit any embedding retrieval scenario. We recommend deploying Milvus using Kubernetes for optimal availability and elasticity.
+Milvus also supports data sharding, streaming data ingestion, dynamic schema, search combine vector and scalar data, multi-vetor and hybrid search, sparse vector and many other advanced functions. The platform offers performance on demand and can be optimized to suit any embedding retrieval scenario. We recommend deploying Milvus using Kubernetes for optimal availability and elasticity.
Milvus adopts a shared-storage architecture featuring storage and computing disaggregation and horizontal scalability for its computing nodes. Following the principle of data plane and control plane disaggregation, Milvus comprises [four layers](four_layers.md): access layer, coordinator service, worker node, and storage. These layers are mutually independent when it comes to scaling or disaster recovery.
-
+
## What's next
diff --git a/preview/site/en/reference/architecture/data_processing.md b/preview/site/en/reference/architecture/data_processing.md
index c8ced6d2d..151ea8ddf 100644
--- a/preview/site/en/reference/architecture/data_processing.md
+++ b/preview/site/en/reference/architecture/data_processing.md
@@ -1,6 +1,7 @@
---
id: data_processing.md
summary: Learn about the data processing procedure in Milvus.
+title: Data Processing
---
# Data Processing
diff --git a/preview/site/en/reference/architecture/four_layers.md b/preview/site/en/reference/architecture/four_layers.md
index a63d88c36..647012db4 100644
--- a/preview/site/en/reference/architecture/four_layers.md
+++ b/preview/site/en/reference/architecture/four_layers.md
@@ -1,6 +1,7 @@
---
id: four_layers.md
summary: Storage/computing disaggregation structure in Milvus.
+title: Storage/Computing Disaggregation
---
# Storage/Computing Disaggregation
@@ -20,7 +21,7 @@ Composed of a group of stateless proxies, the access layer is the front layer of
The coordinator service assigns tasks to the worker nodes and functions as the system's brain. The tasks it takes on include cluster topology management, load balancing, timestamp generation, data declaration, and data management.
-There are four coordinator types: root coordinator (root coord), data coordinator (data coord), query coordinator (query coord), and index coordinator (index coord).
+There are three coordinator types: root coordinator (root coord), data coordinator (data coord), and query coordinator (query coord).
### Root coordinator (root coord)
@@ -32,15 +33,11 @@ Query coord manages topology and load balancing for the query nodes, and handoff
### Data coordinator (data coord)
-Data coord manages topology of the data nodes, maintains metadata, and triggers flush, compact, and other background data operations.
-
-### Index coordinator (index coord)
-
-Index coord manages topology of the index nodes, builds index, and maintains index metadata.
+Data coord manages topology of the data nodes and index nodes, maintains metadata, and triggers flush, compact, and index building and other background data operations.
## Worker nodes
-The arms and legs. Worker nodes are dumb executors that follow instructions from the coordinator service and execute data manipulation language (DML) commands from the proxy. Worker nodes are stateless thanks to separation of storage and computation, and can facilitate system scale-out and disaster recovery when deployed on Kubenetes. There are three types of worker nodes:
+The arms and legs. Worker nodes are dumb executors that follow instructions from the coordinator service and execute data manipulation language (DML) commands from the proxy. Worker nodes are stateless thanks to separation of storage and computation, and can facilitate system scale-out and disaster recovery when deployed on Kubernetes. There are three types of worker nodes:
### Query node
@@ -60,7 +57,7 @@ Storage is the bone of the system, responsible for data persistence. It comprise
### Meta storage
-Meta storage stores snapshots of metadata such as collection schema, node status, and message consumption checkpoints. Storing metadata demands extremely high availability, strong consistency, and transaction support, so Milvus chose etcd for meta store. Milvus also uses etcd for service registration and health check.
+Meta storage stores snapshots of metadata such as collection schema, and message consumption checkpoints. Storing metadata demands extremely high availability, strong consistency, and transaction support, so Milvus chose etcd for meta store. Milvus also uses etcd for service registration and health check.
### Object storage
@@ -68,7 +65,7 @@ Object storage stores snapshot files of logs, index files for scalar and vector
### Log broker
-The log broker is a pub-sub system that supports playback. It is responsible for streaming data persistence, execution of reliable asynchronous queries, event notification, and return of query results. It also ensures integrity of the incremental data when the worker nodes recover from system breakdown. Milvus cluster uses Pulsar as log broker; Milvus standalone uses RocksDB as log broker. Besides, the log broker can be readily replaced with streaming data storage platforms such as Kafka and Pravega.
+The log broker is a pub-sub system that supports playback. It is responsible for streaming data persistence and event notification. It also ensures integrity of the incremental data when the worker nodes recover from system breakdown. Milvus cluster uses Pulsar as log broker; Milvus standalone uses RocksDB as log broker. Besides, the log broker can be readily replaced with streaming data storage platforms such as Kafka.
Milvus is built around log broker and follows the "log as data" principle, so Milvus does not maintain a physical table but guarantees data reliability through logging persistence and snapshot logs.
diff --git a/preview/site/en/reference/architecture/main_components.md b/preview/site/en/reference/architecture/main_components.md
index 035604fb0..402d00562 100644
--- a/preview/site/en/reference/architecture/main_components.md
+++ b/preview/site/en/reference/architecture/main_components.md
@@ -1,6 +1,7 @@
---
id: main_components.md
summary: Learn about the main components in Milvus standalone and cluster.
+title: Main Components
---
# Main Components
@@ -13,15 +14,15 @@ There are two modes for running Milvus: Standalone and Cluster. These two modes
- **Milvus:** The core functional component.
-- **etcd:** The metadata engine, which accesses and stores metadata of Milvus' internal components, including proxies, index nodes, and more.
+- **Meta Store:** The metadata engine, which accesses and stores metadata of Milvus' internal components, including proxies, index nodes, and more.
-- **MinIO:** The storage engine, which is responsible for data persistence for Milvus.
+- **Object Storage:** The storage engine, which is responsible for data persistence for Milvus.
## Milvus cluster
-**Milvus cluster** includes eight microservice components and three third-party dependencies. All microservices can be deployed on Kubernetes, independently from each other.
+**Milvus cluster** includes seven microservice components and three third-party dependencies. All microservices can be deployed on Kubernetes, independently from each other.
### Microservice components
@@ -29,16 +30,15 @@ There are two modes for running Milvus: Standalone and Cluster. These two modes
- Proxy
- Query coord
- Query node
-- Index coord
+- Data coord
- Index node
-- Data coord
- Data node
### Third-party dependencies
-- **etcd:** Stores metadata for various components in the cluster.
-- **MinIO:** Responsible for data persistence of large files in the cluster, such as index and binary log files.
-- **Pulsar:** Manages logs of recent mutation operations, outputs streaming log, and provides log publish-subscribe services.
+- **Meta Store:** Stores metadata for various components in the cluster, e.g. etcd.
+- **Object Storage:** Responsible for data persistence of large files in the cluster, such as index and binary log files, e.g. S3
+- **Log Broker:** Manages logs of recent mutation operations, outputs streaming log, and provides log publish-subscribe services, e.g. Pulsar.
diff --git a/preview/site/en/reference/array_data_type.md b/preview/site/en/reference/array_data_type.md
new file mode 100644
index 000000000..ac0792aef
--- /dev/null
+++ b/preview/site/en/reference/array_data_type.md
@@ -0,0 +1,1297 @@
+---
+id: array_data_type.md
+title: Use Array Fields
+---
+
+# Use Array Fields
+
+This guide explains how to use the array fields, such as inserting array values as well as searching and querying in array fields with basic and advanced operators.
+
+## Overview
+
+Milvus provides support for arrays as one of the field data types. If you want to add an array field to the schema, you have to set the data type for its element and the maximum number of elements that this field can contain. This indicates that an array in a Milvus collection should always have elements of the same data type.
+
+For example, the following code snippet generates a random dataset containing an array field.
+
+
+
+```python
+colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
+data = []
+
+for i in range(1000):
+ current_color = random.choice(colors)
+ current_tag = random.randint(1000, 9999)
+ current_coord = [ random.randint(0, 40) for _ in range(random.randint(3, 5)) ]
+ data.append({
+ "id": i,
+ "vector": [ random.uniform(-1, 1) for _ in range(5) ],
+ "color": current_color,
+ "color_tag": current_tag,
+ "color_coord": current_coord,
+ })
+
+print(data[0])
+```
+
+```java
+List
+
+- Elements in an array field should be of the same data type.
+- The number of elements in an array field should be less than or equal to the specified maximum capacity of the array field.
+
+
+
+## Define array field
+
+To define an array field, simply follow the same procedure as defining fields of other data types.
+
+
+
+For more information on parameters, refer to [`MilvusClient`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md), [`create_schema()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md), [`add_field()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/CollectionSchema/add_field.md), [`add_index()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/add_index.md), [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md), and [`get_load_state()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/get_load_state.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`MilvusClientV2`](https://milvus.io/api-reference/java/v2.4.x/v2/Client/MilvusClientV2.md), [`createSchema()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createSchema.md), [`addField()`](https://milvus.io/api-reference/java/v2.4.x/v2/CollectionSchema/addField.md), [`IndexParam`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/IndexParam.md), [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md), and [`getLoadState()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/getLoadState.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`MilvusClient`](https://milvus.io/api-reference/node/v2.4.x/Client/MilvusClient.md), [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md), and [`getLoadState()`](https://milvus.io/api-reference/node/v2.4.x/Management/getLoadState.md) in the SDK reference.
+
+
+
+
+
+```python
+import random, time
+from pymilvus import MilvusClient, DataType
+
+CLUSTER_ENDPOINT = "http://localhost:19530"
+
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri=CLUSTER_ENDPOINT
+)
+
+# 2. Create a collection
+schema = MilvusClient.create_schema(
+ auto_id=False,
+ enable_dynamic_field=False,
+)
+
+schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
+schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=5)
+schema.add_field(field_name="color", datatype=DataType.VARCHAR, max_length=512)
+schema.add_field(field_name="color_tag", datatype=DataType.INT64)
+# highlight-next-line
+schema.add_field(field_name="color_coord", datatype=DataType.ARRAY, element_type=DataType.INT64, max_capacity=5)
+
+index_params = MilvusClient.prepare_index_params()
+
+index_params.add_index(
+ field_name="id",
+ index_type="STL_SORT"
+)
+
+index_params.add_index(
+ field_name="vector",
+ index_type="IVF_FLAT",
+ metric_type="L2",
+ params={"nlist": 1024}
+)
+
+client.create_collection(
+ collection_name="test_collection",
+ schema=schema,
+ index_params=index_params
+)
+
+res = client.get_load_state(
+ collection_name="test_collection"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "state": "
+
+For more information on parameters, refer to [`insert()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/insert.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`insert()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/insert.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`insert()`](https://milvus.io/api-reference/node/v2.4.x/Vector/insert.md) in the SDK reference.
+
+
+
+
+
+```python
+# 3. Insert data
+res = client.insert(
+ collection_name="test_collection",
+ data=data
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 1000,
+# "ids": [
+# 0,
+# 1,
+# 2,
+# 3,
+# 4,
+# 5,
+# 6,
+# 7,
+# 8,
+# 9,
+# "(990 more items hidden)"
+# ]
+# }
+
+time.sleep(5)
+```
+
+```java
+import com.alibaba.fastjson.JSONObject;
+
+import io.milvus.v2.service.vector.request.InsertReq;
+import io.milvus.v2.service.vector.response.InsertResp;
+
+// 3.1 Insert data into the collection
+InsertReq insertReq = InsertReq.builder()
+ .collectionName("test_collection")
+ .data(data)
+ .build();
+
+InsertResp insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 1000}
+
+Thread.sleep(5000); // wait for a while to ensure the data is indexed
+```
+
+```javascript
+res = await client.insert({
+ collection_name: "test_collection",
+ data: data,
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 1000
+//
+
+await sleep(5000)
+```
+
+## Basic scalar filtering
+
+Once all of your data has been added, you can conduct searches and queries using the elements in the array field in the same manner as you would with a standard scalar field.
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/search.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/search.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/node/v2.4.x/Vector/search.md) in the SDK reference.
+
+
+
+
+
+```python
+# 4. Basic search with the array field
+query_vectors = [ [ random.uniform(-1, 1) for _ in range(5) ]]
+
+res = client.search(
+ collection_name="test_collection",
+ data=query_vectors,
+ filter="color_coord[0] < 10",
+ search_params={
+ "metric_type": "L2",
+ "params": {"nprobe": 16}
+ },
+ output_fields=["id", "color", "color_tag", "color_coord"],
+ limit=3
+)
+
+print(res)
+
+# Output
+#
+# [
+# [
+# {
+# "id": 993,
+# "distance": 0.1538649946451187,
+# "entity": {
+# "color_coord": [
+# 5,
+# 37,
+# 39,
+# 18
+# ],
+# "id": 993,
+# "color": "black",
+# "color_tag": 6785
+# }
+# },
+# {
+# "id": 452,
+# "distance": 0.2353954315185547,
+# "entity": {
+# "color_coord": [
+# 2,
+# 27,
+# 34,
+# 32,
+# 30
+# ],
+# "id": 452,
+# "color": "brown",
+# "color_tag": 2075
+# }
+# },
+# {
+# "id": 862,
+# "distance": 0.27913951873779297,
+# "entity": {
+# "color_coord": [
+# 0,
+# 19,
+# 0,
+# 26
+# ],
+# "id": 862,
+# "color": "brown",
+# "color_tag": 1787
+# }
+# }
+# ]
+# ]
+```
+
+```java
+// 4. Basic search with an Array field
+
+QueryReq queryReq = QueryReq.builder()
+ .collectionName("test_collection")
+ .filter("color_coord[0] in [7, 8, 9]")
+ .outputFields(Arrays.asList("id", "color", "color_tag", "color_coord"))
+ .limit(3L)
+ .build();
+
+QueryResp queryResp = client.query(queryReq);
+
+System.out.println(JSONObject.toJSON(queryResp));
+
+// Output:
+// {"queryResults": [
+// {"entity": {
+// "color": "orange",
+// "color_tag": 2464,
+// "id": 18,
+// "color_coord": [
+// 9,
+// 30
+// ]
+// }},
+// {"entity": {
+// "color": "pink",
+// "color_tag": 2602,
+// "id": 22,
+// "color_coord": [
+// 8,
+// 34,
+// 16
+// ]
+// }},
+// {"entity": {
+// "color": "pink",
+// "color_tag": 1243,
+// "id": 42,
+// "color_coord": [
+// 9,
+// 20
+// ]
+// }}
+// ]}
+```
+
+```javascript
+// 4. Basic search with the array field
+const query_vectors = [Array(5).fill(0).map(() => Math.random())]
+
+res = await client.search({
+ collection_name: "test_collection",
+ data: query_vectors,
+ filter: "color_coord[0] < 10",
+ output_fields: ["id", "color", "color_tag", "color_coord"],
+ limit: 3
+})
+
+console.log(JSON.stringify(res.results, null, 4))
+
+// Output
+//
+// [
+// {
+// "score": 2.015894889831543,
+// "id": "260",
+// "color": "green",
+// "color_tag": "5320",
+// "color_coord": [
+// "1",
+// "7",
+// "33",
+// "13",
+// "23"
+// ]
+// },
+// {
+// "score": 2.006500720977783,
+// "id": "720",
+// "color": "green",
+// "color_tag": "4939",
+// "color_coord": [
+// "0",
+// "19",
+// "5",
+// "30",
+// "15"
+// ]
+// },
+// {
+// "score": 1.9539016485214233,
+// "id": "243",
+// "color": "red",
+// "color_tag": "2403",
+// "color_coord": [
+// "4"
+// ]
+// }
+// ]
+//
+```
+
+## Advanced filtering
+
+As what we have in a JSON field, Milvus also provides advanced filtering operators for arrays, namely `ARRAY_CONTAINS`, `ARRAY_CONTAINS_ALL`, `ARRAY_CONTAINS_ANY`, and `ARRAY_LENGTH`.
+
+- Filters all entities having a `10` in their color coordinators.
+
+
+
+ ```python
+ # 5. Advanced search within the array field
+
+ res = client.query(
+ collection_name="test_collection",
+ filter="ARRAY_CONTAINS(color_coord, 10)",
+ output_fields=["id", "color", "color_tag", "color_coord"],
+ limit=3
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "id": 21,
+ # "color": "white",
+ # "color_tag": 4202,
+ # "color_coord": [
+ # 10,
+ # 5,
+ # 5
+ # ]
+ # },
+ # {
+ # "id": 31,
+ # "color": "grey",
+ # "color_tag": 7386,
+ # "color_coord": [
+ # 8,
+ # 10,
+ # 23,
+ # 7,
+ # 31
+ # ]
+ # },
+ # {
+ # "id": 45,
+ # "color": "purple",
+ # "color_tag": 6126,
+ # "color_coord": [
+ # 0,
+ # 10,
+ # 24
+ # ]
+ # }
+ # ]
+ ```
+
+ ```java
+ // 5. Advanced search within an Array field
+ queryReq = QueryReq.builder()
+ .collectionName("test_collection")
+ .filter("ARRAY_CONTAINS(color_coord, 10)")
+ .outputFields(Arrays.asList("id", "color", "color_tag", "color_coord"))
+ .limit(3)
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [
+ // {"entity": {
+ // "color": "blue",
+ // "color_tag": 4337,
+ // "id": 17,
+ // "color_coord": [
+ // 11,
+ // 33,
+ // 10,
+ // 20
+ // ]
+ // }},
+ // {"entity": {
+ // "color": "white",
+ // "color_tag": 5219,
+ // "id": 25,
+ // "color_coord": [
+ // 10,
+ // 15
+ // ]
+ // }},
+ // {"entity": {
+ // "color": "red",
+ // "color_tag": 7120,
+ // "id": 35,
+ // "color_coord": [
+ // 19,
+ // 10,
+ // 10,
+ // 14
+ // ]
+ // }}
+ // ]}
+ ```
+
+ ```javascript
+ // 5. Advanced search within the array field
+ res = await client.search({
+ collection_name: "test_collection",
+ data: query_vectors,
+ filter: "ARRAY_CONTAINS(color_coord, 10)",
+ output_fields: ["id", "color", "color_tag", "color_coord"],
+ limit: 3
+ })
+
+ console.log(JSON.stringify(res.results, null, 4))
+
+ // Output
+ //
+ // [
+ // {
+ // "score": 1.7962548732757568,
+ // "id": "696",
+ // "color": "red",
+ // "color_tag": "1798",
+ // "color_coord": [
+ // "33",
+ // "10",
+ // "37"
+ // ]
+ // },
+ // {
+ // "score": 1.7126177549362183,
+ // "id": "770",
+ // "color": "red",
+ // "color_tag": "1962",
+ // "color_coord": [
+ // "21",
+ // "23",
+ // "10"
+ // ]
+ // },
+ // {
+ // "score": 1.6707111597061157,
+ // "id": "981",
+ // "color": "yellow",
+ // "color_tag": "3100",
+ // "color_coord": [
+ // "28",
+ // "39",
+ // "10",
+ // "6"
+ // ]
+ // }
+ // ]
+ //
+ ```
+
+- Filters all entities having a `7` and an `8` in their color coordinators.
+
+
+
+ ```python
+ res = client.query(
+ collection_name="test_collection",
+ filter="ARRAY_CONTAINS_ALL(color_coord, [7, 8])",
+ output_fields=["id", "color", "color_tag", "color_coord"],
+ limit=3
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "color": "grey",
+ # "color_tag": 7386,
+ # "color_coord": [
+ # 8,
+ # 10,
+ # 23,
+ # 7,
+ # 31
+ # ],
+ # "id": 31
+ # },
+ # {
+ # "color": "purple",
+ # "color_tag": 7823,
+ # "color_coord": [
+ # 38,
+ # 8,
+ # 36,
+ # 38,
+ # 7
+ # ],
+ # "id": 258
+ # },
+ # {
+ # "color": "purple",
+ # "color_tag": 6356,
+ # "color_coord": [
+ # 34,
+ # 32,
+ # 11,
+ # 8,
+ # 7
+ # ],
+ # "id": 348
+ # }
+ # ]
+ ```
+
+ ```java
+ queryReq = QueryReq.builder()
+ .collectionName("test_collection")
+ .filter("ARRAY_CONTAINS_ALL(color_coord, [7, 8, 9])")
+ .outputFields(Arrays.asList("id", "color", "color_tag", "color_coord"))
+ .limit(3)
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [{"entity": {
+ // "color": "red",
+ // "color_tag": 6986,
+ // "id": 423,
+ // "color_coord": [
+ // 26,
+ // 7,
+ // 8,
+ // 9
+ // ]
+ // }}]}
+ ```
+
+ ```javascript
+ res = await client.search({
+ collection_name: "test_collection",
+ data: query_vectors,
+ filter: "ARRAY_CONTAINS_ALL(color_coord, [7, 8])",
+ output_fields: ["id", "color", "color_tag", "color_coord"],
+ limit: 3
+ })
+
+ console.log(JSON.stringify(res.results, null, 4))
+
+ // Output
+ //
+ // [
+ // {
+ // "score": 0.8267516493797302,
+ // "id": "913",
+ // "color": "brown",
+ // "color_tag": "8897",
+ // "color_coord": [
+ // "39",
+ // "31",
+ // "8",
+ // "29",
+ // "7"
+ // ]
+ // },
+ // {
+ // "score": 0.6889009475708008,
+ // "id": "826",
+ // "color": "blue",
+ // "color_tag": "4903",
+ // "color_coord": [
+ // "7",
+ // "25",
+ // "5",
+ // "12",
+ // "8"
+ // ]
+ // },
+ // {
+ // "score": 0.5851659774780273,
+ // "id": "167",
+ // "color": "blue",
+ // "color_tag": "1550",
+ // "color_coord": [
+ // "8",
+ // "27",
+ // "7"
+ // ]
+ // }
+ // ]
+ //
+ ```
+
+- Filters all entities having either 7, 8, or 9 in their color coordinators.
+
+
+
+ ```python
+ res = client.query(
+ collection_name="test_collection",
+ filter="ARRAY_CONTAINS_ANY(color_coord, [7, 8, 9])",
+ output_fields=["id", "color", "color_tag", "color_coord"],
+ limit=3
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "id": 0,
+ # "color": "green",
+ # "color_tag": 9212,
+ # "color_coord": [
+ # 37,
+ # 36,
+ # 36,
+ # 7,
+ # 9
+ # ]
+ # },
+ # {
+ # "id": 5,
+ # "color": "blue",
+ # "color_tag": 9643,
+ # "color_coord": [
+ # 8,
+ # 20,
+ # 20,
+ # 11
+ # ]
+ # },
+ # {
+ # "id": 12,
+ # "color": "blue",
+ # "color_tag": 3075,
+ # "color_coord": [
+ # 29,
+ # 7,
+ # 17
+ # ]
+ # }
+ # ]
+ ```
+
+ ```java
+ queryReq = QueryReq.builder()
+ .collectionName("test_collection")
+ .filter("ARRAY_CONTAINS_ANY(color_coord, [7, 8, 9])")
+ .outputFields(Arrays.asList("id", "color", "color_tag", "color_coord"))
+ .limit(3)
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [
+ // {"entity": {
+ // "color": "orange",
+ // "color_tag": 2464,
+ // "id": 18,
+ // "color_coord": [
+ // 9,
+ // 30
+ // ]
+ // }},
+ // {"entity": {
+ // "color": "pink",
+ // "color_tag": 2602,
+ // "id": 22,
+ // "color_coord": [
+ // 8,
+ // 34,
+ // 16
+ // ]
+ // }},
+ // {"entity": {
+ // "color": "pink",
+ // "color_tag": 1243,
+ // "id": 42,
+ // "color_coord": [
+ // 9,
+ // 20
+ // ]
+ // }}
+ // ]}
+ ```
+
+ ```javascript
+ res = await client.search({
+ collection_name: "test_collection",
+ data: query_vectors,
+ filter: "ARRAY_CONTAINS_ANY(color_coord, [7, 8, 9])",
+ output_fields: ["id", "color", "color_tag", "color_coord"],
+ limit: 3
+ })
+
+ console.log(JSON.stringify(res.results, null, 4))
+
+ // Output
+ //
+ // [
+ // {
+ // "score": 2.015894889831543,
+ // "id": "260",
+ // "color": "green",
+ // "color_tag": "5320",
+ // "color_coord": [
+ // "1",
+ // "7",
+ // "33",
+ // "13",
+ // "23"
+ // ]
+ // },
+ // {
+ // "score": 1.783075213432312,
+ // "id": "593",
+ // "color": "orange",
+ // "color_tag": "4079",
+ // "color_coord": [
+ // "8",
+ // "19"
+ // ]
+ // },
+ // {
+ // "score": 1.7713876962661743,
+ // "id": "874",
+ // "color": "blue",
+ // "color_tag": "7029",
+ // "color_coord": [
+ // "14",
+ // "8",
+ // "15"
+ // ]
+ // }
+ // ]
+ //
+ ```
+
+- Filters entities that have exactly four elements.
+
+
+
+ ```python
+ res = client.query(
+ collection_name="test_collection",
+ filter="ARRAY_LENGTH(color_coord) == 4",
+ output_fields=["id", "color", "color_tag", "color_coord"],
+ limit=3
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "id": 1,
+ # "color": "pink",
+ # "color_tag": 6708,
+ # "color_coord": [
+ # 15,
+ # 36,
+ # 38,
+ # 2
+ # ]
+ # },
+ # {
+ # "id": 4,
+ # "color": "green",
+ # "color_tag": 5386,
+ # "color_coord": [
+ # 13,
+ # 32,
+ # 35,
+ # 5
+ # ]
+ # },
+ # {
+ # "id": 5,
+ # "color": "blue",
+ # "color_tag": 9643,
+ # "color_coord": [
+ # 8,
+ # 20,
+ # 20,
+ # 11
+ # ]
+ # }
+ # ]
+ ```
+
+ ```java
+ queryReq = QueryReq.builder()
+ .collectionName("test_collection")
+ .filter("ARRAY_LENGTH(color_coord) == 4")
+ .outputFields(Arrays.asList("id", "color", "color_tag", "color_coord"))
+ .limit(3)
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [
+ // {"entity": {
+ // "color": "green",
+ // "color_tag": 2984,
+ // "id": 2,
+ // "color_coord": [
+ // 27,
+ // 31,
+ // 23,
+ // 29
+ // ]
+ // }},
+ // {"entity": {
+ // "color": "black",
+ // "color_tag": 6867,
+ // "id": 4,
+ // "color_coord": [
+ // 37,
+ // 3,
+ // 30,
+ // 33
+ // ]
+ // }},
+ // {"entity": {
+ // "color": "brown",
+ // "color_tag": 3464,
+ // "id": 10,
+ // "color_coord": [
+ // 31,
+ // 38,
+ // 21,
+ // 28
+ // ]
+ // }}
+ // ]}
+ ```
+
+ ```javascript
+ res = await client.search({
+ collection_name: "test_collection",
+ data: query_vectors,
+ filter: "ARRAY_LENGTH(color_coord) == 4",
+ output_fields: ["id", "color", "color_tag", "color_coord"],
+ limit: 3
+ })
+
+ console.log(JSON.stringify(res.results, null, 4))
+
+ // Output
+ //
+ // [
+ // {
+ // "score": 2.0404388904571533,
+ // "id": "439",
+ // "color": "orange",
+ // "color_tag": "7096",
+ // "color_coord": [
+ // "27",
+ // "34",
+ // "26",
+ // "39"
+ // ]
+ // },
+ // {
+ // "score": 1.9059759378433228,
+ // "id": "918",
+ // "color": "purple",
+ // "color_tag": "2903",
+ // "color_coord": [
+ // "28",
+ // "19",
+ // "36",
+ // "35"
+ // ]
+ // },
+ // {
+ // "score": 1.8385567665100098,
+ // "id": "92",
+ // "color": "yellow",
+ // "color_tag": "4693",
+ // "color_coord": [
+ // "1",
+ // "23",
+ // "2",
+ // "3"
+ // ]
+ // }
+ // ]
+ //
+ ```
\ No newline at end of file
diff --git a/preview/site/en/reference/benchmark.md b/preview/site/en/reference/benchmark.md
index a331a70e9..e381ae55c 100644
--- a/preview/site/en/reference/benchmark.md
+++ b/preview/site/en/reference/benchmark.md
@@ -1,16 +1,18 @@
---
id: benchmark.md
-summary: Learn about the benchmark result of Milvus.
+summary: Learn about the benchmark result of Milvus.
+title: Milvus 2.2 Benchmark Test Report
---
# Milvus 2.2 Benchmark Test Report
-This report shows the major test results of Milvus 2.2.0. It aims to provide a picture of Milvus 2.2.0 search performance, especially in the capability to scale up and scale out.
+This report shows the major test results of Milvus 2.2.0. It aims to provide a picture of Milvus 2.2.0 search performance, especially in the capability to scale up and scale out.
- 
+
+
+
+ We have recently run a benchmark against Milvus 2.2.3 and have the following key findings:
@@ -20,13 +22,13 @@ This report shows the major test results of Milvus 2.2.0. It aims to provide a pFor details, welcome referring to this whitepaper and related benchmark test code.
+For details, welcome referring to this whitepaper and related benchmark test code.
-For example, 10 / 2 * 5 will be evaluated as (10 / 2) and the result multiplied by 5.
+
+For example, 10 / 2 * 5 will be evaluated as (10 / 2) and the result multiplied by 5.
+
When a lower precedence operation should be processed first, it should be enclosed within parentheses.
-For example, 30 / 2 + 8. This is normally evaluated as 30 divided by 2 then 8 added to the result. If you want to divide by 2 + 8, it should be written as 30 / (2 + 8).
-
+For example, 30 / 2 + 8. This is normally evaluated as 30 divided by 2 then 8 added to the result. If you want to divide by 2 + 8, it should be written as 30 / (2 + 8).
-Parentheses can be nested within expressions. Innermost parenthetical expressions are evaluated first.
+
+
Milvus only supports deleting entities with clearly specified primary keys, which can be achieved merely with the term expression
```
@@ -212,17 +231,98 @@ VARCHAR not in ["str1", "str2"]
"200+300 < int64 <= 500+500"
```
-7. MatchOp (prefix matching)
+7. MatchOp
```
VARCHAR like "prefix%"
+VARCHAR like "%suffix"
+VARCHAR like "%middle%"
+VARCHAR like "_suffix"
```
-## What's next
+8. JsonArrayOp
-Now that you know how bitsets work in Milvus, you might also want to:
+- `JSON_CONTAINS(identifier, JsonExpr)`
+
+ If the JSON expression of a `JSON_CONTAINS` (the second argument) statement is a list, the identifier (the first argument) should be list of list. Otherwise, the statement always evaluates to False.
+
+ ```python
+ # {"x": [1,2,3]}
+ json_contains(x, 1) # ==> true
+ json_contains(x, "a") # ==> false
+
+ # {"x": [[1,2,3], [4,5,6], [7,8,9]]}
+ json_contains(x, [1,2,3]) # ==> true
+ json_contains(x, [3,2,1]) # ==> false
+ ```
-- Learn how to conduct a [Hybrid Search](hybridsearch.md).
-- Learn how to [use strings to filter](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md) your search results.
+- `JSON_CONTAINS_ALL(identifier, JsonExpr)`
+ The JSON expression in a `JSON_CONTAINS_ALL` statement should always be a list.
+
+ ```python
+ # {"x": [1,2,3,4,5,7,8]}
+ json_contains_all(x, [1,2,8]) # ==> true
+ json_contains_all(x, [4,5,6]) # ==> false 6 is not exists
+ ```
+
+- `JSON_CONTAINS_ANY(identifier, JsonExpr)`
+
+ The JSON expression in a `JSON_CONTAINS_ANY` statement should always be a list. Otherwise, it acts the same as `JSON_CONTAINS`.
+
+ ```python
+ # {"x": [1,2,3,4,5,7,8]}
+ json_contains_any(x, [1,2,8]) # ==> true
+ json_contains_any(x, [4,5,6]) # ==> true
+ json_contains_any(x, [6,9]) # ==> false
+ ```
+
+9. ArrayOp
+
+- `ARRAY_CONTAINS(identifier, ArrayExpr)`
+
+ If the array expression of an `ARRAY_CONTAINS` (the second argument) statement is a list, the identifier (the first argument) should be list of list. Otherwise, the statement always evaluates to False.
+
+ ```python
+ # 'int_array': [1,2,3]
+ array_contains(int_array, 1) # ==> true
+ array_contains(int_array, "a") # ==> false
+ ```
+
+- `ARRAY_CONTAINS_ALL(identifier, ArrayExpr)`
+
+ The array expression in an `ARRAY_CONTAINS_ALL` statement should always be a list.
+
+ ```python
+ # "int_array": [1,2,3,4,5,7,8]
+ array_contains_all(int_array, [1,2,8]) # ==> true
+ array_contains_all(int_array, [4,5,6]) # ==> false 6 is not exists
+ ```
+
+- `ARRAY_CONTAINS_ANY(identifier, ArrayExpr)`
+
+ The array expression in an `ARRAY_CONTAINS_ANY` statement should always be a list. Otherwise, it acts the same as `ARRAY_CONTAINS`.
+
+ ```python
+ # "int_array": [1,2,3,4,5,7,8]
+ array_contains_any(int_array, [1,2,8]) # ==> true
+ array_contains_any(int_array, [4,5,6]) # ==> true
+ array_contains_any(int_array, [6,9]) # ==> false
+ ```
+
+- `ARRAY_LENGTH(identifier)`
+
+ Check the number of elements in an array.
+
+ ```python
+ # "int_array": [1,2,3,4,5,7,8]
+ array_length(int_array) # ==> 7
+ ```
+
+## What's next
+
+Now that you know how bitsets work in Milvus, you might also want to:
+- Learn how to conduct a [Hybrid Search](multi-vector-search.md).
+- Learn how to [use strings to filter](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md) your search results.
+- Learn how to [use dynamic fields in building boolean expressions](enable-dynamic-field.md).
diff --git a/preview/site/en/reference/consistency.md b/preview/site/en/reference/consistency.md
index ce981e249..cdb2240e9 100644
--- a/preview/site/en/reference/consistency.md
+++ b/preview/site/en/reference/consistency.md
@@ -1,6 +1,7 @@
---
id: consistency.md
summary: Learn about the four levels of consistency in Milvus.
+title: Consistency
---
# Consistency
@@ -10,7 +11,7 @@ This topic introduces the four levels of consistency in Milvus and their best-su
Consistency in a distributed database specifically refers to the property that ensures every node or replica has the same view of data when writing or reading data at a given time.
-Milvus supports four consistency levels: strong, bounded staleness, session, and eventually. The default consistency level in Milvus is bounded staleness. You can easily tune the consistency level when conducting a [vector similarity search](search.md) or [query](query.md) to make it best suit your application.
+Milvus supports four consistency levels: strong, bounded staleness, session, and eventually. The default consistency level in Milvus is bounded staleness. You can easily tune the consistency level when conducting a [single-vector search](single-vector-search.md), [hybrid search](multi-vector-search.md) or [query](get-and-scalar-query.md) to make it best suit your application.
## Consistency levels
@@ -64,11 +65,12 @@ A GuaranteeTs serves to inform query nodes that a search or query request will n
- **Eventually**: GuaranteeTs is set to a very small value to skip the consistency check. Query nodes search immediately on the existing data view.
-See [How GuaranteeTs Works](https://github.com/milvus-io/milvus/blob/f3f46d3bb2dcae2de0bdb7bc0f7b20a72efceaab/docs/developer_guides/how-guarantee-ts-works.md) and [All The Things You Should Know About Time Synchronization](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/milvus_timesync_en.md?from=from_parent_mindnote) for more information about the mechanism behind ensuring different levels of consistency in Milvus.
+See [How GuaranteeTs Works](https://github.com/milvus-io/milvus/blob/f3f46d3bb2dcae2de0bdb7bc0f7b20a72efceaab/docs/developer_guides/how-guarantee-ts-works.md) for more information about the mechanism behind ensuring different levels of consistency in Milvus.
## What's next
- Learn how to tune consistency level when:
- - [conducting a vector similarity search](search.md)
- - [conducting a vector query](query.md)
+ - [conducting a single-vector search](single-vector-search.md)
+ - [conducting a hybrid search](multi-vector-search.md)
+ - [conducting a scalar query](get-and-scalar-query.md)
diff --git a/preview/site/en/reference/coordinator_ha.md b/preview/site/en/reference/coordinator_ha.md
index 92f80ce68..413eb13dd 100644
--- a/preview/site/en/reference/coordinator_ha.md
+++ b/preview/site/en/reference/coordinator_ha.md
@@ -1,11 +1,12 @@
---
id: coordinator_ha.md
summary: Learn about the motivation and procedure for Milvus coordinators to work in active standby.
+title: Coordinator HA
---
# Coordinator HA
-As shown in the [Mivlus architecture](architecture_overview), Milvus consists of many components and has them work in a distributed manner. Among all the components, Milvus ensures the high availability of the workers through [scaling up and scaling out](scaleout) of the nodes, making coordinators the only weak link in the chain.
+As shown in the [Milvus architecture](architecture_overview.md), Milvus consists of many components and has them work in a distributed manner. Among all the components, Milvus ensures the high availability of the workers through [scaling up and scaling out](scaleout.md) of the nodes, making coordinators the only weak link in the chain.
## Overview
@@ -15,9 +16,79 @@ In the 2.2.3 release, Milvus implements high availability for coordinators to ma
The figure above illustrates how coordinators work in the active-standby mode. When a pair of coordinators start, they register with etcd using their server ID and compete for the active role. The coordinator who succeeds in leasing the active role from the etcd will start serving, and the other coordinator in the pair will remain on standby, watching the active role and ready to serve if the active coordinator dies.
-## Observe coordinators' behaviors
-
-To observe coordinators' behaviors, you can
+## Enable coordinator HA
+
+### With Helm
+
+To start multiple coordinators and have them work in active-standby mode, you should make the following changes to your `values.yaml` file.
+
+- Set `xxxCoordinator.replicas` to `2`.
+- Set `xxxCoordinator.activeStandby.enabled` to `true`.
+
+The following code snippet uses RootCoord as an example. You can do the same to coordinators of other types.
+
+```yaml
+rootCoordinator:
+ enabled: true
+ # You can set the number of replicas greater than 1 only if you also need to set activeStandby.enabled to true.
+ replicas: 2 # Otherwise, remove this configuration item.
+ resources: {}
+ nodeSelector: {}
+ affinity: {}
+ tolerations: []
+ extraEnv: []
+ heaptrack:
+ enabled: false
+ profiling:
+ enabled: false # Enable live profiling
+ activeStandby:
+ enabled: true # Set this to true to have RootCoordinators work in active-standby mode.
+```
+
+### With Docker
+
+To start multiple coordinators and have them work in active-standby mode, you can add some definitions to the `docker-compose` file that you use to start your Milvus cluster.
+
+The following code snippet uses RootCoord as an example. You can do the same to coordinators of other types.
+
+```yaml
+ rootcoord:
+ container_name: milvus-rootcoord
+ image: milvusdb/milvus:v2.2.3
+ command: ["milvus", "run", "rootcoord"]
+ environment:
+ ETCD_ENDPOINTS: etcd:2379
+ MINIO_ADDRESS: minio:9000
+ PULSAR_ADDRESS: pulsar://pulsar:6650
+ ROOT_COORD_ADDRESS: rootcoord:53100
+ # add ROOT_COORD_ENABLE_ACTIVE_STANDBY to enable active standby
+ ROOT_COORD_ENABLE_ACTIVE_STANDBY: true
+ depends_on:
+ - "etcd"
+ - "pulsar"
+ - "minio"
+
+# add the following to have RootCoords work in active-standby mode
+# rootcoord-1:
+# container_name: milvus-rootcoord-1
+# image: milvusdb/milvus:v2.2.3
+# command: ["milvus", "run", "rootcoord"]
+# environment:
+# ETCD_ENDPOINTS: etcd:2379
+# MINIO_ADDRESS: minio:9000
+# PULSAR_ADDRESS: pulsar://pulsar:6650
+# ROOT_COORD_ADDRESS: rootcoord-1:53100
+# # add ROOT_COORD_ENABLE_ACTIVE_STANDBY to enable active standby
+# ROOT_COORD_ENABLE_ACTIVE_STANDBY: true
+# depends_on:
+# - "etcd"
+# - "pulsar"
+# - "minio"
+```
+
+### With Mac/Linux shell
+
+To start multiple coordinators and have them work in active-standby mode, you can
1. Download the Milvus source code to your local drive, and [start up a Milvus cluster from the source code](https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md) as follows:
@@ -84,10 +155,9 @@ To observe coordinators' behaviors, you can
Coordinator HA is disabled by default. You can enable this feature manually by changing the following items in your Milvus configuration file.
-- [rootCoord.activeStandby.enabled](configure_rootcoord#rootCoordactiveStandbyenabled)
-- [queryCoord.activeStandby.enabled](configure_querycoord#queryCoordactiveStandbyenabled)
-- [dataCoord.activeStandby.enabled](configure_datacoord#dataCoordactiveStandbyenabled)
-- [indexCoord.activeStandby.enabled](configure_indexcoord#indexCoordativeStandbyenabled)
+- [rootCoord.activeStandby.enabled](configure_rootcoord.md#rootCoordactiveStandbyenabled)
+- [queryCoord.activeStandby.enabled](configure_querycoord.md#queryCoordactiveStandbyenabled)
+- [dataCoord.activeStandby.enabled](configure_datacoord.md#dataCoordactiveStandbyenabled)
## Limits
diff --git a/preview/site/en/reference/disk_index.md b/preview/site/en/reference/disk_index.md
index d82f834af..841e61f05 100644
--- a/preview/site/en/reference/disk_index.md
+++ b/preview/site/en/reference/disk_index.md
@@ -2,13 +2,14 @@
id: disk_index.md
related_key: disk_index
summary: Disk index mechanism in Milvus.
+title: On-disk Index
---
# On-disk Index
This article introduces an on-disk indexing algorithm named DiskANN. Based on Vamana graphs, DiskANN powers efficient searches within large datasets.
-To improve query performance, you can [specify an index type](build_index.md) for each vector field.
+To improve query performance, you can [specify an index type](index-vector-fields.md) for each vector field.
in.
+
Currently, a vector field only supports one index type. Milvus automatically deletes the old index when switching the index type.
@@ -17,14 +18,18 @@ Currently, a vector field only supports one index type. Milvus automatically del
## Prerequisites
To use DiskANN, note that
-- You have run `make disk_index=ON` when you compile Milvus from the source.
-- Your Milvus instance runs on Ubuntu 18.04.6 or a later release.
-- The path **${MILVUS_ROOT_PATH}/milvus/data** has been mounted to an NVMe SSD for full performance.
+- DiskANN is enabled by default. If you prefer in-memory index over on-disk index, you are advised to disable this feature for a better performance.
+ - To disable it, you can change `queryNode.enableDisk` to `false` in your milvus configuration file.
+ - To enable it again, you can set `queryNode.enableDisk` to `true`.
+- The Milvus instance runs on Ubuntu 18.04.6 or a later release.
+- The Milvus data path should be mounted to an NVMe SSD for full performance:
+ - For a Milvus Standalone instance, the data path should be **/var/lib/milvus/data** in the container where the instance runs.
+ - For a Milvus Cluster instance, the data path should be **/var/lib/milvus/data** in the containers where the QueryNodes and IndexNodes run.
## Limits
To use DiskANN, ensure that you
-- Use only float vectors with at least 32 dimensions in your data.
+- Use only float vectors with at least 1 dimensions in your data.
- Use only Euclidean Distance (L2) or Inner Product (IP) to measure the distance between vectors.
## Index and search settings
@@ -37,8 +42,7 @@ To use DiskANN, ensure that you
| Parameter | Description | Range |
| ------------- | ----------------------------------- | ----------------------------------------------- |
- | `k` | Number of closest vectors to return | [1, 12768] |
- | `search_list` | Size of the candidate list, a larger size offers a higher recall rate with degraded performance. | [k, min( 10 * k, 65535)] for k > 20
[k, 200] for k <= 20 | + | `search_list` | Size of the candidate list, a larger size offers a higher recall rate with degraded performance. | [topk, int32_max] | ## DiskANN-related Milvus configurations @@ -50,9 +54,7 @@ DiskIndex: MaxDegree: 56 SearchListSize: 100 PQCodeBugetGBRatio: 0.125 - BuildNumThreadsRatio: 1.0 SearchCacheBudgetGBRatio: 0.125 - LoadNumThreadRatio: 8.0 BeamWidthRatio: 4.0 ... ``` @@ -60,36 +62,17 @@ DiskIndex: | Parameter | Description | Value Range | Default Value | | --- | --- | --- | --- | | `MaxDegree` | Maximum degree of the Vamana graph.
A larger value offers a higher recall rate but increases the size of and time to build the index. | [1, 512] | 56 | -| `SearchListSize` | Size of the candidate list.
A larger value increases the time spent on building the index but offers a higher recall rate.
Set it to a value smaller than `MaxDegree` unless you need to reduce the index-building time. | [1, ∞] | 100 | +| `SearchListSize` | Size of the candidate list.
A larger value increases the time spent on building the index but offers a higher recall rate.
Set it to a value smaller than `MaxDegree` unless you need to reduce the index-building time. | [1, int32_max] | 100 | | `PQCodeBugetGBRatio` | Size limit on the PQ code.
A larger value offers a higher recall rate but increases memory usage. | (0.0, 0.25] | 0.125 | -| `BuildNumThreadsRatio` | Ratio between the number of threads used to build the index and the number of CPUs. | [1.0, 128.0 / CPU number] | 1.0 | | `SearchCacheBudgetGBRatio` | Ratio of cached node numbers to raw data.
A larger value improves index-building performance with increased memory usage. | [0.0, 0.3) | 0.10 | -| `LoadNumThreadRatio` | Ratio between the number of threads used to load index/search and the number of CPUs. For details, refer to the first item in [References and Facts](disk_index.md#references-and-facts). | [1, 65536 / 32 / CPU number] | 8.0 | | `BeamWidthRatio` | Ratio between the maximum number of IO requests per search iteration and CPU number. | [1, max(128 / CPU number, 16)] | 4.0 | +## Troubleshooting -## References and Facts +- How to deal with the `io_setup() failed; returned -11, errno=11:Resource temporarily unavailable` error? -- On the maximum number of DiskANN threads + The Linux kernel provides the Asynchronous non-blocking I/O (AIO) feature that allows a process to initiate multiple I/O operations simultaneously without having to wait for any of them to complete. This helps boost performance for applications that can overlap processing and I/O. - DiskANN uses Linux AIO to read and write the disk. The maximum number of concurrent I/O requests for each DiskANN thread is 32, and that limit on Linux is 65536. Therefore, the maximum number of DiskANN threads should be no greater than 65532 / 32 ≈ 2047. + The performance can be tuned using the `/proc/sys/fs/aio-max-nr` virtual file in the proc file system. The `aio-max-nr` parameter determines the maximum number of allowable concurrent requests. - Note that other processes may also involve concurrent I/O operations on Linux, which is usually less than 10000. Therefore, when setting `LoadNumThreadRatio`, use a value smaller than the maximum number of DiskANN threads to calculate its upper bound. - - For your reference, - - - To check the limit on the Linux system, run - - ```Shell - sudo sysctl -a | grep fs.aio-max-nr - ``` - - - To raise the limit, run - - ```Shell - sudo sysctl -w fs.aio-max-nr=X - ``` - -- On the time spent on building and loading the DiskANN index - - Building the DiskANN index is time-consuming. For a SIFT dataset comprising 25,000,000 entries of 128-dimensional vectors, the size of the dataset is about 12.5 GB. The time spent on building an index for the dataset is about 40 minutes and that spent on loading the built index is about 13.2 minutes. + The `aio-max-nr` defaults to `65535`, you can set it up to `10485760`. diff --git a/preview/site/en/reference/glossary.md b/preview/site/en/reference/glossary.md index fea0f4f88..7484ea857 100644 --- a/preview/site/en/reference/glossary.md +++ b/preview/site/en/reference/glossary.md @@ -1,98 +1,208 @@ --- id: glossary.md -title: Glossary +title: Terminology --- -# Glossary +# Terminology -This topic explains some of the core concepts in the Milvus vector database. +## AutoID -## Bitset +AutoID is an attribute of the primary field that determines whether to enable AutoIncrement for the primary field. The value of AutoID is defined based on a timestamp. For more information, refer to [create_schema](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md). -In Milvus, bitsets are arrays of bit numbers `0` and `1` that can be used to represent certain data compactly and efficiently as opposed to in ints, floats, or chars. A bit number is `0` by default and is only set to `1` if it meets certain requirements. +## Auto Index + +Milvus automatically decides the most appropriate index type and params for a specific field based on empirical data. This is ideal for situations when you do not need to control the specific index params. For more information, refer to [add_index](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/add_index.md). + +## Attu + +[Attu](https://github.com/zilliztech/attu) is an all-in-one administration tool for Milvus that significantly reduces the complexity and cost of managing the system. + +## Birdwatcher + +[Birdwatcher](birdwatcher_overview.md) is a debugging tool for Milvus that connects to etcd, allowing you to monitor the status of the Milvus server and make adjustments in real-time. It also supports etcd file backups, aiding developers in troubleshooting. + +## Bulk Writer + +[Bulk Writer](https://milvus.io/api-reference/pymilvus/v2.4.x/DataImport/LocalBulkWriter/LocalBulkWriter.md) is a data processing tool provided by Milvus SDKs (e.g. PyMilvus, Java SDK) , designed to convert raw datasets into a format compatible with Milvus for efficient importing. + +## Bulk Insert + +[Bulk Insert](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md) is an API that enhances writing performance by allowing multiple files to be imported in a single request, optimizing operations with large datasets. + +## Cardinal + +Cardinal, developed by Zilliz Cloud, is a cutter-edge vector search algorithm that delivers unparalleled search quality and performance. With its innovative design and extensive optimizations, Cardinal outperforms Knowhere by several times to an order of magnitude while adaptively handling diverse production scenarios, such as varying K sizes, high filtering, different data distributions, and so on. ## Channel -There are two different channels in Milvus. They are [PChannel](#PChannel) and [VChannel](#VChannel). Each PChannel corresponds to a topic for log storage. While each VChannel corresponds a shard in a collection. +Milvus utilizes two types of channels, [PChannel](https://milvus.io/docs/glossary.md#PChannel) and [VChannel](https://milvus.io/docs/glossary.md#VChannel). Each PChannel corresponds to a topic for log storage, while each VChannel corresponds to a shard in a collection. ## Collection -A collection in Milvus is equivalent to a table in a relational database management system (RDBMS). In Milvus, collections are used to store and manage entities. + +In Milvus, a collection is equivalent to a table in a relational database management system (RDBMS). Collections are major logical objects used to store and manage entities. For more information, refer to [Manage Collections](manage-collections.md). ## Dependency -A dependency is a program that another program relies on to work. Milvus' dependencies include etcd (stores meta data), MinIO or S3 (object storage), and Pulsar (manages snapshot logs). + +A dependency is a program that another program relies on to work. Milvus' dependencies include etcd (stores meta data), MinIO or S3 (object storage), and Pulsar (manages snapshot logs). For more information, refer to [Manage Dependencies](https://milvus.io/docs/manage_dependencies.md#Manage-Dependencies). + +## Dynamic schema + +Dynamic schema allows you to insert entities with new fields into a collection without modifying the existing schema. This means that you can insert data without knowing the full schema of a collection and can include fields that are not yet defined. You can enable this schema-free capability by enableing the dynamic field when creating a collection. For more information, refer to [Enable Dynamic Field](enable-dynamic-field.md). + +## Embeddings + +Milvus offers built-in embedding functions that work with popular embedding providers. Before creating a collection in Milvus, you can use these functions to generate embeddings for your datasets, streamlining the process of preparing data and vector searches. To create embeddings in action, refer to [Using PyMilvus's Model To Generate Text Embeddings](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/model/embedding_functions.ipynb). ## Entity -An entity consists of a group of fields that represent real world objects. Each entity in Milvus is represented by a unique primary key. -
[k, 200] for k <= 20 | + | `search_list` | Size of the candidate list, a larger size offers a higher recall rate with degraded performance. | [topk, int32_max] | ## DiskANN-related Milvus configurations @@ -50,9 +54,7 @@ DiskIndex: MaxDegree: 56 SearchListSize: 100 PQCodeBugetGBRatio: 0.125 - BuildNumThreadsRatio: 1.0 SearchCacheBudgetGBRatio: 0.125 - LoadNumThreadRatio: 8.0 BeamWidthRatio: 4.0 ... ``` @@ -60,36 +62,17 @@ DiskIndex: | Parameter | Description | Value Range | Default Value | | --- | --- | --- | --- | | `MaxDegree` | Maximum degree of the Vamana graph.
A larger value offers a higher recall rate but increases the size of and time to build the index. | [1, 512] | 56 | -| `SearchListSize` | Size of the candidate list.
A larger value increases the time spent on building the index but offers a higher recall rate.
Set it to a value smaller than `MaxDegree` unless you need to reduce the index-building time. | [1, ∞] | 100 | +| `SearchListSize` | Size of the candidate list.
A larger value increases the time spent on building the index but offers a higher recall rate.
Set it to a value smaller than `MaxDegree` unless you need to reduce the index-building time. | [1, int32_max] | 100 | | `PQCodeBugetGBRatio` | Size limit on the PQ code.
A larger value offers a higher recall rate but increases memory usage. | (0.0, 0.25] | 0.125 | -| `BuildNumThreadsRatio` | Ratio between the number of threads used to build the index and the number of CPUs. | [1.0, 128.0 / CPU number] | 1.0 | | `SearchCacheBudgetGBRatio` | Ratio of cached node numbers to raw data.
A larger value improves index-building performance with increased memory usage. | [0.0, 0.3) | 0.10 | -| `LoadNumThreadRatio` | Ratio between the number of threads used to load index/search and the number of CPUs. For details, refer to the first item in [References and Facts](disk_index.md#references-and-facts). | [1, 65536 / 32 / CPU number] | 8.0 | | `BeamWidthRatio` | Ratio between the maximum number of IO requests per search iteration and CPU number. | [1, max(128 / CPU number, 16)] | 4.0 | +## Troubleshooting -## References and Facts +- How to deal with the `io_setup() failed; returned -11, errno=11:Resource temporarily unavailable` error? -- On the maximum number of DiskANN threads + The Linux kernel provides the Asynchronous non-blocking I/O (AIO) feature that allows a process to initiate multiple I/O operations simultaneously without having to wait for any of them to complete. This helps boost performance for applications that can overlap processing and I/O. - DiskANN uses Linux AIO to read and write the disk. The maximum number of concurrent I/O requests for each DiskANN thread is 32, and that limit on Linux is 65536. Therefore, the maximum number of DiskANN threads should be no greater than 65532 / 32 ≈ 2047. + The performance can be tuned using the `/proc/sys/fs/aio-max-nr` virtual file in the proc file system. The `aio-max-nr` parameter determines the maximum number of allowable concurrent requests. - Note that other processes may also involve concurrent I/O operations on Linux, which is usually less than 10000. Therefore, when setting `LoadNumThreadRatio`, use a value smaller than the maximum number of DiskANN threads to calculate its upper bound. - - For your reference, - - - To check the limit on the Linux system, run - - ```Shell - sudo sysctl -a | grep fs.aio-max-nr - ``` - - - To raise the limit, run - - ```Shell - sudo sysctl -w fs.aio-max-nr=X - ``` - -- On the time spent on building and loading the DiskANN index - - Building the DiskANN index is time-consuming. For a SIFT dataset comprising 25,000,000 entries of 128-dimensional vectors, the size of the dataset is about 12.5 GB. The time spent on building an index for the dataset is about 40 minutes and that spent on loading the built index is about 13.2 minutes. + The `aio-max-nr` defaults to `65535`, you can set it up to `10485760`. diff --git a/preview/site/en/reference/glossary.md b/preview/site/en/reference/glossary.md index fea0f4f88..7484ea857 100644 --- a/preview/site/en/reference/glossary.md +++ b/preview/site/en/reference/glossary.md @@ -1,98 +1,208 @@ --- id: glossary.md -title: Glossary +title: Terminology --- -# Glossary +# Terminology -This topic explains some of the core concepts in the Milvus vector database. +## AutoID -## Bitset +AutoID is an attribute of the primary field that determines whether to enable AutoIncrement for the primary field. The value of AutoID is defined based on a timestamp. For more information, refer to [create_schema](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md). -In Milvus, bitsets are arrays of bit numbers `0` and `1` that can be used to represent certain data compactly and efficiently as opposed to in ints, floats, or chars. A bit number is `0` by default and is only set to `1` if it meets certain requirements. +## Auto Index + +Milvus automatically decides the most appropriate index type and params for a specific field based on empirical data. This is ideal for situations when you do not need to control the specific index params. For more information, refer to [add_index](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/add_index.md). + +## Attu + +[Attu](https://github.com/zilliztech/attu) is an all-in-one administration tool for Milvus that significantly reduces the complexity and cost of managing the system. + +## Birdwatcher + +[Birdwatcher](birdwatcher_overview.md) is a debugging tool for Milvus that connects to etcd, allowing you to monitor the status of the Milvus server and make adjustments in real-time. It also supports etcd file backups, aiding developers in troubleshooting. + +## Bulk Writer + +[Bulk Writer](https://milvus.io/api-reference/pymilvus/v2.4.x/DataImport/LocalBulkWriter/LocalBulkWriter.md) is a data processing tool provided by Milvus SDKs (e.g. PyMilvus, Java SDK) , designed to convert raw datasets into a format compatible with Milvus for efficient importing. + +## Bulk Insert + +[Bulk Insert](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md) is an API that enhances writing performance by allowing multiple files to be imported in a single request, optimizing operations with large datasets. + +## Cardinal + +Cardinal, developed by Zilliz Cloud, is a cutter-edge vector search algorithm that delivers unparalleled search quality and performance. With its innovative design and extensive optimizations, Cardinal outperforms Knowhere by several times to an order of magnitude while adaptively handling diverse production scenarios, such as varying K sizes, high filtering, different data distributions, and so on. ## Channel -There are two different channels in Milvus. They are [PChannel](#PChannel) and [VChannel](#VChannel). Each PChannel corresponds to a topic for log storage. While each VChannel corresponds a shard in a collection. +Milvus utilizes two types of channels, [PChannel](https://milvus.io/docs/glossary.md#PChannel) and [VChannel](https://milvus.io/docs/glossary.md#VChannel). Each PChannel corresponds to a topic for log storage, while each VChannel corresponds to a shard in a collection. ## Collection -A collection in Milvus is equivalent to a table in a relational database management system (RDBMS). In Milvus, collections are used to store and manage entities. + +In Milvus, a collection is equivalent to a table in a relational database management system (RDBMS). Collections are major logical objects used to store and manage entities. For more information, refer to [Manage Collections](manage-collections.md). ## Dependency -A dependency is a program that another program relies on to work. Milvus' dependencies include etcd (stores meta data), MinIO or S3 (object storage), and Pulsar (manages snapshot logs). + +A dependency is a program that another program relies on to work. Milvus' dependencies include etcd (stores meta data), MinIO or S3 (object storage), and Pulsar (manages snapshot logs). For more information, refer to [Manage Dependencies](https://milvus.io/docs/manage_dependencies.md#Manage-Dependencies). + +## Dynamic schema + +Dynamic schema allows you to insert entities with new fields into a collection without modifying the existing schema. This means that you can insert data without knowing the full schema of a collection and can include fields that are not yet defined. You can enable this schema-free capability by enableing the dynamic field when creating a collection. For more information, refer to [Enable Dynamic Field](enable-dynamic-field.md). + +## Embeddings + +Milvus offers built-in embedding functions that work with popular embedding providers. Before creating a collection in Milvus, you can use these functions to generate embeddings for your datasets, streamlining the process of preparing data and vector searches. To create embeddings in action, refer to [Using PyMilvus's Model To Generate Text Embeddings](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/model/embedding_functions.ipynb). ## Entity -An entity consists of a group of fields that represent real world objects. Each entity in Milvus is represented by a unique primary key. -
-You can customize primary keys. If you do not configure manually, Milvus automatically assigns primary keys to entities. If you choose to configure your own customized primary keys, note that Milvus does not support primary key de-duplication for now. Therefore, there can be duplicate primary keys in the same collection.
-
+An entity consists of a group of fields that represent real-world objects. Each entity in Milvus is represented by a unique primary key.
+
+You can customize primary keys. If you do not configure manually, Milvus automatically assigns the primary key to entities. If you choose to customize the primary key, note that Milvus does not support primary key de-duplication for now. Therefore, there can be duplicate primary keys in the same collection. For more information, refer to [Insert Entities](insert-update-delete.md#Insert-entities).
## Field
-Fields are the units that make up entities. Fields can be structured data (e.g., numbers, strings) or vectors.
-
-Starting from Milvus 2.0, scalar field filtering is available!
-
+A field in a Milvus collection is equivalent to a column of table in a RDBMS. Fields can be either scalar fields for structured data (e.g., numbers, strings), or vector fields for embedding vectors.
+
+## Filter
+
+Milvus supports scalar filtering by searching with predicates, allowing you to define [filter conditions](https://milvus.io/docs/boolean.md) within queries and searches to refine results.
+
+## Filtered search
+
+Filtered search applies scalar filters to vector searches, allowing you to refine the search results based on specific criteria. For more information, refer to [Filtered search](single-vector-search.md#Filtered-search).
+
+## Hybrid search
+
+[Hybrid Search](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Collection/hybrid_search.md) is an API for hybrid search since Milvus 2.4.0. You can search multiple vector fields and fusion them. For a vector search combined with scalar field filtering, it is referred to as "filtered search". For more information, refer to [Hybrid Search](multi-vector-search.md).
+
+## Index
+
+A vector index is a reorganized data structure derived from raw data that can greatly accelerate the process of vector similarity search. Milvus supports a wide range of index types for both vector fields and scalar fields. For more information, refer to [Vector index types](https://milvus.io/docs/index.md).
+
+## Kafka-Milvus Connector
+
+[Kafka-Milvus Connector](https://github.com/zilliztech/kafka-connect-milvus) refers to a Kafka sink connector for Milvus. It allows you to stream vector data from Kafka to Milvus.
+
+## Knowhere
+
+[Knowhere](https://milvus.io/docs/knowhere.md#Knowhere) is the core vector execution engine of Milvus which incorporates several vector similarity search libraries including Faiss, Hnswlib, and Annoy. Knowhere is also designed to support heterogeneous computing. It controls on which hardware (CPU or GPU) to execute index building and search requests. This is how Knowhere gets its name - knowing where to execute the operations.
## Log broker
-The log broker is a publish-subscribe system that supports playback. It is responsible for streaming data persistence, execution of reliable asynchronous queries, event notification, and return of query results. It also ensures integrity of the incremental data when the worker nodes recover from system breakdown.
-## Log sequence
-The log sequence records all operations that change collection states in Milvus.
+The [log broker](https://milvus.io/docs/four_layers.md#Log-broker) is a publish-subscribe system that supports playback. It is responsible for streaming data persistence, execution of reliable asynchronous queries, event notification, and return of query results. It also ensures integrity of the incremental data when the worker nodes recover from system breakdown.
## Log snapshot
-A log snapshot is a binary log, a smaller unit in segment that records and handles the updates and changes made to data in the Milvus vector database. Data from a segment is persisted in multiple binlogs. There are three types of binlogs in Milvus: InsertBinlog, DeleteBinlog, and DDLBinlog.
+A log snapshot is a binary log, a smaller unit in segment that records and handles the updates and changes made to data in Milvus. Data from a segment is persisted in multiple binlogs. There are three types of binlogs in Milvus: InsertBinlog, DeleteBinlog, and DDLBinlog. For more information, refer to [Meta storage](https://milvus.io/docs/four_layers.md#Meta-storage).
## Log subscriber
-Log subscribers subscribe to the log sequence to update the local data and provides services in the form of read-only copies.
+
+Log subscribers subscribe to the log sequence to update the local data and provide services in the form of read-only copies.
## Message storage
-Message storage is the log storage engine of Milvus.
+Message storage is the log storage engine of Milvus. Milvus supports Kafka or Pulsa as message storage. For more information, refer to [Configure Message Storage](https://milvus.io/docs/message_storage_operator.md#Configure-Message-Storage-with-Milvus-Operator).
+
+## Metric type
+
+Similarity metric types are used to measure similarities between vectors. Currently, Milvus supports Euclidean distance (L2), Inner product (IP), Cosine similarity (COSINE), and binary metric types. You can choose the most appropriate metric type based on your scenario. For more information, refer to [Similarity Metrics](https://milvus.io/docs/metric.md).
+
+## Mmap
+
+Memory-mapped files enable efficient data handling by mapping file contents directly into memory. This is especially useful when memory is limited and loading all data is not possible. This technique can boost data capacity and maintain performance to a point. However, if the data greatly exceeds memory capacity, search and query speeds could significantly decrease. For more information, refer to [MMap-enabled Data Storage](https://milvus.io/docs/mmap.md).
+
+## Milvus Backup
+
+[Milvus Backup](https://milvus.io/docs/milvus_backup_overview.md#Milvus-Backup) is a tool for creating copies of data, which can be used to restore the original after a data loss event.
+
+## Milvus CDC
+
+[Milvus CDC](https://milvus.io/docs/milvus-cdc-overview.md) (Change data capture) is a user-friendly tool that can capture and synchronize incremental data in Milvus instances. It ensures the reliability of business data by seamlessly transferring it between source and target instances, allowing for easy incremental backup and disaster recovery.
+
+## Milvus CLI
+
+[Milvus Command-Line Interface](https://milvus.io/docs/cli_overview.md) (CLI) is a command-line tool that supports database connection, data operations, and import and export of data. Based on [Milvus Python SDK](https://github.com/milvus-io/pymilvus), it allows the execution of commands through a terminal using interactive command-line prompts.
+
+## Milvus Migration
+
+[Milvus Migration](https://github.com/zilliztech/milvus-migration/) is an open-source tool designed to facilitate the easy migration of data from various data sources into Milvus 2.x.
## Milvus cluster
-In a cluster deployment of Milvus, services are provided by a group of nodes to achieve high availability and easy scalability.
+
+In [cluster deployment](https://milvus.io/docs/install_cluster-milvusoperator.md) of Milvus, services are provided by a group of nodes to achieve high availability and easy scalability.
## Milvus standalone
-In a standalone deployment of Milvus, all operations including data insertion, index building, and vector similarity search are completed in one single process.
-## Normalization
-Normalization refers to the process of converting an embedding (vector) so that its norm equals one. If inner product (IP) is used to calculate embeddings similarities, all embeddings must be normalized. After normalization, inner product equals cosine similarity.
+In [standalone deployment](https://milvus.io/docs/install_standalone-docker.md) of Milvus, all operations including data insertion, index building, and vector similarity search are completed in one single process.
+
+## Multi-Vector
+
+Milvus supports multiple vector fields in one collection since 2.4.0. For more information, refer to [Hybrid Search](multi-vector-search.md).
## Partition
-A partition is a division of a collection. Milvus supports dividing collection data into multiple parts on physical storage. This process is called partitioning, and each partition can contain multiple segments.
+
+A partition is a division of a collection. Milvus supports dividing collection data into multiple parts on physical storage. This process is called partitioning, and each partition can contain multiple segments. For more information, refer to [Manage Partitions](https://milvus.io/docs/manage-partitions.md#Manage-Partitions).
+
+## Partition key
+
+The partition key attribute of a field enables the segregation of entities into distinct partitions based on their partition key values. This grouping ensures that entities sharing the same key value are stored together, which can speed up search operations by allowing the system to bypass irrelevant partitions during queries filtered by the partition key field. For more information, refer to [Use Partition Key](https://milvus.io/docs/use-partition-key.md#Use-Partition-Key).
## PChannel
-PChannel stands for physical channel. Each PChannel corresponds to a topic for log storage. A group of 256 PChannels by default will be assigned to store logs that record data insertion, deletion, and update when the Milvus cluster is started.
+
+PChannel stands for physical channel. Each PChannel corresponds to a topic for log storage. By default, a group of 16 PChannels will be assigned to store logs that record data insertion, deletion, and update when the Milvus cluster is started. For more information, refer to [Message Channel-related Configurations](https://milvus.io/docs/configure_messagechannel.md#Message-Channel-related-Configurations).
+
+## PyMilvus
+
+PyMilvus is a Python SDK of Milvus. Its source code is open-sourced and hosted on [GitHub](https://github.com/milvus-io/pymilvus). You have the flexibility to choose MilvusClient (new version Python SDK) or the original ORM module to talk with Milvus.
+
+## Query
+
+[Query](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/query.md) is an API that conducts scalar filtering with a specified boolean expression as filter. For more information, refer to [Get & Scalar Query](https://milvus.io/docs/get-and-scalar-query.md#Use-Basic-Operators).
+
+## Range search
+
+Range search allows you to find vectors that lie within a specified distance from your search vector. For more information, refer to [Range search](https://milvus.io/docs/single-vector-search.md#Range-search).
## Schema
-Schema is the meta information that defines data type and data property. Each collection has its own collection schema that defines all the fields of a collection, automatic ID (primary key) allocation enablement, and collection description. Also included in collection schemas are field schemas that defines the name, data type, and other properties of a field.
+
+Schema is the meta information that defines the data type and data property. Each collection has its own collection schema that defines all the fields of a collection, automatic ID (primary key) allocation enablement, and collection description. Field schemas are also included in collection schemas, which defines the name, data type, and other properties of a field. For more information, refer to [Manage Schema](https://milvus.io/docs/schema.md#Manage-Schema).
+
+## Search
+
+[Search](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/search.md) is an API that performs an operation to conduct a vector similarity search, requiring vector data for its execution. For more information, refer to [Single-Vector Search](https://milvus.io/docs/single-vector-search.md).
## Segment
-A segment is a data file automatically created by Milvus for holding inserted data. A collection can have multiple segments and a segment can have multiple entities. During vector similarity search, Milvus scans each segment and returns the search results. A segment can be either growing or sealed. A growing segment keeps receiving the newly inserted data till it is sealed. A sealed segment no longer receives any new data, and will be flushed to the object storage, leaving new data to be inserted into a freshly created growing segment. A growing segment will be sealed either because the number of entities it holds reaches the pre-defined threshold, or because the span of "growing" status exceeds the specified limit.
-## Sharding
-Sharding refers to distributing write operations to different nodes to make the most of the parallel computing potential of a Milvus cluster for writing data. By default, a single collection contains two shards. Milvus adopts a sharding method based on primary key hashing. Milvus' development roadmap includes supporting more flexible sharding methods such as random and custom sharding.
+A segment is an automatically created data file that stores inserted data. A collection may contain multiple segments, and each segment can hold numerous entities. During a vector similarity search, Milvus examines each segment to compile search results.
-
-Partitioning works to reduce read load by specifying a partition name, while sharding spreads write load among multiple servers.
-
+There are two types of segments: growing and sealed. A growing segment continues to collect new data until it hits a specific threshold or time limit, after which it becomes sealed. Once sealed, a segment no longer accepts new data and is transferred to object storage. Meanwhile, incoming data is routed to a new growing segment. The transition from a growing to a sealed segment is triggered either by reaching the predefined entity limit or by exceeding the maximum allowed duration in the growing state. For more information, refer to [Design Details](https://milvus.io/docs/replica.md#Design-Details).
+
+## Spark-Milvus Connector
+
+[Spark-Milvus Connector](https://github.com/zilliztech/spark-milvus) provides seamless integration between Apache Spark and Milvus, combining the data processing and machine learning (ML) features of Apache Spark with the vector data storage and search capabilities of Milvus.
+
+## Shard
+
+Milvus enhances data write performance by distributing write operations across multiple nodes using shards, which are organized based on the hashing of primary keys. This leverages the cluster's parallel computing capabilities.
+
+*Partitioning works to reduce read load by specifying a partition name, while sharding spreads write load among multiple servers.*
+
+## Sparse vector
+
+Sparse vectors represent words or phrases using vector embeddings where most elements are zero, with only one non-zero element indicating the presence of a specific word. Sparse vector models, such as SPLADEv2, outperform dense models in out-of-domain knowledge search, keyword-awareness, and interpretability. For more information, refer to [Sparse Vectors](https://milvus.io/docs/sparse_vector.md#Sparse-Vector).
## Unstructured data
-Unstructured data, including images, video, audio, and natural language, is information that doesn't follow a predefined model or manner of organization. This data type accounts for around 80% of the world's data, and can be converted into vectors using various artificial intelligence (AI) and machine learning (ML) models.
+
+Unstructured data, including images, video, audio, and natural language, is information that does not follow a predefined model or manner of organization. This data type accounts for around 80% of the world's data, and can be converted into vectors using various artificial intelligence (AI) and ML models.
## VChannel
-VChannel stands for logical channel. Each VChannel represents a shard in a collection. Each collection will be assigned a group of VChannels for recording data insertion, deletion, and update. VChannels are logically separated but physically share resources.
+[VChannel](https://milvus.io/docs/data_processing.md#Data-insertion) stands for logical channel. Each VChannel represents a shard in a collection. Each collection will be assigned a group of VChannels for recording data insertion, deletion, and update. VChannels are logically separated but physically share resources.
-## Embedding Vector
-An embedding vector is a feature abstraction of unstructured data, such as emails, IoT sensor data, Instagram photos, protein structures, and much more. Mathematically speaking, an embedding vector is an array of floating-point numbers or binaries. Modern embedding techniques are used to convert unstructured data to embedding vectors.
+## Vector
-## Vector index
-A vector index is a reorganized data structure derived from raw data that can greatly accelerate the process of vector similarity search. Milvus supports several [vector index types](index.md).
+An embedding vector is a feature abstraction of unstructured data, such as emails, IoT sensor data, Instagram photos, protein structures, and more. Mathematically speaking, an embedding vector is an array of floating-point numbers or binaries. Modern embedding techniques are used to convert unstructured data to embedding vectors. Milvus support both dense and sparse vector since 2.4.0.
-## Vector similarity search
-Vector similarity search is the process of comparing a vector to a database to find vectors that are most similar to the target search vector. Approximate nearest neighbor (ANN) search algorithms are used to calculate [similarity](metric.md) between vectors.
+## Zilliz Cloud
+Fully-managed Milvus on [Zilliz Cloud](https://zilliz.com/), with more enterprise features and highly optimized performance.
diff --git a/preview/site/en/reference/gpu_index.md b/preview/site/en/reference/gpu_index.md
new file mode 100644
index 000000000..4a1421435
--- /dev/null
+++ b/preview/site/en/reference/gpu_index.md
@@ -0,0 +1,127 @@
+---
+id: gpu_index.md
+related_key: gpu_index
+summary: GPU index mechanism in Milvus.
+title: GPU Index
+---
+
+# GPU Index
+
+Milvus supports various GPU index types to accelerate search performance and efficiency, especially in high-throughput, and high-recall scenarios. This topic provides an overview of the GPU index types supported by Milvus, their suitable use cases, and performance characteristics. For information on building indexes with GPU, refer to [Index with GPU](index-with-gpu.md).
+
+It's important to note that using a GPU index may not necessarily reduce latency compared to using a CPU index. If you want to fully maximize throughput, you will need extremely high request pressure or a large number of query vectors.
+
+
+
+Milvus' GPU support is contributed by Nvidia [RAPIDS](https://rapids.ai/) team. The following are the GPU index types currently supported by Milvus.
+
+## GPU_CAGRA
+
+GPU_CAGRA is a graph-based index optimized for GPUs, Using inference-grade GPUs to run the Milvus GPU version can be more cost-effective compared to using expensive training-grade GPUs.
+
+- Index building parameters
+
+ | Parameter | Description | Default Value |
+ | --------------------------- | ------------------------------------------------------------ | -------------------- |
+ | `intermediate_graph_degree` | Affects recall and build time by determining the graph's degree before pruning. Recommended values are `32` or `64`. | 128 |
+ | `graph_degree` | Affects search performance and recall by setting the graph's degree after pruning. A larger difference between these two degrees results in a longer build time. Its value must be smaller than the value of __intermediate_graph_degree__. | 64 |
+ | `build_algo` | Selects the graph generation algorithm before pruning. Possible values:IVF_PQ: Offers higher quality but slower build time. NN_DESCENT: Provides a quicker build with potentially lower recall. | IVF_PQ |
+ | `cache_dataset_on_device` | Decides whether to cache the original dataset in GPU memory. Possible values:"true": Caches the original dataset to enhance recall by refining search results. "false": Does not cache the original dataset to save gpu memory. | "false" |
+
+- Search parameters
+
+ | Parameter | Description | Default Value |
+ |-------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+ | `itopk_size` | Determines the size of intermediate results kept during the search. A larger value may improve recall at the expense of search performance. It should be at least equal to the final top-k (limit) value and is typically a power of 2 (e.g., 16, 32, 64, 128). | Empty |
+ | `search_width` | Specifies the number of entry points into the CAGRA graph during the search. Increasing this value can enhance recall but may impact search performance（e.g. 1, 2, 4, 8, 16, 32). | Empty |
+ | `min_iterations` / `max_iterations` | Controls the search iteration process. By default, they are set to `0`, and CAGRA automatically determines the number of iterations based on `itopk_size` and `search_width`. Adjusting these values manually can help balance performance and accuracy. | `0` |
+ | `team_size` | Specifies the number of CUDA threads used for calculating metric distance on the GPU. Common values are a power of 2 up to 32 (e.g. 2, 4, 8, 16, 32). It has a minor impact on search performance. The default value is `0`, where Milvus automatically selects the `team_size` based on the vector dimension. | `0` |
+
+* Limits on search
+
+ | Parameter | Range |
+ | --------- | ---------------------------------------------------- |
+ | `top-K` | <= 1024 |
+ | `top-K` | <=max((`itopk_size` + 31)// 32, `search_width`) * 32 |
+
+## GPU_IVF_FLAT
+
+Similar to [IVF_FLAT](https://milvus.io/docs/index.md#IVF_FLAT), GPU_IVF_FLAT also divides vector data into `nlist` cluster units, and then compares distances between the target input vector and the center of each cluster. Depending on the number of clusters the system is set to query (`nprobe`), similarity search results are returned based on comparisons between the target input and the vectors in the most similar cluster(s) only — drastically reducing query time.
+
+By adjusting `nprobe`, an ideal balance between accuracy and speed can be found for a given scenario. Results from the [IVF_FLAT performance test](https://zilliz.com/blog/Accelerating-Similarity-Search-on-Really-Big-Data-with-Vector-Indexing) demonstrate that query time increases sharply as both the number of target input vectors (`nq`), and the number of clusters to search (`nprobe`), increase.
+
+GPU_IVF_FLAT is the most basic IVF index, and the encoded data stored in each unit is consistent with the original data.
+
+When conducting searches, note that you can set the top-K up to 256 for any search against a GPU_IVF_FLAT-indexed collection.
+
+- Index building parameters
+
+ | Parameter | Description | Range | Default Value |
+ | --------- | ----------------------- | ---------- | ------------- |
+ | `nlist` | Number of cluster units | [1, 65536] | `128` |
+ | `cache_dataset_on_device` | Decides whether to cache the original dataset in GPU memory. Possible values:"true": Caches the original dataset to enhance recall by refining search results. "false": Does not cache the original dataset to save gpu memory. | `"true"` `"flase"` | `"false"` |
+
+- Search parameters
+
+ - Common search
+
+ | Parameter | Description | Range | Default Value |
+ | --------- | ------------------------ | --------------- | ------------- |
+ | `nprobe` | Number of units to query | [1, nlist] | `8` |
+
+- Limits on search
+
+ | Parameter | Range |
+ | --------- | --------- |
+ | `top-K` | <= `2048` |
+
+## GPU_IVF_PQ
+
+`PQ` (Product Quantization) uniformly decomposes the original high-dimensional vector space into Cartesian products of `m` low-dimensional vector spaces, and then quantizes the decomposed low-dimensional vector spaces. Instead of calculating the distances between the target vector and the center of all the units, product quantization enables the calculation of distances between the target vector and the clustering center of each low-dimensional space and greatly reduces the time complexity and space complexity of the algorithm.
+
+IVF\_PQ performs IVF index clustering before quantizing the product of vectors. Its index file is even smaller than IVF\_SQ8, but it also causes a loss of accuracy during searching vectors.
+
+
+
+Index building parameters and search parameters vary with Milvus distribution. Select your Milvus distribution first.
+
+When conducting searches, note that you can set the top-K up to 8192 for any search against a GPU_IVF_FLAT-indexed collection.
+
+
+
+- Index building parameters
+
+ | Parameter | Description | Range | Default Value |
+ | --------- | ----------------------------------------- | ------------------- | ------------- |
+ | `nlist` | Number of cluster units | [1, 65536] | `128` |
+ | `m` | Number of factors of product quantization, | `dim mod m or = 0` | `0` |
+ | `nbits` | [Optional] Number of bits in which each low-dimensional vector is stored. | [1, 16] | `8` |
+ | ``cache_dataset_on_device`` | Decides whether to cache the original dataset in GPU memory. Possible values:"true": Caches the original dataset to enhance recall by refining search results. "false": Does not cache the original dataset to save gpu memory. | `"true"` `"false"` | `"false"` |
+
+- Search parameters
+
+ - Common search
+
+ | Parameter | Description | Range | Default Value |
+ | --------- | ------------------------ | --------------- | ------------- |
+ | `nprobe` | Number of units to query | [1, nlist] | `8` |
+
+- Limits on search
+
+ | Parameter | Range |
+ | --------- | --------- |
+ | `top-K` | <= `1024` |
+
+## GPU_BRUTE_FORCE
+
+GPU_BRUTE_FORCE is tailored for cases where extremely high recall is crucial, guaranteeing a recall of 1 by comparing each query with all vectors in the dataset. It only requires the metric type (`metric_type`) and top-k (`limit`) as index building and search parameters.
+
+For GPU_BRUTE_FORCE, no addition index building parameters or search parameters are required.
+
+## Conclusion
+
+Currently, Milvus loads all indexes into GPU memory for efficient search operations. The amount of data that can be loaded depends on the size of the GPU memory:
+
+- **GPU_CAGRA**: Memory usage is approximately 1.8 times that of the original vector data.
+- **GPU_IVF_FLAT** and **GPU_BRUTE_FORCE**: Requires memory equal to the size of the original data.
+- **GPU_IVF_PQ**: Utilizes a smaller memory footprint, which depends on the compression parameter settings.
diff --git a/preview/site/en/reference/index.md b/preview/site/en/reference/index.md
index a1576c9e7..5261c0b0c 100644
--- a/preview/site/en/reference/index.md
+++ b/preview/site/en/reference/index.md
@@ -2,53 +2,70 @@
id: index.md
related_key: index
summary: Index mechanism in Milvus.
+title: In-memory Index
---
# In-memory Index
This topic lists various types of in-memory indexes Milvus supports, scenarios each of them best suits, and parameters users can configure to achieve better search performance. For on-disk indexes, see **[On-disk Index](disk_index.md)**.
-Indexing is the process of efficiently organizing data, and it plays a major role in making similarity search useful by dramatically accelerating time-consuming queries on large datasets.
+Indexing is the process of efficiently organizing data, and it plays a major role in making similarity search useful by dramatically accelerating time-consuming queries on large datasets.
-To improve query performance, you can [specify an index type](build_index.md) for each vector field.
+To improve query performance, you can [specify an index type](index-vector-fields.md) for each vector field.
-
+
Currently, a vector field only supports one index type. Milvus automatically deletes the old index when switching the index type.
-
## ANNS vector indexes
Most of the vector index types supported by Milvus use approximate nearest neighbors search (ANNS) algorithms. Compared with accurate retrieval, which is usually very time-consuming, the core idea of ANNS is no longer limited to returning the most accurate result, but only searching for neighbors of the target. ANNS improves retrieval efficiency by sacrificing accuracy within an acceptable range.
-According to the implementation methods, the ANNS vector index can be divided into four categories:
+According to the implementation methods, the ANNS vector index can be categorized into four types: Tree-based, Graph-based, Hash-based, and Quantization-based.
-- Tree-based index
-- Graph-based index
-- Hash-based index
-- Quantization-based index
+## Indexes supported in Milvus
+Milvus supports various index types, which are categorized by the type of embedding they handle: **floating-point**, **binary**, and **sparse**.
-## Indexes supported in Milvus
+
-According to the suited data type, the supported indexes in Milvus can be divided into two categories:
-- Indexes for floating-point embeddings:
- - For 128-dimensional floating-point embeddings, the storage they take up is 128 * the size of float = 512 bytes. And the [distance metrics](metric.md) used for float-point embeddings are Euclidean distance (L2) and Inner product.
- - This type of indexes include FLAT, IVF_FLAT, IVF_PQ, IVF_SQ8, ANNOY, and HNSW.
-- Indexes for binary embeddings
- - For 128-dimensional binary embeddings, the storage they take up is 128 / 8 = 16 bytes. And the distance metrics used for binary embeddings are Jaccard, Tanimoto, Hamming, Superstructure, and Substructure.
- - This type of indexes include BIN_FLAT and BIN_IVF_FLAT.
+
-The following table classifies the indexes that Milvus supports:
+### Indexes for floating-point embeddings
-
-Floating-point embeddings Binary embeddings
+For 128-dimensional floating-point embeddings, the storage they take up is 128 * the size of float = 512 bytes. And the [distance metrics](metric.md) used for float-point embeddings are Euclidean distance (`L2`) and Inner product (`IP`).
+
+These types of indexes include `FLAT`, `IVF_FLAT`, `IVF_PQ`, `IVF_SQ8`, `HNSW`, and `SCANN` for CPU-based ANN searches.
+
+
+
+
+
+### Indexes for binary embeddings
+
+For 128-dimensional binary embeddings, the storage they take up is 128 / 8 = 16 bytes. And the distance metrics used for binary embeddings are `JACCARD` and `HAMMING`.
+
+This type of indexes include `BIN_FLAT` and `BIN_IVF_FLAT`.
+
+
+
+
+
+### Indexes for sparse embeddings
+
+The distance metric supported for sparse embeddings is `IP` only.
+
+The types of indexes include `SPARSE_INVERTED_INDEX` and `SPARSE_WAND`.
-
+
You can also create a collection with
-
+
+
-### FLAT
+ --reuse-values -f new-values.yaml milvus/milvus
+```
+
+- For Milvus installed using Milvus Operator:
+
+```bash
+# patch.yaml
+spec:
+ config:
+ queryNode:
+ mmap:
+ mmapEnabled: true
+ mmapDirPath: any/valid/path
+
+ kubectl patch milvus --patch-file patch.yaml
+```
+
+- For Milvus installed using Docker:
+
+```bash
+# A new installation script is provided to enable mmap-related settings.
+```
+
+## Limits
+
+- Memory mapping cannot be enabled for a loaded collection, ensure the collection has been released before enabling memory mapping.
+
+- Memory mapping is not supported for DiskANN or GPU-class indexes.
+
+## FAQ
+
+- __In which scenarios is it recommended to enable memory mapping? What are the trade-offs after enabling this feature?__
+
+ Memory mapping is recommended when memory is limited or when performance requirements are moderate. Enabling this feature increases the capacity for data loading. For example, with a configuration of 2 CPUs and 8 GB of memory, enabling memory mapping can allow for up to 4 times more data to be loaded compared to not enabling it. The impact on performance varies:
+
+ - With sufficient memory, the expected performance is similar to that of using only memory.
+
+ - With insufficient memory, the expected performance may degrade.
+
+- __What is the relationship between collection-level and index-level configurations?__
+
+ Collection-level and index-level are not inclusive relationships, collection-level controls whether the original data is mmap-enabled or not, whereas index-level is for vector indexes only.
+
+- __Is there any recommended index type for memory mapping?__
+
+ Yes, HNSW is recommended for enable mmap. We have tested HNSW, IVF_FLAT, IVF_PQ/SQ series indexes before, the performance of IVF series indexes dropped seriously, while the performance drop of turning on mmap for HNSW indexes is still within expectation.
+
+- __What kind of local storage is required for memory mapping?__
+
+ A high-quality disk enhances performance, with NVMe drives being the preferred option.
+
+- __Can scalar data be memory-mapped?__
+
+ Memory mapping can be applied to scalar data, but it is not applicable to indexes built on scalar fields.
+
+- __How is the priority determined for memory mapping configurations across different levels?__
+
+ In Milvus, when memory mapping configurations are explicitly defined across multiple levels, index-level and collection-level configurations share the highest priority, which is then followed by cluster-level configurations.
+
+- __If I upgrade from Milvus 2.3 and have configured the memory mapping directory path, what will happen?__
+
+ If you upgrade from Milvus 2.3 and have configured the memory mapping directory path (`mmapDirPath`), your configuration will be retained, and the default setting for memory mapping enabled (`mmapEnabled`) will be `true`. It's important to migrate the metadata to synchronize the configuration of your existing memory-mapped files. For more details, refer to [Migrate the metadata](https://milvus.io/docs/upgrade_milvus_standalone-docker.md#Migrate-the-metadata).
\ No newline at end of file
diff --git a/preview/site/en/reference/multi_tenancy.md b/preview/site/en/reference/multi_tenancy.md
new file mode 100644
index 000000000..41fd7a44a
--- /dev/null
+++ b/preview/site/en/reference/multi_tenancy.md
@@ -0,0 +1,59 @@
+---
+id: multi_tenancy.md
+related_key: multi-tenancy
+summary: Multi-tenancy in Milvus.
+title: Multi-tenancy strategies
+---
+
+# Multi-tenancy strategies
+
+As ChatGPT gains popularity, more developers are creating their own SaaS services using the CVP (ChatGPT, Vector Database, Prompt) stack. This guide explains how to achieve multi-tenancy on Milvus, one of the most widely-used vector databases in the world, to keep up with this trend.
+
+Multi-tenancy is an architecture where a single Milvus instance serves multiple tenants. The simplest way to distinguish tenants is by separating their data and resources from those of others. Each tenant has their own dedicated resources or shares resources with others to manage Milvus objects like databases, collections, and partitions. Based on these objects, there are corresponding methods for achieving Milvus multi-tenancy.
+
+## Database-oriented multi-tenancy
+
+Since Milvus version 2.2.9, the object database is now available. You can create multiple databases in a single Milvus cluster. This new feature makes it possible to achieve database-oriented multi-tenancy by assigning a database for each tenant, so that they can create their own collections and partitions to make the most out of their data. However, this strategy ensures data isolation and search performance for tenants, but resources may be wasted on idle tenants.
+
+## Collection-oriented multi-tenancy
+
+There are two possible ways to achieve collection-oriented multi-tenancy.
+
+### One collection for all tenants
+
+Using a single collection to implement multi-tenancy by adding a tenant field to distinguish between tenants is a simple option. When conducting ANN searches for a specific tenant, add a filter expression to filter out all entities that belong to other tenants. This is the simplest way to achieve multi-tenancy. However, be aware that the filter's performance may become the bottleneck of ANN searches.
+
+### One collection per tenant
+
+Another approach is to create a collection for each tenant to store its own data, instead of storing the data of all tenants in a single collection. This provides better data isolation and query performance. However, keep in mind that this approach requires more investment in resource scheduling, operational capability, and costs and may be not applicable if the number of tenants exceeds the maximum number of collections that a single Milvus cluster supports.
+
+## Partition-oriented multi-tenancy
+
+There are also two possible ways to achieve partition-oriented multi-tenancy:
+
+### One partition per tenant
+
+Managing a single collection is much easier than managing multiple ones. Instead of creating multiple collections, consider assigning a partition for each tenant to achieve flexible data isolation and memory management. The search performance of partition-oriented multi-tenancy is much better than collection-oriented multi-tenancy. However, note that the number of tenants of the collection should not exceed the maximum number of partitions a collection can hold.
+
+### Partition-key-based multi-tenancy
+
+Milvus 2.2.9 introduces a new feature named partition key. Upon the creation of a collection, nominate a tenant field and make it the partition key field. Milvus will store entities in a partition according to the values in the partition key field. When conducting ANN searches, Milvus changes to a partition based on the specified partition key, filters entities according to the partition key, and searches among the filtered entities.
+
+
+
+This strategy lifts the limit on the maximum number of tenants that a Milvus collection can support and greatly simplifies resource management because Milvus automatically manages partitions for you.
+
+To recap, you can use either or some of the multi-tenancy strategies above to form your own solution. The following table makes comparisons among these strategies in terms of data isolation, search performance, and maximum number of tenants.
+
+| | Data isolation | Search perf. | Max. num. of tenants | Recommend scenarios |
+|---------------------------|----------------|--------------|----------------------|---------------------------------------------------------------------------------------------|
+| Database oriented | Strong | Strong | 64 |For those that require collections to vary with projects, especially suitable for data isolation between departments in your organization. |
+| One collection for all | Weak | Medium | N/A |For those that have limited resources and are insensitive to data isolation. |
+| One collection per tenant | Strong | Strong | Less than 10,000 |For those that have less than 10,000 tenants per cluster. |
+| One partition per tenant | Medium | Strong | 4,096 |For those that have less than 4,096 tenants per collection. |
+| Partition-key-based | Medium | Strong | 10,000,000+ |For those that predict a rapid tenant increase into millions. |
+
+## What's next
+
+[Manage Databases](manage_databases.md)
+[Schema](schema.md)
diff --git a/preview/site/en/reference/references.md b/preview/site/en/reference/references.md
deleted file mode 100644
index 7cd55f051..000000000
--- a/preview/site/en/reference/references.md
+++ /dev/null
@@ -1,31 +0,0 @@
----
-id: references.md
-title: References
----
-
-# References
-
-Unfamiliar with the features of Milvus? Not sure about the meaning of some terms? You are in the right place to find a detailed explanation of the architecture, features, terms, and many other things you may want to know about Milvus.
-
-- [Architecture](architecture.md): An anatomy of the architecture, components, data processing workflow of Milvus.
-
-- [Benchmark](benchmark.md): A benchmark test report covering the performances of data insertion, index building, and vector similarity search in Milvus.
-
-- [Boolean Expression Rules](boolean.md): The boolean expression rules and operators for scalar filtering in Milvus.
-
-- [Consistency](consistency.md): Milvus provides four levels of consistency for vector similarity search. Learn about the consistency levels and their best-suited scenarios.
-
-- [Glossary](glossary.md): The most frequently used terminologies in Milvus.
-
-- [In-Memory Replica](replica.md): The in-memory replica mechanism enables multiple segment replications in the working memory to improve performance and availability.
-
-- [Schema](schema.md): Schema is used to define the properties of a collection and the fields within.
-
-- [Similarity Metrics](metric.md): Milvus supports multiple similarity metrics. Learn how to choose the right metrics that best suit your application.
-
-- [System Configurations](system_configuration.md): A full list of system parameters that can be flexibly configured to best suit your need.
-
-- [Time Travel](timetravel_ref.md): A detailed explanation of the Time Travel feature and the mechanism behind.
-
-- [Vector Index](index.md): A full list of the various types of indexes supported by Milvus. Learn about the best-suited scenarios of each index and the configurable index parameters.
-
diff --git a/preview/site/en/reference/replica.md b/preview/site/en/reference/replica.md
index aa83326ec..4db418ccf 100644
--- a/preview/site/en/reference/replica.md
+++ b/preview/site/en/reference/replica.md
@@ -1,12 +1,15 @@
---
id: replica.md
summary: Learn about in-memory replica in Milvus.
+title: In-Memory Replica
---
# In-Memory Replica
This topic introduces the in-memory replica (replication) mechanism in Milvus that enables multiple segment replications in the working memory to improve performance and availability.
+For information on how to configure in-memory replicas, refer to [Query Node-related Configurations](configure_querynode.md#queryNodereplicas).
+
## Overview
@@ -67,13 +70,8 @@ The caches on the proxy are not always up-to-date. Some segments or channels may
A segment will be ignored if the proxy still cannot find it after updating the cache. This could happen if the segment has been compacted.
-If the cache is not accurate, the proxy may miss some segments. Query nodes with DML channels (growing segments) return search responses along with a list of reliable segments that the proxy can compare and update the cache with.
+If the cache is not accurate, the proxy may miss some segments. Query nodes with DML channels (growing segments) return search responses along with a list of reliable segments that the proxy can compare and update the cache with.
### Enhancement
The proxy cannot allocate search requests to query nodes completely equally and query nodes may have different resources to serve search requests. To avoid a long-tailed distribution of resources, the proxy will assign active segments on other query nodes to an idle query node that also has these segments.
-
-## What's next
-
-- Learn how to [load a collection](load_collection.md) as multiple replicas.
-- Learn how to [load a partition](load_partition.md) as multiple replicas.
diff --git a/preview/site/en/reference/reranking.md b/preview/site/en/reference/reranking.md
new file mode 100644
index 000000000..2a420c528
--- /dev/null
+++ b/preview/site/en/reference/reranking.md
@@ -0,0 +1,82 @@
+---
+id: reranking.md
+summary: This topic covers the reranking process, explaining its significance and implementation of two reranking methods.
+title: Reranking
+---
+
+# Reranking
+
+Milvus enables hybrid search capabilities using the [hybrid_search()](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Collection/hybrid_search.md) API, incorporating sophisticated reranking strategies to refine search results from multiple `AnnSearchRequest` instances. This topic covers the reranking process, explaining its significance and implementation of different reranking strategies in Milvus.
+
+## Overview
+
+The following figure illustrates the execution of a hybrid search in Milvus and highlights the role of reranking in the process.
+
+
+
+Reranking in hybrid search is a crucial step that consolidates results from several vector fields, ensuring the final output is relevant and accurately prioritized. Currently, Milvus offers these reranking strategies:
+
+- `WeightedRanker`: This approach merges results by calculating a weighted average of scores (or vector distances) from different vector searches. It assigns weights based on the significance of each vector field.
+
+- `RRFRanker`: This strategy combines results based on their ranks across different vector columns.
+
+## Weighted Scoring (WeightedRanker)
+
+The `WeightedRanker` strategy assigns different weights to results from each vector retrieval route based on the significance of each vector field. This reranking strategy is applied when the significance of each vector field varies, allowing you to emphasize certain vector fields over others by assigning them higher weights. For example, in a multimodal search, the text description might be considered more important than the color distribution in images.
+
+WeightedRanker’s basic process is as follows:
+
+- **Collect Scores During Retrieval**: Gather results and their scores from different vector retrieval routes.
+- **Score Normalization**: Normalize the scores from each route to a [0,1] range, where values closer to 1 indicate higher relevance. This normalization is crucial due to score distributions varying with different metric types. For instance, the distance for IP ranges from [-∞,+∞], while the distance for L2 ranges from [0,+∞]. Milvus employs the `arctan` function, transforming values to the [0,1] range to provide a standardized basis for different metric types.
+
+ 
+
+- **Weight Allocation**: Assign a weight `w𝑖` to each vector retrieval route. Users specify the weights, which reflect the data source's reliability, accuracy, or other pertinent metrics. Each weight ranges from [0,1].
+- **Score Fusion**: Calculate a weighted average of the normalized scores to derive the final score. The results are then ranked based on these highest to lowest scores to generate the final sorted results.
+
+
+
+To use this strategy, apply a `WeightedRanker` instance and set weight values by passing in a variable number of numeric arguments.
+
+```python
+from pymilvus import WeightedRanker
+
+# Use WeightedRanker to combine results with specified weights
+rerank = WeightedRanker(0.8, 0.8, 0.7)
+```
+
+Note that:
+
+- Each weight value ranges from 0 (least important) to 1 (most important), influencing the final aggregated score.
+
+- The total number of weight values provided in `WeightedRanker` should equal the number of `AnnSearchRequest` instances you have created earlier.
+
+- It is worth noting that due to the different measurements of the different metric types, we normalize the distances of the recall results so that they lie in the interval [0,1], where 0 means different and 1 means similar. The final score will be the sum of the weight values and distances.
+
+## Reciprocal Rank Fusion (RRFRanker)
+
+RRF is a data fusion method that combines ranking lists based on the reciprocal of their ranks. It is an effective way to balance the influence of each vector field, especially when there is no clear precedence of importance. This strategy is typically used when you want to give equal consideration to all vector fields or when there is uncertainty about the relative importance of each field.
+
+RRF’s basic process is as follows:
+
+- **Collect Rankings During Retrieval**: Retrievers across multiple vector fields retrieve and sort results.
+- **Rank Fusion**: The RRF algorithm weighs and combines the ranks from each retriever. The formula is as follows:
+
+ 
+
+ Here, 𝑁 represents the number of different retrieval routes, rank𝑖(𝑑) is the rank position of retrieved document 𝑑 by the 𝑖th retriever, and 𝑘 is a smoothing parameter, typically set to 60.
+- **Comprehensive Ranking**: Re-rank the retrieved results based on the combined scores to produce the final results.
+
+To use this strategy, apply an `RRFRanker` instance.
+
+```python
+from pymilvus import RRFRanker
+
+# Default k value is 60
+ranker = RRFRanker()
+
+# Or specify k value
+ranker = RRFRanker(k=100)
+```
+
+RRF allows balancing influence across fields without specifying explicit weights. The top matches agreed upon by multiple fields will be prioritized in the final ranking.
diff --git a/preview/site/en/reference/scalar_index.md b/preview/site/en/reference/scalar_index.md
index 12c592c4b..c9a9caec8 100644
--- a/preview/site/en/reference/scalar_index.md
+++ b/preview/site/en/reference/scalar_index.md
@@ -2,43 +2,69 @@
id: scalar_index.md
related_key: scalar_index
summary: Scalar index in Milvus.
+title: Scalar Index
---
# Scalar Index
-Milvus supports [hybrid searches](hybridsearch.md) using both scalar and vector fields. To speed up searching among entities by scalar fields, Milvus introduced scalar field indexing in version 2.1.0. This article helps you understand scalar field indexing in Milvus.
+Milvus supports filtered searches combining both scalar and vector fields. To enhance the efficiency of searches involving scalar fields, Milvus introduced scalar field indexing starting from version 2.1.0. This article provides an overview of scalar field indexing in Milvus, helping you understand its significance and implementation.
## Overview
-Once conducting vector similarity searches in Milvus, you can use logical operators to organize scalar fields into boolean expressions.
+Once conducting vector similarity searches in Milvus, you can use logical operators to organize scalar fields into boolean expressions.
When Milvus receives a search request with such a boolean expression, it parses the boolean expression into an abstract syntax tree (AST) to generate a physical plan for attribute filtering. Milvus then applies the physical plan in each segment to generate a [bitset](bitset.md) as the filtering result and includes the result as a vector search parameter to narrow down the search scope. In this case, the speed of vector searches relies heavily on the speed of attribute filtering.
-Scalar field indexing is a way of ensuring the speed of attribute filtering by sorting scalar field values in a particular way to accelerate information retrieval.
+Scalar field indexing is a way of ensuring the speed of attribute filtering by sorting scalar field values in a particular way to accelerate information retrieval.
## Scalar field indexing algorithms
-Milvus implements scalar field indexing with the goal of low memory usage, high filtering efficiency, and short loading time.
+Milvus aims to achieve low memory usage, high filtering efficiency, and short loading time with its scalar field indexing algorithms. These algorithms are categorized into two main types: [auto indexing](#auto-indexing) and [inverted indexing](#inverted-indexing).
-Specifically, indexing algorithms for scalar fields vary with field data types. However, you do not need to set the index type manually when indexing a scalar field. The default indexing algorithm applies.
+### Auto indexing
-The following table lists the data types that Milvus supports and their corresponding default indexing algorithms.
+Milvus automatically creates an auto index for a scalar field based on its data type, without requiring manual intervention. Auto indexing is suitable for prefix match queries and frequent retrieval scenarios.
-| Data type | Default indexing algorithm |
-| ----------------------------- | --------------------------------- |
-| BOOL | STL sort |
-| VARCHAR | MARISA-trie |
-| INT16 | STL sort |
-| INT32 | STL sort |
-| INT64 | STL sort |
-| FLOAT | STL sort |
-| DOUBLE | STL sort |
+The following table lists the data types that Milvus supports and their corresponding auto indexing algorithms.
+
+| Data type | Auto indexing algorithm |
+|-----------|----------------------------|
+| VARCHAR | Inverted index |
+| INT8 | Inverted index |
+| INT16 | Inverted index |
+| INT32 | Inverted index |
+| INT64 | Inverted index |
+| FLOAT | Inverted index |
+| DOUBLE | Inverted index |
+
+### Inverted indexing
+
+Inverted indexing offers a flexible way to create an index for a scalar field by manually specifying index parameters. This method works well for various scenarios, including point queries, pattern match queries, full-text searches, JSON searches, Boolean searches, and even prefix match queries.
+
+The inverted indexes implemented in Milvus are powered by [Tantivy](https://github.com/quickwit-oss/tantivy), a full-text search engine library. Tantivy ensures that inverted indexing in Milvus is both efficient and fast.
+
+An inverted index has two main components: a term dictionary and an inverted list. The term dictionary includes all tokenized words sorted alphabetically, while the inverted list contains the list of documents where each word appears. This setup makes point queries and range queries much faster and more efficient than brute-force searches.
+
+
+
+The advantages of using an inverted index are particularly evident in the following operations:
+
+- **Point query**: For example, when searching for documents containing the word **Milvus**, the process begins by checking if **Milvus** is present in the term dictionary. If it is not found, no documents contain the word. However, if it is found, the inverted list associated with **Milvus** is retrieved, indicating the documents that contain the word. This method is far more efficient than a brute-force search through a million documents, as the sorted term dictionary significantly reduces the time complexity of finding the word **Milvus**.
+- **Range query**: The efficiency of range queries, such as finding documents with words alphabetically greater than **very**, is also enhanced by the sorted term dictionary. This approach is more efficient than a brute-force search, providing quicker and more accurate results.
+
+### Test results
+
+To demonstrate the performance improvements provided by scalar indexes in Milvus, an experiment was conducted comparing the performance of several expressions using inverted indexing and brute-force search on raw data.
+
+The experiment involved testing various expressions under two conditions: with an inverted index and with a brute-force search. To ensure fairness, the same data distribution was maintained across tests, using the same collection each time. Before each test, the collection was released, and the index was dropped and rebuilt. Additionally, a warm query was performed before each test to minimize the impact of cold and hot data, and each query was executed multiple times to ensure accuracy.
+
+For a dataset of **1 million** records, using an **inverted index** can provide up to a **30x** performance improvement for point queries. The performance gains can be even more significant for larger datasets.
## Performance recommandations
-To take full advantage of Milvus' capability in scalar field indexing and unleash its power in vector similarity searches. You may need a model to estimate the size of memory required based on the data you have.
+To take full advantage of Milvus' capability in scalar field indexing and unleash its power in vector similarity searches, you may need a model to estimate the size of memory required based on the data you have.
The following tables list the estimation functions for all the data types that Milvus supports.
@@ -66,10 +92,10 @@ The following tables list the estimation functions for all the data types that M
## What's next
-- To index a scalar field, read [Build an Index on Scalars](build_scalar_index.md).
+- To index a scalar field, read [Build an Index on Scalars](index-scalar-fields.md).
- To learn more about the related terms and rules mentioned above, read
- [Bitset](bitset.md)
- - [Hybrid search](hybridsearch.md)
+ - [Hybrid search](multi-vector-search.md)
- [Boolean expression rules](boolean.md)
- - [Supported data types](schema.md#Supported-data-type)
\ No newline at end of file
+ - [Supported data types](schema.md#Supported-data-type)
diff --git a/preview/site/en/reference/schema.md b/preview/site/en/reference/schema.md
index 8e4010048..40dc745ae 100644
--- a/preview/site/en/reference/schema.md
+++ b/preview/site/en/reference/schema.md
@@ -1,16 +1,17 @@
---
id: schema.md
summary: Learn how to define a schema in Milvus.
+title: Manage Schema
---
-# Schema
+# Manage Schema
This topic introduces schema in Milvus. Schema is used to define the properties of a collection and the fields within.
## Field schema
-A field schema is the logical definition of a field. It is the first thing you need to define before defining a [collection schema](#Collection-schema) and [creating a collection](create_collection.md).
+A field schema is the logical definition of a field. It is the first thing you need to define before defining a [collection schema](#Collection-schema) and [managing collections](manage-collections.md).
Milvus supports only one primary key field in a collection.
@@ -26,29 +27,44 @@ Milvus supports only one primary key field in a collection.
- name
+ Name of the field in the collection to create
Data type: String.
Mandatory
- dtype
+ Data type of the field
Mandatory
- description
+ Description of the field
Data type: String.
Optional
- is_primary
+ Whether to set the field as the primary key field or not
Data type: Boolean (
Mandatory for the primary key field
+
+ Switch to enable or disable automatic ID (primary key) allocation.
+
+
+ Maximum length of strings allowed to be inserted.
+ [1, 65,535]
+
- dim
+ Dimension of the vector
- Data type: Integer ∈[1, 32768].
Mandatory for the vector field
+ Data type: Integer ∈[1, 32768].
Mandatory for a dense vector field. Omit for a sparse vector field.
+
+
+ Whether this field is a partition-key field.
+ Data type: Boolean (
@@ -56,16 +72,34 @@ Milvus supports only one primary key field in a collection.
### Create a field schema
+To reduce the complexity in data inserts, Milvus allows you to specify a default value for each scalar field during field schema creation, excluding the primary key field. This indicates that if you leave a field empty when inserting data, the default value you specified for this field applies.
+
+Create a regular field schema:
+
```python
from pymilvus import FieldSchema
id_field = FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, description="primary id")
age_field = FieldSchema(name="age", dtype=DataType.INT64, description="age")
embedding_field = FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=128, description="vector")
+
+# The following creates a field and use it as the partition key
+position_field = FieldSchema(name="position", dtype=DataType.VARCHAR, max_length=256, is_partition_key=True)
```
+Create a field schema with default field values:
+
+```python
+from pymilvus import FieldSchema
+fields = [
+ FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
+ # configure default value `25` for field `age`
+ FieldSchema(name="age", dtype=DataType.INT64, default_value=25, description="age"),
+ embedding_field = FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=128, description="vector")
+]
+```
-### Supported data type
+### Supported data types
`DataType` defines the kind of data a field contains. Different fields support different data types.
@@ -81,14 +115,23 @@ embedding_field = FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim
- FLOAT: numpy.float32
- DOUBLE: numpy.double
- VARCHAR: VARCHAR
+ - JSON: [JSON](use-json-fields.md)
+ - Array: [Array](array_data_type.md)
+
+ JSON as a composite data type is available. A JSON field comprises key-value pairs. Each key is a string, and a value can be a number, string, boolean value, array, or list. For details, refer to [JSON: a new data type](use-json-fields.md).
+
- Vector field supports:
- - BINARY_VECTOR: Binary vector
- - FLOAT_VECTOR: Float vector
+ - BINARY_VECTOR: Stores binary data as a sequence of 0s and 1s, used for compact feature representation in image processing and information retrieval.
+ - FLOAT_VECTOR: Stores 32-bit floating-point numbers, commonly used in scientific computing and machine learning for representing real numbers.
+ - FLOAT16_VECTOR: Stores 16-bit half-precision floating-point numbers, used in deep learning and GPU computations for memory and bandwidth efficiency.
+ - BFLOAT16_VECTOR: Stores 16-bit floating-point numbers with reduced precision but the same exponent range as Float32, popular in deep learning for reducing memory and computational requirements without significantly impacting accuracy.
+ - SPARSE_FLOAT_VECTOR: Stores a list of non-zero elements and their corresponding indices, used for representing sparse vectors. For more information, refer to [Sparse Vectors](sparse_vector.md).
-## Collection schema
+ Milvus supports multiple vector fields in a collection. For more information, refer to [Hybrid Search](multi-vector-search.md).
-A collection schema is the logical definition of a collection. Usually you need to define the [field schema](#Field-schema) before defining a collection schema and [creating a collection](create_collection.md).
+## Collection schema
+A collection schema is the logical definition of a collection. Usually you need to define the [field schema](#Field-schema) before defining a collection schema and [managing collections](manage-collections.md).
### Collection schema properties
@@ -102,19 +145,24 @@ A collection schema is the logical definition of a collection. Usually you need
- field
+ Fields in the collection to create
Mandatory
- description
+ Description of the collection
Data type: String.
Optional
- auto_id
- Whether to enable Automatic ID (primary key) allocation or not
- Data type: Boolean (
Optional
+ Name of a field that is designed to act as the partition key.
+ Data type: String.
Optional
+
+
+ Whether to enable dynamic schema or not
+ Data type: Boolean (
Optional, defaults to
For details on dynamic schema, refer to Dynamic Schema and the user guides for managing collections.
@@ -130,7 +178,12 @@ from pymilvus import FieldSchema, CollectionSchema
id_field = FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, description="primary id")
age_field = FieldSchema(name="age", dtype=DataType.INT64, description="age")
embedding_field = FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=128, description="vector")
-schema = CollectionSchema(fields=[id_field, age_field, embedding_field], auto_id=False, description="desc of a collection")
+
+# Enable partition key on a field if you need to implement multi-tenancy based on the partition-key field
+position_field = FieldSchema(name="position", dtype=DataType.VARCHAR, max_length=256, is_partition_key=True)
+
+# Set enable_dynamic_field to True if you need to use dynamic fields.
+schema = CollectionSchema(fields=[id_field, age_field, embedding_field], auto_id=False, enable_dynamic_field=True, description="desc of a collection")
```
Create a collection with the schema specified:
@@ -141,8 +194,13 @@ collection_name1 = "tutorial_1"
collection1 = Collection(name=collection_name1, schema=schema, using='default', shards_num=2)
```
| Supported index | @@ -104,31 +121,101 @@ The following table classifies the indexes that Milvus supports:Graph-based index |
|
||
|---|---|---|---|---|
| ANNOY | -Tree-based index | +SCANN | +Quantization-based index |
|
+
+
+
+
+
+
+| Supported index | +Classification | +Scenario | +
|---|---|---|
| BIN_FLAT | +Quantization-based index | +
|
+
| BIN_IVF_FLAT | +Quantization-based index | +
|
+
+
+
+
+
+
+
+
+| Supported index | +Classification | +Scenario | +
|---|---|---|
| SPARSE_INVERTED_INDEX | +Inverted index | +
|
+
| SPARSE_WAND | +Inverted index | +
|
+
+
+### FLAT
For vector similarity search applications that require perfect accuracy and depend on relatively small (million-scale) datasets, the FLAT index is a good choice. FLAT does not compress vectors, and is the only index that can guarantee exact search results. Results from FLAT can also be used as a point of comparison for results produced by other indexes that have less than 100% recall.
-FLAT is accurate because it takes an exhaustive approach to search, which means for each query the target input is compared to every vector in a dataset. This makes FLAT the slowest index on our list, and poorly suited for querying massive vector data. There are no parameters for the FLAT index in Milvus, and using it does not require data training or additional storage.
+FLAT is accurate because it takes an exhaustive approach to search, which means for each query the target input is compared to every set of vectors in a dataset. This makes FLAT the slowest index on our list, and poorly suited for querying massive vector data. There are no parameters required for the FLAT index in Milvus, and using it does not need data training.
- Search parameters
@@ -138,56 +225,68 @@ FLAT is accurate because it takes an exhaustive approach to search, which means
### IVF_FLAT
-
IVF_FLAT divides vector data into `nlist` cluster units, and then compares distances between the target input vector and the center of each cluster. Depending on the number of clusters the system is set to query (`nprobe`), similarity search results are returned based on comparisons between the target input and the vectors in the most similar cluster(s) only — drastically reducing query time.
By adjusting `nprobe`, an ideal balance between accuracy and speed can be found for a given scenario. Results from the [IVF_FLAT performance test](https://zilliz.com/blog/Accelerating-Similarity-Search-on-Really-Big-Data-with-Vector-Indexing) demonstrate that query time increases sharply as both the number of target input vectors (`nq`), and the number of clusters to search (`nprobe`), increase.
IVF_FLAT is the most basic IVF index, and the encoded data stored in each unit is consistent with the original data.
- - Index building parameters
-
- | Parameter | Description | Range |
- | --------- | ----------------------- | ---------- |
- | `nlist` | Number of cluster units | [1, 65536] |
+- Index building parameters
+ | Parameter | Description | Range | Default Value |
+ | --------- | ----------------------- | ---------- | ------------- |
+ | `nlist` | Number of cluster units | [1, 65536] | 128 |
- Search parameters
- | Parameter | Description | Range |
- | --------- | ------------------------ | ----------------------------------------------- |
- | `nprobe` | Number of units to query | CPU: [1, nlist]
GPU: [1, min(2048, nlist)] | + - Common search -### IVF_SQ8 + | Parameter | Description | Range | Default Value | + |----------------------------|---------------------------------------------------------|------------|---------------| + | `nprobe` | Number of units to query | [1, nlist] | 8 | + + - Range search + | Parameter | Description | Range | Default Value | + |----------------------------|---------------------------------------------------------|------------|---------------| + | `max_empty_result_buckets` | Maximum number of buckets not returning any search results.
This is a range-search parameter and terminates the search process whilst the number of consecutive empty buckets reaches the specified value.
Increasing this value can improve recall rate at the cost of increased search time. | [1, 65535] | 2 | -IVF_FLAT does not perform any compression, so the index files it produces are roughly the same size as the original, raw non-indexed vector data. For example, if the original 1B SIFT dataset is 476 GB, its IVF_FLAT index files will be slightly larger (~470 GB). Loading all the index files into memory will consume 470 GB of storage. +### IVF_SQ8 -When disk, CPU, or GPU memory resources are limited, IVF_SQ8 is a better option than IVF_FLAT. This index type can convert each FLOAT (4 bytes) to UINT8 (1 byte) by performing scalar quantization. This reduces disk, CPU, and GPU memory consumption by 70–75%. For the 1B SIFT dataset, the IVF_SQ8 index files require just 140 GB of storage. +IVF_FLAT does not perform any compression, so the index files it produces are roughly the same size as the original, raw non-indexed vector data. For example, if the original 1B SIFT dataset is 476 GB, its IVF_FLAT index files will be slightly smaller (~470 GB). Loading all the index files into memory will consume 470 GB of storage. +When disk, CPU, or GPU memory resources are limited, IVF_SQ8 is a better option than IVF_FLAT. This index type can convert each FLOAT (4 bytes) to UINT8 (1 byte) by performing Scalar Quantization (SQ). This reduces disk, CPU, and GPU memory consumption by 70–75%. For the 1B SIFT dataset, the IVF_SQ8 index files require just 140 GB of storage. - - Index building parameters +- Index building parameters | Parameter | Description | Range | | --------- | ----------------------- | ---------- | | `nlist` | Number of cluster units | [1, 65536] | - - Search parameters - | Parameter | Description | Range | - | --------- | ------------------------ | ----------------------------------------------- | - | `nprobe` | Number of units to query | CPU: [1, nlist]
GPU: [1, min(2048, nlist)] | + - Common search -### IVF_PQ + | Parameter | Description | Range | Default Value | + | --------- | ------------------------ | --------------- | ------------- | + | `nprobe` | Number of units to query | [1, nlist] | 8 | + + - Range search + | Parameter | Description | Range | Default Value | + |----------------------------|---------------------------------------------------------|------------|---------------| + | `max_empty_result_buckets` | Maximum number of buckets not returning any search results.
This is a range-search parameter and terminates the search process whilst the number of consecutive empty buckets reaches the specified value.
Increasing this value can improve recall rate at the cost of increased search time. | [1, 65535] | 2 | + +### IVF_PQ `PQ` (Product Quantization) uniformly decomposes the original high-dimensional vector space into Cartesian products of `m` low-dimensional vector spaces, and then quantizes the decomposed low-dimensional vector spaces. Instead of calculating the distances between the target vector and the center of all the units, product quantization enables the calculation of distances between the target vector and the clustering center of each low-dimensional space and greatly reduces the time complexity and space complexity of the algorithm. IVF\_PQ performs IVF index clustering before quantizing the product of vectors. Its index file is even smaller than IVF\_SQ8, but it also causes a loss of accuracy during searching vectors.
This is a range-search parameter and terminates the search process whilst the number of consecutive empty buckets reaches the specified value.
Increasing this value can improve recall rate at the cost of increased search time. | [1, 65535] | 2 | -HNSW (Hierarchical Navigable Small World Graph) is a graph-based indexing algorithm. It builds a multi-layer navigation structure for an image according to certain rules. In this structure, the upper layers are more sparse and the distances between nodes are farther; the lower layers are denser and the distances between nodes are closer. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then enters the next layer to begin another search. After multiple iterations, it can quickly approach the target position. +### SCANN -In order to improve performance, HNSW limits the maximum degree of nodes on each layer of the graph to `M`. In addition, you can use `efConstruction` (when building index) or `ef` (when searching targets) to specify a search range. +SCANN (Score-aware quantization loss) is similar to IVF_PQ in terms of vector clustering and product quantization. What makes them different lies in the implementation details of product quantization and the use of SIMD (Single-Instruction / Multi-data) for efficient calculation. - Index building parameters - | Parameter | Description | Range | - | ---------------- | -------------------------- | -------- | - | `M` | Maximum degree of the node | [4, 64] | - | `efConstruction` | Search scope | [8, 512] | + | Parameter | Description | Range | + |-----------------|----------------------------------------------|----------------------------------------| + | `nlist` | Number of cluster units | [1, 65536] | + | `with_raw_data` | Whether to include the raw data in the index | `True` or `False`. Defaults to `True`. | +
This is a range-search parameter and terminates the search process whilst the number of consecutive empty buckets reaches the specified value.
Increasing this value can improve recall rate at the cost of increased search time. | [1, 65535] | 2 | -- `n_trees` is provided during build time and affects the build time and the index size. A larger value will give more accurate results, but larger indexes. +### HNSW -- `search_k` is provided in runtime and affects the search performance. A larger value will give more accurate results, but will take longer time to return. - -If `search_k` is not provided, it will default to `n * n_trees` where `n` is the number of approximate nearest neighbors. Otherwise, `search_k` and `n_trees` are roughly independent, i.e. the value of `n_trees` will not affect search time if `search_k` is held constant and vice versa. Basically it's recommended to set `n_trees` as large as possible given the amount of memory you can afford, and it's recommended to set `search_k` as large as possible given the time constraints you have for the queries. +HNSW (Hierarchical Navigable Small World Graph) is a graph-based indexing algorithm. It builds a multi-layer navigation structure for an image according to certain rules. In this structure, the upper layers are more sparse and the distances between nodes are farther; the lower layers are denser and the distances between nodes are closer. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then enters the next layer to begin another search. After multiple iterations, it can quickly approach the target position. +In order to improve performance, HNSW limits the maximum degree of nodes on each layer of the graph to `M`. In addition, you can use `efConstruction` (when building index) or `ef` (when searching targets) to specify a search range. - Index building parameters - | Parameter | Description | Range | - | --------- | ---------------------------------------- | --------- | - | `n_trees` | The number of trees. | [1, 1024] | + | Parameter | Description | Range | + | ---------------- | -------------------------- | ------------ | + | `M` | M defines tha maximum number of outgoing connections in the graph. Higher M leads to higher accuracy/run_time at fixed ef/efConstruction. | [2, 2048] | + | `efConstruction` | ef_construction controls index search speed/build speed tradeoff. Increasing the efConstruction parameter may enhance index quality, but it also tends to lengthen the indexing time. | [1, int_max] | - Search parameters - | Parameter | Description | Range | - | ---------- | ------------------------------------------------------------ | ------------------------------- | - | `search_k` | The parameters that controls the search scope. | [k, inf] | + | Parameter | Description | Range | + | --------- | ------------ | ---------------- | + | `ef` | Parameter controlling query time/accuracy trade-off. Higher `ef` leads to more accurate but slower search. | [`top_k`, int_max] |
-GPU: [1, min(2048, nlist)] | + - Common search -### IVF_SQ8 + | Parameter | Description | Range | Default Value | + |----------------------------|---------------------------------------------------------|------------|---------------| + | `nprobe` | Number of units to query | [1, nlist] | 8 | + + - Range search + | Parameter | Description | Range | Default Value | + |----------------------------|---------------------------------------------------------|------------|---------------| + | `max_empty_result_buckets` | Maximum number of buckets not returning any search results.
This is a range-search parameter and terminates the search process whilst the number of consecutive empty buckets reaches the specified value.
Increasing this value can improve recall rate at the cost of increased search time. | [1, 65535] | 2 | -IVF_FLAT does not perform any compression, so the index files it produces are roughly the same size as the original, raw non-indexed vector data. For example, if the original 1B SIFT dataset is 476 GB, its IVF_FLAT index files will be slightly larger (~470 GB). Loading all the index files into memory will consume 470 GB of storage. +### IVF_SQ8 -When disk, CPU, or GPU memory resources are limited, IVF_SQ8 is a better option than IVF_FLAT. This index type can convert each FLOAT (4 bytes) to UINT8 (1 byte) by performing scalar quantization. This reduces disk, CPU, and GPU memory consumption by 70–75%. For the 1B SIFT dataset, the IVF_SQ8 index files require just 140 GB of storage. +IVF_FLAT does not perform any compression, so the index files it produces are roughly the same size as the original, raw non-indexed vector data. For example, if the original 1B SIFT dataset is 476 GB, its IVF_FLAT index files will be slightly smaller (~470 GB). Loading all the index files into memory will consume 470 GB of storage. +When disk, CPU, or GPU memory resources are limited, IVF_SQ8 is a better option than IVF_FLAT. This index type can convert each FLOAT (4 bytes) to UINT8 (1 byte) by performing Scalar Quantization (SQ). This reduces disk, CPU, and GPU memory consumption by 70–75%. For the 1B SIFT dataset, the IVF_SQ8 index files require just 140 GB of storage. - - Index building parameters +- Index building parameters | Parameter | Description | Range | | --------- | ----------------------- | ---------- | | `nlist` | Number of cluster units | [1, 65536] | - - Search parameters - | Parameter | Description | Range | - | --------- | ------------------------ | ----------------------------------------------- | - | `nprobe` | Number of units to query | CPU: [1, nlist]
GPU: [1, min(2048, nlist)] | + - Common search -### IVF_PQ + | Parameter | Description | Range | Default Value | + | --------- | ------------------------ | --------------- | ------------- | + | `nprobe` | Number of units to query | [1, nlist] | 8 | + + - Range search + | Parameter | Description | Range | Default Value | + |----------------------------|---------------------------------------------------------|------------|---------------| + | `max_empty_result_buckets` | Maximum number of buckets not returning any search results.
This is a range-search parameter and terminates the search process whilst the number of consecutive empty buckets reaches the specified value.
Increasing this value can improve recall rate at the cost of increased search time. | [1, 65535] | 2 | + +### IVF_PQ `PQ` (Product Quantization) uniformly decomposes the original high-dimensional vector space into Cartesian products of `m` low-dimensional vector spaces, and then quantizes the decomposed low-dimensional vector spaces. Instead of calculating the distances between the target vector and the center of all the units, product quantization enables the calculation of distances between the target vector and the clustering center of each low-dimensional space and greatly reduces the time complexity and space complexity of the algorithm. IVF\_PQ performs IVF index clustering before quantizing the product of vectors. Its index file is even smaller than IVF\_SQ8, but it also causes a loss of accuracy during searching vectors.
+
Index building parameters and search parameters vary with Milvus distribution. Select your Milvus distribution first.
+
- Index building parameters
@@ -195,107 +294,85 @@ Index building parameters and search parameters vary with Milvus distribution. S
| Parameter | Description | Range |
| --------- | ----------------------------------------- | ------------------- |
| `nlist` | Number of cluster units | [1, 65536] |
- | `m` | Number of factors of product quantization | `dim` ≡ 0 (mod `m`) |
- | `nbits` | [Optional] Number of bits in which each low-dimensional vector is stored. | [1, 16] (8 by default) |
+ | `m` | Number of factors of product quantization | `dim mod m == 0` |
+ | `nbits` | [Optional] Number of bits in which each low-dimensional vector is stored. | [1, 64] (8 by default) |
- Search parameters
- | Parameter | Description | Range |
- | --------- | ------------------------ | ---------- |
- | `nprobe` | Number of units to query | [1, nlist] |
+ - Common search
+ | Parameter | Description | Range | Default Value |
+ | --------- | ------------------------ | --------------- | ------------- |
+ | `nprobe` | Number of units to query | [1, nlist] | 8 |
+ - Range search
-### HNSW
+ | Parameter | Description | Range | Default Value |
+ |----------------------------|---------------------------------------------------------|------------|---------------|
+ | `max_empty_result_buckets` | Maximum number of buckets not returning any search results.This is a range-search parameter and terminates the search process whilst the number of consecutive empty buckets reaches the specified value.
Increasing this value can improve recall rate at the cost of increased search time. | [1, 65535] | 2 | -HNSW (Hierarchical Navigable Small World Graph) is a graph-based indexing algorithm. It builds a multi-layer navigation structure for an image according to certain rules. In this structure, the upper layers are more sparse and the distances between nodes are farther; the lower layers are denser and the distances between nodes are closer. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then enters the next layer to begin another search. After multiple iterations, it can quickly approach the target position. +### SCANN -In order to improve performance, HNSW limits the maximum degree of nodes on each layer of the graph to `M`. In addition, you can use `efConstruction` (when building index) or `ef` (when searching targets) to specify a search range. +SCANN (Score-aware quantization loss) is similar to IVF_PQ in terms of vector clustering and product quantization. What makes them different lies in the implementation details of product quantization and the use of SIMD (Single-Instruction / Multi-data) for efficient calculation. - Index building parameters - | Parameter | Description | Range | - | ---------------- | -------------------------- | -------- | - | `M` | Maximum degree of the node | [4, 64] | - | `efConstruction` | Search scope | [8, 512] | + | Parameter | Description | Range | + |-----------------|----------------------------------------------|----------------------------------------| + | `nlist` | Number of cluster units | [1, 65536] | + | `with_raw_data` | Whether to include the raw data in the index | `True` or `False`. Defaults to `True`. | +
-- Search parameters
+ Unlike IVF_PQ, default values apply to `m` and `nbits` for optimized performance.
- | Parameter | Description | Range |
- | --------- | ------------ | ---------------- |
- | `ef` | Search scope | [`top_k`, 32768] |
+
+- Search parameters
-### ANNOY
+ - Common search
+ | Parameter | Description | Range | Default value |
+ | --------- | ------------------------ | ---------- | ------------- |
+ | `nprobe` | Number of units to query | [1, nlist] | |
+ | `reorder_k` | Number of candidate units to query | [`top_k`, ∞] | |
-ANNOY (Approximate Nearest Neighbors Oh Yeah) is an index that uses a hyperplane to divide a high-dimensional space into multiple subspaces, and then stores them in a tree structure.
+ - Range search
-There are just two main parameters needed to tune ANNOY: the number of trees `n_trees` and the number of nodes to inspect during searching `search_k`.
+ | Parameter | Description | Range | Default Value |
+ |----------------------------|---------------------------------------------------------|------------|---------------|
+ | `max_empty_result_buckets` | Maximum number of buckets not returning any search results.This is a range-search parameter and terminates the search process whilst the number of consecutive empty buckets reaches the specified value.
Increasing this value can improve recall rate at the cost of increased search time. | [1, 65535] | 2 | -- `n_trees` is provided during build time and affects the build time and the index size. A larger value will give more accurate results, but larger indexes. +### HNSW -- `search_k` is provided in runtime and affects the search performance. A larger value will give more accurate results, but will take longer time to return. - -If `search_k` is not provided, it will default to `n * n_trees` where `n` is the number of approximate nearest neighbors. Otherwise, `search_k` and `n_trees` are roughly independent, i.e. the value of `n_trees` will not affect search time if `search_k` is held constant and vice versa. Basically it's recommended to set `n_trees` as large as possible given the amount of memory you can afford, and it's recommended to set `search_k` as large as possible given the time constraints you have for the queries. +HNSW (Hierarchical Navigable Small World Graph) is a graph-based indexing algorithm. It builds a multi-layer navigation structure for an image according to certain rules. In this structure, the upper layers are more sparse and the distances between nodes are farther; the lower layers are denser and the distances between nodes are closer. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then enters the next layer to begin another search. After multiple iterations, it can quickly approach the target position. +In order to improve performance, HNSW limits the maximum degree of nodes on each layer of the graph to `M`. In addition, you can use `efConstruction` (when building index) or `ef` (when searching targets) to specify a search range. - Index building parameters - | Parameter | Description | Range | - | --------- | ---------------------------------------- | --------- | - | `n_trees` | The number of trees. | [1, 1024] | + | Parameter | Description | Range | + | ---------------- | -------------------------- | ------------ | + | `M` | M defines tha maximum number of outgoing connections in the graph. Higher M leads to higher accuracy/run_time at fixed ef/efConstruction. | [2, 2048] | + | `efConstruction` | ef_construction controls index search speed/build speed tradeoff. Increasing the efConstruction parameter may enhance index quality, but it also tends to lengthen the indexing time. | [1, int_max] | - Search parameters - | Parameter | Description | Range | - | ---------- | ------------------------------------------------------------ | ------------------------------- | - | `search_k` | The parameters that controls the search scope. | [k, inf] | + | Parameter | Description | Range | + | --------- | ------------ | ---------------- | + | `ef` | Parameter controlling query time/accuracy trade-off. Higher `ef` leads to more accurate but slower search. | [`top_k`, int_max] |
-
-
-
+
| Supported index | -Classification | -Scenario | -
|---|---|---|
| BIN_FLAT | -N/A | -
-
|
-
| BIN_IVF_FLAT | -Quantization-based index | -
-
|
-
### BIN_FLAT
-
This index is exactly the same as FLAT except that this can only be used for binary embeddings.
For vector similarity search applications that require perfect accuracy and depend on relatively small (million-scale) datasets, the BIN_FLAT index is a good choice. BIN_FLAT does not compress vectors, and is the only index that can guarantee exact search results. Results from BIN_FLAT can also be used as a point of comparison for results produced by other indexes that have less than 100% recall.
-BIN_FLAT is accurate because it takes an exhaustive approach to search, which means for each query the target input is compared to every vector in a dataset. This makes BIN_FLAT the slowest index on our list, and poorly suited for querying massive vector data. There are no parameters for the BIN_FLAT index in Milvus, and using it does not require data training or additional storage.
+BIN_FLAT is accurate because it takes an exhaustive approach to search, which means for each query the target input is compared to vectors in a dataset. This makes BIN_FLAT the slowest index on our list, and poorly suited for querying massive vector data. There are no parameters for the BIN_FLAT index in Milvus, and using it does not require data training or additional storage.
- Search parameters
@@ -305,7 +382,6 @@ BIN_FLAT is accurate because it takes an exhaustive approach to search, which me
### BIN_IVF_FLAT
-
This index is exactly the same as IVF_FLAT except that this can only be used for binary embeddings.
BIN_IVF_FLAT divides vector data into `nlist` cluster units, and then compares distances between the target input vector and the center of each cluster. Depending on the number of clusters the system is set to query (`nprobe`), similarity search results are returned based on comparisons between the target input and the vectors in the most similar cluster(s) only — drastically reducing query time.
@@ -314,25 +390,68 @@ By adjusting `nprobe`, an ideal balance between accuracy and speed can be found
BIN_IVF_FLAT is the most basic BIN_IVF index, and the encoded data stored in each unit is consistent with the original data.
- - Index building parameters
+- Index building parameters
| Parameter | Description | Range |
| --------- | ----------------------- | ---------- |
| `nlist` | Number of cluster units | [1, 65536] |
+- Search parameters
+
+ - Common search
+
+ | Parameter | Description | Range | Default Value |
+ | --------- | ------------------------ | --------------- | ------------- |
+ | `nprobe` | Number of units to query | [1, nlist] | 8 |
+
+ - Range search
+
+ | Parameter | Description | Range | Default Value |
+ |----------------------------|---------------------------------------------------------|------------|---------------|
+ | `max_empty_result_buckets` | Maximum number of buckets not returning any search results.
This is a range-search parameter and terminates the search process whilst the number of consecutive empty buckets reaches the specified value.
Increasing this value can improve recall rate at the cost of increased search time. | [1, 65535] | 2 | + +
+
+This is a range-search parameter and terminates the search process whilst the number of consecutive empty buckets reaches the specified value.
Increasing this value can improve recall rate at the cost of increased search time. | [1, 65535] | 2 | + +
+
+### SPARSE_INVERTED_INDEX
+
+Each dimension maintains a list of vectors that have a non-zero value at that dimension. During search, Milvus iterates through each dimension of the query vector and computes scores for vectors that have non-zero values in those dimensions.
+
+- Index building parameters
+
+ | Parameter | Description | Range |
+ | ---------------- | -------------------------- | ------------ |
+ | `drop_ratio_build` | The proportion of small vector values that are excluded during the indexing process. This option allows fine-tuning of the indexing process, making a trade-off between efficiency and accuracy by disregarding small values when building the index. | [0, 1] |
- Search parameters
- | Parameter | Description | Range |
- | --------- | ------------------------ | ----------------------------------------------- |
- | `nprobe` | Number of units to query | CPU: [1, nlist]
GPU: [1, min(2048, nlist)] | + | Parameter | Description | Range | + |---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------| + | `drop_ratio_search` | The proportion of small vector values that are excluded during the search process. This option allows fine-tuning of the search process by specifying the ratio of the smallest values in the query vector to ignore. It helps balance search precision and performance. The smaller the value set for `drop_ratio_search`, the less these small values contribute to the final score. By ignoring some small values, search performance can be improved with minimal impact on accuracy. | [0, 1] | + +### SPARSE_WAND +This index shares similarities with `SPARSE_INVERTED_INDEX`, while it utilizes the [Weak-AND](https://dl.acm.org/doi/10.1145/956863.956944) algorithm to further reduce the number of full IP distance evaluations during the search process. + +Based on our testing, `SPARSE_WAND` generally outperforms other methods in terms of speed. However, its performance can deteriorate rapidly as the density of the vectors increases. To address this issue, introducing a non-zero `drop_ratio_search` can significantly enhance performance while only incurring minimal accuracy loss. For more information, refer to [Sparse Vector](sparse_vector.md). + +- Index building parameters + + | Parameter | Description | Range | + | ---------------- | -------------------------- | ------------ | + | `drop_ratio_build` | The proportion of small vector values that are excluded during the indexing process. This option allows fine-tuning of the indexing process, making a trade-off between efficiency and accuracy by disregarding small values when building the index. | [0, 1] | + +- Search parameters + + | Parameter | Description | Range | + |---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------| + | `drop_ratio_search` | The proportion of small vector values that are excluded during the search process. This option allows fine-tuning of the search process by specifying the ratio of the smallest values in the query vector to ignore. It helps balance search precision and performance. The smaller the value set for `drop_ratio_search`, the less these small values contribute to the final score. By ignoring some small values, search performance can be improved with minimal impact on accuracy. | [0, 1] |
## FAQ
-
GPU: [1, min(2048, nlist)] | + | Parameter | Description | Range | + |---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------| + | `drop_ratio_search` | The proportion of small vector values that are excluded during the search process. This option allows fine-tuning of the search process by specifying the ratio of the smallest values in the query vector to ignore. It helps balance search precision and performance. The smaller the value set for `drop_ratio_search`, the less these small values contribute to the final score. By ignoring some small values, search performance can be improved with minimal impact on accuracy. | [0, 1] | + +### SPARSE_WAND +This index shares similarities with `SPARSE_INVERTED_INDEX`, while it utilizes the [Weak-AND](https://dl.acm.org/doi/10.1145/956863.956944) algorithm to further reduce the number of full IP distance evaluations during the search process. + +Based on our testing, `SPARSE_WAND` generally outperforms other methods in terms of speed. However, its performance can deteriorate rapidly as the density of the vectors increases. To address this issue, introducing a non-zero `drop_ratio_search` can significantly enhance performance while only incurring minimal accuracy loss. For more information, refer to [Sparse Vector](sparse_vector.md). + +- Index building parameters + + | Parameter | Description | Range | + | ---------------- | -------------------------- | ------------ | + | `drop_ratio_build` | The proportion of small vector values that are excluded during the indexing process. This option allows fine-tuning of the indexing process, making a trade-off between efficiency and accuracy by disregarding small values when building the index. | [0, 1] | + +- Search parameters + + | Parameter | Description | Range | + |---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------| + | `drop_ratio_search` | The proportion of small vector values that are excluded during the search process. This option allows fine-tuning of the search process by specifying the ratio of the smallest values in the query vector to ignore. It helps balance search precision and performance. The smaller the value set for `drop_ratio_search`, the less these small values contribute to the final score. By ignoring some small values, search performance can be improved with minimal impact on accuracy. | [0, 1] |
What is the difference between FLAT index and IVF_FLAT index?
IVF_FLAT index divides a vector space into nlist clusters. If you keep the default value of nlist as 16384, Milvus compares the distances between the target vector and the centers of all 16384 clusters to get nprobe nearest clusters. Then Milvus compares the distances between the target vector and the vectors in the selected clusters to get the nearest vectors. Unlike IVF_FLAT, FLAT directly compares the distances between the target vector and each and every vector.
@@ -345,7 +464,6 @@ See
-Floating point embeddings Binary embeddings
-
+ Floating point embeddings
+ Binary embeddings
+ Sparse embeddings
 supported in Milvus.
diff --git a/preview/site/en/reference/knowhere.md b/preview/site/en/reference/knowhere.md
index 5326c12e3..ed06e11f0 100644
--- a/preview/site/en/reference/knowhere.md
+++ b/preview/site/en/reference/knowhere.md
@@ -1,6 +1,7 @@
---
id: knowhere.md
summary: Learn about Knowhere in Milvus.
+title: Knowhere
---
# Knowhere
@@ -81,7 +82,7 @@ However, for code consistency, `IDMAP` also inherits from the `VecIndex` class w
#### IVF indexes
-
@@ -20,14 +22,14 @@ The following table shows how these widely used similarity metrics fit with vari
@@ -39,18 +41,33 @@ The following table shows how these widely used similarity metrics fit with vari
+
+
+
+| Similarity Metrics | +Metric Types | Index Types | |
|---|---|---|---|
| Distance Metrics | +Metric Types | Index Types |
|---|---|---|
+
+
+
+
@@ -58,7 +75,6 @@ The following table shows how these widely used similarity metrics fit with vari
-
### Euclidean distance (L2)
Essentially, Euclidean distance measures the length of a segment that connects 2 points.
@@ -67,7 +83,7 @@ The formula for Euclidean distance is as follows:
-where **a** = (a1, a2,..., an) and **b** = (b1, b2,..., bn) are two points in n-dimensional Euclidean space
+where **a** = (a0, a1,..., an-1) and **b** = (b0, b0,..., bn-1) are two points in n-dimensional Euclidean space
It's the most commonly used distance metric and is very useful when the data are continuous.
@@ -81,46 +97,43 @@ The IP distance between two embeddings are defined as follows:
-Where A and B are embeddings, `||A||` and `||B||` are the norms of A and B.
-
-IP is more useful if you are more interested in measuring the orientation but not the magnitude of the vectors.
+IP is more useful if you need to compare non-normalized data or when you care about magnitude and angle.
| Metric Types | +Index Types | +||
|---|---|---|---|
| BIN_FLAT | +IP | +
- If you use IP to calculate embeddings similarities, you must normalize your embeddings. After normalization, the inner product equals cosine similarity.
-
+ If you apply the IP distance metric to normalized embeddings, the result will be equivalent to calculating the cosine similarity between the embeddings.
+
+
Suppose X' is normalized from embedding X:
-The correlation between the two embeddings is as follows:
+The correlation between the two embeddings is as follows:
-### Jaccard distance
+### Cosine Similarity
-Jaccard similarity coefficient measures the similarity between two sample sets and is defined as the cardinality of the intersection of the defined sets divided by the cardinality of the union of them. It can only be applied to finite sample sets.
+Cosine similarity uses the cosine of the angle between two sets of vectors to measure how similar they are. You can think of the two sets of vectors as two line segments that start from the same origin ([0,0,...]) but point in different directions.
-
+To calculate the cosine similarity between two sets of vectors **A = (a0, a1,..., an-1)** and **B = (b0, b1,..., bn-1)**, use the following formula:
-Jaccard distance measures the dissimilarity between data sets and is obtained by subtracting the Jaccard similarity coefficient from 1. For binary variables, Jaccard distance is equivalent to the Tanimoto coefficient.
+
-
+The cosine similarity is always in the interval **[-1, 1]**. For example, two proportional vectors have a cosine similarity of **1**, two orthogonal vectors have a similarity of **0**, and two opposite vectors have a similarity of **-1**. The larger the cosine, the smaller the angle between two vectors, indicating that these two vectors are more similar to each other.
-### Tanimoto distance
+By subtracting their cosine similarity from 1, you can get the cosine distance between two vectors.
-For binary variables, the Tanimoto coefficient is equivalent to Jaccard distance:
-
-
+### Jaccard distance
-In Milvus, the Tanimoto coefficient is only applicable for a binary variable, and for binary variables, the Tanimoto coefficient ranges from 0 to +1 (where +1 is the highest similarity).
+Jaccard similarity coefficient measures the similarity between two sample sets and is defined as the cardinality of the intersection of the defined sets divided by the cardinality of the union of them. It can only be applied to finite sample sets.
-For binary variables, the formula of Tanimoto distance is:
+
-
+Jaccard distance measures the dissimilarity between data sets and is obtained by subtracting the Jaccard similarity coefficient from 1. For binary variables, Jaccard distance is equivalent to the Tanimoto coefficient.
-The value ranges from 0 to +infinity.
+
### Hamming distance
@@ -130,35 +143,33 @@ For example, suppose there are two strings, 1101 1001 and 1001 1101.
11011001 ⊕ 10011101 = 01000100. Since, this contains two 1s, the Hamming distance, d (11011001, 10011101) = 2.
-### Superstructure
+### Structural Similarity
+
+When a chemical structure occurs as a part of a larger chemical structure, the former is called a substructure and the latter is called a superstructure. For example, ethanol is a substructure of acetic acid, and acetic acid is a superstructure of ethanol.
-The Superstructure is used to measure the similarity of a chemical structure and its superstructure. When the value equals 0, this means the chemical structure in the database is the superstructure of the target chemical structure.
+Structural similarity is used to determine whether two chemical formulae are similar to each other in that one is the superstructure or substructure of the other.
-Superstructure similarity can be measured by:
+To determine whether A is a superstructure of B, use the following formula:
-
+
-Where
+Where:
-- B is the superstructure of A
-- NA specifies the number of bits in the fingerprint of molecular A.
-- NB specifies the number of bits in the fingerprint of molecular B.
-- NAB specifies the number of shared bits in the fingerprint of molecular A and B.
+- A is the binary representation of a chemical formula to be retrieved
+- B is the binary representation of a chemical formula in the database
-### Substructure
+Once it returns `0`, **A** is not a superstructure of **B**. Otherwise, the result is the other way around.
-The Substructure is used to measure the similarity of a chemical structure and its substructure. When the value equals 0, this means the chemical structure in the database is the substructure of the target chemical structure.
+To determine whether A is a substructure of B, use the following formula:
-Substructure similarity can be measured by:
+
-
+Where:
-Where
+- A is the binary representation of a chemical formula to be retrieved
+- B is the binary representation of a chemical formula in the database
-- B is the substructure of A
-- NA specifies the number of bits in the fingerprint of molecular A.
-- NB specifies the number of bits in the fingerprint of molecular B.
-- NAB specifies the number of shared bits in the fingerprint of molecular A and B.
+Once it returns `0`, **A** is not a substructure of **B**. Otherwise, the result is the other way around.
## FAQ
diff --git a/preview/site/en/reference/mmap.md b/preview/site/en/reference/mmap.md
new file mode 100644
index 000000000..a4b8c3cbc
--- /dev/null
+++ b/preview/site/en/reference/mmap.md
@@ -0,0 +1,136 @@
+---
+id: mmap.md
+summary: MMap enables more data in a single node.
+title: MMap-enabled Data Storage
+---
+
+# MMap-enabled Data Storage
+
+In Milvus, memory-mapped files allow for direct mapping of file contents into memory. This feature enhances memory efficiency, particularly in situations where available memory is scarce but complete data loading is infeasible. This optimization mechanism can increase data capacity while ensuring performance up to a certain limit; however, when the amount of data exceeds memory by too much, search and query performance may suffer serious degradation, so please choose to turn this feature on or off as appropriate.
+
+## Configure memory mapping
+
+Starting with Milvus 2.4, you have the flexibility to adjust the static configuration file to configure default memory mapping settings for the entire cluster before deployment. Additionally, there's the option for you to dynamically alter parameters to fine-tune memory mapping settings at both the cluster and index levels. Looking ahead, future updates will extend memory mapping capabilities to include field-level configurations.
+
+### Before cluster deployment: global configuration
+
+Before you deploy a cluster, __cluster-level__ settings apply memory mapping across your entire cluster. This ensures all new objects will automatically adhere to these configurations. It's important to note that modifying these settings requires a restart of the cluster to become effective.
+
+To adjust your cluster's memory mapping settings, edit the `configs/milvus.yaml` file. Within this file, you can specify whether to enable memory mapping by default and determine the directory path for storing memory-mapped files. If the path (`mmapDirPath`) is left unspecified, the system defaults to storing memory-mapped files in `{localStorage.path}/mmap`. For more information, refer to [Local Storage-related Configurations](https://milvus.io/docs/configure_localstorage.md#localStoragepath).
+
+```yaml
+# This parameter was set in configs/milvus.yaml
+...
+queryNode:
+ mmap:
+ # Set memory mapping property for whole cluster
+ mmapEnabled: false | true
+ # Set memory-mapped directory path, if you leave mmapDirPath unspecified, the memory-mapped files will be stored in {localStorage.path}/ mmap by default.
+ mmapDirPath: any/valid/path
+....
+```
+
+### During cluster operation: dynamic configuration
+
+During cluster runtime, you can dynamically adjust memory mapping settings at either the collection or index level.
+
+At the __collection level__, memory mapping is applied to all unindexed raw data within a collection, excluding primary keys, timestamps, and row IDs. This approach is particularly suited for comprehensive management of large datasets.
+
+For dynamic adjustments to memory mapping settings within a collection, utilize the `set_properties()` method. Here, you can toggle `mmap.enabled` between `True` or `False` as needed.
+
+```python
+# Get existing collection
+collection = Collection("test_collection") # Replace with your collection name
+
+# Set memory mapping property to True or Flase
+collection.set_properties({'mmap.enabled': True})
+```
+
+For __index-level__ settings, memory mapping can be specifically applied to vector indexes without affecting other data types. This feature is invaluable for collections that require optimized performance for vector searches.
+
+To enable or disable memory mapping for an index within a collection, call the `alter_index()` method, specifying the target index name in `index_name` and setting `mmap.enabled` to `True` or `False`.
+
+```python
+collection.alter_index(
+ index_name="vector_index", # Replace with your vector index name
+ extra_params={"mmap.enabled": True} # Enable memory mapping for index
+)
+```
+
+## Customize storage path in different deployments
+
+Memory-mapped files default to the `/mmap` directory within `localStorage.path`. Here's how to customize this setting across various deployment methods:
+
+- For Milvus installed using Helm Chart:
+
+```bash
+# new-values.yaml
+extraConfigFiles:
+ user.yaml: |+
+ queryNode:
+ mmap:
+ mmapEnabled: true
+ mmapDirPath: any/valid/path
+
+helm upgrade
+
+Reranking in hybrid search is a crucial step that consolidates results from several vector fields, ensuring the final output is relevant and accurately prioritized. Currently, Milvus offers these reranking strategies:
+
+- `WeightedRanker`: This approach merges results by calculating a weighted average of scores (or vector distances) from different vector searches. It assigns weights based on the significance of each vector field.
+
+- `RRFRanker`: This strategy combines results based on their ranks across different vector columns.
+
+## Weighted Scoring (WeightedRanker)
+
+The `WeightedRanker` strategy assigns different weights to results from each vector retrieval route based on the significance of each vector field. This reranking strategy is applied when the significance of each vector field varies, allowing you to emphasize certain vector fields over others by assigning them higher weights. For example, in a multimodal search, the text description might be considered more important than the color distribution in images.
+
+WeightedRanker’s basic process is as follows:
+
+- **Collect Scores During Retrieval**: Gather results and their scores from different vector retrieval routes.
+- **Score Normalization**: Normalize the scores from each route to a [0,1] range, where values closer to 1 indicate higher relevance. This normalization is crucial due to score distributions varying with different metric types. For instance, the distance for IP ranges from [-∞,+∞], while the distance for L2 ranges from [0,+∞]. Milvus employs the `arctan` function, transforming values to the [0,1] range to provide a standardized basis for different metric types.
+
+
+
+- **Weight Allocation**: Assign a weight `w𝑖` to each vector retrieval route. Users specify the weights, which reflect the data source's reliability, accuracy, or other pertinent metrics. Each weight ranges from [0,1].
+- **Score Fusion**: Calculate a weighted average of the normalized scores to derive the final score. The results are then ranked based on these highest to lowest scores to generate the final sorted results.
+
+
+
+To use this strategy, apply a `WeightedRanker` instance and set weight values by passing in a variable number of numeric arguments.
+
+```python
+from pymilvus import WeightedRanker
+
+# Use WeightedRanker to combine results with specified weights
+rerank = WeightedRanker(0.8, 0.8, 0.7)
+```
+
+Note that:
+
+- Each weight value ranges from 0 (least important) to 1 (most important), influencing the final aggregated score.
+
+- The total number of weight values provided in `WeightedRanker` should equal the number of `AnnSearchRequest` instances you have created earlier.
+
+- It is worth noting that due to the different measurements of the different metric types, we normalize the distances of the recall results so that they lie in the interval [0,1], where 0 means different and 1 means similar. The final score will be the sum of the weight values and distances.
+
+## Reciprocal Rank Fusion (RRFRanker)
+
+RRF is a data fusion method that combines ranking lists based on the reciprocal of their ranks. It is an effective way to balance the influence of each vector field, especially when there is no clear precedence of importance. This strategy is typically used when you want to give equal consideration to all vector fields or when there is uncertainty about the relative importance of each field.
+
+RRF’s basic process is as follows:
+
+- **Collect Rankings During Retrieval**: Retrievers across multiple vector fields retrieve and sort results.
+- **Rank Fusion**: The RRF algorithm weighs and combines the ranks from each retriever. The formula is as follows:
+
+ 
+
+ Here, 𝑁 represents the number of different retrieval routes, rank𝑖(𝑑) is the rank position of retrieved document 𝑑 by the 𝑖th retriever, and 𝑘 is a smoothing parameter, typically set to 60.
+- **Comprehensive Ranking**: Re-rank the retrieved results based on the combined scores to produce the final results.
+
+To use this strategy, apply an `RRFRanker` instance.
+
+```python
+from pymilvus import RRFRanker
+
+# Default k value is 60
+ranker = RRFRanker()
+
+# Or specify k value
+ranker = RRFRanker(k=100)
+```
+
+RRF allows balancing influence across fields without specifying explicit weights. The top matches agreed upon by multiple fields will be prioritized in the final ranking.
diff --git a/preview/site/en/reference/scalar_index.md b/preview/site/en/reference/scalar_index.md
index 12c592c4b..c9a9caec8 100644
--- a/preview/site/en/reference/scalar_index.md
+++ b/preview/site/en/reference/scalar_index.md
@@ -2,43 +2,69 @@
id: scalar_index.md
related_key: scalar_index
summary: Scalar index in Milvus.
+title: Scalar Index
---
# Scalar Index
-Milvus supports [hybrid searches](hybridsearch.md) using both scalar and vector fields. To speed up searching among entities by scalar fields, Milvus introduced scalar field indexing in version 2.1.0. This article helps you understand scalar field indexing in Milvus.
+Milvus supports filtered searches combining both scalar and vector fields. To enhance the efficiency of searches involving scalar fields, Milvus introduced scalar field indexing starting from version 2.1.0. This article provides an overview of scalar field indexing in Milvus, helping you understand its significance and implementation.
## Overview
-Once conducting vector similarity searches in Milvus, you can use logical operators to organize scalar fields into boolean expressions.
+Once conducting vector similarity searches in Milvus, you can use logical operators to organize scalar fields into boolean expressions.
When Milvus receives a search request with such a boolean expression, it parses the boolean expression into an abstract syntax tree (AST) to generate a physical plan for attribute filtering. Milvus then applies the physical plan in each segment to generate a [bitset](bitset.md) as the filtering result and includes the result as a vector search parameter to narrow down the search scope. In this case, the speed of vector searches relies heavily on the speed of attribute filtering.
-Scalar field indexing is a way of ensuring the speed of attribute filtering by sorting scalar field values in a particular way to accelerate information retrieval.
+Scalar field indexing is a way of ensuring the speed of attribute filtering by sorting scalar field values in a particular way to accelerate information retrieval.
## Scalar field indexing algorithms
-Milvus implements scalar field indexing with the goal of low memory usage, high filtering efficiency, and short loading time.
+Milvus aims to achieve low memory usage, high filtering efficiency, and short loading time with its scalar field indexing algorithms. These algorithms are categorized into two main types: [auto indexing](#auto-indexing) and [inverted indexing](#inverted-indexing).
-Specifically, indexing algorithms for scalar fields vary with field data types. However, you do not need to set the index type manually when indexing a scalar field. The default indexing algorithm applies.
+### Auto indexing
-The following table lists the data types that Milvus supports and their corresponding default indexing algorithms.
+Milvus automatically creates an auto index for a scalar field based on its data type, without requiring manual intervention. Auto indexing is suitable for prefix match queries and frequent retrieval scenarios.
-| Data type | Default indexing algorithm |
-| ----------------------------- | --------------------------------- |
-| BOOL | STL sort |
-| VARCHAR | MARISA-trie |
-| INT16 | STL sort |
-| INT32 | STL sort |
-| INT64 | STL sort |
-| FLOAT | STL sort |
-| DOUBLE | STL sort |
+The following table lists the data types that Milvus supports and their corresponding auto indexing algorithms.
+
+| Data type | Auto indexing algorithm |
+|-----------|----------------------------|
+| VARCHAR | Inverted index |
+| INT8 | Inverted index |
+| INT16 | Inverted index |
+| INT32 | Inverted index |
+| INT64 | Inverted index |
+| FLOAT | Inverted index |
+| DOUBLE | Inverted index |
+
+### Inverted indexing
+
+Inverted indexing offers a flexible way to create an index for a scalar field by manually specifying index parameters. This method works well for various scenarios, including point queries, pattern match queries, full-text searches, JSON searches, Boolean searches, and even prefix match queries.
+
+The inverted indexes implemented in Milvus are powered by [Tantivy](https://github.com/quickwit-oss/tantivy), a full-text search engine library. Tantivy ensures that inverted indexing in Milvus is both efficient and fast.
+
+An inverted index has two main components: a term dictionary and an inverted list. The term dictionary includes all tokenized words sorted alphabetically, while the inverted list contains the list of documents where each word appears. This setup makes point queries and range queries much faster and more efficient than brute-force searches.
+
+
+
+The advantages of using an inverted index are particularly evident in the following operations:
+
+- **Point query**: For example, when searching for documents containing the word **Milvus**, the process begins by checking if **Milvus** is present in the term dictionary. If it is not found, no documents contain the word. However, if it is found, the inverted list associated with **Milvus** is retrieved, indicating the documents that contain the word. This method is far more efficient than a brute-force search through a million documents, as the sorted term dictionary significantly reduces the time complexity of finding the word **Milvus**.
+- **Range query**: The efficiency of range queries, such as finding documents with words alphabetically greater than **very**, is also enhanced by the sorted term dictionary. This approach is more efficient than a brute-force search, providing quicker and more accurate results.
+
+### Test results
+
+To demonstrate the performance improvements provided by scalar indexes in Milvus, an experiment was conducted comparing the performance of several expressions using inverted indexing and brute-force search on raw data.
+
+The experiment involved testing various expressions under two conditions: with an inverted index and with a brute-force search. To ensure fairness, the same data distribution was maintained across tests, using the same collection each time. Before each test, the collection was released, and the index was dropped and rebuilt. Additionally, a warm query was performed before each test to minimize the impact of cold and hot data, and each query was executed multiple times to ensure accuracy.
+
+For a dataset of **1 million** records, using an **inverted index** can provide up to a **30x** performance improvement for point queries. The performance gains can be even more significant for larger datasets.
## Performance recommandations
-To take full advantage of Milvus' capability in scalar field indexing and unleash its power in vector similarity searches. You may need a model to estimate the size of memory required based on the data you have.
+To take full advantage of Milvus' capability in scalar field indexing and unleash its power in vector similarity searches, you may need a model to estimate the size of memory required based on the data you have.
The following tables list the estimation functions for all the data types that Milvus supports.
@@ -66,10 +92,10 @@ The following tables list the estimation functions for all the data types that M
## What's next
-- To index a scalar field, read [Build an Index on Scalars](build_scalar_index.md).
+- To index a scalar field, read [Build an Index on Scalars](index-scalar-fields.md).
- To learn more about the related terms and rules mentioned above, read
- [Bitset](bitset.md)
- - [Hybrid search](hybridsearch.md)
+ - [Hybrid search](multi-vector-search.md)
- [Boolean expression rules](boolean.md)
- - [Supported data types](schema.md#Supported-data-type)
\ No newline at end of file
+ - [Supported data types](schema.md#Supported-data-type)
diff --git a/preview/site/en/reference/schema.md b/preview/site/en/reference/schema.md
index 8e4010048..40dc745ae 100644
--- a/preview/site/en/reference/schema.md
+++ b/preview/site/en/reference/schema.md
@@ -1,16 +1,17 @@
---
id: schema.md
summary: Learn how to define a schema in Milvus.
+title: Manage Schema
---
-# Schema
+# Manage Schema
This topic introduces schema in Milvus. Schema is used to define the properties of a collection and the fields within.
## Field schema
-A field schema is the logical definition of a field. It is the first thing you need to define before defining a [collection schema](#Collection-schema) and [creating a collection](create_collection.md).
+A field schema is the logical definition of a field. It is the first thing you need to define before defining a [collection schema](#Collection-schema) and [managing collections](manage-collections.md).
Milvus supports only one primary key field in a collection.
@@ -26,29 +27,44 @@ Milvus supports only one primary key field in a collection.
nameMandatory
dtypedescriptionOptional
is_primarytrue or false).Mandatory for the primary key field
auto_id (Mandatory for primary key field)True or Falsemax_length (Mandatory for VARCHAR field)dimMandatory for the vector field
Mandatory for a dense vector field. Omit for a sparse vector field.
is_partition_keytrue or false).fielddescriptionOptional
true or false).Optional
partition_key_fieldOptional
enable_dynamic_fieldtrue or false).Optional, defaults to
False.For details on dynamic schema, refer to Dynamic Schema and the user guides for managing collections.
- You can define the shard number with
+
+ - You can define the shard number with shards_num and in which Milvus server you wish to create a collection by specifying the alias in using.
- shards_num.
+ - You can define the Milvus server on which you wish to create a collection by specifying the alias in using.
+ - You can enable the partition key feature on a field by setting is_partition_key to True on the field if you need to implement [partition-key-based multi-tenancy](multi_tenancy.md).
+ - You can enable dynamic schema by setting enable_dynamic_field to True in the collection schema if you need to [enable dynamic field](enable-dynamic-field.md).
+
+You can also create a collection with
Collection.construct_from_dataframe, which automatically generates a collection schema from DataFrame and creates a collection.
@@ -150,19 +208,22 @@ You can also create a collection with Collection.construct_from_dataframe<
```python
import pandas as pd
df = pd.DataFrame({
- "id": [i for i in range(nb)],
- "age": [random.randint(20, 40) for i in range(nb)],
- "embedding": [[random.random() for _ in range(dim)] for _ in range(nb)]
- })
+ "id": [i for i in range(nb)],
+ "age": [random.randint(20, 40) for i in range(nb)],
+ "embedding": [[random.random() for _ in range(dim)] for _ in range(nb)],
+ "position": "test_pos"
+})
+
collection, ins_res = Collection.construct_from_dataframe(
- 'my_collection',
- df,
- primary_field='id',
- auto_id=False
- )
+ 'my_collection',
+ df,
+ primary_field='id',
+ auto_id=False
+ )
```
## What's next
-- Learn how to prepare schema when [creating a collection](create_collection.md).
-
+- Learn how to prepare schema when [managing collections](manage-collections.md).
+- Read more about [dynamic schema](enable-dynamic-field.md).
+- Read more about partition-key in [Multi-tenancy](multi_tenancy.md).
diff --git a/preview/site/en/reference/sparse_vector.md b/preview/site/en/reference/sparse_vector.md
new file mode 100644
index 000000000..245992260
--- /dev/null
+++ b/preview/site/en/reference/sparse_vector.md
@@ -0,0 +1,308 @@
+---
+id: sparse_vector.md
+summary: Learn how to use sparse vectors in Milvus.
+title: Sparse Vector
+---
+
+# Sparse Vector
+
+Sparse vectors represent words or phrases using vector embeddings where most elements are zero, with only one non-zero element indicating the presence of a specific word. Sparse vector models, such as [SPLADEv2](https://arxiv.org/abs/2109.10086), outperform dense models in out-of-domain knowledge search, keyword-awareness, and interpretability. They are particularly useful in information retrieval, natural language processing, and recommendation systems, where combining sparse vectors for recall with a large model for ranking can significantly improve retrieval results.
+
+In Milvus, the use of sparse vectors follows a similar workflow to that of dense vectors. It involves creating a collection with a sparse vector column, inserting data, creating an index, and conducting similarity searches and scalar queries.
+
+In this tutorial, you will learn how to:
+
+- Prepare sparse vector embeddings;
+- Create a collection with a sparse vector field;
+- Insert entities with sparse vector embeddings;
+- Index the collection and perform ANN search on sparse vectors.
+
+To see sparse vectors in action, refer to [hello_sparse.py](https://github.com/milvus-io/pymilvus/blob/master/examples/milvus_client/sparse.py).
+
+
+ notes
+ Currently, the support for sparse vectors is a beta feature in 2.4.0, with plans to make it generally available in 3.0.0.
+
+
+## Prepare sparse vector embeddings
+
+To use sparse vectors in Milvus, prepare vector embeddings in one of the supported formats:
+
+- __Sparse Matrices__: Utilize the [scipy.sparse](https://docs.scipy.org/doc/scipy/reference/sparse.html#module-scipy.sparse) class family to represent your sparse embeddings. This method is efficient for handling large-scale, high-dimensional data.
+
+- __List of Dictionaries__: Represent each sparse embedding as a dictionary, structured as `{dimension_index: value, ...}`, where each key-value pair represents the dimension index and its corresponding value.
+
+ Example:
+
+ ```python
+ {2: 0.33, 98: 0.72, ...}
+ ```
+
+- __List of Iterables of Tuples__: Similar to the list of dictionaries, but use an iterable of tuples, `[(dimension_index, value)]`, to specify only the non-zero dimensions and their values.
+
+ Example:
+
+ ```python
+ [(2, 0.33), (98, 0.72), ...]
+ ```
+
+The following example prepares sparse embeddings by generating a random sparse matrix for 10,000 entities, each with 10,000 dimensions and a sparsity density of 0.005.
+
+```python
+# Prepare entities with sparse vector representation
+import numpy as np
+import random
+
+rng = np.random.default_rng()
+
+num_entities, dim = 10000, 10000
+
+# Generate random sparse rows with an average of 25 non-zero elements per row
+entities = [
+ {
+ "scalar_field": rng.random(),
+ # To represent a single sparse vector row, you can use:
+ # - Any of the scipy.sparse sparse matrices class family with shape[0] == 1
+ # - Dict[int, float]
+ # - Iterable[Tuple[int, float]]
+ "sparse_vector": {
+ d: rng.random() for d in random.sample(range(dim), random.randint(20, 30))
+ },
+ }
+ for _ in range(num_entities)
+]
+
+# print the first entity to check the representation
+print(entities[0])
+
+# Output:
+# {
+# 'scalar_field': 0.520821523849214,
+# 'sparse_vector': {
+# 5263: 0.2639375518635271,
+# 3573: 0.34701499565746674,
+# 9637: 0.30856525997853057,
+# 4399: 0.19771651149001523,
+# 6959: 0.31025067641541815,
+# 1729: 0.8265339135915016,
+# 1220: 0.15303302147479103,
+# 7335: 0.9436728846033107,
+# 6167: 0.19929870545596562,
+# 5891: 0.8214617920371853,
+# 2245: 0.7852255053773395,
+# 2886: 0.8787982039149889,
+# 8966: 0.9000606703940665,
+# 4910: 0.3001170013981104,
+# 17: 0.00875671667413136,
+# 3279: 0.7003425473001098,
+# 2622: 0.7571360018373428,
+# 4962: 0.3901879090102064,
+# 4698: 0.22589525720196246,
+# 3290: 0.5510228492587324,
+# 6185: 0.4508413201390492
+# }
+# }
+```
+
+
+
+notes
+
+The vector dimensions must be of Python int or numpy.integer type, and the values must be of Python float or numpy.floating type.
+
+
+
+To generate embeddings, you can also use the `model` package built in the PyMilvus library, which offers a range of embedding functions. For details, refer to [Embeddings](embeddings.md).
+
+## Create a collection with a sparse vector field
+
+To create a collection with a sparse vector field, set the __datatype__ of the sparse vector field to __DataType.SPARSE_FLOAT_VECTOR__. Unlike dense vectors, there is no need to specify a dimension for sparse vectors.
+
+```python
+from pymilvus import MilvusClient, DataType
+
+# Create a MilvusClient instance
+client = MilvusClient(uri="http://localhost:19530")
+
+# Create a collection with a sparse vector field
+schema = client.create_schema(
+ auto_id=True,
+ enable_dynamic_fields=True,
+)
+
+schema.add_field(field_name="pk", datatype=DataType.VARCHAR, is_primary=True, max_length=100)
+schema.add_field(field_name="scalar_field", datatype=DataType.DOUBLE)
+# For sparse vector, no need to specify dimension
+schema.add_field(field_name="sparse_vector", datatype=DataType.SPARSE_FLOAT_VECTOR) # set `datatype` to `SPARSE_FLOAT_VECTOR`
+
+client.create_collection(collection_name="test_sparse_vector", schema=schema)
+```
+
+For details on common collection parameters, refer to [create_collection()
+](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md).
+
+## Insert entities with sparse vector embeddings
+
+To insert entities with sparse vector embeddings, simply pass the list of entities to the [`insert()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/insert.md) method.
+
+```python
+# Insert entities
+client.insert(collection_name="test_sparse_vector", data=entities)
+```
+
+## Index the collection
+
+Before performing similarity searches, create an index for the collection. For more information on index types and parameters, refer to [add_index()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/add_index.md) and [create_index()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/create_index.md).
+
+```python
+# Index the collection
+
+# Prepare index params
+index_params = client.prepare_index_params()
+
+index_params.add_index(
+ field_name="sparse_vector",
+ index_name="sparse_inverted_index",
+ index_type="SPARSE_INVERTED_INDEX", # the type of index to be created. set to `SPARSE_INVERTED_INDEX` or `SPARSE_WAND`.
+ metric_type="IP", # the metric type to be used for the index. Currently, only `IP` (Inner Product) is supported.
+ params={"drop_ratio_build": 0.2}, # the ratio of small vector values to be dropped during indexing.
+)
+
+# Create index
+client.create_index(collection_name="test_sparse_vector", index_params=index_params)
+```
+
+For index building on sparse vectors, take note of the following:
+
+- `index_type`: The type of index to be built. Possible options for sparse vectors:
+
+ - `SPARSE_INVERTED_INDEX`: An inverted index that maps each dimension to its non-zero vectors, facilitating direct access to relevant data during searches. Ideal for datasets with sparse but high-dimensional data.
+
+ - `SPARSE_WAND`: Utilizes the Weak-AND (WAND) algorithm to quickly bypass unlikely candidates, focusing evaluation on those with higher ranking potential. Treats dimensions as terms and vectors as documents, speeding up searches in large, sparse datasets.
+
+- `metric_type`: Only `IP` (Inner Product) distance metric is supported for sparse vectors.
+
+- `params.drop_ratio_build`: The index parameter used specifically for sparse vectors. It controls the proportion of small vector values that are excluded during the indexing process. This parameter enables fine-tuning of the trade-off between efficiency and accuracy by disregarding small values when constructing the index. For instance, if `drop_ratio_build = 0.3`, during the index construction, all values from all sparse vectors are gathered and sorted. The smallest 30% of these values are not included in the index, thereby reducing the computational workload during search.
+
+For more information, refer to [In-memory Index](index.md).
+
+## Perform ANN search
+
+After the collection is indexed and loaded into memory, use the [`search()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/search.md) method to retrieve the relevant documents based on the query.
+
+```python
+# Load the collection into memory
+client.load_collection(collection_name="test_sparse_vector")
+
+# Perform ANN search on sparse vectors
+
+# for demo purpose we search for the last inserted vector
+query_vector = entities[-1]["sparse_vector"]
+
+search_params = {
+ "metric_type": "IP",
+ "params": {"drop_ratio_search": 0.2}, # the ratio of small vector values to be dropped during search.
+}
+
+search_res = client.search(
+ collection_name="test_sparse_vector",
+ data=[query_vector],
+ limit=3,
+ output_fields=["pk", "scalar_field"],
+ search_params=search_params,
+)
+
+for hits in search_res:
+ for hit in hits:
+ print(f"hit: {hit}")
+
+# Output:
+# hit: {'id': '448458373272710786', 'distance': 7.220192909240723, 'entity': {'pk': '448458373272710786', 'scalar_field': 0.46767865218233806}}
+# hit: {'id': '448458373272708317', 'distance': 1.2287548780441284, 'entity': {'pk': '448458373272708317', 'scalar_field': 0.7315987515699472}}
+# hit: {'id': '448458373272702005', 'distance': 0.9848432540893555, 'entity': {'pk': '448458373272702005', 'scalar_field': 0.9871869181562156}}
+```
+
+When configuring search parameters, take note of the following:
+
+- `params.drop_ratio_search`: The search parameter used specifically for sparse vectors. This option allows fine-tuning of the search process by specifying the ratio of the smallest values in the query vector to ignore. It helps balance search precision and performance. The smaller the value set for `drop_ratio_search`, the less these small values contribute to the final score. By ignoring some small values, search performance can be improved with minimal impact on accuracy.
+
+## Perform scalar queries
+
+In addition to ANN search, Milvus also supports scalar queries on sparse vectors. These queries allow you to retrieve documents based on a scalar value associated with the sparse vector. For more information on parameters, refer to [query()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/query.md).
+
+Filter entities with __scalar_field__ greater than 3:
+
+```python
+# Perform a query by specifying filter expr
+filter_query_res = client.query(
+ collection_name="test_sparse_vector",
+ filter="scalar_field > 0.999",
+)
+
+print(filter_query_res[:2])
+
+# Output:
+# [{'pk': '448458373272701862', 'scalar_field': 0.9994093623822689, 'sparse_vector': {173: 0.35266244411468506, 400: 0.49995484948158264, 480: 0.8757831454277039, 661: 0.9931875467300415, 1040: 0.0965644046664238, 1728: 0.7478245496749878, 2365: 0.4351981580257416, 2923: 0.5505295395851135, 3181: 0.7396837472915649, 3848: 0.4428485333919525, 4701: 0.39119353890419006, 5199: 0.790219783782959, 5798: 0.9623121619224548, 6213: 0.453134149312973, 6341: 0.745091438293457, 6775: 0.27766478061676025, 6875: 0.017947908490896225, 8093: 0.11834774166345596, 8617: 0.2289179265499115, 8991: 0.36600416898727417, 9346: 0.5502803921699524}}, {'pk': '448458373272702421', 'scalar_field': 0.9990218525410719, 'sparse_vector': {448: 0.587817907333374, 1866: 0.0994109958410263, 2438: 0.8672442436218262, 2533: 0.8063794374465942, 2595: 0.02122959867119789, 2828: 0.33827054500579834, 2871: 0.1984412521123886, 2938: 0.09674275666475296, 3154: 0.21552987396717072, 3662: 0.5236313343048096, 3711: 0.6463911533355713, 4029: 0.4041993021965027, 7143: 0.7370485663414001, 7589: 0.37588241696357727, 7776: 0.436136394739151, 7962: 0.06377989053726196, 8385: 0.5808192491531372, 8592: 0.8865005970001221, 8648: 0.05727503448724747, 9071: 0.9450633525848389, 9161: 0.146037295460701, 9358: 0.1903032660484314, 9679: 0.3146636486053467, 9974: 0.8561339378356934, 9991: 0.15841573476791382}}]
+```
+
+Filter entities by primary key:
+
+```python
+# primary keys of entities that satisfy the filter
+pks = [ret["pk"] for ret in filter_query_res]
+
+# Perform a query by primary key
+pk_query_res = client.query(
+ collection_name="test_sparse_vector", filter=f"pk == '{pks[0]}'"
+)
+
+print(pk_query_res)
+
+# Output:
+# [{'scalar_field': 0.9994093623822689, 'sparse_vector': {173: 0.35266244411468506, 400: 0.49995484948158264, 480: 0.8757831454277039, 661: 0.9931875467300415, 1040: 0.0965644046664238, 1728: 0.7478245496749878, 2365: 0.4351981580257416, 2923: 0.5505295395851135, 3181: 0.7396837472915649, 3848: 0.4428485333919525, 4701: 0.39119353890419006, 5199: 0.790219783782959, 5798: 0.9623121619224548, 6213: 0.453134149312973, 6341: 0.745091438293457, 6775: 0.27766478061676025, 6875: 0.017947908490896225, 8093: 0.11834774166345596, 8617: 0.2289179265499115, 8991: 0.36600416898727417, 9346: 0.5502803921699524}, 'pk': '448458373272701862'}]
+```
+
+## Limits
+
+When using sparse vectors in Milvus, consider the following limits:
+
+- Currently, only the __IP__ distance metric is supported for sparse vectors.
+
+- For sparse vector fields, only the __SPARSE_INVERTED_INDEX__ and __SPARSE_WAND__ index types are supported.
+
+- Currently, [range search](https://milvus.io/docs/single-vector-search.md#Range-search), [grouping search](https://milvus.io/docs/single-vector-search.md#Grouping-search), and [search iterator](https://milvus.io/docs/with-iterators.md#Search-with-iterator) are not supported for sparse vectors.
+
+## FAQ
+
+- __What distance metric is supported for sparse vectors?__
+
+ Sparse vectors only support the Inner Product (IP) distance metric due to the high dimensionality of sparse vectors, which makes L2 distance and cosine distance impractical.
+
+- __Can you explain the difference between SPARSE_INVERTED_INDEX and SPARSE_WAND, and how do I choose between them?__
+
+ __SPARSE_INVERTED_INDEX__ is a traditional inverted index, while __SPARSE_WAND__ uses the [Weak-AND](https://dl.acm.org/doi/10.1145/956863.956944) algorithm to reduce the number of full IP distance evaluations during search. __SPARSE_WAND__ is typically faster, but its performance can decline with increasing vector density. To choose between them, conduct experiments and benchmarks based on your specific dataset and use case.
+
+- __How should I choose the drop_ratio_build and drop_ratio_search parameters?__
+
+ The choice of __drop_ratio_build__ and __drop_ratio_search__ depends on the characteristics of your data and your requirements for search latency/throughput and accuracy.
+
+- __What data types are supported for sparse embeddings?__
+
+ The dimension part must be an unsigned 32-bit integer, and the value part can be a non-negative 32-bit floating-point number.
+
+- __Can the dimension of a sparse embedding be any discrete value within the uint32 space?__
+
+ Yes, with one exception. The dimension of a sparse embedding can be any value in the range of `[0, maximum of uint32)`. This means you cannot use the maximum value of uint32.
+
+- __Are searches on growing segments conducted through an index or by brute force?__
+
+ Searches on growing segments are conducted through an index of the same type as the sealed segment index. For new growing segments before the index is built, a brute force search is used.
+
+- __Is it possible to have both sparse and dense vectors in a single collection?__
+
+ Yes, with multiple vector type support, you can create collections with both sparse and dense vector columns and perform hybrid searches on them.
+
+- __What are the requirements for sparse embeddings to be inserted or searched?__
+
+ Sparse embeddings must have at least one non-zero value, and vector indices must be non-negative.
+
diff --git a/preview/site/en/reference/sys_config/configure_common.md b/preview/site/en/reference/sys_config/configure_common.md
index 81999347f..88608b9fc 100644
--- a/preview/site/en/reference/sys_config/configure_common.md
+++ b/preview/site/en/reference/sys_config/configure_common.md
@@ -3,6 +3,7 @@ id: configure_common.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure common parameters of Milvus.
+title: Common Configurations
---
# Common Configurations
@@ -65,3 +66,19 @@ Under this section, you can configure the default names of partition and index,
+## `common.ttMsgEnabled`
+
+
+
+
+ Description
+ Default Value
+
+
+
+
+ Whether to disable the internal time messaging mechanism for the system. If disabled (set to `false`), the system will not allow DML operations, including insertion, deletion, queries, and searches. This helps Milvus-CDC synchronize incremental data.
+ false
+
+
+
\ No newline at end of file
diff --git a/preview/site/en/reference/sys_config/configure_datacoord.md b/preview/site/en/reference/sys_config/configure_datacoord.md
index 5240dd30e..61c69e220 100644
--- a/preview/site/en/reference/sys_config/configure_datacoord.md
+++ b/preview/site/en/reference/sys_config/configure_datacoord.md
@@ -3,6 +3,7 @@ id: configure_datacoord.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure data coordinator of Milvus.
+title: Data Coordinator-related Configurations
---
# Data Coordinator-related Configurations
@@ -206,7 +207,7 @@ Under this section, you can configure data coord address, segment settings, comp
Switch value to control if to enable garbage collection to clear the discarded data in MinIO or S3 service.
- false
+ true
@@ -358,4 +359,3 @@ Under this section, you can configure data coord address, segment settings, comp
-
diff --git a/preview/site/en/reference/sys_config/configure_datanode.md b/preview/site/en/reference/sys_config/configure_datanode.md
index 485f54f17..f41c960e6 100644
--- a/preview/site/en/reference/sys_config/configure_datanode.md
+++ b/preview/site/en/reference/sys_config/configure_datanode.md
@@ -3,6 +3,7 @@ id: configure_datanode.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure data node of Milvus.
+title: Data Node-related Configurations
---
# Data Node-related Configurations
diff --git a/preview/site/en/reference/sys_config/configure_etcd.md b/preview/site/en/reference/sys_config/configure_etcd.md
index 9de4881b2..fcc732218 100644
--- a/preview/site/en/reference/sys_config/configure_etcd.md
+++ b/preview/site/en/reference/sys_config/configure_etcd.md
@@ -3,15 +3,18 @@ id: configure_etcd.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure etcd for Milvus.
+title: etcd-related Configurations
---
# etcd-related Configurations
-This topic introduces the etcd-related configurations of Milvus.
+This topic introduces the etcd-related configurations of Milvus. etcd is the metadata engine supporting Milvus' metadata storage and access.
-etcd is the metadata engine supporting Milvus' metadata storage and access.
+In this section, you can configure etcd endpoints, relevant key prefixes, etc.
-Under this section, you can configure etcd endpoints, relevant key prefixes, etc.
+
+To share an etcd instance among multiple Milvus instances, you need to change etcd.rootPath to a unique value for each of the Milvus instances. For details, refer to Operation FAQs.
+
## `etcd.endpoints`
@@ -48,9 +51,10 @@ Under this section, you can configure etcd endpoints, relevant key prefixes, etc
by-dev
@@ -100,3 +104,4 @@ Under this section, you can configure etcd endpoints, relevant key prefixes, etc
+
diff --git a/preview/site/en/reference/sys_config/configure_indexcoord.md b/preview/site/en/reference/sys_config/configure_indexcoord.md
deleted file mode 100644
index b1a6e13a6..000000000
--- a/preview/site/en/reference/sys_config/configure_indexcoord.md
+++ /dev/null
@@ -1,152 +0,0 @@
----
-id: configure_indexcoord.md
-related_key: configure
-group: system_configuration.md
-summary: Learn how to configure index coordinator of Milvus.
----
-
-# Index Coordinator-related Configurations
-
-This topic introduces the index coordinator-related configurations of Milvus.
-
-Index coordinator (index coord) manages topology of the index nodes, and maintains index metadata.
-
-Under this section, you can configure index coord address, etc.
-
-
-## `indexCoord.address`
-
-
-
-
- Description
- Default Value
-
-
-
-
-
-
- localhost
-
-
-
-
-
-## `indexCoord.port`
-
-
-
-
- Description
- Default Value
-
-
-
-
- TCP port of index coordinator.
- 31000
-
-
-
-
-## `indexCoord.activeStandby.enabled`
-
-
-
-
- Description
- Default Value
-
-
-
-
-
- Whether the indexCoord works in active-standby mode.
-
- false
-
-
-
-
-## `indexCoord.grpc.serverMaxRecvSize`
-
-
-
-
- Description
- Default Value
-
-
-
-
-
-
- 2147483647
-
-
-
-
-## `indexCoord.grpc.serverMaxSendSize`
-
-
-
-
- Description
- Default Value
-
-
-
-
-
-
- 2147483647
-
-
-
-
-## `indexCoord.grpc.clientMaxRecvSize`
-
-
-
-
- Description
- Default Value
-
-
-
-
-
-
- 104857600
-
-
-
-
-## `indexCoord.grpc.clientMaxSendSize`
-
-
-
-
- Description
- Default Value
-
-
-
-
-
-
- 104857600
-
-
-
diff --git a/preview/site/en/reference/sys_config/configure_indexnode.md b/preview/site/en/reference/sys_config/configure_indexnode.md
index c15f8f160..5cedf6c0a 100644
--- a/preview/site/en/reference/sys_config/configure_indexnode.md
+++ b/preview/site/en/reference/sys_config/configure_indexnode.md
@@ -3,6 +3,7 @@ id: configure_indexnode.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure index node of Milvus.
+title: Index Node-related Configurations
---
# Index Node-related Configurations
diff --git a/preview/site/en/reference/sys_config/configure_kafka.md b/preview/site/en/reference/sys_config/configure_kafka.md
new file mode 100644
index 000000000..27a5af631
--- /dev/null
+++ b/preview/site/en/reference/sys_config/configure_kafka.md
@@ -0,0 +1,145 @@
+---
+id: configure_kafka.md
+related_key: configure
+group: system_configuration.md
+summary: Learn how to configure Kafka for Milvus cluster.
+title: Kafka-related Configurations
+---
+
+# Kafka-related Configurations
+
+This topic introduces the Kafka-related configurations of Milvus.
+
+Kafka is the underlying engine supporting Milvus cluster's reliable storage and publication/subscription of message streams.
+
+Under this section, you can configure Kafka producer, consumer, sasl information and etc.
+
+## `kafka.producer.client.id`
+
+
+
+
+ Description
+ Default Value
+
+
+
+
+
+
+
+
+
+
+## `kafka.consumer.client.id`
+
+
+
+
+ Description
+ Default Value
+
+
+
+
+
+
+
+
+
+
+## `kafka.brokerList`
+
+
+
+
+ Description
+ Default Value
+
+
+
+
+
+
+
+
+
+
+## `kafka.saslUsername`
+
+
+
+
+ Description
+ Default Value
+
+
+
+
+
+
+
+
+
+
+## `kafka.saslPassword`
+
+
+
+
+ Description
+ Default Value
+
+
+
+
+
+
+
+
+
+
+## `kafka.saslMechanisms`
+
+
+
+
+ Description
+ Default Value
+
+
+
+
+
+
+
+
+
+
+## `kafka.securityProtocol`
+
+
+
+
+ Description
+ Default Value
+
+
+
+
+
+
+
+
+
diff --git a/preview/site/en/reference/sys_config/configure_knowhere.md b/preview/site/en/reference/sys_config/configure_knowhere.md
index c58fb5a83..1d2c57258 100644
--- a/preview/site/en/reference/sys_config/configure_knowhere.md
+++ b/preview/site/en/reference/sys_config/configure_knowhere.md
@@ -3,14 +3,13 @@ id: configure_knowhere.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure Knowhere-related parameters of Milvus.
+title: Knowhere-related Configurations
---
# Knowhere-related Configurations
This topic introduces the knowhere-related configurations of Milvus.
-[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus.
-
Under this section, you can configure the default SIMD instruction set type of the system.
## `knowhere.simdType`
diff --git a/preview/site/en/reference/sys_config/configure_localstorage.md b/preview/site/en/reference/sys_config/configure_localstorage.md
index 35f5e967d..504291c89 100644
--- a/preview/site/en/reference/sys_config/configure_localstorage.md
+++ b/preview/site/en/reference/sys_config/configure_localstorage.md
@@ -3,6 +3,7 @@ id: configure_localstorage.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure local storage of Milvus.
+title: Local Storage-related Configurations
---
# Local Storage-related Configurations
diff --git a/preview/site/en/reference/sys_config/configure_log.md b/preview/site/en/reference/sys_config/configure_log.md
index 838536190..f6161167a 100644
--- a/preview/site/en/reference/sys_config/configure_log.md
+++ b/preview/site/en/reference/sys_config/configure_log.md
@@ -3,6 +3,7 @@ id: configure_log.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure log of Milvus.
+title: Log-related Configurations
---
# Log-related Configurations
diff --git a/preview/site/en/reference/sys_config/configure_messagechannel.md b/preview/site/en/reference/sys_config/configure_messagechannel.md
index 3ba65eee4..77b32eb6a 100644
--- a/preview/site/en/reference/sys_config/configure_messagechannel.md
+++ b/preview/site/en/reference/sys_config/configure_messagechannel.md
@@ -3,6 +3,7 @@ id: configure_messagechannel.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure message channels of Milvus.
+title: Message Channel-related Configurations
---
# Message Channel-related Configurations
@@ -11,6 +12,11 @@ This topic introduces the message channel-related configurations of Milvus.
Under this section, you can configure the message channel name prefixes and component subscription name prefixes.
+
+
## `msgChannel.chanNamePrefix.cluster`
@@ -25,8 +31,8 @@ Under this section, you can configure the message channel name prefixes and comp
"by-dev"
diff --git a/preview/site/en/reference/sys_config/configure_minio.md b/preview/site/en/reference/sys_config/configure_minio.md
index 317f500ba..9870f56a6 100644
--- a/preview/site/en/reference/sys_config/configure_minio.md
+++ b/preview/site/en/reference/sys_config/configure_minio.md
@@ -3,6 +3,7 @@ id: configure_minio.md
related_key: configure
group: system_configuration.md
summary: Learn how to configure MinIO for Milvus.
+title: MinIO-related Configurations
---
# MinIO-related Configurations
@@ -11,7 +12,11 @@ This topic introduces the MinIO-related configurations of Milvus.
Milvus supports MinIO and Amazon S3 as the storage engine for data persistence of insert log files and index files. Whereas MinIO is the de facto standard for S3 compatibility, you can configure S3 parameters directly under MinIO section.
-Under this section, you can configure MinIO or S3 address, relevant access keys, etc.
+In this section, you can configure MinIO or S3 address, relevant access keys, etc.
+
+
+To share a MinIO instance among multiple Milvus instances, you need to change minio.bucketName or minio.rootPath to a unique value for each of the Milvus instances. For details, refer to Operation FAQs.
+
## `minio.address`
@@ -73,9 +78,9 @@ Under this section, you can configure MinIO or S3 address, relevant access keys,
minioadmin
@@ -97,9 +102,9 @@ Under this section, you can configure MinIO or S3 address, relevant access keys,
minioadmin
@@ -120,7 +125,6 @@ Under this section, you can configure MinIO or S3 address, relevant access keys,
false
@@ -143,6 +147,7 @@ Under this section, you can configure MinIO or S3 address, relevant access keys,
| Description | +Default Value | +
|---|---|
| + Listening port of the NATS server. + | +4222 |
+
| Description | +Default Value | +
|---|---|
| + JetStream storage path. + | +/var/lib/milvus/nats |
+
| Description | +Default Value | +
|---|---|
| + Maximum size of the file storage. + | +17179869184 (16 GB) |
+
| Description | +Default Value | +
|---|---|
| + Maximum size of the payload per message in bytes. + | +8388608 (8 MB) |
+
| Description | +Default Value | +
|---|---|
| + Maximum buffer size per client connection in bytes. + | +67108864 (64 MB) |
+
| Description | +Default Value | +
|---|---|
| + Timeout duration for NATs to initialize in milliseconds. + | +4000 (4 seconds) |
+
| Description | +Default Value | +
|---|---|
| + Whether to enable debug logs + | +false |
+
| Description | +Default Value | +
|---|---|
| + Whether to include timestamps in debug logs. + | +true |
+
| Description | +Default Value | +
|---|---|
|
+ |
+ N/A | +
| Description | +Default Value | +
|---|---|
+ 0, no limit applies. |
+ 0 |
+
| Description | +Default Value | +
|---|---|
| + Maximum age per message in the P-channel in minutes. + | +4320 (3 days) |
+
| Description | +Default Value | +
|---|---|
|
+ |
+ N/A | +
| Description | +Default Value | +
|---|---|
|
+ |
+ N/A | +
| Description | +Default Value | +
|---|---|
| The maximum number of vector fields that can be specified in a collection. Value range: [1, 10]. | +4 | +
| Description | +Default Value | +
|---|---|
| Whether to enable the access log feature. | +False | +
| Description | +Default Value | +
|---|---|
| The name of the access log file. If you leave this parameter empty, access logs will be printed to stdout. | +Empty string | +
| Description | +Default Value | +
|---|---|
The local folder path where the access log file is stored. This parameter can be specified when proxy.accessLog.filename is not empty. |
+ /tmp/milvus_access | +
| Description | +Default Value | +
|---|---|
|
+ |
+ 64 | +
| Description | +Default Value | +
|---|---|
|
+ |
+ 0 | +
| Description | +Default Value | +
|---|---|
| The maximum number of sealed access log files that can be retained. If the number of sealed access log files exceeds this limit, the oldest one will be deleted. | +8 | +
| Description | +Default Value | +
|---|---|
Whether to upload local access log files to MinIO. This parameter can be specified when proxy.accessLog.filename is not empty. |
+ False | +
| Description | +Default Value | +
|---|---|
| The path of the object storage for uploading access log files. | +access_log/ | +
| Description | +Default Value | +
|---|---|
The time interval allowed for uploading access log files. If the upload time of a log file exceeds this interval, the file will be deleted. Setting the value to 0 disables this feature. |
+ 0 | +
| Description | +Default Value | +
|---|---|
| The log format with dynamic metrics. This format applies to all methods by default. For more information about metrics, see Configure Access Logs. | +Empty string | +
| Description | +Default Value | +
|---|---|
The custom log format that you configure for specific methods. This parameter works together with proxy.accessLog.<custom_formatter_name>.methods. |
+ Empty string | +
| Description | +Default Value | +
|---|---|
The methods to which the custom formatter applies. This parameter works together with proxy.accessLog.<custom_formatter_name>.format. |
+ Empty string | +
+To share a Pulsar instance with multi-tenancy enabled among multiple Milvus instances, you need to change
+To share a Pulsar instance with multi-tenancy disabled among multiple Milvus instances, you need to change
+For details, refer to Operation FAQs.
+
+
## `pulsar.address`
@@ -60,6 +67,26 @@ Under this section, you can configure Pulsar address, the message size, etc.
+## `pulsar.webport`
+
+pulsar.tenant or pulsar.namespace to a unique value for each of the Milvus instances. msgChannel.chanNamePrefix.cluster to a unique value for each of the Milvus instances.| Description | +Default Value | +
|---|---|
|
+ |
+ 80 | +
| Description | +Default Value | +
|---|---|
+ msgChannel.chanNamePrefix.cluster to the different value. For details, see Operation FAQs. |
+ public | +
| Description | +Default Value | +
|---|---|
+ msgChannel.chanNamePrefix.cluster to the different value. For details, see Operation FAQs. |
+ default | +
| Description | +Default Value | +
|---|---|
| + The number of in-memory replicas of data segments that are created on query nodes when a collection is loaded. In a Standalone deployment, the maximum value is 1. For more information, refer to In-Memory Replica. + | +1 | +
| Row count by which Segcore divides a segment into chunks. | -32768 | +1024 | + + +
| Description | +Default Value | +
|---|---|
|
+ Whether to create a temporary index for growing segments and sealed segments not yet indexed, improving search performance. +
|
+ true |
| Description | +Default Value | +
|---|---|
| Maximum number of collections per database. | +64 | +
| Description | +Default Value | +
|---|---|
+ 5 indicates that Milvus only allows data insertion to any collection at the rate of 5 MB/s.quotaAndLimits.dml.enabled to true at the same time. |
+ ∞ | +
- 0.1 indicates that Milvus only allows data insertion at the rate of 0.1 MB/s.0.1 indicates that Milvus only allows data deletion at the rate of 0.1 MB/s.quotaAndLimits.dml.enabled to true at the same time. |
+ ∞ | +
| Description | +Default Value | +
|---|---|
+ 0.1 indicates that Milvus only allows data deletion from any collection at the rate of 0.1 MB/s.quotaAndLimits.dml.enabled to true at the same time. |
∞ | @@ -283,6 +343,27 @@ Some of them are used to set backpressure signals that force Milvus to lower the
| Description | +Default Value | +
|---|---|
+ 100 indicates that Milvus only allows searching 100 vectors per second per collection no matter whether these 100 vectors are all in one search or scattered across multiple searches.quotaAndLimits.dql.enabled to true at the same time. |
+ ∞ | +
| Description | +Default Value | +
|---|---|
+ 100 indicates that Milvus only allows 100 queries per collection per second.quotaAndLimits.dql.enabled to true at the same time. |
+ ∞ | +
300 indicates that Milvus reduces the DML request rate as the delay increases and drops all DML requests once the delay reaches the set maximum in milliseconds.300 indicates that Milvus reduces the DML request rate as the delay increases and drops all DML requests once the delay reaches the set maximum in seconds.quotaAndLimits.limitWriting.ttProtection.enabled to true at the same time. |
300 | @@ -481,6 +583,27 @@ Some of them are used to set backpressure signals that force Milvus to lower the
| Description | +Default Value | +
|---|---|
+ 8192 indicates that Milvus drops all DML requests in a collection as the size of binlog of the collection reaches the set value.quotaAndLimits.limitWriting.diskProtection.enabled to true at the same time. |
+ ∞ | +
| False | -
| Description | +Default Value | +
|---|---|
|
+ |
+ 86400 | +
| Description | +Default Value | +
|---|---|
|
+ |
+ 0.06 | +
Bug Fixes
+- **Grouping Search**: Categorical aggregation enhances document-level recall for Retrieval-Augmented Generation (RAG) applications. For details, refer to [Grouping Search](https://milvus.io/docs/single-vector-search.md#Grouping-search). -- Fixed the issue that the proxy doesn't update the cache of shard leaders due to some types of errors. ([#21320](https://github.com/milvus-io/milvus/pull/21320)) -- Fixed the issue that the loaded info is not cleaned for released collections/partitions. ([#21321](https://github.com/milvus-io/milvus/pull/21321)) -- Fixed the issue that the load count is not cleared on time. ([#21314](https://github.com/milvus-io/milvus/pull/21314)) +- **Inverted Index** and **Fuzzy Matching**: These capabilities improve keyword retrieval for scalar fields. For details, refer to [Index Scalar Fields](index-scalar-fields.md) and [Filtered Search](single-vector-search.md#filtered-search). -## v2.2.1 -Release date: 15 December, 2022 +### New Features -| Milvus version | Python SDK version | Java SDK version | Go SDK version | Node.js SDK version | -| -------------- | ------------------ | ---------------- | -------------- | ------------------- | -| 2.2.1 | 2.2.1 | 2.2.1 | 2.2.0 | 2.2.1 | +#### GPU Index - CAGRA -Milvus 2.2.1 is a minor fix of Milvus 2.2.0. It supports authentication and TLS on all dependencies, optimizes the performance ludicrously on searches and fixes some critical issues. With tremendous contribution from the community, this release managed to resolve over 280 issues, so please try the new release and give us feedback on stability, performance and ease of use. +We would like to express our sincere gratitude to the NVIDIA team for their invaluable contribution to CAGRA, a state-of-the-art (SoTA) GPU-based graph index that can be used online. -New Features
+Unlike previous GPU indices, CAGRA demonstrates overwhelming superiority even in small batch queries, an area where CPU indices traditionally excel. In addition, CAGRA's performance in large batch queries and index construction speed, domains where GPU indices already shine, is truly unparalleled. -- Supports Pulsa tenant and authentication. ([#20762](https://github.com/milvus-io/milvus/pull/20762)) -- Supports TLS in etcd config source. ([#20910](https://github.com/milvus-io/milvus/pull/20910)) +Example code can be found in [example_gpu_cagra.py](https://github.com/milvus-io/pymilvus/blob/2.4/examples/example_gpu_cagra.py). -Performance
+#### Sparse Vector (Beta) -After upgrading the Knowhere vector engine and changing the parallelism strategy, Milvus 2.2.1 improves search performance by over 30%. +In this release, we are introducing a new type of vector field called sparse vector. Sparse vectors are different from their dense counterparts as they tend to have several magnitude higher number of dimensions with only a handful being non-zero. This feature offers better interpretability due to its term-based nature and can be more effective in certain domains. Learned sparse models such as SPLADEv2/BGE-M3 have proven to be very useful for common first-stage ranking tasks. The main use case for this new feature in Milvus is to allow efficient approximate semantic nearest neighbor search over sparse vectors generated by neural models such as SPLADEv2/BGE-M3 and statistics models such as the BM25 algorithm. Milvus now supports effective and high-performance storage, indexing, and searching (MIPS, Maximum Inner Product Search) of sparse vectors. -Optimizes the scheduler, and increases merge tasks probability. ([#20931](https://github.com/milvus-io/milvus/pull/20931)) +Example code can be found in [hello_sparse.py](https://github.com/milvus-io/pymilvus/blob/2.4/examples/hello_sparse.py). -Bug Fixes
+#### Multi Embedding & Hybrid Search -- Fixed term filtering failures on indexed scalar fields. ([#20840](https://github.com/milvus-io/milvus/pull/20840)) -- Fixed the issue that only partial data returned upon QueryNode restarts. ([#21139](https://github.com/milvus-io/milvus/pull/21139))([#20976](https://github.com/milvus-io/milvus/pull/20976)) -- Fixed IndexNode panic upon failures to create an index. ([#20826](https://github.com/milvus-io/milvus/pull/20826)) -- Fixed endless BinaryVector compaction and generation of data on Minio. ([#21119](https://github.com/milvus-io/milvus/pull/21119)) ([#20971](https://github.com/milvus-io/milvus/pull/20971)) -- Fixed the issue that `meta_cache` of proxy partially updates. ([#21232](https://github.com/milvus-io/milvus/pull/21232)) -- Fixed slow segment loading due to staled checkpoints. ([#21150](https://github.com/milvus-io/milvus/pull/21150)) -- Fixed concurrently loaded Casbin model causing concurrent write operations. ([#21132](https://github.com/milvus-io/milvus/pull/21132))([#21145](https://github.com/milvus-io/milvus/pull/21145))([#21073](https://github.com/milvus-io/milvus/pull/21073)) -- Forbade garbage-collecting index meta when creating an index. ([#21024](https://github.com/milvus-io/milvus/pull/21024)) -- Fixed a bug that the index data can not be garbage-collected because `ListWithPrefix` from Minio with recursive is false. ([#21040](https://github.com/milvus-io/milvus/pull/21040)) -- Fixed an issue that an error code is returned when a query expression does not match any results. ([#21066](https://github.com/milvus-io/milvus/pull/21066)) -- Fixed search failures on disk index when `search_list` equals to `limit`. ([#21114](https://github.com/milvus-io/milvus/pull/21114)) -- Filled collection schema after DataCoord restarts. ([#21164](https://github.com/milvus-io/milvus/pull/21164)) -- Fixed an issue that the compaction handler may double release and hang. ([#21019](https://github.com/milvus-io/milvus/pull/21019)) -- [restapi] Fixed precision loss for Int64 fields upon insert requests. ([#20827](https://github.com/milvus-io/milvus/pull/20827)) -- Increased `MaxWatchDuration` and make it configurable to prevent shards with large data loads from timing out. ([#21010](https://github.com/milvus-io/milvus/pull/21010)) -- Fixed the issue that the compaction target segment `rowNum` is always 0. ([#20941](https://github.com/milvus-io/milvus/pull/20941)) -- Fixed the issue that IndexCoord deletes segment index by mistake because IndexMeta is not stored in time. ([#21058](https://github.com/milvus-io/milvus/pull/21058)) -- Fixed the issue that DataCoord crushes if auto-compaction is disabled. ([#21079](https://github.com/milvus-io/milvus/pull/21079)) -- Fixed the issue that searches on growing segments even though the segments are indexed. ([#21215](https://github.com/milvus-io/milvus/pull/21215)) +Multi-vector support is the cornerstone for applications that require multi-model data processing or a mix of dense and sparse vectors. With multi-vector support, now you can: -Improvements
+- Store vector embeddings generated for unstructured text, image, or audio samples from multiple models. +- Conduct ANN searches that include multiple vectors of each entity. +- Customize search strategies by assigning weights to different embedding models. +- Experiment with various embedding models to find the optimal model combination. -- Refined logs and the default log level is set to INFO. -- Fixed incorrect metrics and refined the metric dashboard. -- Made TopK limit configurable ([#21155](https://github.com/milvus-io/milvus/pull/21155)) +Multi-vector support allows storing, indexing, and applying reranking strategies to multiple vector fields of different types, such as FLOAT_VECTOR and SPARSE_FLOAT_VECTOR, in a collection. Currently, two reranking strategies are available: **Reciprocal Rank Fusion (RRF)** and **Average Weighted Scoring**. Both strategies combine the search results from different vector fields into a unified result set. The first strategy prioritizes the entities that consistently appear in the search results of different vector fields, while the other strategy assigns weights to the search results of each vector field to determine their importance in the final result set. -Breaking changes
+Example code can be found in [hybrid_search.py](https://github.com/milvus-io/pymilvus/blob/2.4/examples/hybrid_search.py). -Milvus now limits each RPC to 64 MB to avoid OOM and generating large message packs. +#### Inverted Index and Fuzzy Match -## v2.2.0 -Release date: 18 November, 2022 +In previous releases of Milvus, memory-based binary search indexes and Marisa Trie indexes were used for scalar field indexing. However, these methods were memory-intensive. The latest release of Milvus now employs the Tantivy-based inverted index, which can be applied to all numeric and string data types. This new index dramatically improves scalar query performance, reducing the query of keywords in strings by ten times. In addition, The inverted index consumes less memory, thanks to additional optimizations in data compression and Memory-mapped storage (MMap) mechanism of the internal indexing structure. -| Milvus version | Python SDK version | Java SDK version | Go SDK version | Node.js SDK version | -| -------------- | ------------------ | ---------------- | -------------- | ------------------- | -| 2.2.0 | 2.2.0 | 2.2.1 | 2.2.0 | 2.2.0 | +This release also supports fuzzy matches in scalar filtering using prefixes, infixes, and suffixes. -Milvus 2.2.0 introduces many new features including support for Disk-based approximate nearest neighbor (ANN) algorithm, bulk insertion of entities from files, and role-based access control (RBAC) for an improved security. In addition, this major release also ushers in a new era for vector search with enhanced stability, faster search speed, and more flexible scalability. +Example code can be found in [inverted_index_example.py](https://github.com/milvus-io/pymilvus/blob/2.4/examples/inverted_index_example.py) and [fuzzy_match.py](https://github.com/milvus-io/pymilvus/blob/2.4/examples/fuzzy_match.py). -Breaking changes
+#### Grouping Search -Since metadata storage is refined and API usage is normalized, Milvus 2.2 is *not* fully compatible with earlier releases. Read [this guide](upgrade_milvus_cluster-helm.md) to learn how to safely upgrade from Milvus 2.1.x to 2.2.0. +You can now aggregate the search results by the values in a specific scalar field. This helps RAG applications to implement document-level recall. Consider a collection of documents, each document splits into various passages. Each passage is represented by one vector embedding and belongs to one document. To find the most relevant documents instead of scattering passages, you can include the group_by_field argument in the search() operation to group results by the document ID. -Features
+Example code can be found in [example_group_by.py](https://github.com/milvus-io/pymilvus/blob/2.4/examples/example_group_by.py). -- Support for bulk insertion of entities from files -Milvus now offers a new set of bulk insertion APIs to make data insertion more efficient. You can now upload entities in a Json file directly to Milvus. See [Insert Entities from Files](bulk_insert.md) for details. +#### Float16 and BFloat- Vector DataType -- Query result pagination -To avoid massive search and query results returned in a single RPC, Milvus now supports configuring offset and filtering results with keywords in searches and queries. See [Search](search.md) and [Query](query.md) for details. +Machine learning and neural networks often use half-precision data types, such as Float16 and BFloat- While these data types can improve query efficiency and reduce memory usage, they come with a tradeoff of reduced accuracy. With this release, Milvus now supports these data types for vector fields. +Example code can be found in [float16_example.py](https://github.com/milvus-io/pymilvus/blob/2.4/examples/float16_example.py) and [bfloat16_example.py](https://github.com/milvus-io/pymilvus/blob/2.4/examples/bfloat16_example.py). -- Role-based access control (RBAC) -Like other traditional databases, Milvus now supports RBAC so that you can manages users, roles and privileges. See [Enable RBAC](rbac.md) for details. +### Upgraded Architecture -- Quotas and limits -Quota is a new mechanism that protects the system from OOM and crash under a burst of traffic. By imposing quota limitations, you can limit ingestion rate, search rate, etc. See [Quota and Limitation Configurations](configure_quota_limits.md) for details. +#### L0 Segment +This release includes a new segment called L0 Segment, designed to record deleted data. This segment periodically compacts stored deleted records and splits them into sealed segments, reducing the number of data flushes required for small deletions and leaving a small storage footprint. With this mechanism, Milvus completely separates data compactions from data flushes, enhancing the performance of delete and upsert operations. -- Time to live (TTL) at a collection level -In prior releases, we only support configuring TTL at a cluster level. Milvus 2.2.0 now supports configuring collection TTL when you create or modify a collection. After setting TTL for a collection, the entities in this collection automatically expires after the specified period of time. See [Create a collection](create_collection.md) or [Modify a collection](modify_collection.md) for details. +#### Refactored BulkInsert +This release also introduces improved bulk-insert logic. This allows you to import multiple files in a single bulk-insert request. With the refactored version, both the performance and stability of bulk insert have seen significant improvements. The user experience has also been enhanced, such as fine-tuned rate limiting and more user-friendly error messages. In addition, you can easily access the bulk-insert endpoints through Milvus' RESTful API. -- Support for disk-based approximate nearest neighbor search (ANNS) indexes (Beta) -Traditionally, you need to load the entire index into memory before search. Now with DiskANN, an SSD-resident and Vamana graph-based ANNS algorithm, you can directly search on large-scale datasets and save up to 10 times the memory. +#### Memory-mapped Storage +Milvus uses memory-mapped storage (MMap) to optimize its memory usage. Instead of loading file content directly into memory, this mechanism maps the file content into memory. This approach comes with a tradeoff of performance degradation. By enabling MMap for an HNSW-indexed collection on a host with 2 CPUs and 8 GB RAM, you can load 4x more data with less than 10% performance degradation. -- Data backup (Beta) -Thanks to the contribution from [Zilliz](https://zilliz.com/), Milvus 2.2.0 now provides [a tool](https://github.com/zilliztech/milvus-backup) to back up and restore data. The tool can be used either in a command line or an API server for data security. - +In addition, this release also allows dynamic and fine-grained control over MMap without the need to restart Milvus. -Bug fixes and stability
+For details, refer to [MMap Storage](mmap.md). -- Implements query coord V2, which handles all channel/segment allocation in a fully event-driven and asynchronous mode. Query coord V2 address all issues of stuck searches and accelerates failure recovery. -- Root coord and index coord are refactored for more elegant handling of errors and better task scheduling. -- Fixes the issue of invalid RocksMQ retention mechanism when Milvus Standalone restarts. -- Meta storage format in etcd is refactored. With the new compression mechanism, etcd kv size is reduced by 10 times and the issues of etcd memory and space are solved. -- Fixes a couple of memory issues when entities are continuously inserted or deleted. +### Others -Improvements
+#### Milvus-CDC -- Performance - - Fixes performance bottleneck to that Milvus can fully utilize all cores when CPU is more than 8 cores. - - Dramatically improves the search throughput and reduce the latency. - - Decreases load speed by processing load in parallel. - -- Observability - - Changes all log levels to `info` by default. - - Added collection-level latency metrics for search, query, insertion, and deletion. +Milvus-CDC is an easy-to-use companion tool to capture and synchronize incremental data between Milvus instances, allowing for easy incremental backup and disaster recovery. In this release, Milvus-CDC has improved stability, and its Change Data Capture (CDC) functionality now becomes generally available. -- Debug tool - - [BirdWatcher](https://github.com/milvus-io/birdwatcher), the debug tool for Milvus, is further optimized as it can now connect to Milvus meta storage and inspect the part of the internal status of the Milvus system. +To learn more about Milvus-CDC, refer to [GitHub repository](https://github.com/zilliztech/milvus-cdc) and [Milvus-CDC Overview](milvus-cdc-overview.md). -Others
+#### Refined MilvusClient Interfaces -- Index and load - - A collection can only be loaded with an index created on it. - - Indexes cannot be created after a collection is loaded. - - A loaded collection must be released before dropping the index created on this collection. - -- Flush - - Flush API, which forces a seal on a growing segment and syncs the segment to object storage, is now exposed to users. Calling `flush()` frequently may affect search performance as too many small segments are created. - - No auto-flush is triggered by any SDK APIs such as `num_entities()`, `create_index()`, etc. - -- Time Travel - - In Milvus 2.2, Time Travel is disabled by default to save disk usage. To enable Time Travel, configure the parameter `common.retentionDuration` manually. +MilvusClient is an easy-to-use alternative to the ORM module. It adopts a purely functional approach to simplify interactions with the server. Instead of maintaining a connection pool, each MilvusClient establishes a gRPC connection to the server. +The MilvusClient module has implemented most of the functionalities of the ORM module. +To learn more about the MilvusClient module, visit [pymilvus](https://github.com/milvus-io/pymilvus) and the [reference documents](/api-reference/pymilvus/v2.4.x/About.md). diff --git a/preview/site/en/rerankers/rerankers-bge.md b/preview/site/en/rerankers/rerankers-bge.md new file mode 100644 index 000000000..a1208dc67 --- /dev/null +++ b/preview/site/en/rerankers/rerankers-bge.md @@ -0,0 +1,80 @@ +--- +id: rerankers-bge.md +order: 2 +summary: Milvus supports BGE reranker models through the `BGERerankFunction` class. This functionality allows you to score the relevance of query-document pairs effectively. +title: BGE +--- + +# BGE + +Milvus supports [BGE reranker models](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/reranker) through the `BGERerankFunction` class. This functionality allows you to score the relevance of query-document pairs effectively. + +To use this feature, install the necessary dependencies: + +```bash +pip install --upgrade pymilvus +pip install "pymilvus[model]" +``` + +Then, instantiate the `BGERerankFunction`: + +```python +from pymilvus.model.reranker import BGERerankFunction + +# Define the rerank function +bge_rf = BGERerankFunction( + model_name="BAAI/bge-reranker-v2-m3", # Specify the model name. Defaults to `BAAI/bge-reranker-v2-m3`. + device="cpu" # Specify the device to use, e.g., 'cpu' or 'cuda:0' +) +``` + +**Parameters** + +- `model_name` (*string*) + + The name of the model to use. You can specify any of the available BGE reranker model names, for example, `BAAI/bge-reranker-base`, `BAAI/bge-reranker-large`, etc. If you leave this parameter unspecified, `BAAI/bge-reranker-v2-m3` will be used. For a list of available models, refer to [Model List](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/llm_reranker#model-list). + +- `device` (*string*) + + Optional. The device to use for running the model. If not specified, the model will be run on the CPU. You can specify `cpu` for the CPU and `cuda:n` for the nth GPU device. + +Then, use the following code to rerank documents based on the query: + +```python +query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?" + +documents = [ + "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.", + "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.", + "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.", + "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems." +] + +results = bge_rf( + query=query, + documents=documents, + top_k=3, +) + +for result in results: + print(f"Index: {result.index}") + print(f"Score: {result.score:.6f}") + print(f"Text: {result.text}\n") +``` + +The expected output is similar to the following: + +```python +Index: 1 +Score: 0.991162 +Text: The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals. + +Index: 0 +Score: 0.032697 +Text: In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence. + +Index: 3 +Score: 0.006515 +Text: The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems. +``` + diff --git a/preview/site/en/rerankers/rerankers-cohere.md b/preview/site/en/rerankers/rerankers-cohere.md new file mode 100644 index 000000000..f646e807e --- /dev/null +++ b/preview/site/en/rerankers/rerankers-cohere.md @@ -0,0 +1,79 @@ +--- +id: rerankers-cohere.md +order: 3 +summary: Milvus supports Cohere reranker models through the `CohereRerankFunction` class. This functionality allows you to score the relevance of query-document pairs effectively. +title: Rerankers Cohere +--- + +# Cohere + +Milvus supports [Cohere](https://docs.cohere.com/docs/rerank-2)[ reranker models](https://docs.cohere.com/docs/rerank-2) through the `CohereRerankFunction` class. This functionality allows you to score the relevance of query-document pairs effectively. + +To use this feature, install the necessary dependencies: + +```bash +pip install --upgrade pymilvus +pip install "pymilvus[model]" +``` + +Then, instantiate the `CohereRerankFunction`: + +```python +from pymilvus.model.reranker import CohereRerankFunction + +# Define the rerank function +cohere_rf = CohereRerankFunction( + model_name="rerank-english-v3.0", # Specify the model name. Defaults to `rerank-english-v2.0`. + api_key=COHERE_API_KEY # Replace with your Cohere API key +) +``` + +**Parameters** + +- `model_name` (*string*) + + The name of the model to use. You can specify any of the available Cohere reranker model names, for example, `rerank-english-v3.0`, `rerank-multilingual-v3.0`, etc. If you leave this parameter unspecified, `rerank-english-v2.0` will be used. For a list of available models, refer to [Rerank](https://docs.cohere.com/docs/rerank-2). + +- `api_key` (*string*) + + The API key for accessing the Cohere API. For information on how to create an API key, refer to [Cohere dashboard](https://dashboard.cohere.com/api-keys). + +Then, use the following code to rerank documents based on the query: + +```python +query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?" + +documents = [ + "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.", + "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.", + "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.", + "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems." +] + +results = cohere_rf( + query=query, + documents=documents, + top_k=3, +) + +for result in results: + print(f"Index: {result.index}") + print(f"Score: {result.score:.6f}") + print(f"Text: {result.text}\n") +``` + +The expected output is similar to the following: + +```python +Index: 1 +Score: 0.99691266 +Text: The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals. + +Index: 3 +Score: 0.8578872 +Text: The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems. + +Index: 0 +Score: 0.3589146 +Text: In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence. +``` diff --git a/preview/site/en/rerankers/rerankers-cross-encoder.md b/preview/site/en/rerankers/rerankers-cross-encoder.md new file mode 100644 index 000000000..eb45f2976 --- /dev/null +++ b/preview/site/en/rerankers/rerankers-cross-encoder.md @@ -0,0 +1,79 @@ +--- +id: rerankers-cross-encoder.md +order: 4 +summary: Milvus supports Cross Encoder reranker models through the `CrossEncoderRerankFunction` class. This functionality allows you to score the relevance of query-document pairs effectively. +title: Cross Encoder +--- + +# Cross Encoder + +Milvus supports [Cross-Encoders](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/reranker) through the `CrossEncoderRerankFunction` class. This functionality allows you to score the relevance of query-document pairs effectively. + +To use this feature, install the necessary dependencies: + +```bash +pip install --upgrade pymilvus +pip install "pymilvus[model]" +``` + +Then, instantiate the `CrossEncoderRerankFunction`: + +```python +from pymilvus.model.reranker import CrossEncoderRerankFunction + +# Define the rerank function +ce_rf = CrossEncoderRerankFunction( + model_name="cross-encoder/ms-marco-MiniLM-L-6-v2", # Specify the model name. + device="cpu" # Specify the device to use, e.g., 'cpu' or 'cuda:0' +) +``` + +**Parameters**: + +- `model_name` (*string*) + + The name of the model to use. You can specify any of the available Cross-Encoder model names, for example, `cross-encoder/ms-marco-TinyBERT-L-2-v2`, `cross-encoder/ms-marco-MiniLM-L-2-v2`, etc. If you leave this parameter unspecified, an empty string will be used. For a list of available models, refer to [Pretrained Cross-Encoders](https://www.sbert.net/docs/pretrained_cross-encoders.html#). + +- `device` (*string*) + + The device to use for running the model. You can specify `cpu` for the CPU and `cuda:n` for the nth GPU device. + +Then, use the following code to rerank documents based on the query: + +```python +query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?" + +documents = [ + "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.", + "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.", + "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.", + "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems." +] + +results = ce_rf( + query=query, + documents=documents, + top_k=3, +) + +for result in results: + print(f"Index: {result.index}") + print(f"Score: {result.score:.6f}") + print(f"Text: {result.text}\n") +``` + +The expected output is similar to the following: + +```python +Index: 1 +Score: 6.250533 +Text: The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals. + +Index: 0 +Score: -2.954602 +Text: In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence. + +Index: 3 +Score: -4.771512 +Text: The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems. +``` diff --git a/preview/site/en/rerankers/rerankers-jina.md b/preview/site/en/rerankers/rerankers-jina.md new file mode 100644 index 000000000..9646228ee --- /dev/null +++ b/preview/site/en/rerankers/rerankers-jina.md @@ -0,0 +1,78 @@ +--- +id: rerankers-jina.md +order: 6 +summary: Milvus supports Jina reranker model through the `JinaRerankFunction` class. This functionality allows you to score the relevance of query-document pairs effectively. +title: Jina AI - Rerankers +--- + +# Jina AI + +Milvus supports [Jina AI reranker models](https://jina.ai/reranker/) through the JinaRerankFunction class. This functionality allows you to score the relevance of query-document pairs effectively. + +To use this feature, install the necessary dependencies: + +```bash +pip install --upgrade pymilvus +pip install "pymilvus[model]" +``` + +Then, instantiate the `JinaRerankFunction`: + +```python +from pymilvus.model.reranker import JinaRerankFunction + +jina_rf = JinaRerankFunction( + model_name="jina-reranker-v2-base-multilingual", # Defaults to `jina-reranker-v2-base-multilingual` + api_key=JINAAI_API_KEY +) +``` + +**Parameters**: + +- `model_name` (*string*) + + The name of the Jina AI reranker model to use for encoding. If you leave this parameter unspecified, `jina-reranker-v2-base-multilingual` will be used. For a list of available models, refer to [Jina AI Rerankers](https://jina.ai/reranker/#apiform). + +- `api_key` (*string*) + + The API key for accessing the Jina AI API. + +Then, use the following code to rerank documents based on the query: + +```python +query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?" + +documents = [ + "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.", + "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.", + "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.", + "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems." +] + +results = jina_rf( + query=query, + documents=documents, + top_k=3, +) + +for result in results: + print(f"Index: {result.index}") + print(f"Score: {result.score:.6f}") + print(f"Text: {result.text}\n") +``` + +The expected output is similar to the following: + +```python +Index: 1 +Score: 0.937096 +Text: The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals. + +Index: 3 +Score: 0.354210 +Text: The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems. + +Index: 0 +Score: 0.349866 +Text: In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence. +``` diff --git a/preview/site/en/rerankers/rerankers-overview.md b/preview/site/en/rerankers/rerankers-overview.md new file mode 100644 index 000000000..7ccdd4349 --- /dev/null +++ b/preview/site/en/rerankers/rerankers-overview.md @@ -0,0 +1,210 @@ +--- +id: rerankers-overview.md +order: 1 +summary: PyMilvus model library integrates rerank functions to optimize the order of results returned from initial searches. +title: Rerankers Overview +--- + +# Rerankers Overview + +In the realm of information retrieval and generative AI, a reranker is an essential tool that optimizes the order of results from initial searches. Rerankers differ from traditional [embedding models](embeddings.md) by taking a query and document as input and directly returning a similarity score instead of embeddings. This score indicates the relevance between the input query and document. + +Rerankers are often employed after the first stage retrieval, typically done via vector Approximate Nearest Neighbor (ANN) techniques. While ANN searches are efficient at fetching a broad set of potentially relevant results, they might not always prioritize results in terms of actual semantic closeness to the query. Here, rerankers is used to optimize the results order using deeper contextual analyses, often leveraging advanced machine learning models like BERT or other Transformer-based models. By doing this, rerankers can dramatically enhance the accuracy and relevance of the final results presented to the user. + +PyMilvus model library integrates rerank functions to optimize the order of results returned from initial searches. After you retrieved nearest embedings from Milvus, you can leverage these reranking tools to refine search results to enhance the precision of search outcomes. + +| Rerank Function | API or Open-sourced | +| --------------- | ------------------- | +| [BGE](https://milvus.io/api-reference/pymilvus/v2.4.x/Rerankers/BGERerankFunction/BGERerankFunction.md) | Open-sourced | +| [Cross Encoder](https://milvus.io/api-reference/pymilvus/v2.4.x/Rerankers/CrossEncoderRerankFunction/CrossEncoderRerankFunction.md) | Open-sourced | +| [Voyage](https://milvus.io/api-reference/pymilvus/v2.4.x/Rerankers/VoyageRerankFunction/VoyageRerankFunction.md) | API | +| [Cohere](https://milvus.io/api-reference/pymilvus/v2.4.x/Rerankers/CohereRerankFunction/CohereRerankFunction.md) | API | +| [Jina AI](https://milvus.io/api-reference/pymilvus/v2.4.x/Rerankers/JinaRerankFunction/JinaRerankFunction.md) | API | + +
+
+- Before using open-source rerankers, make sure to download and install all required dependencies and models.
+
+- For API-based rerankers, get an API key from the provider and set it in the appropriate environment variables or arguments.
+
+
+
+## Example 1: Use BGE rerank function to rerank documents according to a query
+
+In this example, we demonstrate how to rerank search results using the [BGE reranker](rerankers-bge.md) based on a specific query.
+
+To use a reranker with [PyMilvus model](https://github.com/milvus-io/milvus-model) library, start by installing the PyMilvus model library along with the model subpackage that contains all necessary reranking utilities:
+
+```bash
+pip install pymilvus[model]
+# or pip install "pymilvus[model]" for zsh.
+```
+
+To use the BGE reranker, first import the `BGERerankFunction` class:
+
+```python
+from pymilvus.model.reranker import BGERerankFunction
+```
+
+Then, create a `BGERerankFunction` instance for reranking:
+
+```python
+bge_rf = BGERerankFunction(
+ model_name="BAAI/bge-reranker-v2-m3", # Specify the model name. Defaults to `BAAI/bge-reranker-v2-m3`.
+ device="cpu" # Specify the device to use, e.g., 'cpu' or 'cuda:0'
+)
+```
+
+To rerank documents based on a query, use the following code:
+
+```python
+query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?"
+
+documents = [
+ "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.",
+ "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.",
+ "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.",
+ "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems."
+]
+
+bge_rf(query, documents)
+```
+
+The expected output is similar to the following:
+
+```python
+[RerankResult(text="The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.", score=0.9911615761470803, index=1),
+ RerankResult(text="In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.", score=0.0326971950177779, index=0),
+ RerankResult(text='The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.', score=0.006514905766152258, index=3),
+ RerankResult(text='In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.', score=0.0042116724917325935, index=2)]
+```
+
+## Example 2: Use a reranker to enhance relevance of search results
+
+In this guide, we'll explore how to utilize the `search()` method in PyMilvus for conducting similarity searches, and how to enhance the relevance of the search results using a reranker. Our demonstration will use the following dataset:
+
+```python
+entities = [
+ {'doc_id': 0, 'doc_vector': [-0.0372721,0.0101959,...,-0.114994], 'doc_text': "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence."},
+ {'doc_id': 1, 'doc_vector': [-0.00308882,-0.0219905,...,-0.00795811], 'doc_text': "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals."},
+ {'doc_id': 2, 'doc_vector': [0.00945078,0.00397605,...,-0.0286199], 'doc_text': 'In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.'},
+ {'doc_id': 3, 'doc_vector': [-0.0391119,-0.00880096,...,-0.0109257], 'doc_text': 'The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.'}
+]
+```
+
+__Dataset components__:
+
+- `doc_id`: Unique identifier for each document.
+- `doc_vector`: Vector embeddings representing the document. For guidance on generating embeddings, refer to [Embeddings](embeddings.md).
+- `doc_text`: Text content of the document.
+
+### Preparations
+
+Before initiating a similarity search, you need to establish a connection with Milvus, create a collection, and prepare and insert data into that collection. The following code snippet illustrates these preliminary steps.
+
+```python
+from pymilvus import MilvusClient, DataType
+
+client = MilvusClient(
+ uri="http://10.102.6.214:19530" # replace with your own Milvus server address
+)
+
+client.drop_collection('test_collection')
+
+# define schema
+
+schema = client.create_schema(auto_id=False, enabel_dynamic_field=True)
+
+schema.add_field(field_name="doc_id", datatype=DataType.INT64, is_primary=True, description="document id")
+schema.add_field(field_name="doc_vector", datatype=DataType.FLOAT_VECTOR, dim=384, description="document vector")
+schema.add_field(field_name="doc_text", datatype=DataType.VARCHAR, max_length=65535, description="document text")
+
+# define index params
+
+index_params = client.prepare_index_params()
+
+index_params.add_index(field_name="doc_vector", index_type="IVF_FLAT", metric_type="IP", params={"nlist": 128})
+
+# create collection
+
+client.create_collection(collection_name="test_collection", schema=schema, index_params=index_params)
+
+# insert data into collection
+
+client.insert(collection_name="test_collection", data=entities)
+
+# Output:
+# {'insert_count': 4, 'ids': [0, 1, 2, 3]}
+```
+
+### Conduct a similarity search
+
+After data insertion, perform similarity searches using the `search` method.
+
+```python
+# search results based on our query
+
+res = client.search(
+ collection_name="test_collection",
+ data=[[-0.045217834, 0.035171617, ..., -0.025117004]], # replace with your query vector
+ limit=3,
+ output_fields=["doc_id", "doc_text"]
+)
+
+for i in res[0]:
+ print(f'distance: {i["distance"]}')
+ print(f'doc_text: {i["entity"]["doc_text"]}')
+```
+
+The expected output is similar to the following:
+
+```python
+distance: 0.7235960960388184
+doc_text: The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.
+distance: 0.6269873976707458
+doc_text: In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.
+distance: 0.5340118408203125
+doc_text: The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.
+```
+
+### Use a reranker to enhance search results
+
+Then, improve the relevance of your search results with a reranking step. In this example, we use `CrossEncoderRerankFunction` built in PyMilvus to rerank the results for improved accuracy.
+
+```python
+# use reranker to rerank search results
+
+from pymilvus.model.reranker import CrossEncoderRerankFunction
+
+ce_rf = CrossEncoderRerankFunction(
+ model_name="cross-encoder/ms-marco-MiniLM-L-6-v2", # Specify the model name.
+ device="cpu" # Specify the device to use, e.g., 'cpu' or 'cuda:0'
+)
+
+reranked_results = ce_rf(
+ query='What event in 1956 marked the official birth of artificial intelligence as a discipline?',
+ documents=[
+ "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.",
+ "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.",
+ "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.",
+ "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems."
+ ],
+ top_k=3
+)
+
+# print the reranked results
+for result in reranked_results:
+ print(f'score: {result.score}')
+ print(f'doc_text: {result.text}')
+```
+
+The expected output is similar to the following:
+
+```python
+score: 6.250532627105713
+doc_text: The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.
+score: -2.9546022415161133
+doc_text: In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.
+score: -4.771512031555176
+doc_text: The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.
+```
diff --git a/preview/site/en/rerankers/rerankers-voyage.md b/preview/site/en/rerankers/rerankers-voyage.md
new file mode 100644
index 000000000..25ad50a3c
--- /dev/null
+++ b/preview/site/en/rerankers/rerankers-voyage.md
@@ -0,0 +1,79 @@
+---
+id: rerankers-voyage.md
+order: 5
+summary: Milvus supports Voyage reranker model through the `VoyageRerankFunction` class. This functionality allows you to score the relevance of query-document pairs effectively.
+title: Rerankers Voyage
+---
+
+# Voyage
+
+Milvus supports [Voyage reranker model](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/reranker) through the `VoyageRerankFunction` class. This functionality allows you to score the relevance of query-document pairs effectively.
+
+To use this feature, install the necessary dependencies:
+
+```bash
+pip install --upgrade pymilvus
+pip install "pymilvus[model]"
+```
+
+Then, instantiate the `VoyageRerankFunction`:
+
+```python
+from pymilvus.model.reranker import VoyageRerankFunction
+
+# Define the rerank function
+voyage_rf = VoyageRerankFunction(
+ model_name="rerank-lite-1", # Specify the model name. Defaults to `rerank-lite-1`.
+ api_key=VOYAGE_API_KEY # Replace with your Voyage API key
+)
+```
+
+**Parameters**:
+
+- `model_name` (*string*)
+
+ The name of the Voyage model to use for encoding. If you leave this parameter unspecified, `rerank-lite-1` will be used. For a list of available models, refer to [Rerankers](https://docs.voyageai.com/docs/reranker).
+
+- `api_key` (*string*)
+
+ The API key for accessing the Voyage API. For information on how to create an API key, refer to [API Key and Python Client](https://docs.voyageai.com/docs/api-key-and-installation).
+
+Then, use the following code to rerank documents based on the query:
+
+```python
+query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?"
+
+documents = [
+ "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.",
+ "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.",
+ "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.",
+ "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems."
+]
+
+results = voyage_rf(
+ query=query,
+ documents=documents,
+ top_k=3,
+)
+
+for result in results:
+ print(f"Index: {result.index}")
+ print(f"Score: {result.score:.6f}")
+ print(f"Text: {result.text}\n")
+```
+
+The expected output is similar to the following:
+
+```python
+Index: 1
+Score: 0.898438
+Text: The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.
+
+Index: 3
+Score: 0.718750
+Text: The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.
+
+Index: 0
+Score: 0.679688
+Text: In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.
+```
diff --git a/preview/site/en/tutorials/audio_similarity_search.md b/preview/site/en/tutorials/audio_similarity_search.md
index a21b5e58f..3c3550fff 100644
--- a/preview/site/en/tutorials/audio_similarity_search.md
+++ b/preview/site/en/tutorials/audio_similarity_search.md
@@ -1,13 +1,12 @@
---
id: audio_similarity_search.md
-summary: Build an audio similarity search system with Milvus.
+summary: Build an audio similarity search system with Milvus.
+title: Audio Similarity Search
---
# Audio Similarity Search
This tutorial demonstrates how to use Milvus, the open-source vector database to build an audio similarity search system.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/audio/audio_similarity_search/audio_similarity_search.ipynb)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/tree/master/solutions/audio/audio_similarity_search/quick_deploy)
The ML model and third-party software used include:
- PANNs (Large-Scale Pretrained Audio Neural Networks)
diff --git a/preview/site/en/tutorials/build-rag-with-milvus.md b/preview/site/en/tutorials/build-rag-with-milvus.md
new file mode 100644
index 000000000..fa281696d
--- /dev/null
+++ b/preview/site/en/tutorials/build-rag-with-milvus.md
@@ -0,0 +1,278 @@
+---
+id: build-rag-with-milvus.md
+summary: build rag with milvus
+title: Build RAG with Milvus
+---
+
+# Build RAG with Milvus
+
+
+
+In this tutorial, we will show you how to build a RAG(Retrieval-Augmented Generation) pipeline with Milvus.
+
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+
+
+## Preparation
+### Dependencies and Environment
+
+
+```python
+$ pip install --upgrade pymilvus openai requests tqdm
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**. (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+We will use OpenAI as the LLM in this example. You should prepare the [api key](https://platform.openai.com/docs/quickstart) `OPENAI_API_KEY` as an environment variable.
+
+
+```python
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+```
+
+### Prepare the data
+
+We use the [Milvus development guide](https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md) to be as the private knowledge in our RAG, which is a good data source for a simple RAG pipeline.
+
+Download it and save it as a local text file.
+
+
+```python
+import json
+import urllib.request
+
+url = "https://raw.githubusercontent.com/milvus-io/milvus/master/DEVELOPMENT.md"
+file_path = "./Milvus_DEVELOPMENT.md"
+
+if not os.path.exists(file_path):
+ urllib.request.urlretrieve(url, file_path)
+```
+
+We simply use "# " to separate the content in the file, which can roughly separate the content of each main part of the markdown file.
+
+
+```python
+with open(file_path, "r") as file:
+ file_text = file.read()
+
+text_lines = file_text.split("# ")
+```
+
+### Prepare the Embedding Model
+
+We initialize the OpenAI client to prepare the embedding model.
+
+
+```python
+from openai import OpenAI
+
+openai_client = OpenAI()
+```
+
+Define a function to generate text embeddings using OpenAI client. We use the [text-embedding-3-small](https://platform.openai.com/docs/guides/embeddings) model as an example.
+
+
+```python
+def emb_text(text):
+ return (
+ openai_client.embeddings.create(input=text, model="text-embedding-3-small")
+ .data[0]
+ .embedding
+ )
+```
+
+Generate a test embedding and print its dimension and first few elements.
+
+
+```python
+test_embedding = emb_text("This is a test")
+embedding_dim = len(test_embedding)
+print(embedding_dim)
+print(test_embedding[:10])
+```
+
+ 1536
+ [0.009907577186822891, -0.0055520725436508656, 0.006800490897148848, -0.0380667969584465, -0.018235687166452408, -0.04122573509812355, -0.007634099572896957, 0.03221159428358078, 0.0189057644456625, 9.491520904703066e-05]
+
+
+## Load data into Milvus
+
+### Create the Collection
+
+
+```python
+from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+```
+
+
+
+As for the argument of `MilvusClient`:
+
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+Check if the collection already exists and drop it if it does.
+
+
+```python
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+```
+
+Create a new collection with specified parameters.
+
+If we don't specify any field information, Milvus will automatically create a default `id` field for primary key, and a `vector` field to store the vector data. A reserved JSON field is used to store non-schema-defined fields and their values.
+
+
+```python
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+```
+
+### Insert data
+Iterate through the text lines, create embeddings, and then insert the data into Milvus.
+
+Here is a new field `text`, which is a non-defined field in the collection schema. It will be automatically added to the reserved JSON dynamic field, which can be treated as a normal field at a high level.
+
+
+```python
+from tqdm import tqdm
+
+data = []
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": emb_text(line), "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+```
+
+ Creating embeddings: 100%|█| 47/47 [00:16<00:00,
+
+
+
+
+
+ {'insert_count': 47,
+ 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46],
+ 'cost': 0}
+
+
+
+## Build RAG
+
+### Retrieve data for a query
+
+Let's define a query question about the content of the development guide documentation.
+
+
+```python
+question = "what is the hardware requirements specification if I want to build Milvus and run from source code?"
+```
+
+Search for the question in the collection and retrieve the semantic top-3 matches.
+
+
+```python
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[
+ emb_text(question)
+ ], # Use the `emb_text` function to convert the question to an embedding vector
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+```
+
+Let's take a look at the search results of the query
+
+
+
+```python
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+```
+
+ [
+ [
+ "Hardware Requirements\n\nThe following specification (either physical or virtual machine resources) is recommended for Milvus to build and run from source code.\n\n```\n- 8GB of RAM\n- 50GB of free disk space\n```\n\n##",
+ 0.7718855142593384
+ ],
+ [
+ "Building Milvus on a local OS/shell environment\n\nThe details below outline the hardware and software requirements for building on Linux and MacOS.\n\n##",
+ 0.7328144907951355
+ ],
+ [
+ "Software Requirements\n\nAll Linux distributions are available for Milvus development. However a majority of our contributor worked with Ubuntu or CentOS systems, with a small portion of Mac (both x86_64 and Apple Silicon) contributors. If you would like Milvus to build and run on other distributions, you are more than welcome to file an issue and contribute!\n\nHere's a list of verified OS types where Milvus can successfully build and run:\n\n- Debian/Ubuntu\n- Amazon Linux\n- MacOS (x86_64)\n- MacOS (Apple Silicon)\n\n##",
+ 0.6443224549293518
+ ]
+ ]
+
+
+### Use LLM to get a RAG response
+
+Convert the retrieved documents into a string format.
+
+
+```python
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+```
+
+Define system and user prompts for the Lanage Model. This prompt is assembled with the retrieved documents from Milvus.
+
+
+```python
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in
-The ML models and third-party software used include:
-- YOLOv3
-- ResNet-50
-- MySQL
-- [Towhee](https://towhee.io/)
+In this notebook, we will show you how to use Milvus to search for similar images in a dataset. We will use a subset of the [ImageNet](https://www.image-net.org/) dataset, then search for an image of an Afghan hound to demonstrate this.
-
-Major search engines like Google already give users the option to search by image. Additionally, e-commerce platforms have realized the benefits this functionality offers online shoppers, with Amazon incorporating image search into its smartphone applications.
+## Dataset Preparation
+First, we need to load the dataset and unextract it for further processing.
-
-In this tutorial, you will learn how to build a reverse image search system that can detect image patterns and return similar images to the one you upload. To build such an image similarity search system, download PASCAL VOC image set which contains 17125 images of 20 categories. Alternatively, you can prepare your own image datasets. Use YOLOv3 for object detection and ResNet-50 for image feature extraction. After going through the two ML models, images are converted into 256-dimensional vectors. Then store the vectors in Milvus and a unique ID for each vector is automatically generated by Milvus. MySQL is then used to map the vector IDs to the images in the dataset. Whenever you upload a new image to the image search system, it will be converted into a new vector and compared against the vectors previously stored in Milvus. Milvus then returns the IDs of the most similar vectors and you can query the corresponding images in MySQL.
+```python
+!wget https://github.com/milvus-io/pymilvus-assets/releases/download/imagedata/reverse_image_search.zip
+!unzip -q -o reverse_image_search.zip
+```
-
+## Prequisites
-
+To run this notebook, you need to have the following dependencies installed:
+- pymilvus>=2.4.2
+- timm
+- torch
+- numpy
+- sklearn
+- pillow
-
+To run Colab, we provide the handy commands to install the necessary dependencies.
+
+
+```python
+$ pip install pymilvus --upgrade
+$ pip install timm
+```
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**. (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+## Define the Feature Extractor
+Then, we need to define a feature extractor which extracts embedding from an image using timm's ResNet-34 model.
+
+
+```python
+import torch
+from PIL import Image
+import timm
+from sklearn.preprocessing import normalize
+from timm.data import resolve_data_config
+from timm.data.transforms_factory import create_transform
+
+
+class FeatureExtractor:
+ def __init__(self, modelname):
+ # Load the pre-trained model
+ self.model = timm.create_model(
+ modelname, pretrained=True, num_classes=0, global_pool="avg"
+ )
+ self.model.eval()
+
+ # Get the input size required by the model
+ self.input_size = self.model.default_cfg["input_size"]
+
+ config = resolve_data_config({}, model=modelname)
+ # Get the preprocessing function provided by TIMM for the model
+ self.preprocess = create_transform(**config)
+
+ def __call__(self, imagepath):
+ # Preprocess the input image
+ input_image = Image.open(imagepath).convert("RGB") # Convert to RGB if needed
+ input_image = self.preprocess(input_image)
+
+ # Convert the image to a PyTorch tensor and add a batch dimension
+ input_tensor = input_image.unsqueeze(0)
+
+ # Perform inference
+ with torch.no_grad():
+ output = self.model(input_tensor)
+
+ # Extract the feature vector
+ feature_vector = output.squeeze().numpy()
+
+ return normalize(feature_vector.reshape(1, -1), norm="l2").flatten()
+```
+
+## Create a Milvus Collection
+Then we need to create Milvus collection to store the image embeddings
+
+
+```python
+from pymilvus import MilvusClient
+
+# Set up a Milvus client
+client = MilvusClient(uri="example.db")
+# Create a collection in quick setup mode
+if client.has_collection(collection_name="image_embeddings"):
+ client.drop_collection(collection_name="image_embeddings")
+client.create_collection(
+ collection_name="image_embeddings",
+ vector_field_name="vector",
+ dimension=512,
+ auto_id=True,
+ enable_dynamic_field=True,
+ metric_type="COSINE",
+)
+```
+
+
+
+As for the argument of `MilvusClient`:
+
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+## Insert the Embeddings to Milvus
+We will extract embeddings of each image using the ResNet34 model and insert images from the training set into Milvus.
+
+
+```python
+import os
+
+extractor = FeatureExtractor("resnet34")
+
+root = "./train"
+insert = True
+if insert is True:
+ for dirpath, foldername, filenames in os.walk(root):
+ for filename in filenames:
+ if filename.endswith(".JPEG"):
+ filepath = dirpath + "/" + filename
+ image_embedding = extractor(filepath)
+ client.insert(
+ "image_embeddings",
+ {"vector": image_embedding, "filename": filepath},
+ )
+```
+
+
+```python
+from IPython.display import display
+
+query_image = "./test/Afghan_hound/n02088094_4261.JPEG"
+
+results = client.search(
+ "image_embeddings",
+ data=[extractor(query_image)],
+ output_fields=["filename"],
+ search_params={"metric_type": "COSINE"},
+)
+images = []
+for result in results:
+ for hit in result[:10]:
+ filename = hit["entity"]["filename"]
+ img = Image.open(filename)
+ img = img.resize((150, 150))
+ images.append(img)
+
+width = 150 * 5
+height = 150 * 2
+concatenated_image = Image.new("RGB", (width, height))
+
+for idx, img in enumerate(images):
+ x = idx % 5
+ y = idx // 5
+ concatenated_image.paste(img, (x * 150, y * 150))
+display("query")
+display(Image.open(query_image).resize((150, 150)))
+display("results")
+display(concatenated_image)
+```
+
+
+ 'query'
+
+
+
+
+
+
+
+
+
+ 'results'
+
+
+
+
+
+
+
+
+We can see that most of the images are from the same category as the search image, which is the Afghan hound. This means that we found similar images to the search image.
diff --git a/preview/site/en/tutorials/molecular_similarity_search.md b/preview/site/en/tutorials/molecular_similarity_search.md
deleted file mode 100644
index a82294dc3..000000000
--- a/preview/site/en/tutorials/molecular_similarity_search.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-id: molecular_similarity_search.md
-summary: Build a molecular similarity search system with Milvus.
----
-
-# Molecular Similarity Search
-
-This tutorial demonstrates how to use Milvus, the open-source vector database, to build a molecular similarity search system.
-- [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/medical/molecular_search)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/tree/master/solutions/medical/molecular_similarity_search/quick_deploy)
-- [Try demo](https://milvus.io/milvus-demos/)
-
-The third-party software used include:
-- RDKit
-- MySQL
-- [Towhee](https://towhee.io/)
-
-- -Drug discovery is an important part of new medicine research and development. The process of drug discovery includes target selection and confirmation. When fragments or lead compounds are discovered, researchers usually search for similar compounds in internal or commercial compound libraries in order to discover structure-activity relationship (SAR), compound availability. Ultimately, they will evaluate the potential of the lead compounds to be optimized to candidate compounds. In order to discover available compounds from billion-scale compound libraries, chemical fingerprint is usually retrieved for substructure search and molecule similarity search. - -
- -In this tutorial, you will learn how to build a molecular similarity search system that can retrieve the substructure, superstructure, and similar structure of a particular molecule. RDKit is an open-source cheminformatics software that can convert molecule structures into vectors. Then, the vectors are stored in Milvus and Milvus can perform similarity search on vectors. Milvus also automatically generates a unique ID for each vector. The mapping of vector IDs and structure of molecules are stored in MySQL. - -
- - - - diff --git a/preview/site/en/tutorials/question_answering_system.md b/preview/site/en/tutorials/question_answering_system.md index 6d8c62522..dec6aafce 100644 --- a/preview/site/en/tutorials/question_answering_system.md +++ b/preview/site/en/tutorials/question_answering_system.md @@ -1,13 +1,13 @@ --- id: question_answering_system.md -summary: Build a question answering system with Milvus. +summary: Build a question answering system with Milvus. +title: Question Answering System --- # Question Answering System This tutorial demonstrates how to use Milvus, the open-source vector database, to build a question answering (QA) system. - [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/nlp/question_answering) -- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/question_answering_system/quick_deploy) - [Try online demo](https://milvus.io/milvus-demos/) The ML model and third-party software used include: diff --git a/preview/site/en/tutorials/recommendation_system.md b/preview/site/en/tutorials/recommendation_system.md index 7fcb612d6..cee4bf2db 100644 --- a/preview/site/en/tutorials/recommendation_system.md +++ b/preview/site/en/tutorials/recommendation_system.md @@ -1,13 +1,12 @@ --- id: recommendation_system.md -summary: Build a personalized recommender system with Milvus. +summary: Build a personalized recommender system with Milvus. +title: Recommender System --- # Recommender System This tutorial demonstrates how to use Milvus, the open-source vector database, to build a recommendation system. -- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/recommender_system.ipynb) -- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/quick_deploy) The ML model and third-party software used include: - PaddlePaddle diff --git a/preview/site/en/tutorials/text_image_search.md b/preview/site/en/tutorials/text_image_search.md index ea4cb53f2..bc38dda35 100644 --- a/preview/site/en/tutorials/text_image_search.md +++ b/preview/site/en/tutorials/text_image_search.md @@ -1,6 +1,7 @@ --- id: text_image_search.md -summary: Build a text to image search engine with Milvus. +summary: Build a text to image search engine with Milvus. +title: Text to Image Search Engine --- # Text to Image Search Engine diff --git a/preview/site/en/tutorials/text_search_engine.md b/preview/site/en/tutorials/text_search_engine.md index f99bb749a..f223d1579 100644 --- a/preview/site/en/tutorials/text_search_engine.md +++ b/preview/site/en/tutorials/text_search_engine.md @@ -1,6 +1,7 @@ --- id: text_search_engine.md -summary: Build a text search engine with Milvus. +summary: Build a text search engine with Milvus. +title: Text Search Engine --- # Text Search Engine diff --git a/preview/site/en/tutorials/video_similarity_search.md b/preview/site/en/tutorials/video_similarity_search.md index 4292487d3..7071bc136 100644 --- a/preview/site/en/tutorials/video_similarity_search.md +++ b/preview/site/en/tutorials/video_similarity_search.md @@ -1,13 +1,13 @@ --- id: video_similarity_search.md -summary: Build a video similarity search system with Milvus. +summary: Build a video similarity search system with Milvus. +title: Video Similarity Search --- # Video Similarity Search This tutorial demonstrates how to use Milvus, the open-source vector database, to build a video similarity search system. - [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/video/reverse_video_search) -- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/video/video_similarity_search/quick_deploy) The ML models and third-party software used include: - OpenCV diff --git a/preview/site/en/userGuide/build_index.md b/preview/site/en/userGuide/build_index.md deleted file mode 100644 index d9ddd48cc..000000000 --- a/preview/site/en/userGuide/build_index.md +++ /dev/null @@ -1,536 +0,0 @@ ---- -id: build_index.md -related_key: create index -summary: Learn how to build an index for vectors in Milvus. ---- - -# Build an Index on Vectors - -This guide describes how to build an index on vectors in Milvus. - -Vector indexes are an organizational unit of metadata used to accelerate [vector similarity search](search.md). Without the index built on vectors, Milvus will perform a brute-force search by default. - -See [Vector Index](index.md) for more information about the mechanism and varieties of vector indexes. - -
-
-By default, Milvus does not index a segment with less than 1,024 rows. To change this parameter, configure
-
-The following example builds a 1024-cluster IVF_FLAT index with Euclidean distance (L2) as the similarity metric. You can choose the index and metrics that suit your scenario. See [Similarity Metrics](metric.md) for more information.
-
-## Prepare index parameter
-
-Prepare the index parameters as follows:
-
-
-
-
-```python
-index_params = {
- "metric_type":"L2",
- "index_type":"IVF_FLAT",
- "params":{"nlist":1024}
-}
-```
-
-```javascript
-const index_params = {
- metric_type: "L2",
- index_type: "IVF_FLAT",
- params: JSON.stringify({ nlist: 1024 }),
-};
-```
-
-```go
-idx, err := entity.NewIndexIvfFlat( // NewIndex func
- entity.L2, // metricType
- 1024, // ConstructParams
-)
-if err != nil {
- log.Fatal("fail to create ivf flat index parameter:", err.Error())
-}
-```
-
-```java
-final IndexType INDEX_TYPE = IndexType.IVF_FLAT; // IndexType
-final String INDEX_PARAM = "{\"nlist\":1024}"; // ExtraParam
-```
-
-```shell
-create index
-
-Collection name (book): book
-
-The name of the field to create an index for (book_intro): book_intro
-
-Index type (FLAT, IVF_FLAT, IVF_SQ8, IVF_PQ, RNSG, HNSW, ANNOY): IVF_FLAT
-
-Index metric type (L2, IP, HAMMING, TANIMOTO): L2
-
-Index params nlist: 1024
-
-Timeout []:
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/index' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "field_name": "book_intro",
- "extra_params":[
- {"key": "metric_type", "value": "L2"},
- {"key": "index_type", "value": "IVF_FLAT"},
- {"key": "params", "value": "{\"nlist\":1024}"}
- ]
- }'
-```
-
-rootCoord.minSegmentSizeToEnableIndex in milvus.yaml.
-
-| Parameter | -Description | -Options | -
|---|---|---|
metric_type |
- Type of metrics used to measure the similarity of vectors. | -For floating point vectors:
-
|
-
index_type |
- Type of index used to accelerate the vector search. | -For floating point vectors:
-
|
-
params |
- Building parameter(s) specific to the index. | -See In-memory Index and On-disk Index for more information. | -
| * DISKANN has certain prerequisites to meet. For details, see On-disk Index. | -||
| Parameter | -Description | -Option | -
|---|---|---|
metric_type |
- Type of metrics used to measure the similarity of vectors. | -For floating point vectors:
-
|
-
index_type |
- Type of index used to accelerate the vector search. | -For floating point vectors:
-
|
-
params |
- Building parameter(s) specific to the index. | -See In-memory Index and On-disk Index for more information. | -
| Parameter | -Description | -Options | -
|---|---|---|
NewIndex func |
- Function to create entity. Index according to different index types. | -For floating point vectors:
-
|
-
metricType |
- Type of metrics used to measure the similarity of vectors. | -For floating point vectors:
-
|
-
ConstructParams |
- Building parameter(s) specific to the index. | -See In-memory Index and On-disk Index for more information. | -
| * DISKANN has certain prerequisites to meet. For details, see On-disk Index. | -||
| Parameter | -Description | -Options | -
|---|---|---|
IndexType |
- Type of index used to accelerate the vector search. | -For floating point vectors:
-
|
-
ExtraParam |
- Building parameter(s) specific to the index. | -See In-memory Index and On-disk Index for more information. | -
| * DISKANN has certain prerequisites to meet. For details, see On-disk Index. | -||
| Option | -Description | -
|---|---|
| --help | -Displays help for using the command. | -
| Parameter | -Description | -Options | -
|---|---|---|
collection_name |
- Name of the collection to build the index on. | -|
field_name |
- Name of the vector field to build the index on. | -|
metric_type |
- Type of metrics used to measure the similarity of vectors. | -For floating point vectors:
-
|
-
index_type |
- Type of index used to accelerate the vector search. | -For floating point vectors:
-
|
-
params |
- Building parameter(s) specific to the index. | -See In-memory Index for more information. | -
| Parameter | -Description | -
|---|---|
field_name |
- Name of the vector field to build index on. | -
index_params |
- Parameters of the index to build. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to build index in. | -
field_name |
- Name of the vector field to build index on. | -
extra_params |
- Parameters of the index to build. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to build index on. | -
fieldName |
- Name of the vector field to build index on. | -
entity.Index |
- Parameters of the index to build. | -
async |
- Switch to control sync/async behavior. The deadline of context is not applied in sync building process. | -
-
-- Do not add any field that does not exist in the target collection, and do not miss any field that the schema of the target collection defines.
-- Use the correct types of values in each field. For example, use integers in integer fields, floats in float fields, strings in varchar fields, and float arrays in vector fields.
-- Do not include an auto-generated primary key in the JSON file.
-- For binary vectors, use uint8 arrays. Each uint8 value represents 8 dimensions, and the value must be between 0 and 255. For example, `[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]` is a 16-dimensional binary vector and should be written as `[128, 7]` in the JSON file.
-
-
-
-### Column-based NumPy files
-
-As an alternative to the row-based JSON file mentioned above, you can also use NumPy arrays to organize each column of a dataset in a separate file. In this case, use the field name of each column to name the NumPy file.
-
-```python
-import numpy
-numpy.save('book_id.npy', numpy.array([101, 102, 103, 104, 105]))
-numpy.save('word_count.npy', numpy.array([13, 25, 7, 12, 34]))
-arr = numpy.array([[1.1, 1.2],
- [2.1, 2.2],
- [3.1, 3.2],
- [4.1, 4.2],
- [5.1, 5.2]])
-numpy.save('book_intro.npy', arr)
-```
-
-
-
-- Use the field name of each column to name the NumPy file. Do not add files named after a field that does not exist in the target collection. There should be one NumPy file for each field.
-- Use the correct value type when creating NumPy arrays. For details, refer to [these examples](#Create-NumPy-files).
-
-
-
-## Insert entities from files
-
-### 1. Upload data files
-
-You can use either MinIO or the local hard disk for storage in Milvus.
-
-
-Using the local hard disk for storage is only available in Milvus Standalone.
-
-
-- To use MinIO for storage, upload data files to the bucket defined by `minio.bucketName` in the `milvus.yml` configuration file .
-- For local storage, copy the data files into a directory of the local disk.
-
-
-### 2. Insert entities
-
-To facilitate data import from files, Milvus offers a bulk-insert API in various flavors. In PyMilvus, you can use the [`do_bulk_insert()`](https://milvus.io/api-reference/pymilvus/v2.2.2/Utility/do_bulk_insert().md) method. As to the Java SDK, use the [`bulkInsert`](https://milvus.io/api-reference/java/v2.2.3/BulkInsert/bulkInsert().md) method.
-
-In this method, you need to set the name of the target collection as **collection_name** and the list of files [prepared in the previous step](#Prepare-the-data-file) as **files**. Optionally, you can specify the name of a specific partition as **partition_name** in the target collection so that Milvus imports the data from the files listed only into this partition.
-
-- For a row-based JSON file, parameter **files** should be a one-member list containing the path to the JSON file.
-
-
-
- ```python
- from pymilvus import utility
- task_id = utility.do_bulk_insert(
- collection_name="book",
- partition_name="2022",
- files=["test.json"]
- )
- ```
-
- ```java
- import io.milvus.param.bulkinsert.BulkInsertParam;
- import io.milvus.response.BulkInsertResponseWrapper;
- import io.milvus.grpc.ImportResponse;
- import io.milvus.param.R;
-
- BulkInsertParam param = BulkInsertParam.newBuilder()
- .withCollectionName("book")
- .withPartitionName("2022")
- .addFile("test.json")
- .build()
- R
-
- When setting the file paths, note that
-
- - If you upload the data file to a MinIO instance, a valid file path should be relative to the root bucket defined in **"milvus.yml"**, such as **"data/book_id.npy"**.
- - If you upload the data file to the local hard drive, a valid file path should be an absolute path such as **"/tmp/data/book_id.npy"**.
-
- If you have a lot of files to process, consider [creating multiple data-import tasks and have them run in parallel](#Import-multiple-NumPy-files-in-parallel).
-
-
-
-
-
-## List tasks
-
-### Check task state
-
-Since the bulk-insert API is asynchronous, you might need to check whether a data-import task is complete. Milvus provides a **BulkInsertState** object to hold the details of a data-import task and you can use the get-bulk-insert-state API to retrieve this object using the programming language of your choice.
-
-In the flavor of PyMilvus, you can use [`get_bulk_insert_state()`](https://milvus.io/api-reference/pymilvus/v2.2.2/Utility/get_bulk_insert_state().md). For Java SDK, use [`getBulkInsertState()`](https://milvus.io/api-reference/java/v2.2.3/BulkInsert/getBulkInsertState().md).
-
-
-
-```python
-task = utility.get_bulk_insert_state(task_id=task_id)
-print("Task state:", task.state_name())
-print("Imported files:", task.files)
-print("Collection name:", task.collection_name)
-print("Partition name:", task.partition_name)
-print("Start time:", task.create_time_str)
-print("Imported row count:", task.row_count)
-print("Entities ID array generated by this task:", task.ids())
-
-if task.state == BulkInsertState.ImportFailed:
- print("Failed reason:", task.failed_reason)
-```
-
-```java
-import io.milvus.param.bulkinsert.GetBulkInsertStateParam;
-import io.milvus.response.GetBulkInsertStateWrapper;
-import io.milvus.grpc.GetImportStateResponse;
-import io.milvus.grpc.ImportState;
-import io.milvus.param.R;
-
-GetBulkInsertStateParam param = GetBulkInsertStateParam.newBuilder()
- .withTask(task_id)
- .build()
-Rtask.failed_reason to understand why the task fails. |
-| Started | 2 | The task has been dispatched to a data node and will be executed soon. |
-| Persisted | 5 | New data segments have been generated and persisted. |
-| Completed | 6 | The metadata has been updated for the new segments. |
-| Failed and cleaned | 7 | The task fails and all temporary data generated by this task are cleared. |
-
-
-### List all tasks
-
-Milvus also offers a list-bulk-insert-tasks API for you to list all data-import tasks. In this method, you need to specify a collection name so that Milvus lists all tasks that import data into this collection. Optionally, you can specify a limit for the maximum number of tasks to return.
-
-
-
-```python
-tasks = utility.list_bulk_insert_tasks(collection_name="book", limit=10)
-for task in tasks:
- print(task)
-```
-
-```java
-import io.milvus.param.bulkinsert.ListBulkInsertTasksParam;
-import io.milvus.grpc. ListImportTasksResponse;
-import io.milvus.grpc.GetImportStateResponse;
-import io.milvus.grpc.ImportState;
-import io.milvus.param.R;
-
-ListBulkInsertTasksParam param = ListBulkInsertTasksParam.newBuilder()
- .withCollectionName("book")
- .build()
-R-├── task_1 -│ └── book_id.npy -│ └── word_count.npy -│ └── book_intro.npy -├── task_2 -│ └── book_id.npy -│ └── word_count.npy -│ └── book_intro.npy -- -You can create multiple data-import tasks as follows - -``` -task_1 = utility.do_bulk_insert( - collection_name="book", - files=["task_1/book_id.npy", "task_1/word_count.npy", "task_1/book_intro.npy"] -) -task_2 = utility.do_bulk_insert( - collection_name="book", - files=["task_2/book_id.npy", "task_2/word_count.npy", "task_2/book_intro.npy"] -) -``` - - -### Check data searchability - -After a data-import task is complete, Milvus persists the imported data into segments and sends these segments to the index nodes for index-building. During the index-building process, these segments are unavailable for searches. Once such a process is complete, you need to call the load API again to load these segments into the query nodes. These segments will then be ready for searches. - -1. Check the index-building progress - - PyMilvus provides a utility method to wait for the index-building process to complete. - - ``` - utility.wait_for_index_building_complete(collection_name) - ``` - - In other SDKs, you can use the describe-index API to check the index-building progress. - - ``` - while (true) { - R
-
-
-
-
- The `_refresh` parameter is `false` by default. Do not set it to `true` when you load a collection for the first time.
-
-
-
-
-
-
-
-## What's next
-
-Learn more basic operations of Milvus:
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
diff --git a/preview/site/en/userGuide/calculate_distance.md b/preview/site/en/userGuide/calculate_distance.md
deleted file mode 100644
index 5e0893069..000000000
--- a/preview/site/en/userGuide/calculate_distance.md
+++ /dev/null
@@ -1,375 +0,0 @@
----
-id: calculate_distance.md
-related_key: calculate distance
-summary: Learn how to calculate distance between vectors with Milvus.
----
-
-# Calculate Distance Between Vectors
-
-This topic describes how to calculate distance between vectors with Milvus.
-
-Milvus searches most similar vectors based on the distance calculation of vectors. Vice versa, you can use Milvus to calculate the distance between vectors using distance metrics that suit specific scenario. See [Similarity Metrics](metric.md) for more information.
-
-The following example simulates the scenarios when you want to calculate the distance between vectors in the collection and some other vectors.
-
-## Prepare vectors
-
-Prepare the vectors used for calculation.
-
-
-
- The `withRefresh()` method is optional. Do not call it with a `Boolean.TRUE` when you load a collection for the first time.
-
-
-
-Vectors to be calculated must agree in vector type and dimension.
-
-
-
-
-
-```python
-vectors_left = {
- "ids": [0, 1],
- "collection": "book",
- "partition": "_default",
- "field": "book_intro"
-}
-import random
-external_vectors = [[random.random() for _ in range(2)] for _ in range(4)]
-vectors_right = {"float_vectors": external_vectors}
-```
-
-```javascript
-// Node User Guide will be ready soon.
-```
-
-```go
-// GO User Guide will be ready soon.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-// CLI User Guide will be ready soon.
-```
-
-```curl
-vectors_left='{
- "dim": 2,
- "ids": {
- "id_array": [1,2],
- "collection_name": "book",
- "partition_names": ["_default"],
- "field_name": "book_intro"
- }
-}'
-vectors_right='{
- "dim": 2,
- "vectors": [1,2,3,4,5,6,7,8] # The numbers in the list will be automatically split into four vectors.
-}'
-```
-
-| Parameter | -Description | -
|---|---|
vectors_left and vectors_right |
- Vectors on the left and right side of the operator. Dict type that can be represented as {"ids": [primary_key_1, primary_key_2, ... primary_key_n], "collection": "collection_name", "partition": "partition_name", "field": "vector_field_name"}, {"float_vectors": [[1.0, 2.0], [3.0, 4.0], ... [9.0, 10.0]]}, or {"bin_vectors": [b'', b'N', ... b'Ê']}. |
-
ids |
- List of primary key of entities that in the collection. | -
collection |
- Name of the collection that holds the entities. | -
partition |
- Name of the partition that holds the entities. | -
field |
- Name of the vector field in the collection. | -
float_vectors or bin_vectors |
- Type of the vectors. | -
| Parameter | -Description | -Option | -
|---|---|---|
dim |
- Dimension of the vector. | -N/A | -
id_array |
- List of the primary keys of entities in the collection. | -N/A | -
collection_name |
- Name of the collection that holds the entities. | -N/A | -
partition_names |
- Names of the partitions that hold the entities. | -N/A | -
field_name |
- Name of the vector field in the collection. | -N/A | -
vectors |
- Temporarily only floating-point vectors are supported. | -N/A | -
| Parameter | -Description | -Option | -
|---|---|---|
params |
- Calculation parameters. | -N/A | -
metric |
- Metric types used for calculation. | -For floating-point vectors:
-
|
-
dim |
- Dimension of the vector. | -N/A | -
| Parameter | -Description | -Option | -
|---|---|---|
metric |
- Metric types used for calculation. | -For floating-point vectors:
-
|
-
-Output:
-
-```json
-{"status":{},"Array":{"FloatDist":{"data":[3,7,11,15,4,10,16,22]}}}
-```
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
- - [Search with Time Travel](timetravel.md)
-
diff --git a/preview/site/en/userGuide/check_collection.md b/preview/site/en/userGuide/check_collection.md
deleted file mode 100644
index aabf1f0fa..000000000
--- a/preview/site/en/userGuide/check_collection.md
+++ /dev/null
@@ -1,534 +0,0 @@
----
-id: check_collection.md
-related_key: collection
-summary: Learn how to check collection information in Milvus.
----
-
-# Check Collection Information
-
-This topic describes how to check the information of the collection in Milvus.
-
-## Check if a collection exists
-
-Verify if a collection exists in Milvus.
-
-
-
-
-```python
-from pymilvus import utility
-utility.has_collection("book")
-```
-
-```javascript
-await milvusClient.collectionManager.hasCollection({
- collection_name: "book",
-});
-```
-
-```go
-hasColl, err := milvusClient.HasCollection(
- context.Background(), // ctx
- collectionName, // CollectionName
-)
-if err != nil {
- log.Fatal("failed to check whether collection exists:", err.Error())
-}
-log.Println(hasColl)
-```
-
-```java
-R
-Output:
-
-```json
-{
- "status":{},
- "value":true
-}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to check. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to check. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to check. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to check. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to check. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to check. | -
-Output:
-
-```json
-{
- "status": {},
- "schema": {
- "name": "book",
- "description": "Test book search",
- "fields": [
- {
- "fieldID": 100,
- "name": "book_id",
- "is_primary_key": true,
- "description": "book id",
- "data_type": 5
- },
- {
- "fieldID": 101,
- "name": "book_intro",
- "description": "embedded vector of book introduction",
- "data_type": 101,
- "type_params": [
- {
- "key": "dim",
- "value": "2"
- }
- ]
- }
- ]
- },
- "collectionID": 434240188610972993,
- "virtual_channel_names": [
- "by-dev-rootcoord-dml_0_434240188610972993v0",
- "by-dev-rootcoord-dml_1_434240188610972993v1"
- ],
- "physical_channel_names": [
- "by-dev-rootcoord-dml_0",
- "by-dev-rootcoord-dml_1"
- ],
- "created_timestamp": 434240188610772994,
- "created_utc_timestamp": 1656494860118,
- "shards_num": 2,
- "consistency_level": 1
-}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
| schema | -The schema of the collection. | -
| description | -The description of the collection. | -
| name | -The name of the collection. | -
| is_empty | -A boolean value that indicates whether the collection is empty. | -
| num_entities | -The number of entities in the collection. | -
| primary_field | -The primary field of the collection. | -
| properties | -Currently, only the property of collection.ttl.seconds is shown. Collection time to live (TTL) is the expiration time of data in a collection. |
-
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to check. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to check. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to check. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to check. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to check. | -
-Output:
-
-```json
-{
- "status": {},
- "collection_names": [
- "book"
- ],
- "collection_ids": [
- 434240188610972993
- ],
- "created_timestamps": [
- 434240188610772994
- ],
- "created_utc_timestamps": [
- 1656494860118
- ]
-}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
-Output:
-
-```json
-{"status":{},"value":true}
-```
-
-
-
-
-| Parameter | -Description | -
|---|---|
partition_name |
- Name of the partition to check. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to check. | -
partition_name |
- Name of the partition to check. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to check. | -
partitionName |
- Name of the partition to check. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to check. | -
| -p | -Name of the partition to check. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to check. | -
partition_name |
- Name of the partition to check. | -
-Output:
-
-```json
-{
- "status": {},
- "partition_names": [
- "_default",
- "novel"
- ],
- "partitionIDs": [
- 434261413928632322,
- 434261764795531265
- ],
- "created_timestamps": [
- 434261413928632323,
- 434261764795531266
- ],
- "created_utc_timestamps": [
- 1656575828280,
- 1656577166731
- ]
-}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to check. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to check. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to check. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to check. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to check. | -
-A collection alias is globally unique, hence you cannot assign the same alias to different collections. However, you can assign multiple aliases to one collection.
-
-
-The following example is based on the alias `publication`.
-
-## Create a collection alias
-
-Specify an alias for a collection.
-
-
-
-
-```python
-from pymilvus import utility
-utility.create_alias(
- collection_name = "book",
- alias = "publication"
-)
-```
-
-```javascript
-await milvusClient.collectionManager.createAlias({
- collection_name: "book",
- alias: "publication",
-});
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-milvusClient.createAlias(
- CreateAliasParam.newBuilder()
- .withCollectionName("book")
- .withAlias("publication")
- .build()
-);
-```
-
-```shell
-create alias -c book -a publication
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/alias' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "alias":"publication"
- }'
-```
-```curl
-# Output:
-{}
-```
-
-| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to create alias on. | -
alias |
- Collection alias to create. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to create alias on. | -
alias |
- Collection alias to create. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to create alias on. | -
Alias |
- Collection alias to create. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to create alias on. | -
| -a | -Collection alias to create. | -
| -A (Optional) | -Flag to transfer the alias to a specified collection. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to create alias on. | -
alias |
- Collection alias to create. | -
| Parameter | -Description | -
|---|---|
alias |
- Collection alias to drop. | -
| Parameter | -Description | -
|---|---|
alias |
- Collection alias to drop. | -
| Parameter | -Description | -
|---|---|
Alias |
- Collection alias to drop. | -
| Option | -Description | -
|---|---|
| -a | -Collection alias to drop. | -
| Parameter | -Description | -
|---|---|
alias |
- Collection alias to drop. | -
-Output:
-
-```json
-{}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to alter alias to. | -
alias |
- Collection alias to alter. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to alter alias to. | -
alias |
- Collection alias to alter. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to alter alias to. | -
Alias |
- Collection alias to alter. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to alter alias to. | -
| -a | -Collection alias to alter. | -
| -A | -Flag to transfer the alias to a specified collection. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to alter alias to. | -
alias |
- Collection alias to alter. | -
-To ensure accuracy of searches with Time Travel, Milvus retains the data operation log within the span specified in
-
-## Compact data manually
-
-Compaction requests are processed asynchronously because they are usually time-consuming.
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.compact()
-```
-
-```javascript
-const res = await milvusClient.collectionManager.compact({
- collection_name: "book",
-});
-const compactionID = res.compactionID;
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-Rcommon.retentionDuration. Therefore, data operated within this period will not be compacted.
-
-Output:
-
-```json
-{"status":{},"compactionID":434262132129005569}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to compact data. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to compact data. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to compact data. | -
-Output:
-
-```json
-{"status":{},"state":2}
-```
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Insert data into Milvus](insert_data.md)
- - [Create a partition](create_partition.md)
- - [Build an index for vectors](build_index.md)
- - [Conduct a vector search](search.md)
- - [Conduct a hybrid search](hybridsearch.md)
diff --git a/preview/site/en/userGuide/create_collection.md b/preview/site/en/userGuide/create_collection.md
deleted file mode 100644
index 6935a5e00..000000000
--- a/preview/site/en/userGuide/create_collection.md
+++ /dev/null
@@ -1,735 +0,0 @@
----
-id: create_collection.md
-related_key: create collection
-summary: Learn how to create a collection in Milvus.
----
-
-# Create a Collection
-
-This topic describes how to create a collection in Milvus.
-
-A collection consists of one or more partitions. While creating a new collection, Milvus creates a default partition `_default`. See [Glossary - Collection](glossary.md#Collection) for more information.
-
-The following example builds a two-[shard](glossary.md#Sharding) collection named `book`, with a primary key field named `book_id`, an `INT64` scalar field named `word_count`, and a two-dimensional floating-point vector field named `book_intro`. Real applications will likely use much higher dimensional vectors than the example.
-
-
-## Prepare Schema
-
-
-The collection to create must contain a primary key field and a vector field. INT64 and String are supported data type on primary key field.
-
-
-
-First, prepare necessary parameters, including field schema, collection schema, and collection name.
-
-
-
-
-```python
-from pymilvus import CollectionSchema, FieldSchema, DataType
-book_id = FieldSchema(
- name="book_id",
- dtype=DataType.INT64,
- is_primary=True,
-)
-book_name = FieldSchema(
- name="book_name",
- dtype=DataType.VARCHAR,
- max_length=200,
-)
-word_count = FieldSchema(
- name="word_count",
- dtype=DataType.INT64,
-)
-book_intro = FieldSchema(
- name="book_intro",
- dtype=DataType.FLOAT_VECTOR,
- dim=2
-)
-schema = CollectionSchema(
- fields=[book_id, book_name, word_count, book_intro],
- description="Test book search"
-)
-collection_name = "book"
-```
-
-```javascript
-const params = {
- collection_name: "book",
- description: "Test book search"
- fields: [
- {
- name: "book_intro",
- description: "",
- data_type: 101, // DataType.FloatVector
- type_params: {
- dim: "2",
- },
- },
- {
- name: "book_id",
- data_type: 5, //DataType.Int64
- is_primary_key: true,
- description: "",
- },
- {
- name: "word_count",
- data_type: 5, //DataType.Int64
- description: "",
- },
- ],
-};
-```
-
-```go
-var (
- collectionName = "book"
- )
-schema := &entity.Schema{
- CollectionName: collectionName,
- Description: "Test book search",
- Fields: []*entity.Field{
- {
- Name: "book_id",
- DataType: entity.FieldTypeInt64,
- PrimaryKey: true,
- AutoID: false,
- },
- {
- Name: "word_count",
- DataType: entity.FieldTypeInt64,
- PrimaryKey: false,
- AutoID: false,
- },
- {
- Name: "book_intro",
- DataType: entity.FieldTypeFloatVector,
- TypeParams: map[string]string{
- "dim": "2",
- },
- },
- },
-}
-```
-
-```java
-FieldType fieldType1 = FieldType.newBuilder()
- .withName("book_id")
- .withDataType(DataType.Int64)
- .withPrimaryKey(true)
- .withAutoID(false)
- .build();
-FieldType fieldType2 = FieldType.newBuilder()
- .withName("word_count")
- .withDataType(DataType.Int64)
- .build();
-FieldType fieldType3 = FieldType.newBuilder()
- .withName("book_intro")
- .withDataType(DataType.FloatVector)
- .withDimension(2)
- .build();
-CreateCollectionParam createCollectionReq = CreateCollectionParam.newBuilder()
- .withCollectionName("book")
- .withDescription("Test book search")
- .withShardsNum(2)
- .addFieldType(fieldType1)
- .addFieldType(fieldType2)
- .addFieldType(fieldType3)
- .build();
-```
-
-```shell
-create collection -c book -f book_id:INT64:book_id -f word_count:INT64:word_count -f book_intro:FLOAT_VECTOR:2 -p book_id
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/collection' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "schema": {
- "autoID": false,
- "description": "Test book search",
- "fields": [
- {
- "name": "book_id",
- "description": "book id",
- "is_primary_key": true,
- "autoID": false,
- "data_type": 5
- },
- {
- "name": "word_count",
- "description": "count of words",
- "is_primary_key": false,
- "data_type": 5
- },
- {
- "name": "book_intro",
- "description": "embedded vector of book introduction",
- "data_type": 101,
- "is_primary_key": false,
- "type_params": [
- {
- "key": "dim",
- "value": "2"
- }
- ]
- }
- ],
- "name": "book"
- }
- }'
-```
-
-
-Output:
-
-```json
-{}
-```
-
-
-
-| Parameter | -Description | -Option | -
|---|---|---|
FieldSchema |
- Schema of the fields within the collection to create. Refer to Schema for more information. | -N/A | -
name |
- Name of the field to create. | -N/A | -
dtype |
- Data type of the field to create. | -For primary key field:
-
|
-
is_primary (Mandatory for primary key field) |
- Switch to control if the field is primary key field. | -True or False |
-
auto_id (Mandatory for primary key field) |
- Switch to enable or disable automatic ID (primary key) allocation. | -True or False |
-
max_length (Mandatory for VARCHAR field) |
- Maximum length of strings allowed to be inserted. | -[1, 65,535] | -
dim (Mandatory for vector field) |
- Dimension of the vector. | -[1, 32,768] | -
description (Optional) |
- Description of the field. | -N/A | -
CollectionSchema |
- Schema of the collection to create. Refer to Schema for more information. | -N/A | -
fields |
- Fields of the collection to create. | -N/A | -
description (Optional) |
- Description of the collection to create. | -N/A | -
collection_name |
- Name of the collection to create. | -N/A | -
| Parameter | -Description | -Option | -
|---|---|---|
collectionName |
- Name of the collection to create. | -N/A | -
description |
- Description of the collection to create. | -N/A | -
Fields |
- Schema of the fields within the collection to create. Refer to Schema for more information. | -N/A | -
Name |
- Name of the field to create. | -N/A | -
DataType |
- Data type of the field to create. | -For primary key field:
-
|
-
PrimaryKey (Mandatory for primary key field) |
- Switch to control if the field is primary key field. | -True or False |
-
AutoID (Mandatory for primary key field) |
- Switch to enable or disable Automatic ID (primary key) allocation. | -True or False |
-
dim (Mandatory for vector field) |
- Dimension of the vector. | -[1, 32768] | -
| Parameter | -Description | -Option | -
|---|---|---|
collection_name |
- Name of the collection to create. | -N/A | -
description |
- Description of the collection to create. | -N/A | -
fields |
- Schema of the filed and the collection to create. | -Refer to Schema for more information. | -
data_type |
- Data type of the filed to create. | -Refer to data type reference number for more information. | -
is_primary (Mandatory for primary key field) |
- Switch to control if the field is primary key field. | -True or False |
-
auto_id |
- Switch to enable or disable Automatic ID (primary key) allocation. | -True or False |
-
dim (Mandatory for vector field) |
- Dimension of the vector. | -[1, 32768] | -
description (Optional) |
- Description of the field. | -N/A | -
| Parameter | -Description | -Option | -
|---|---|---|
Name |
- Name of the field to create. | -N/A | -
Description |
- Description of the field to create. | -N/A | -
DataType |
- Data type of the field to create. | -For primary key field:
-
|
-
PrimaryKey (Mandatory for primary key field) |
- Switch to control if the field is primary key field. | -True or False |
-
AutoID |
- Switch to enable or disable Automatic ID (primary key) allocation. | -True or False |
-
Dimension (Mandatory for vector field) |
- Dimension of the vector. | -[1, 32768] | -
CollectionName |
- Name of the collection to create. | -N/A | -
Description (Optional) |
- Description of the collection to create. | -N/A | -
ShardsNum |
- Number of the shards for the collection to create. | -[1,64] | -
| Option | -Description | -
|---|---|
| -c | -The name of the collection. | -
| -f (Multiple) | -The field schema in the <fieldName>:<dataType>:<dimOfVector/desc> format. |
-
| -p | -The name of the primary key field. | -
| -a (Optional) | -Flag to generate IDs automatically. | -
| -d (Optional) | -The description of the collection. | -
| Parameter | -Description | -Option | -
|---|---|---|
collection_name |
- Name of the collection to create. | -N/A | -
name (schema) |
- Must be the same as collection_name, this duplicated field is kept for historical reasons. |
- Same as collection_name |
-
autoID (schema) |
- Switch to enable or disable Automatic ID (primary key) allocation. | -True or False |
-
description (schema) |
- Description of the collection to create. | -N/A | -
fields |
- Schema of the fields within the collection to create. Refer to Schema for more information. | -N/A | -
name(field) |
- Name of the field to create. | -N/A | -
description (field) |
- Description of the collection to create. | -N/A | -
is_primary_key(Mandatory for primary key field) |
- Switch to control if the field is primary key field. | -True or False |
-
autoID (field)(Mandatory for primary key field) |
- Switch to enable or disable Automatic ID (primary key) allocation. | -True or False |
-
data_type |
- Data type of the field to create. | -
- Enums:
- 1: "Bool", - 2: "Int8", - 3: "Int16", - 4: "Int32", - 5: "Int64", - 10: "Float", - 11: "Double", - 20: "String", - 21: "VarChar", - 100: "BinaryVector", - 101: "FloatVector", - - For primary key field: -
|
-
dim (Mandatory for vector field) |
- Dimension of the vector. | -[1, 32,768] | -
| Parameter | -Description | -Option | -
|---|---|---|
using (optional) |
- By specifying the server alias here, you can choose in which Milvus server you create a collection. | -N/A | -
shards_num (optional) |
- Number of the shards for the collection to create. | -[1,256] | -
properties: collection.ttl.seconds (optional) |
- Collection time to live (TTL) is the expiration time of a collection. Data in an expired collection will be cleaned up and will not be involved in searches or queries. Specify TTL in the unit of seconds. | -The value should be 0 or greater. 0 means TTL is disabled. | -
| Parameter | -Description | -Option | -
|---|---|---|
ctx |
- Context to control API invocation process. | -N/A | -
shardNum |
- Number of the shards for the collection to create. | -[1,256] | -
| Parameter | -Description | -
|---|---|
partition_name |
- Name of the partition to create. | -
description (optional) |
- Description of the partition to create. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to create a partition in. | -
partition_name |
- Name of the partition to create. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to create a partition in. | -
partitionName |
- Name of the partition to create. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to create a partition in. | -
PartitionName |
- Name of the partition to create. | -
| Option | -Description | -
|---|---|
| -c | -The name of the collection. | -
| -p | -The partition name. | -
| -d (Optional) | -The description of the partition. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to create a partition in. | -
partition_name |
- Name of the partition to create. | -
+
+The following code snippet creates a simple collection with the given schema. For more information on parameters, refer to [`create_schema()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md) and [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md) in the SDK reference.
+
+
+
+
+
+The following code snippet creates a simple collection with the given schema. For more information on parameters, refer to [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v1/Collection/createCollection.md) in the SDK reference.
+
+
+
+
+
+```python
+client = MilvusClient("http://localhost:19530")
+
+schema = MilvusClient.create_schema(
+ auto_id=False,
+ enable_dynamic_field=True
+)
+
+schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
+schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=768)
+schema.add_field(field_name="scalar_1", datatype=DataType.VARCHAR, max_length=512)
+schema.add_field(field_name="scalar_2", datatype=DataType.INT64)
+
+client.create_collection(
+ collection_name="quick_setup",
+ schema=schema
+)
+```
+
+```java
+import io.milvus.client.MilvusServiceClient;
+import io.milvus.param.ConnectParam;
+import io.milvus.grpc.DataType;
+import io.milvus.param.collection.CollectionSchemaParam;
+import io.milvus.param.collection.CollectionSchemaParam;
+import io.milvus.param.collection.FieldType;
+
+final MilvusServiceClient milvusClient = new MilvusServiceClient(
+ConnectParam.newBuilder()
+ .withUri("localhost:19530")
+ .withToken("root:Milvus")
+ .build()
+);
+
+// Define schema for the target collection
+FieldType id = FieldType.newBuilder()
+ .withName("id")
+ .withDataType(DataType.Int64)
+ .withPrimaryKey(true)
+ .withAutoID(false)
+ .build();
+
+FieldType vector = FieldType.newBuilder()
+ .withName("vector")
+ .withDataType(DataType.FloatVector)
+ .withDimension(768)
+ .build();
+
+FieldType scalar1 = FieldType.newBuilder()
+ .withName("scalar_1")
+ .withDataType(DataType.VarChar)
+ .withMaxLength(512)
+ .build();
+
+FieldType scalar2 = FieldType.newBuilder()
+ .withName("scalar_2")
+ .withDataType(DataType.Int64)
+ .build();
+
+CollectionSchemaParam schema = CollectionSchemaParam.newBuilder()
+ .withEnableDynamicField(true)
+ .addFieldType(id)
+ .addFieldType(vector)
+ .addFieldType(scalar1)
+ .addFieldType(scalar2)
+ .build();
+
+// Create a collection with the given schema
+milvusClient.createCollection(CreateCollectionParam.newBuilder()
+ .withCollectionName("quick_setup")
+ .withSchema(schema)
+ .build()
+);
+```
+
+## Import data
+
+To import the prepared data, you have to create an import job as follows:
+
+```
+export MILVUS_URI="localhost:19530"
+
+curl --request POST "http://${MILVUS_URI}/v2/vectordb/jobs/import/create" \
+--header "Content-Type: application/json" \
+--data-raw '{
+ "files": [
+ [
+ "/8ca44f28-47f7-40ba-9604-98918afe26d1/1.parquet"
+ ],
+ [
+ "/8ca44f28-47f7-40ba-9604-98918afe26d1/2.parquet"
+ ]
+ ],
+ "collectionName": "quick_setup"
+}'
+```
+
+The request body contains two fields:
+
+- `collectionName`
+
+ The name of the target collection.
+
+- `files`
+
+ A list of lists of file paths relative to the root path of the Milvus bucket on the MioIO instance started along with your Milvus instance. Possible sub-lists are as follows:
+
+ - **JSON files**
+
+ If the prepared file is in JSON format, **each sub-list should contain the path to a single prepared JSON file**.
+
+ ```
+ [
+ "/d1782fa1-6b65-4ff3-b05a-43a436342445/1.json"
+ ],
+ ```
+
+ - **Parquet files**
+
+ If the prepared file is in Parquet format, **each sub-list should contain the path to a single prepared parquet file**.
+
+ ```
+ [
+ "/a6fb2d1c-7b1b-427c-a8a3-178944e3b66d/1.parquet"
+ ]
+
+The possible return is as follows:
+
+```
+{
+ "code": 200,
+ "data": {
+ "jobId": "448707763884413158"
+ }
+}
+```
+
+## Check import progress
+
+Once you get an import job ID, you can check the import progress as follows:
+
+```
+export MILVUS_URI="localhost:19530"
+
+curl --request POST "http://${MILVUS_URI}/v2/vectordb/jobs/import/get_progress" \
+--header "Content-Type: application/json" \
+--data-raw '{
+ "jobId": "449839014328146739"
+}'
+```
+
+The possible response is as follows:
+
+```
+{
+ "code": 200,
+ "data": {
+ "collectionName": "quick_setup",
+ "completeTime": "2024-05-18T02:57:13Z",
+ "details": [
+ {
+ "completeTime": "2024-05-18T02:57:11Z",
+ "fileName": "id:449839014328146740 paths:\"/8ca44f28-47f7-40ba-9604-98918afe26d1/1.parquet\" ",
+ "fileSize": 31567874,
+ "importedRows": 100000,
+ "progress": 100,
+ "state": "Completed",
+ "totalRows": 100000
+ },
+ {
+ "completeTime": "2024-05-18T02:57:11Z",
+ "fileName": "id:449839014328146741 paths:\"/8ca44f28-47f7-40ba-9604-98918afe26d1/2.parquet\" ",
+ "fileSize": 31517224,
+ "importedRows": 100000,
+ "progress": 100,
+ "state": "Completed",
+ "totalRows": 200000
+ }
+ ],
+ "fileSize": 63085098,
+ "importedRows": 200000,
+ "jobId": "449839014328146739",
+ "progress": 100,
+ "state": "Completed",
+ "totalRows": 200000
+ }
+}
+```
+
+## List Import Jobs
+
+You can list all import jobs relative to a specific collection as follows:
+
+```
+export MILVUS_URI="localhost:19530"
+
+curl --request POST "http://${MILVUS_URI}/v2/vectordb/jobs/import/list" \
+--header "Content-Type: application/json" \
+--data-raw '{
+ "collectionName": "quick_setup"
+}'
+```
+
+The possible values are as follows:
+
+```
+{
+ "code": 200,
+ "data": {
+ "records": [
+ {
+ "collectionName": "quick_setup",
+ "jobId": "448761313698322011",
+ "progress": 50,
+ "state": "Importing"
+ }
+ ]
+ }
+}
+```
diff --git a/preview/site/en/userGuide/data-import/prepare-source-data.md b/preview/site/en/userGuide/data-import/prepare-source-data.md
new file mode 100644
index 000000000..e1a87c146
--- /dev/null
+++ b/preview/site/en/userGuide/data-import/prepare-source-data.md
@@ -0,0 +1,437 @@
+---
+id: prepare-source-data.md
+order: 0
+title: Prepare Source Data
+summary: This page discusses something you should consider before you start bulk-inserting data into your collection.
+---
+
+# Prepare Source Data
+
+This page discusses something you should consider before you start bulk-inserting data into your collection.
+
+## Before you start
+
+The target collection requires mapping the source data to its schema. The diagram below shows how acceptable source data is mapped to the schema of a target collection.
+
+
+
+You should carefully examine your data and design the schema of the target collection accordingly.
+
+Taking the JSON data in the above diagram as an example, there are two entities in the rows list, each row having six fields. The collection schema selectively includes four: **id**, **vector**, **scalar_1**, and **scalar_2**.
+
+There are two more things to consider when designing the schema:
+
+- **Whether to enable AutoID**
+
+ The **id** field serves as the primary field of the collection. To make the primary field automatically increment, you can enable **AutoID** in the schema. In this case, you should exclude the **id** field from each row in the source data.
+
+- **Whether to enable dynamic fields**
+
+ The target collection can also store fields not included in its pre-defined schema if the schema enables dynamic fields. The **$meta** field is a reserved JSON field to hold dynamic fields and their values in key-value pairs. In the above diagram, the fields **dynamic_field_1** and **dynamic_field_2** and the values will be saved as key-value pairs in the **$meta** field.
+
+The following code shows how to set up the schema for the collection illustrated in the above diagram.
+
+
+
+To obtain more information, refer to [`create_schema()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md) and [`add_field()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/CollectionSchema/add_field.md) in the SDK reference.
+
+
+
+
+
+To obtain more information, refer to [`CollectionSchema`](https://milvus.io/api-reference/java/v2.4.x/v2/CollectionSchema/CollectionSchema.md) in the SDK reference.
+
+
+
+
+
+```python
+from pymilvus import MilvusClient, DataType
+
+# You need to work out a collection schema out of your dataset.
+schema = MilvusClient.create_schema(
+ auto_id=False,
+ enable_dynamic_field=True
+)
+
+schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
+schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=768)
+schema.add_field(field_name="scalar_1", datatype=DataType.VARCHAR, max_length=512)
+schema.add_field(field_name="scalar_2", datatype=DataType.INT64)
+
+schema.verify()
+```
+
+```java
+import io.milvus.grpc.DataType;
+import io.milvus.param.collection.CollectionSchemaParam;
+import io.milvus.param.collection.FieldType;
+
+// Define schema for the target collection
+FieldType id = FieldType.newBuilder()
+ .withName("id")
+ .withDataType(DataType.Int64)
+ .withPrimaryKey(true)
+ .withAutoID(false)
+ .build();
+
+FieldType vector = FieldType.newBuilder()
+ .withName("vector")
+ .withDataType(DataType.FloatVector)
+ .withDimension(768)
+ .build();
+
+FieldType scalar1 = FieldType.newBuilder()
+ .withName("scalar_1")
+ .withDataType(DataType.VarChar)
+ .withMaxLength(512)
+ .build();
+
+FieldType scalar2 = FieldType.newBuilder()
+ .withName("scalar_2")
+ .withDataType(DataType.Int64)
+ .build();
+
+CollectionSchemaParam schema = CollectionSchemaParam.newBuilder()
+ .withEnableDynamicField(true)
+ .addFieldType(id)
+ .addFieldType(vector)
+ .addFieldType(scalar1)
+ .addFieldType(scalar2)
+ .build();
+```
+
+## Set up BulkWriter
+
+**BulkWriter** is a tool designed to convert raw datasets into a format suitable for importing via the RESTful Import API. It offers two types of writers:
+
+- **LocalBulkWriter**: Reads the designated dataset and transforms it into an easy-to-use format.
+- **RemoteBulkWriter**: Performs the same task as the LocalBulkWriter but additionally transfers the converted data files to a specified remote object storage bucket.
+
+**RemoteBulkWriter** differs from **LocalBulkWriter** in that **RemoteBulkWriter** transfers the converted data files to a target object storage bucket.
+
+### Set up LocalBulkWriter
+
+A **LocalBulkWriter** appends rows from the source dataset and commits them to a local file of the specified format.
+
+
+
+```python
+from pymilvus.bulk_writer import LocalBulkWriter, BulkFileType
+# Use `from pymilvus import LocalBulkWriter, BulkFileType`
+# when you use pymilvus earlier than 2.4.2
+
+writer = LocalBulkWriter(
+ schema=schema,
+ local_path='.',
+ segment_size=512 * 1024 * 1024, # Default value
+ file_type=BulkFileType.PARQUET
+)
+```
+
+```java
+import io.milvus.bulkwriter.LocalBulkWriter;
+import io.milvus.bulkwriter.LocalBulkWriterParam;
+import io.milvus.bulkwriter.common.clientenum.BulkFileType;
+
+LocalBulkWriterParam localBulkWriterParam = LocalBulkWriterParam.newBuilder()
+ .withCollectionSchema(schema)
+ .withLocalPath(".")
+ .withChunkSize(512 * 1024 * 1024)
+ .withFileType(BulkFileType.PARQUET)
+ .build();
+
+LocalBulkWriter localBulkWriter = new LocalBulkWriter(localBulkWriterParam);
+```
+
+
+
+When creating a **LocalBulkWriter**, you should:
+
+- Reference the created schema in `schema`.
+- Set `local_path` to the output directory.
+- Set `file_type` to the output file type.
+- If your dataset contains a large number of records, you are advised to segment your data by setting `segment_size` to a proper value.
+
+For details on parameter settings, refer to [LocalBulkWriter](https://milvus.io/api-reference/pymilvus/v2.4.x/DataImport/LocalBulkWriter/LocalBulkWriter.md) in the SDK reference.
+
+
+
+
+
+When creating a **LocalBulkWriter**, you should:
+
+- Reference the created schema in `CollectionSchema()`.
+- Set the output directory in `withLocalPath()`.
+- Set the output file type in `withFileType()`.
+- If your dataset contains a large number of records, you are advised to segment your data by setting `withChunkSize()` to a proper value.
+
+For details on parameter settings, refer to LocalBulkWriter in the SDK reference.
+
+
+
+### Set up RemoteBulkWriter
+
+Instead of committing appended data to a local file, a **RemoteBulkWriter** commits them to a remote bucket. Therefore, you should set up a **ConnectParam** object before creating a **RemoteBulkWriter**.
+
+
+
+
+```python
+from pymilvus.bulk_writer import RemoteBulkWriter
+# Use `from pymilvus import RemoteBulkWriter`
+# when you use pymilvus earlier than 2.4.2
+
+# Third-party constants
+ACCESS_KEY="minioadmin"
+SECRET_KEY="minioadmin"
+BUCKET_NAME="milvus-bucket"
+
+# Connections parameters to access the remote bucket
+conn = RemoteBulkWriter.S3ConnectParam(
+ endpoint="localhost:9000", # the default MinIO service started along with Milvus
+ access_key=ACCESS_KEY,
+ secret_key=SECRET_KEY,
+ bucket_name=BUCKET_NAME,
+ secure=False
+)
+```
+
+```java
+import io.milvus.bulkwriter.common.clientenum.BulkFileType;
+import io.milvus.bulkwriter.connect.S3ConnectParam;
+import io.milvus.bulkwriter.connect.StorageConnectParam;
+
+String ACCESS_KEY = "minioadmin";
+String SECRET_KEY = "minioadmin";
+String BUCKET_NAME = "milvus-bucket";
+
+StorageConnectParam storageConnectParam = S3ConnectParam.newBuilder()
+ .withEndpoint(MINIO_URI)
+ .withAccessKey(ACCESS_KEY)
+ .withSecretKey(SECRET_KEY)
+ .withBucketName(BUCKET_NAME)
+ .build();
+```
+
+Once the connection parameters are ready, you can reference it in the **RemoteBulkWriter** as follows:
+
+
+
+```python
+from pymilvus.bulk_writer import BulkFileType
+# Use `from pymilvus import BulkFileType`
+# when you use pymilvus earlier than 2.4.2
+
+writer = RemoteBulkWriter(
+ schema=schema,
+ remote_path="/",
+ connect_param=conn,
+ file_type=BulkFileType.PARQUET
+)
+```
+
+```java
+import io.milvus.bulkwriter.RemoteBulkWriter;
+import io.milvus.bulkwriter.RemoteBulkWriterParam;
+
+RemoteBulkWriterParam remoteBulkWriterParam = RemoteBulkWriterParam.newBuilder()
+ .withCollectionSchema(schema)
+ .withConnectParam(storageConnectParam)
+ .withChunkSize(512 * 1024 * 1024)
+ .withRemotePath("/")
+ .withFileType(BulkFileType.PARQUET)
+ .build();
+
+RemoteBulkWriter remoteBulkWriter = new RemoteBulkWriter(remoteBulkWriterParam);
+```
+
+
+
+The parameters for creating a **RemoteBulkWriter** are barely the same as those for a **LocalBulkWriter**, except `connect_param`. For details on parameter settings, refer to [RemoteBulkWriter](https://milvus.io/api-reference/pymilvus/v2.4.x/DataImport/RemoteBulkWriter/RemoteBulkWriter.md) and [ConnectParam](https://milvus.io/api-reference/pymilvus/v2.4.x/DataImport/RemoteBulkWriter/S3ConnectParam.md) in the SDK reference.
+
+
+
+
+
+The parameters for creating a **RemoteBulkWriter** are barely the same as those for a **LocalBulkWriter**, except `StorageConnectParam`. For details on parameter settings, refer to RemoteBulkWriter and StorageConnectParam in the SDK reference.
+
+
+
+## Start writing
+
+
+
+A **BulkWriter** has two methods: `append_row()` adds a row from a source dataset, and `commit()` commits added rows to a local file or a remote bucket.
+
+
+
+
+
+A **BulkWriter** has two methods: `appendRow()` adds a row from a source dataset, and `commit()` commits added rows to a local file or a remote bucket.
+
+
+
+For demonstration purposes, the following code appends randomly generated data.
+
+
+
+```python
+import random
+import string
+
+def generate_random_str(length=5):
+ letters = string.ascii_uppercase
+ digits = string.digits
+
+ return ''.join(random.choices(letters + digits, k=length))
+
+for i in range(10000):
+ writer.append_row({
+ "id": i,
+ "vector": [random.uniform(-1, 1) for _ in range(768)],
+ "scalar_1": generate_random_str(random.randint(1, 20)),
+ "scalar_2": random.randint(0, 100)
+ })
+
+writer.commit()
+```
+
+```java
+import com.alibaba.fastjson.JSONObject;
+
+for (int i = 0; i < 10000; i++) {
+ JSONObject json = new JSONObject();
+ json.put("id", i);
+ json.put("vector", get_random_vector(768));
+ json.put("scalar_1", get_random_string(20));
+ json.put("scalar_2", (long) (Math.random() * 100));
+
+ // localBulkWriter.appendRow(json);
+ remoteBulkWriter.appendRow(json);
+}
+
+// localBulkWriter.commit(false);
+remoteBulkWriter.commit(false);
+```
+
+Since the schema defined permits dynamic fields, you can also include non-schema-defined fields in the data to insert as follows.
+
+
+
+```python
+import random
+import string
+
+def generate_random_string(length=5):
+ letters = string.ascii_uppercase
+ digits = string.digits
+
+ return ''.join(random.choices(letters + digits, k=length))
+
+for i in range(10000):
+ writer.append_row({
+ "id": i,
+ "vector":[random.uniform(-1, 1) for _ in range(768)],
+ "scalar_1": generate_random_string(),
+ "scalar_2": random.randint(0, 100),
+ "dynamic_field_1": random.choice([True, False]),
+ "dynamic_field_2": random.randint(0, 100)
+ })
+
+writer.commit()
+```
+
+```java
+for (int i = 0; i < 10000; i++) {
+ JSONObject json = new JSONObject();
+ json.put("id", i);
+ json.put("vector", get_random_vector(768));
+ json.put("scalar_1", get_random_string(20));
+ json.put("scalar_2", (long) (Math.random() * 100));
+ json.put("dynamic_field_1", get_random_boolean());
+ json.put("dynamic_field_2", (long) (Math.random() * 100));
+
+ // localBulkWriter.appendRow(json);
+ remoteBulkWriter.appendRow(json);
+}
+
+// localBulkWriter.commit(false);
+remoteBulkWriter.commit(false);
+```
+
+## Verify the results
+
+
+
+To check the results, you can get the actual output path by printing the `batch_files` property of the writer.
+
+
+
+
+
+To check the results, you can get the actual output path by printing the `getBatchFiles()` method of the writer.
+
+
+
+
+
+```python
+print(writer.batch_files)
+
+# [['d4220a9e-45be-4ccb-8cb5-bf09304b9f23/1.parquet'],
+# ['d4220a9e-45be-4ccb-8cb5-bf09304b9f23/2.parquet']]
+```
+
+```java
+// localBulkWriter.getBatchFiles();
+remoteBulkWriter.getBatchFiles();
+
+//
+
+// Close the BulkWriter
+try {
+ localBulkWriter.close();
+ remoteBulkWriter.close();
+} catch (Exception e) {
+ // TODO: handle exception
+ e.printStackTrace();
+}
+```
+
+**BulkWriter** generates a UUID, creates a sub-folder using the UUID in the provided output directory, and places all generated files in the sub-folder. [Click here](https://assets.zilliz.com/bulk_writer.zip) to download the prepared sample data.
+
+Possible folder structures are as follows:
+
+```bash
+# JSON
+├── folder
+│ └── 45ae1139-1d87-4aff-85f5-0039111f9e6b
+│ └── 1.json
+
+# Parquet
+├── folder
+│ └── 45ae1139-1d87-4aff-85f5-0039111f9e6b
+│ └── 1.parquet
+```
diff --git a/preview/site/en/userGuide/delete_data.md b/preview/site/en/userGuide/delete_data.md
deleted file mode 100644
index e065abcde..000000000
--- a/preview/site/en/userGuide/delete_data.md
+++ /dev/null
@@ -1,242 +0,0 @@
----
-id: delete_data.md
-related_key: delete
-summary: Learn how to delete data in Milvus.
----
-
-# Delete Entities
-
-This topic describes how to delete entities in Milvus.
-
-Milvus supports deleting entities by primary key filtered with boolean expression.
-
-
-
-
-
-
-
-## Prepare boolean expression
-
-Prepare the boolean expression that filters the entities to delete.
-
-Milvus only supports deleting entities with clearly specified primary keys, which can be achieved merely with the term expression `in`. Other operators can be used only in query or scalar filtering in vector search. See [Boolean Expression Rules](boolean.md) for more information.
-
-The following example filters data with primary key values of `0` and `1`.
-
-
-
-
-```python
-expr = "book_id in [0,1]"
-```
-
-```javascript
-const expr = "book_id in [0,1]";
-```
-
-```java
-private static final String DELETE_EXPR = "book_id in [0,1]";
-```
-
-```shell
-delete entities -c book
-The expression to specify entities to be deleted： book_id in [0,1]
-```
-
-```curl
-"expr" = "book_id in [0,1]"
-```
-
--
-
- Deleted entities can still be retrieved immediately after the deletion if the consistency level is set lower than
Strong.
- - Entities deleted beyond the pre-specified span of time for Time Travel cannot be retrieved again. -
- Frequent deletion operations will impact the system performance. -
| Option | -Description | -
|---|---|
| -c | -The name of the collection. | -
| -p (Optional) | -The name of the partition that the entities belong to. | -
-Output:
-
-```json
-{
- "status":{},
- "IDs":{"IdField":{"IntId":{"data":[0,1]}}},
- "delete_cnt":2,
- "timestamp":434262178115092482
-}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
expr |
- Boolean expression that specifies the entities to delete. | -
partition_name (optional) |
- Name of the partition to delete entities from. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to delete entities from. | -
expr |
- Boolean expression that specifies the entities to delete. | -
partition_name (optional) |
- Name of the partition to delete entities from. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to delete entities from. | -
expr |
- Boolean expression that specifies the entities to delete. | -
PartitionName (optional) |
- Name of the partition to delete entities from. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to delete entities from. | -
expr |
- Boolean expression that specifies the entities to delete. | -
-Dropping a collection irreversibly deletes all data within it.
-
-
-
-
-
-
-```python
-from pymilvus import utility
-utility.drop_collection("book")
-```
-
-```javascript
-await milvusClient.collectionManager.dropCollection({ collection_name: "book",});
-```
-
-```go
-err = milvusClient.DropCollection(
- context.Background(), // ctx
- "book", // CollectionName
-)
-if err != nil {
- log.Fatal("fail to drop collection:", err.Error())
-}
-```
-
-```java
-milvusClient.dropCollection(
- DropCollectionParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-```
-
-```shell
-delete collection -c book
-```
-
-```curl
-curl -X 'DELETE' \
- 'http://localhost:9091/api/v1/collection' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book"
- }'
-```
-
-
-Output:
-
-```json
-{}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to drop. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to drop. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to drop. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to drop. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to drop. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to drop. | -
-Dropping an index irreversibly removes all corresponding index files.
-
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.drop_index()
-```
-
-```javascript
-await milvusClient.indexManager.dropIndex({
- collection_name: "book",
-});
-```
-
-```go
-err = milvusClient.DropIndex(
- context.Background(), // ctx
- "book", // CollectionName
- "book_intro", // fieldName
-)
-if err != nil {
- log.Fatal("fail to drop index:", err.Error())
-}
-```
-
-```java
-milvusClient.dropIndex(
- DropIndexParam.newBuilder()
- .withCollectionName("book")
- .withFieldName("book_intro")
- .build()
-);
-```
-
-```shell
-delete index -c book
-```
-
-```curl
-curl -X 'DELETE' \
- 'http://localhost:9091/api/v1/index' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "field_name": "book_intro"
- }'
-```
-
-| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to drop index from. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to drop index on. | -
fieldName |
- Name of the vector field to drop index on. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to drop index on. | -
FieldName |
- Name of the vector field to drop index on. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to drop index from. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to drop index on. | -
field_name |
- Name of the vector field to drop index on. | -
-Dropping a partition irreversibly deletes all data within it.
-
-
-
-
-
-
-```python
-from pymilvus import Collection
-collection.drop_partition("novel")
-```
-
-```javascript
-await milvusClient.partitionManager.dropPartition({
- collection_name: "book",
- partition_name: "novel",
-});
-```
-
-```go
-err := milvusClient.DropPartition(
- context.Background(), // ctx
- "book", // CollectionName
- "novel", // partitionName
-)
-if err != nil {
- log.Fatal("fail to drop partition:", err.Error())
-}
-```
-
-```java
-milvusClient.dropPartition(
- DropPartitionParam.newBuilder()
- .withCollectionName("book")
- .withPartitionName("novel")
- .build()
-);
-```
-
-```shell
-delete partition -c book -p novel
-```
-
-```curl
-curl -X 'DELETE' \
- 'http://localhost:9091/api/v1/partition' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "partition_name": "novel"
- }'
-```
-
-| Parameter | -Description | -
|---|---|
partition_name |
- Name of the partition to drop. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to drop partition from. | -
partition_name |
- Name of the partition to drop. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to drop a partition in. | -
partitionName |
- Name of the partition to drop. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to drop a partition in. | -
PartitionName |
- Name of the partition to drop. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to drop partition from. | -
| -p | -Name of the partition to drop. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to drop partition from. | -
partition_name |
- Name of the partition to drop. | -
+
+For more information on parameters, refer to [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md) in the SDK reference.
+
+
+
+
+
+```python
+import random, time
+from pymilvus import connections, MilvusClient, DataType
+
+SERVER_ADDR = "http://localhost:19530"
+
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri=SERVER_ADDR
+)
+
+# 2. Create a collection
+schema = MilvusClient.create_schema(
+ auto_id=False,
+ # highlight-next-line
+ enable_dynamic_field=True,
+)
+
+schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
+schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=5)
+
+index_params = MilvusClient.prepare_index_params()
+
+index_params.add_index(
+ field_name="id",
+ index_type="STL_SORT"
+)
+
+index_params.add_index(
+ field_name="vector",
+ index_type="IVF_FLAT",
+ metric_type="L2",
+ params={"nlist": 1024}
+)
+
+client.create_collection(
+ collection_name="test_collection",
+ schema=schema,
+ index_params=index_params
+)
+
+res = client.get_load_state(
+ collection_name="test_collection"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "state": "
+
+For more information on parameters, refer to [`insert()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/insert.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`insert()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/insert.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`insert()`](https://milvus.io/api-reference/node/v2.4.x/Vector/insert.md) in the SDK reference.
+
+
+
+
+
+```python
+res = client.insert(
+ collection_name="test_collection",
+ data=data,
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 1000,
+# "ids": [
+# 0,
+# 1,
+# 2,
+# 3,
+# 4,
+# 5,
+# 6,
+# 7,
+# 8,
+# 9,
+# "(990 more items hidden)"
+# ]
+# }
+
+time.sleep(5)
+```
+
+```java
+// 3.1 Insert data into the collection
+InsertReq insertReq = InsertReq.builder()
+ .collectionName("customized_setup")
+ .data(data)
+ .build();
+
+InsertResp insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 1000}
+
+Thread.sleep(5000);
+```
+
+```javascript
+res = await client.insert({
+ collection_name: "test_collection",
+ data: data,
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 1000
+//
+
+await sleep(5000)
+```
+
+## Search with dynamic fields
+
+If you have created the collection with the dynamic field enabled and inserted non-schema-defined fields, you can use these fields in the filter expression of a search or a query as follows.
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/search.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/search.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/node/v2.4.x/Vector/search.md) in the SDK reference.
+
+
+
+
+
+```python
+# 4. Search with dynamic fields
+query_vectors = [[0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]]
+
+res = client.search(
+ collection_name="test_collection",
+ data=query_vectors,
+ filter="color in [\"red\", \"green\"]",
+ search_params={"metric_type": "L2", "params": {"nprobe": 10}},
+ limit=3
+)
+
+print(res)
+
+# Output
+#
+# [
+# [
+# {
+# "id": 863,
+# "distance": 0.188413605093956,
+# "entity": {
+# "id": 863,
+# "color_tag": "red_2371"
+# }
+# },
+# {
+# "id": 799,
+# "distance": 0.29188022017478943,
+# "entity": {
+# "id": 799,
+# "color_tag": "red_2235"
+# }
+# },
+# {
+# "id": 564,
+# "distance": 0.3492690920829773,
+# "entity": {
+# "id": 564,
+# "color_tag": "red_9186"
+# }
+# }
+# ]
+# ]
+```
+
+```java
+// 4. Search with non-schema-defined fields
+List| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to load data into. | -
is_row_based |
- Boolean value to indicate if the file is row-based. | -
files |
- List of file names to load into Milvus. | -
partition_name (optional) |
- Name of the partition to insert data into. | -
+
+For preparations, use [`MilvusClient`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) to connect to the Milvus server and [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md) to create a collection in a quick-setup mode.
+
+
+
+
+
+For preparations, use [`MilvusClientV2`](https://milvus.io/api-reference/java/v2.4.x/v2/Client/MilvusClientV2.md) to connect to the Milvus server and [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md) to create a collection in a quick-setup mode.
+
+
+
+
+
+For preparations, use [`MilvusClient`](https://milvus.io/api-reference/node/v2.4.x/Client/MilvusClient.md) to connect to the Milvus server and [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md) to create a collection in a quick-setup mode.
+
+
+
+
+
+```python
+from pymilvus import MilvusClient
+
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri="http://localhost:19530"
+)
+
+# 2. Create a collection
+client.create_collection(
+ collection_name="quick_setup",
+ dimension=5,
+ metric_type="IP"
+)
+```
+
+```java
+import io.milvus.v2.client.ConnectConfig;
+import io.milvus.v2.client.MilvusClientV2;
+import io.milvus.v2.service.collection.request.CreateCollectionReq;
+
+String CLUSTER_ENDPOINT = "http://localhost:19530";
+
+// 1. Connect to Milvus server
+ConnectConfig connectConfig = ConnectConfig.builder()
+ .uri(CLUSTER_ENDPOINT)
+ .build();
+
+MilvusClientV2 client = new MilvusClientV2(connectConfig);
+
+// 2. Create a collection in quick setup mode
+CreateCollectionReq quickSetupReq = CreateCollectionReq.builder()
+ .collectionName("quick_setup")
+ .dimension(5)
+ .metricType("IP")
+ .build();
+
+client.createCollection(quickSetupReq);
+```
+
+```javascript
+const { MilvusClient, DataType, sleep } = require("@zilliz/milvus2-sdk-node")
+
+const address = "http://localhost:19530"
+
+// 1. Set up a Milvus Client
+client = new MilvusClient({address});
+
+// 2. Create a collection in quick setup mode
+await client.createCollection({
+ collection_name: "quick_setup",
+ dimension: 5,
+ metric_type: "IP"
+});
+```
+
+
+
+
+
+## Insert entities
+
+To insert entities, you need to organize the data into a list of dictionaries, where each dictionary represents an entity. Each dictionary contains the keys corresponding to both pre-defined and dynamic fields in the target collection.
+
+notes
+ +The collection generated in the above code contains only two fields: id (as the primary key) and vector (as the vector field), with auto_id and enable_dynamic_field settings enabled by default. When inserting data,
-
+
You do not need to include id in the data to be inserted, because the primary field automatically increments as data is inserted.
+Non-schema-defined fields will be saved as key-value pairs in a reserved JSON field named $meta.
+
+
+To insert entities into a collection, use the [`insert()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/insert.md) method.
+
+
+
+
+
+To insert entities into a collection, use the [`insert()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/insert.md) method.
+
+
+
+
+
+
+To insert entities into a collection, use the [`insert()`](https://milvus.io/api-reference/node/v2.4.x/Vector/insert.md) method.
+
+
+
+
+
+```python
+# 3. Insert some data
+data=[
+ {"id": 0, "vector": [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592], "color": "pink_8682"},
+ {"id": 1, "vector": [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, 0.2614474506242501, 0.838729485096104], "color": "red_7025"},
+ {"id": 2, "vector": [0.43742130801983836, -0.5597502546264526, 0.6457887650909682, 0.7894058910881185, 0.20785793220625592], "color": "orange_6781"},
+ {"id": 3, "vector": [0.3172005263489739, 0.9719044792798428, -0.36981146090600725, -0.4860894583077995, 0.95791889146345], "color": "pink_9298"},
+ {"id": 4, "vector": [0.4452349528804562, -0.8757026943054742, 0.8220779437047674, 0.46406290649483184, 0.30337481143159106], "color": "red_4794"},
+ {"id": 5, "vector": [0.985825131989184, -0.8144651566660419, 0.6299267002202009, 0.1206906911183383, -0.1446277761879955], "color": "yellow_4222"},
+ {"id": 6, "vector": [0.8371977790571115, -0.015764369584852833, -0.31062937026679327, -0.562666951622192, -0.8984947637863987], "color": "red_9392"},
+ {"id": 7, "vector": [-0.33445148015177995, -0.2567135004164067, 0.8987539745369246, 0.9402995886420709, 0.5378064918413052], "color": "grey_8510"},
+ {"id": 8, "vector": [0.39524717779832685, 0.4000257286739164, -0.5890507376891594, -0.8650502298996872, -0.6140360785406336], "color": "white_9381"},
+ {"id": 9, "vector": [0.5718280481994695, 0.24070317428066512, -0.3737913482606834, -0.06726932177492717, -0.6980531615588608], "color": "purple_4976"}
+]
+
+res = client.insert(
+ collection_name="quick_setup",
+ data=data
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 10,
+# "ids": [
+# 0,
+# 1,
+# 2,
+# 3,
+# 4,
+# 5,
+# 6,
+# 7,
+# 8,
+# 9
+# ]
+# }
+```
+
+```java
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import io.milvus.v2.service.vector.request.InsertReq;
+import io.milvus.v2.service.vector.response.InsertResp;
+
+// 3. Insert some data
+List
+
+- Upsert operations will not update the primary keys.
+- Upsert operations does not support collections with
+
+autoID enabled.
+- If you plan to use the upsert operation instead of insert for large-scale data ingestion (e.g. millions of vectors), be aware that this can lead to high memory consumption on Milvus data nodes.
+
+
+
+To upsert entities, use the [`upsert()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/upsert.md) method.
+
+
+
+
+
+To upsert entities, use the [`upsert()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/insert.md) method.
+
+
+
+
+
+To upsert entities, use the [`upsert()`](https://milvus.io/api-reference/node/v2.4.x/Vector/upsert.md) method.
+
+
+
+
+
+```python
+# 5. Upsert some data
+data=[
+ {"id": 0, "vector": [-0.619954382375778, 0.4479436794798608, -0.17493894838751745, -0.4248030059917294, -0.8648452746018911], "color": "black_9898"},
+ {"id": 1, "vector": [0.4762662251462588, -0.6942502138717026, -0.4490002642657902, -0.628696575798281, 0.9660395877041965], "color": "red_7319"},
+ {"id": 2, "vector": [-0.8864122635045097, 0.9260170474445351, 0.801326976181461, 0.6383943392381306, 0.7563037341572827], "color": "white_6465"},
+ {"id": 3, "vector": [0.14594326235891586, -0.3775407299900644, -0.3765479013078812, 0.20612075380355122, 0.4902678929632145], "color": "orange_7580"},
+ {"id": 4, "vector": [0.4548498669607359, -0.887610217681605, 0.5655081329910452, 0.19220509387904117, 0.016513983433433577], "color": "red_3314"},
+ {"id": 5, "vector": [0.11755001847051827, -0.7295149788999611, 0.2608115847524266, -0.1719167007897875, 0.7417611743754855], "color": "black_9955"},
+ {"id": 6, "vector": [0.9363032158314308, 0.030699901477745373, 0.8365910312319647, 0.7823840208444011, 0.2625222076909237], "color": "yellow_2461"},
+ {"id": 7, "vector": [0.0754823906014721, -0.6390658668265143, 0.5610517334334937, -0.8986261118798251, 0.9372056764266794], "color": "white_5015"},
+ {"id": 8, "vector": [-0.3038434006935904, 0.1279149203380523, 0.503958664270957, -0.2622661156746988, 0.7407627307791929], "color": "purple_6414"},
+ {"id": 9, "vector": [-0.7125086947677588, -0.8050968321012257, -0.32608864121785786, 0.3255654958645424, 0.26227968923834233], "color": "brown_7231"}
+]
+
+res = client.upsert(
+ collection_name='quick_setup',
+ data=data
+)
+
+print(res)
+
+# Output
+#
+# {
+# "upsert_count": 10
+# }
+```
+
+```java
+// 5. Upsert some data
+data = Arrays.asList(
+ new JSONObject(Map.of("id", 0L, "vector", Arrays.asList(-0.619954382375778f, 0.4479436794798608f, -0.17493894838751745f, -0.4248030059917294f, -0.8648452746018911f), "color", "black_9898")),
+ new JSONObject(Map.of("id", 1L, "vector", Arrays.asList(0.4762662251462588f, -0.6942502138717026f, -0.4490002642657902f, -0.628696575798281f, 0.9660395877041965f), "color", "red_7319")),
+ new JSONObject(Map.of("id", 2L, "vector", Arrays.asList(-0.8864122635045097f, 0.9260170474445351f, 0.801326976181461f, 0.6383943392381306f, 0.7563037341572827f), "color", "white_6465")),
+ new JSONObject(Map.of("id", 3L, "vector", Arrays.asList(0.14594326235891586f, -0.3775407299900644f, -0.3765479013078812f, 0.20612075380355122f, 0.4902678929632145f), "color", "orange_7580")),
+ new JSONObject(Map.of("id", 4L, "vector", Arrays.asList(0.4548498669607359f, -0.887610217681605f, 0.5655081329910452f, 0.19220509387904117f, 0.016513983433433577f), "color", "red_3314")),
+ new JSONObject(Map.of("id", 5L, "vector", Arrays.asList(0.11755001847051827f, -0.7295149788999611f, 0.2608115847524266f, -0.1719167007897875f, 0.7417611743754855f), "color", "black_9955")),
+ new JSONObject(Map.of("id", 6L, "vector", Arrays.asList(0.9363032158314308f, 0.030699901477745373f, 0.8365910312319647f, 0.7823840208444011f, 0.2625222076909237f), "color", "yellow_2461")),
+ new JSONObject(Map.of("id", 7L, "vector", Arrays.asList(0.0754823906014721f, -0.6390658668265143f, 0.5610517334334937f, -0.8986261118798251f, 0.9372056764266794f), "color", "white_5015")),
+ new JSONObject(Map.of("id", 8L, "vector", Arrays.asList(-0.3038434006935904f, 0.1279149203380523f, 0.503958664270957f, -0.2622661156746988f, 0.7407627307791929f), "color", "purple_6414")),
+ new JSONObject(Map.of("id", 9L, "vector", Arrays.asList(-0.7125086947677588f, -0.8050968321012257f, -0.32608864121785786f, 0.3255654958645424f, 0.26227968923834233f), "color", "brown_7231"))
+);
+
+UpsertReq upsertReq = UpsertReq.builder()
+ .collectionName("quick_setup")
+ .data(data)
+ .build();
+
+UpsertResp upsertResp = client.upsert(upsertReq);
+
+System.out.println(JSONObject.toJSON(upsertResp));
+
+// Output:
+// {"upsertCnt": 10}
+```
+
+```javascript
+// 5. Upsert some data
+data = [
+ {id: 0, vector: [-0.619954382375778, 0.4479436794798608, -0.17493894838751745, -0.4248030059917294, -0.8648452746018911], color: "black_9898"},
+ {id: 1, vector: [0.4762662251462588, -0.6942502138717026, -0.4490002642657902, -0.628696575798281, 0.9660395877041965], color: "red_7319"},
+ {id: 2, vector: [-0.8864122635045097, 0.9260170474445351, 0.801326976181461, 0.6383943392381306, 0.7563037341572827], color: "white_6465"},
+ {id: 3, vector: [0.14594326235891586, -0.3775407299900644, -0.3765479013078812, 0.20612075380355122, 0.4902678929632145], color: "orange_7580"},
+ {id: 4, vector: [0.4548498669607359, -0.887610217681605, 0.5655081329910452, 0.19220509387904117, 0.016513983433433577], color: "red_3314"},
+ {id: 5, vector: [0.11755001847051827, -0.7295149788999611, 0.2608115847524266, -0.1719167007897875, 0.7417611743754855], color: "black_9955"},
+ {id: 6, vector: [0.9363032158314308, 0.030699901477745373, 0.8365910312319647, 0.7823840208444011, 0.2625222076909237], color: "yellow_2461"},
+ {id: 7, vector: [0.0754823906014721, -0.6390658668265143, 0.5610517334334937, -0.8986261118798251, 0.9372056764266794], color: "white_5015"},
+ {id: 8, vector: [-0.3038434006935904, 0.1279149203380523, 0.503958664270957, -0.2622661156746988, 0.7407627307791929], color: "purple_6414"},
+ {id: 9, vector: [-0.7125086947677588, -0.8050968321012257, -0.32608864121785786, 0.3255654958645424, 0.26227968923834233], color: "brown_7231"}
+]
+
+res = await client.upsert({
+ collection_name: "quick_setup",
+ data: data,
+})
+
+console.log(res.upsert_cnt)
+
+// Output
+//
+// 10
+//
+```
+
+### Upsert data in partitions
+
+To upsert data into a specific partition, you can specify the name of the partition in the insert request as follows:
+
+
+
+```python
+# 6. Upsert data in partitions
+data=[
+ {"id": 10, "vector": [0.06998888224297328, 0.8582816610326578, -0.9657938677934292, 0.6527905683627726, -0.8668460657158576], "color": "black_3651"},
+ {"id": 11, "vector": [0.6060703043917468, -0.3765080534566074, -0.7710758854987239, 0.36993888322346136, 0.5507513364206531], "color": "grey_2049"},
+ {"id": 12, "vector": [-0.9041813104515337, -0.9610546012461163, 0.20033003106083358, 0.11842506351635174, 0.8327356724591011], "color": "blue_6168"},
+ {"id": 13, "vector": [0.3202914977909075, -0.7279137773695252, -0.04747830871620273, 0.8266053056909548, 0.8277957187455489], "color": "blue_1672"},
+ {"id": 14, "vector": [0.2975811497890859, 0.2946936202691086, 0.5399463833894609, 0.8385334966677529, -0.4450543984655133], "color": "pink_1601"},
+ {"id": 15, "vector": [-0.04697464305600074, -0.08509022265734134, 0.9067184632552001, -0.2281912685064822, -0.9747503428652762], "color": "yellow_9925"},
+ {"id": 16, "vector": [-0.9363075919673911, -0.8153981031085669, 0.7943039120490902, -0.2093886809842529, 0.0771191335807897], "color": "orange_9872"},
+ {"id": 17, "vector": [-0.050451522820639916, 0.18931572752321935, 0.7522886192190488, -0.9071793089474034, 0.6032647330692296], "color": "red_6450"},
+ {"id": 18, "vector": [-0.9181544231141592, 0.6700755998126806, -0.014174674636136642, 0.6325780463623432, -0.49662222164032976], "color": "purple_7392"},
+ {"id": 19, "vector": [0.11426945899602536, 0.6089190684002581, -0.5842735738352236, 0.057050610092692855, -0.035163433018196244], "color": "pink_4996"}
+]
+
+res = client.upsert(
+ collection_name="quick_setup",
+ data=data,
+ partition_name="partitionA"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "upsert_count": 10
+# }
+```
+
+```java
+import io.milvus.v2.service.vector.request.UpsertReq;
+import io.milvus.v2.service.vector.response.UpsertResp;
+
+// 6. Upsert data in parition
+
+data = Arrays.asList(
+ new JSONObject(Map.of("id", 10L, "vector", Arrays.asList(0.06998888224297328f, 0.8582816610326578f, -0.9657938677934292f, 0.6527905683627726f, -0.8668460657158576f), "color", "black_3651")),
+ new JSONObject(Map.of("id", 11L, "vector", Arrays.asList(0.6060703043917468f, -0.3765080534566074f, -0.7710758854987239f, 0.36993888322346136f, 0.5507513364206531f), "color", "grey_2049")),
+ new JSONObject(Map.of("id", 12L, "vector", Arrays.asList(-0.9041813104515337f, -0.9610546012461163f, 0.20033003106083358f, 0.11842506351635174f, 0.8327356724591011f), "color", "blue_6168")),
+ new JSONObject(Map.of("id", 13L, "vector", Arrays.asList(0.3202914977909075f, -0.7279137773695252f, -0.04747830871620273f, 0.8266053056909548f, 0.8277957187455489f), "color", "blue_1672")),
+ new JSONObject(Map.of("id", 14L, "vector", Arrays.asList(0.2975811497890859f, 0.2946936202691086f, 0.5399463833894609f, 0.8385334966677529f, -0.4450543984655133f), "color", "pink_1601")),
+ new JSONObject(Map.of("id", 15L, "vector", Arrays.asList(-0.04697464305600074f, -0.08509022265734134f, 0.9067184632552001f, -0.2281912685064822f, -0.9747503428652762f), "color", "yellow_9925")),
+ new JSONObject(Map.of("id", 16L, "vector", Arrays.asList(-0.9363075919673911f, -0.8153981031085669f, 0.7943039120490902f, -0.2093886809842529f, 0.0771191335807897f), "color", "orange_9872")),
+ new JSONObject(Map.of("id", 17L, "vector", Arrays.asList(-0.050451522820639916f, 0.18931572752321935f, 0.7522886192190488f, -0.9071793089474034f, 0.6032647330692296f), "color", "red_6450")),
+ new JSONObject(Map.of("id", 18L, "vector", Arrays.asList(-0.9181544231141592f, 0.6700755998126806f, -0.014174674636136642f, 0.6325780463623432f, -0.49662222164032976f), "color", "purple_7392")),
+ new JSONObject(Map.of("id", 19L, "vector", Arrays.asList(0.11426945899602536f, 0.6089190684002581f, -0.5842735738352236f, 0.057050610092692855f, -0.035163433018196244f), "color", "pink_4996"))
+);
+
+upsertReq = UpsertReq.builder()
+ .collectionName("quick_setup")
+ .data(data)
+ .partitionName("partitionA")
+ .build();
+
+upsertResp = client.upsert(upsertReq);
+
+System.out.println(JSONObject.toJSON(upsertResp));
+
+// Output:
+// {"upsertCnt": 10}
+```
+
+```javascript
+// 6. Upsert data in partitions
+data = [
+ {id: 10, vector: [0.06998888224297328, 0.8582816610326578, -0.9657938677934292, 0.6527905683627726, -0.8668460657158576], color: "black_3651"},
+ {id: 11, vector: [0.6060703043917468, -0.3765080534566074, -0.7710758854987239, 0.36993888322346136, 0.5507513364206531], color: "grey_2049"},
+ {id: 12, vector: [-0.9041813104515337, -0.9610546012461163, 0.20033003106083358, 0.11842506351635174, 0.8327356724591011], color: "blue_6168"},
+ {id: 13, vector: [0.3202914977909075, -0.7279137773695252, -0.04747830871620273, 0.8266053056909548, 0.8277957187455489], color: "blue_1672"},
+ {id: 14, vector: [0.2975811497890859, 0.2946936202691086, 0.5399463833894609, 0.8385334966677529, -0.4450543984655133], color: "pink_1601"},
+ {id: 15, vector: [-0.04697464305600074, -0.08509022265734134, 0.9067184632552001, -0.2281912685064822, -0.9747503428652762], color: "yellow_9925"},
+ {id: 16, vector: [-0.9363075919673911, -0.8153981031085669, 0.7943039120490902, -0.2093886809842529, 0.0771191335807897], color: "orange_9872"},
+ {id: 17, vector: [-0.050451522820639916, 0.18931572752321935, 0.7522886192190488, -0.9071793089474034, 0.6032647330692296], color: "red_6450"},
+ {id: 18, vector: [-0.9181544231141592, 0.6700755998126806, -0.014174674636136642, 0.6325780463623432, -0.49662222164032976], color: "purple_7392"},
+ {id: 19, vector: [0.11426945899602536, 0.6089190684002581, -0.5842735738352236, 0.057050610092692855, -0.035163433018196244], color: "pink_4996"}
+]
+
+res = await client.upsert({
+ collection_name: "quick_setup",
+ data: data,
+ partition_name: "partitionA"
+})
+
+console.log(res.upsert_cnt)
+
+// Output
+//
+// 10
+//
+```
+
+The output is a dictionary containing the statistics on the affected entities. For details on partition operations, refer to [Manage Partitions](manage-partitions.md).
+
+## Delete entities
+
+
+
+If an entity is no longer needed, you can delete it from the collection by using [`delete()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/delete.md).
+
+
+
+
+
+If an entity is no longer needed, you can delete it from the collection by using [`delete()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/delete.md).
+
+
+
+
+
+If an entity is no longer needed, you can delete it from the collection by using [`delete()`](https://milvus.io/api-reference/node/v2.4.x/Vector/delete.md).
+
+
+
+Milvus offers two ways for you to identify the entities to delete.
+
+- __Delete entities by filter.__
+
+
+
+ ```python
+ # 7. Delete entities
+ res = client.delete(
+ collection_name="quick_setup",
+ filter="id in [4,5,6]"
+ )
+
+ print(res)
+
+ # Output
+ #
+ # {
+ # "delete_count": 3
+ # }
+ ```
+
+ ```java
+ import io.milvus.v2.service.vector.request.DeleteReq;
+ import io.milvus.v2.service.vector.response.DeleteResp;
+
+
+ // 7. Delete entities
+
+ DeleteReq deleteReq = DeleteReq.builder()
+ .collectionName("quick_setup")
+ .filter("id in [4, 5, 6]")
+ .build();
+
+ DeleteResp deleteResp = client.delete(deleteReq);
+
+ System.out.println(JSONObject.toJSON(deleteResp));
+
+ // Output:
+ // {"deleteCnt": 3}
+ ```
+
+ ```javascript
+ // 7. Delete entities
+ res = await client.delete({
+ collection_name: "quick_setup",
+ filter: "id in [4,5,6]"
+ })
+
+ console.log(res.delete_cnt)
+
+ // Output
+ //
+ // 3
+ //
+ ```
+
+- __Delete entities by IDs.__
+
+ The following snippets demonstrate how to delete entities by IDs from a specific partition. It also works if you leave the partition name unspecified.
+
+
+
+ ```python
+ res = client.delete(
+ collection_name="quick_setup",
+ ids=[18, 19],
+ partition_name="partitionA"
+ )
+
+ print(res)
+
+ # Output
+ #
+ # {
+ # "delete_count": 2
+ # }
+ ```
+
+ ```java
+ deleteReq = DeleteReq.builder()
+ .collectionName("quick_setup")
+ .ids(Arrays.asList(18L, 19L))
+ .partitionName("partitionA")
+ .build();
+
+ deleteResp = client.delete(deleteReq);
+
+ System.out.println(JSONObject.toJSON(deleteResp));
+
+ // Output:
+ // {"deleteCnt": 2}
+ ```
+
+ ```javascript
+ res = await client.delete({
+ collection_name: "quick_setup",
+ ids: [18, 19],
+ partition_name: "partitionA"
+ })
+
+ console.log(res.delete_cnt)
+
+ // Output
+ //
+ // 2
+ //
+ ```
+
+For details on how to use filter expressions, refer to [Get & Scalar Query](get-and-scalar-query.md).
diff --git a/preview/site/en/userGuide/insert_data.md b/preview/site/en/userGuide/insert_data.md
deleted file mode 100644
index f668d5f8e..000000000
--- a/preview/site/en/userGuide/insert_data.md
+++ /dev/null
@@ -1,393 +0,0 @@
----
-id: insert_data.md
-related_key: insert
-summary: Learn how to insert data in Milvus.
----
-
-# Insert Entities
-
-This topic describes how to insert data in Milvus via client.
-
-You can also migrate data to Milvus with [MilvusDM](migrate_overview.md), an open-source tool designed specifically for importing and exporting data with Milvus.
-
-Milvus 2.1 supports VARCHAR data type on scalar field. When building indexes for VARCHAR-type scalar fields, the default index type is dictionary tree.
-
-The following example inserts 2,000 rows of randomly generated data as the example data (Milvus CLI example uses a pre-built, remote CSV file containing similar data). Real applications will likely use much higher dimensional vectors than the example. You can prepare your own data to replace the example.
-
-## Prepare data
-
-First, prepare the data to insert. Data type of the data to insert must match the schema of the collection, otherwise Milvus will raise exception.
-
-
-
-
-```python
-import random
-data = [
- [i for i in range(2000)],
- [str(i) for i in range(2000)],
- [i for i in range(10000, 12000)],
- [[random.random() for _ in range(2)] for _ in range(2000)],
-]
-```
-
-```javascript
-const data = Array.from({ length: 2000 }, (v,k) => ({
- "book_id": k,
- "word_count": k+10000,
- "book_intro": Array.from({ length: 2 }, () => Math.random()),
-}));
-```
-
-```go
-bookIDs := make([]int64, 0, 2000)
-wordCounts := make([]int64, 0, 2000)
-bookIntros := make([][]float32, 0, 2000)
-for i := 0; i < 2000; i++ {
- bookIDs = append(bookIDs, int64(i))
- wordCounts = append(wordCounts, int64(i+10000))
- v := make([]float32, 0, 2)
- for j := 0; j < 2; j++ {
- v = append(v, rand.Float32())
- }
- bookIntros = append(bookIntros, v)
-}
-idColumn := entity.NewColumnInt64("book_id", bookIDs)
-wordColumn := entity.NewColumnInt64("word_count", wordCounts)
-introColumn := entity.NewColumnFloatVector("book_intro", 2, bookIntros)
-```
-
-```java
-Random ran = new Random();
-List
-Output:
-
-```json
-{
- "status":{},
- "IDs":{
- "IdField":{
- "IntId":{"data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100]
- }
- }
- },
- "succ_index":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99],
- "insert_cnt":100,
- "timestamp":434262073374408706
-}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
data |
- Data to insert into Milvus. | -
partition_name (optional) |
- Name of the partition to insert data into. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to insert data into. | -
partition_name (optional) |
- Name of the partition to insert data into. | -
fields_data |
- Data to insert into Milvus. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to insert data into. | -
partitionName |
- Name of the partition to insert data into. Data will be inserted in the default partition if left blank. | -
columnarData |
- Data to insert into each field. | -
| Parameter | -Description | -
|---|---|
fieldName |
- Name of the field to insert data into. | -
DataType |
- Data type of the field to insert data into. | -
data |
- Data to insert into each field. | -
CollectionName |
- Name of the collection to insert data into. | -
PartitionName (optional) |
- Name of the partition to insert data into. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to insert data into. | -
| -p (Optional) | -Name of the partition to insert data into. | -
| Parameter | -Description | -Option | -
|---|---|---|
collection_name |
- Name of the collection to insert data into. | -N/A | -
fields_data |
- Data to insert into Milvus. | -N/A | -
field_name |
- Name of the field to insert data into. | -N/A | -
type |
- Data type of the field to insert data into. | -
- Enums:
- 1: "Bool", - 2: "Int8", - 3: "Int16", - 4: "Int32", - 5: "Int64", - 10: "Float", - 11: "Double", - 20: "String", - 21: "VarChar", - 100: "BinaryVector", - 101: "FloatVector", - |
-
field |
- The data of one column to be inserted. | -N/A | -
num_rows |
- Number of rows to be inserted. The number should be the same as the length of each field array. |
- N/A | -
-
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.load(replica_number=2)
-```
-
-```javascript
-await milvusClient.collectionManager.loadCollection({
- collection_name: "book",
-});
-```
-
-```go
-err := milvusClient.LoadCollection(
- context.Background(), // ctx
- "book", // CollectionName
- false // async
-)
-if err != nil {
- log.Fatal("failed to load collection:", err.Error())
-}
-```
-
-```java
-milvusClient.loadCollection(
- LoadCollectionParam.newBuilder()
- .withCollectionName("book")
- .build()
-);
-```
-
-```shell
-load -c book
-```
-
-``` curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/collection/load' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book"
- }'
-```
-
--
-
- In current release, volume of the data to load must be under 90% of the total memory resources of all query nodes to reserve memory resources for execution engine. -
- In current release, all on-line query nodes will be divided into multiple replica groups according to the replica number specified by user. All replica groups shall have minimal memory resources to load one replica of the provided collection. Otherwise, an error will be returned. -
-Output:
-
-```json
-{}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
partition_name (optional) |
- Name of the partition to load. | -
replica_number (optional) |
- Number of the replica to load. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to load. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to load. | -
async |
- Switch to control sync/async behavior. The deadline of context is not applied in sync load. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to load. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to load. | -
| -p (Optional/Multiple) | -The name of the partition to load. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to load. | -
-
-
-
-
-
-```python
-from pymilvus import Collection
-collection = Collection("book") # Get an existing collection.
-collection.load(["novel"], replica_number=2)
-
-# Or you can load a partition with the partition as an object
-from pymilvus import Partition
-partition = Partition("novel") # Get an existing partition.
-partition.load(replica_number=2)
-```
-
-```javascript
-await milvusClient.partitionManager.loadPartitions({
- collection_name: "book",
- partition_names: ["novel"],
-});
-```
-
-```go
-err := milvusClient.LoadPartitions(
- context.Background(), // ctx
- "book", // CollectionName
- []string{"novel"}, // partitionNames
- false // async
-)
-if err != nil {
- log.Fatal("failed to load partitions:", err.Error())
-}
-```
-
-```java
-milvusClient.loadPartitions(
- LoadPartitionsParam.newBuilder()
- .withCollectionName("book")
- .withPartitionNames(["novel"])
- .build()
-);
-```
-
-```shell
-load -c book -p novel
-```
-
-``` curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/partitions/load' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "book",
- "partition_names": ["novel"],
- "replica_number": 1
- }'
-```
-
--
-
- In current release, volume of the data to load must be under 90% of the total memory resources of all query nodes to reserve memory resources for execution engine. -
- In current release, all on-line query nodes will be divided into multiple replica groups according to the replica number specified by user. All replica groups shall have minimal memory resources to load one replica of the provided collection. Otherwise, an error will be returned. -
| Parameter | -Description | -
|---|---|
partition_name |
- Name of the partition. | -
replica_number (optional) |
- Number of the replica to load. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to load partitions from. | -
partition_names |
- List of names of the partitions to load. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to load partitions from. | -
partitionNames |
- List of names of the partitions to load. | -
async |
- Switch to control sync/async behavior. The deadline of context is not applied in sync load. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to load partitions from. | -
PartitionNames |
- List of names of the partitions to load. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to load partitions from. | -
| -p (Multiple) | -The name of the partition to load. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to load partitions from. | -
partition_names |
- List of names of the partitions to load. | -
replica_number (optional) |
- Number of the replica to load. | -
+
+For quick setup, use the [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md) method of the [`MilvusClient`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) class to create a collection with the specified name and dimension.
+
+
+
+
+
+For quick setup, use the [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md) method of the [`MilvusClientV2`](https://milvus.io/api-reference/java/v2.4.x/v2/Client/MilvusClientV2.md) class to create a collection with the specified name and dimension.
+
+
+
+
+
+For quick setup, use the [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md) method of the [`MilvusClient`](https://milvus.io/api-reference/node/v2.4.x/Client/MilvusClient.md) class to create a collection with the specified name and dimension.
+
+
+
+
+
+For quick setup, use the [`POST /v2/vectordb/collections/create`](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/Create.md) API endpoint to create a collection with the specified name and dimension.
+
+
+
+
+
+```python
+from pymilvus import MilvusClient, DataType
+
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri="http://localhost:19530"
+)
+
+# 2. Create a collection in quick setup mode
+client.create_collection(
+ collection_name="quick_setup",
+ dimension=5
+)
+
+res = client.get_load_state(
+ collection_name="quick_setup"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "state": "
+
+To set up a schema, use [`create_schema()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md) to create a schema object and [`add_field()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/CollectionSchema/add_field.md) to add fields to the schema.
+
+
+
+
+
+To set up a schema, use [`createSchema()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createSchema.md) to create a schema object and [`addField()`](https://milvus.io/api-reference/java/v2.4.x/v2/CollectionSchema/addField.md) to add fields to the schema.
+
+
+
+
+
+To set up a schema, use [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md).
+
+
+
+
+
+To set up a schema, you need to define a JSON object that follows the schema format as displayed on the [`POST /v2/vectordb/collections/create`](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/Create.md) API endpoint reference page.
+
+
+
+
+
+```python
+# 3. Create a collection in customized setup mode
+
+# 3.1. Create schema
+schema = MilvusClient.create_schema(
+ auto_id=False,
+ enable_dynamic_field=True,
+)
+
+# 3.2. Add fields to schema
+schema.add_field(field_name="my_id", datatype=DataType.INT64, is_primary=True)
+schema.add_field(field_name="my_vector", datatype=DataType.FLOAT_VECTOR, dim=5)
+```
+
+```java
+import io.milvus.v2.common.DataType;
+import io.milvus.v2.service.collection.request.CreateCollectionReq;
+
+// 3. Create a collection in customized setup mode
+
+// 3.1 Create schema
+CreateCollectionReq.CollectionSchema schema = client.createSchema();
+
+// 3.2 Add fields to schema
+schema.addField(AddFieldReq.builder()
+ .fieldName("my_id")
+ .dataType(DataType.Int64)
+ .isPrimaryKey(true)
+ .autoID(false)
+ .build());
+
+schema.addField(AddFieldReq.builder()
+ .fieldName("my_vector")
+ .dataType(DataType.FloatVector)
+ .dimension(5)
+ .build());
+```
+
+```javascript
+// 3. Create a collection in customized setup mode
+// 3.1 Define fields
+const fields = [
+ {
+ name: "my_id",
+ data_type: DataType.Int64,
+ is_primary_key: true,
+ auto_id: false
+ },
+ {
+ name: "my_vector",
+ data_type: DataType.FloatVector,
+ dim: 5
+ },
+]
+```
+
+```shell
+export fields='[{ \
+ "fieldName": "my_id", \
+ "dataType": "Int64", \
+ "isPrimary": true \
+}, \
+{ \
+ "fieldName": "my_vector", \
+ "dataType": "FloatVector", \
+ "elementTypeParams": { \
+ "dim": 5 \
+ } \
+}]'
+```
+
+| Parameter | +Description | +
|---|---|
auto_id |
+ Whether allows the primary field to automatically increment. Setting this to True makes the primary field automatically increment. In this case, the primary field should not be included in the data to insert to avoid errors. |
+
enable_dynamic_field |
+ Whether allows Milvus saves the values of undefined fields in a dynamic field if the data being inserted into the target collection includes fields that are not defined in the collection's schema. When you set this to True, Milvus will create a field called $meta to store any undefined fields and their values from the data that is inserted. |
+
field_name |
+ The name of the field. | +
datatype |
+ The data type of the field. For a list of available data types, refer to DataType. | +
is_primary |
+ Whether the current field is the primary field in a collection. Each collection has only one primary field. A primary field should be of either the DataType.INT64 type or the DataType.VARCHAR type. |
+
dim |
+ The dimension of the vector embeddings. This is mandatory for a field of the DataType.FLOAT_VECTOR, DataType.BINARY_VECTOR, DataType.FLOAT16_VECTOR, or DataType.BFLOAT16_VECTOR type. If you use DataType.SPARSE_FLOAT_VECTOR, omit this parameter. |
+
| Parameter | +Description | +
|---|---|
fieldName |
+ The name of the field. | +
dataType |
+ The data type of the field. For a list of available data types, refer to DataType. | +
isPrimaryKey |
+ Whether the current field is the primary field in a collection. Each collection has only one primary field. A primary field should be of either the DataType.Int64 type or the DataType.VarChar type. |
+
autoID |
+ Whether allows the primary field to automatically increment. Setting this to true makes the primary field automatically increment. In this case, the primary field should not be included in the data to insert to avoid errors. |
+
dimension |
+ The dimension of the vector embeddings. This is mandatory for a field of the DataType.FloatVector, DataType.BinaryVector, DataType.Float16Vector, or DataType.BFloat16Vector type. |
+
| Parameter | +Description | +
|---|---|
name |
+ The name of the field. | +
data_type |
+ The data type of the field. For an enumeration of all available data types, refer to DataType. | +
is_primary_key |
+ Whether the current field is the primary field in a collection. Each collection has only one primary field. A primary field should be of either the DataType.INT64 type or the DataType.VARCHAR type. |
+
auto_id |
+ Whether the primary field automatically increments upon data insertions into this collection. The value defaults to False. Setting this to True makes the primary field automatically increment. Skip this parameter if you need to set up a collection with a customized schema. |
+
dim |
+ The dimensionality of the collection field that holds vector embeddings. The value should be an integer greater than 1 and is usually determined by the model you use to generate vector embeddings. |
+
| Parameter | +Description | +
|---|---|
fieldName |
+ The name of the field to create in the target collection. | +
dataType |
+ The data type of the field values. | +
isPrimary |
+ Whether the current field is the primary field. Setting this to True makes the current field the primary field. |
+
elementTypeParams |
+ Extra field parameters. | +
dim |
+ An optional parameter for FloatVector or BinaryVector fields that determines the vector dimension. | +
+
+To set up index parameters, use [`prepare_index_params()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/prepare_index_params.md) to prepare index parameters and [`add_index()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/add_index.md) to add the index.
+
+
+
+
+
+To set up index parameters, use [IndexParam](https://milvus.io/api-reference/java/v2.4.x/v2/Management/IndexParam.md).
+
+
+
+
+
+To set up index parameters, use [`createIndex()`](https://milvus.io/api-reference/node/v2.4.x/Management/createIndex.md).
+
+
+
+
+
+To set up index parameters, you need to define a JSON object that follows the index parameters format as displayed on the [`POST /v2/vectordb/collections/create`](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/Create.md) API endpoint reference page.
+
+
+
+
+
+```python
+# 3.3. Prepare index parameters
+index_params = client.prepare_index_params()
+
+# 3.4. Add indexes
+index_params.add_index(
+ field_name="my_id",
+ index_type="STL_SORT"
+)
+
+index_params.add_index(
+ field_name="my_vector",
+ index_type="IVF_FLAT",
+ metric_type="IP",
+ params={ "nlist": 128 }
+)
+```
+
+```java
+import io.milvus.v2.common.IndexParam;
+
+// 3.3 Prepare index parameters
+IndexParam indexParamForIdField = IndexParam.builder()
+ .fieldName("my_id")
+ .indexType(IndexParam.IndexType.STL_SORT)
+ .build();
+
+IndexParam indexParamForVectorField = IndexParam.builder()
+ .fieldName("my_vector")
+ .indexType(IndexParam.IndexType.IVF_FLAT)
+ .metricType(IndexParam.MetricType.L2)
+ .extraParams(Map.of("nlist", 1024))
+ .build();
+
+List| Parameter | +Description | +
|---|---|
field_name |
+ The name of the target file to apply this object applies. | +
index_type |
+ The name of the algorithm used to arrange data in the specific field. For applicable algorithms, refer to In-memory Index and On-disk Index. | +
metric_type |
+ The algorithm that is used to measure similarity between vectors. Possible values are IP, L2, COSINE, JACCARD, HAMMING. This is available only when the specified field is a vector field. For more information, refer to Indexes supported in Milvus. | +
params |
+ The fine-tuning parameters for the specified index type. For details on possible keys and value ranges, refer to In-memory Index. | +
| Parameter | +Description | +
|---|---|
fieldName |
+ The name of the target field to apply this IndexParam object applies. | +
indexType |
+ The name of the algorithm used to arrange data in the specific field. For applicable algorithms, refer to In-memory Index and On-disk Index. | +
metricType |
+ The distance metric to use for the index. Possible values are IP, L2, COSINE, JACCARD, HAMMING. | +
extraParams |
+ Extra index parameters. For details, refer to In-memory Index and On-disk Index. | +
| Parameter | +Description | +
|---|---|
field_name |
+ The name of the target field on which an index is to be created. | +
index_type |
+ The name of the algorithm used to arrange data in the specific field. For applicable algorithms, refer to In-memory Index and On-disk Index. | +
metric_type |
+ The algorithm that is used to measure similarity between vectors. Possible values are IP, L2, COSINE, JACCARD, HAMMING. This is available only when the specified field is a vector field. For more information, refer to Indexes supported in Milvus. | +
params |
+ The fine-tuning parameters for the specified index type. For details on possible keys and value ranges, refer to In-memory Index. | +
| Parameter | +Description | +
|---|---|
fieldName |
+ The name of the target field on which an index is to be created. | +
indexName |
+ The name of the index to create. The value defaults to the target field name. | +
metricType |
+ The algorithm that is used to measure similarity between vectors. Possible values are IP, L2, COSINE, JACCARD, HAMMING. This is available only when the specified field is a vector field. For more information, refer to Indexes supported in Milvus. | +
params |
+ The index type and related settings. For details, refer to In-memory Index. | +
params.index_type |
+ The type of the index to create. | +
params.nlist |
+ The number of cluster units. This applies to IVF-related index types. | +
+
+Use [create_collection()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md) to create a collection with the specified schema and index parameters and [get_load_state()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/get_load_state.md) to check the load state of the collection.
+
+
+
+
+
+Use [createCollection()](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md) to create a collection with the specified schema and index parameters and [getLoadState()](https://milvus.io/api-reference/java/v2.4.x/v2/Management/getLoadState.md) to check the load state of the collection.
+
+
+
+
+
+Use [createCollection()](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md) to create a collection with the specified schema and index parameters and [getLoadState()](https://milvus.io/api-reference/node/v2.4.x/Management/getLoadState.md) to check the load state of the collection.
+
+
+
+- __Create a collection with the index loaded simultaneously upon creation.__
+
+
+
+ ```python
+ # 3.5. Create a collection with the index loaded simultaneously
+ client.create_collection(
+ collection_name="customized_setup_1",
+ schema=schema,
+ index_params=index_params
+ )
+
+ time.sleep(5)
+
+ res = client.get_load_state(
+ collection_name="customized_setup_1"
+ )
+
+ print(res)
+
+ # Output
+ #
+ # {
+ # "state": "| Parameter | +Description | +
|---|---|
collection_name |
+ The name of the collection. | +
schema |
+ The schema of this collection. Setting this to None indicates this collection will be created with default settings. To set up a collection with a customized schema, you need to create a CollectionSchema object and reference it here. In this case, Milvus ignores all other schema-related settings carried in the request. |
+
index_params |
+ The parameters for building the index on the vector field in this collection. To set up a collection with a customized schema and automatically load the collection to memory, you need to create an IndexParams object and reference it here. You should at least add an index for the vector field in this collection. You can also skip this parameter if you prefer to set up the index parameters later on. |
+
| Parameter | +Description | +
|---|---|
collectionName |
+ The name of the collection. | +
collectionSchema |
+ The schema of this collection. Leaving it empty indicates this collection will be created with default settings. To set up a collection with a customized schema, you need to create a CollectionSchema object and reference it here. |
+
indexParams |
+ The parameters for building the index on the vector field in this collection. To set up a collection with a customized schema and automatically load the collection to memory, create an IndexParams object with a list of IndexParam objects and reference it here. | +
| Parameter | +Description | +
|---|---|
collection_name |
+ The name of the collection. | +
fields |
+ The fields in the collection. | +
index_params |
+ The index parameters for the collection to create. | +
| Parameter | +Description | +
|---|---|
collectionName |
+ The name of the collection. | +
schema |
+ The schema is responsible for organizing data in the target collection. A valid schema should have multiple fields, which must include a primary key, a vector field, and several scalar fields. | +
schema.autoID |
+ Whether allows the primary field to automatically increment. Setting this to True makes the primary field automatically increment. In this case, the primary field should not be included in the data to insert to avoid errors. Set this parameter in the field with is_primary set to True. | +
schema.enableDynamicField |
+ Whether allows to use the reserved $meta field to hold non-schema-defined fields in key-value pairs. | +
fields |
+ A list of field objects. | +
fields.fieldName |
+ The name of the field to create in the target collection. | +
fields.dataType |
+ The data type of the field values. | +
fields.isPrimary |
+ Whether the current field is the primary field. Setting this to True makes the current field the primary field. | +
fields.elementTypeParams |
+ Extra field parameters. | +
fields.elementTypeParams.dim |
+ An optional parameter for FloatVector or BinaryVector fields that determines the vector dimension. | +
| Parameter | +Description | +
|---|---|
collection_name |
+ The name of the collection. | +
index_params |
+ An IndexParams object containing a list of IndexParam objects. | +
| Parameter | +Description | +
|---|---|
collectionName |
+ The name of the collection. | +
indexParams |
+ A list of IndexParam objects. | +
| Parameter | +Description | +
|---|---|
collection_name |
+ The name of the collection. | +
field_name |
+ The name of the field in which to create an index. | +
index_type |
+ The name of the algorithm used to arrange data in the specific field. For applicable algorithms, refer to In-memory Index and On-disk Index. | +
metric_type |
+ The algorithm that is used to measure similarity between vectors. Possible values are IP, L2, COSINE, JACCARD, HAMMING. This is available only when the specified field is a vector field. For more information, refer to Indexes supported in Milvus. | +
params |
+ The fine-tuning parameters for the specified index type. For details on possible keys and value ranges, refer to In-memory Index. | +
| Parameter | +Description | +
|---|---|
collectionName |
+ The name of the collection. | +
indexParams |
+ The index parameters for the collection to create. | +
indexParams.metricType |
+ The similarity metric type used to build the index. The value defaults to COSINE. | +
indexParams.fieldName |
+ The name of the target field on which an index is to be created. | +
indexParams.indexName |
+ The name of the index to create, the value defaults to the target field name. | +
indexParams.indexConfig.index_type |
+ The type of the index to create. | +
indexParams.indexConfig.nlist |
+ The number of cluster units. This applies to IVF-related index types. | +
+
+To check the details of an existing collection, use [describe_collection()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/describe_collection.md).
+
+
+
+
+
+To check the details of an existing collection, use [describeCollection()](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/describeCollection.md).
+
+
+
+
+
+To check the details of an existing collection, use [describeCollection()](https://milvus.io/api-reference/node/v2.4.x/Collections/describeCollection.md).
+
+
+
+
+
+To view the definition of a collection, you can use the [`POST /v2/vectordb/collections/describe`](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/Describe.md) and the [`POST /v2/vectordb/collections/list`](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/List.md) API endpoints.
+
+
+
+
+
+```python
+# 5. View Collections
+res = client.describe_collection(
+ collection_name="customized_setup_2"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "collection_name": "customized_setup_2",
+# "auto_id": false,
+# "num_shards": 1,
+# "description": "",
+# "fields": [
+# {
+# "field_id": 100,
+# "name": "my_id",
+# "description": "",
+# "type": 5,
+# "params": {},
+# "element_type": 0,
+# "is_primary": true
+# },
+# {
+# "field_id": 101,
+# "name": "my_vector",
+# "description": "",
+# "type": 101,
+# "params": {
+# "dim": 5
+# },
+# "element_type": 0
+# }
+# ],
+# "aliases": [],
+# "collection_id": 448143479230158446,
+# "consistency_level": 2,
+# "properties": {},
+# "num_partitions": 1,
+# "enable_dynamic_field": true
+# }
+
+```
+
+```java
+import io.milvus.v2.service.collection.request.DescribeCollectionReq;
+import io.milvus.v2.service.collection.response.DescribeCollectionResp;
+
+// 4. View collections
+DescribeCollectionReq describeCollectionReq = DescribeCollectionReq.builder()
+ .collectionName("customized_setup_2")
+ .build();
+
+DescribeCollectionResp describeCollectionRes = client.describeCollection(describeCollectionReq);
+
+System.out.println(JSONObject.toJSON(describeCollectionRes));
+
+// Output:
+// {
+// "createTime": 449005822816026627,
+// "collectionSchema": {"fieldSchemaList": [
+// {
+// "autoID": false,
+// "dataType": "Int64",
+// "name": "my_id",
+// "description": "",
+// "isPrimaryKey": true,
+// "maxLength": 65535,
+// "isPartitionKey": false
+// },
+// {
+// "autoID": false,
+// "dataType": "FloatVector",
+// "name": "my_vector",
+// "description": "",
+// "isPrimaryKey": false,
+// "dimension": 5,
+// "maxLength": 65535,
+// "isPartitionKey": false
+// }
+// ]},
+// "vectorFieldName": ["my_vector"],
+// "autoID": false,
+// "fieldNames": [
+// "my_id",
+// "my_vector"
+// ],
+// "description": "",
+// "numOfPartitions": 1,
+// "primaryFieldName": "my_id",
+// "enableDynamicField": true,
+// "collectionName": "customized_setup_2"
+// }
+```
+
+```javascript
+// 5. View Collections
+res = await client.describeCollection({
+ collection_name: "customized_setup_2"
+})
+
+console.log(res)
+
+// Output
+//
+// {
+// virtual_channel_names: [ 'by-dev-rootcoord-dml_13_449007919953017716v0' ],
+// physical_channel_names: [ 'by-dev-rootcoord-dml_13' ],
+// aliases: [],
+// start_positions: [],
+// properties: [],
+// status: {
+// extra_info: {},
+// error_code: 'Success',
+// reason: '',
+// code: 0,
+// retriable: false,
+// detail: ''
+// },
+// schema: {
+// fields: [ [Object], [Object] ],
+// properties: [],
+// name: 'customized_setup_2',
+// description: '',
+// autoID: false,
+// enable_dynamic_field: false
+// },
+// collectionID: '449007919953017716',
+// created_timestamp: '449024569603784707',
+// created_utc_timestamp: '1712892797866',
+// shards_num: 1,
+// consistency_level: 'Bounded',
+// collection_name: 'customized_setup_2',
+// db_name: 'default',
+// num_partitions: '1'
+// }
+//
+```
+
+```shell
+curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/describe" \
+-H "Content-Type: application/json" \
+-d '{
+ "dbName": "default",
+ "collectionName": "test_collection"
+}'
+
+# {
+# "code": 0,
+# "data": {
+# "aliases": [],
+# "autoId": false,
+# "collectionID": 448707763883002014,
+# "collectionName": "test_collection",
+# "consistencyLevel": "Bounded",
+# "description": "",
+# "enableDynamicField": true,
+# "fields": [
+# {
+# "autoId": false,
+# "description": "",
+# "id": 100,
+# "name": "id",
+# "partitionKey": false,
+# "primaryKey": true,
+# "type": "Int64"
+# },
+# {
+# "autoId": false,
+# "description": "",
+# "id": 101,
+# "name": "vector",
+# "params": [
+# {
+# "key": "dim",
+# "value": "5"
+# }
+# ],
+# "partitionKey": false,
+# "primaryKey": false,
+# "type": "FloatVector"
+# }
+# ],
+# "indexes": [
+# {
+# "fieldName": "vector",
+# "indexName": "vector",
+# "metricType": "COSINE"
+# }
+# ],
+# "load": "LoadStateLoaded",
+# "partitionsNum": 1,
+# "properties": [],
+# "shardsNum": 1
+# }
+# }
+```
+
+To list all existing collections, you can do as follows:
+
+
+
+```python
+# 6. List all collection names
+res = client.list_collections()
+
+print(res)
+
+# Output
+#
+# [
+# "customized_setup_2",
+# "quick_setup",
+# "customized_setup_1"
+# ]
+```
+
+```java
+import io.milvus.v2.service.collection.response.ListCollectionsResp;
+
+// 5. List all collection names
+ListCollectionsResp listCollectionsRes = client.listCollections();
+
+System.out.println(listCollectionsRes.getCollectionNames());
+
+// Output:
+// [
+// "customized_setup_2",
+// "quick_setup",
+// "customized_setup_1"
+// ]
+```
+
+```javascript
+// 5. List all collection names
+ListCollectionsResp listCollectionsRes = client.listCollections();
+
+System.out.println(listCollectionsRes.getCollectionNames());
+
+// Output:
+// [
+// "customized_setup_1",
+// "quick_setup",
+// "customized_setup_2"
+// ]
+```
+
+```shell
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/list" \
+-H "Content-Type: application/json" \
+-d '{
+ "dbName": "default"
+}'
+
+# {
+# "code": 0,
+# "data": [
+# "quick_setup",
+# "customized_setup_1",
+# "customized_setup_2"
+# ]
+# }
+```
+
+## Load & Release Collection
+
+During the loading process of a collection, Milvus loads the collection's index file into memory. Conversely, when releasing a collection, Milvus unloads the index file from memory. Before conducting searches in a collection, ensure that the collection is loaded.
+
+### Load a collection
+
+
+
+To load a collection, use the [`load_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/load_collection.md) method, specifying the collection name. You can also set `replica_number` to determine how many in-memory replicas of data segments to create on query nodes when the collection is loaded.
+
+- Milvus Standalone: The maximum allowed value for `replica_number` is 1.
+- Milvus Cluster: The maximum value should not exceed the `queryNode.replicas` set in your Milvus configurations. For additional details, refer to [Query Node-related Configurations](https://milvus.io/docs/configure_querynode.md#Query-Node-related-Configurations).
+
+
+
+
+
+To load a collection, use the [`loadCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/loadCollection.md) method, specifying the collection name.
+
+
+
+
+
+To load a collection, use the [`loadCollection()`](https://milvus.io/api-reference/node/v2.4.x/Management/loadCollection.md) method, specifying the collection name.
+
+
+
+
+
+To load a collection, you can use the [`POST /v2/vectordb/collections/load`](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/Load.md) and the [`POST /v2/vectordb/collections/get_load_state`](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/GetLoadState.md) API endpoints.
+
+
+
+
+
+```python
+# 7. Load the collection
+client.load_collection(
+ collection_name="customized_setup_2",
+ replica_number=1 # Number of replicas to create on query nodes. Max value is 1 for Milvus Standalone, and no greater than `queryNode.replicas` for Milvus Cluster.
+)
+
+res = client.get_load_state(
+ collection_name="customized_setup_2"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "state": "
+
+To release a collection, use the [`release_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/release_collection.md) method, specifying the collection name.
+
+
+
+
+
+To release a collection, use the [`releaseCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/releaseCollection.md) method, specifying the collection name.
+
+
+
+
+
+To release a collection, use the [`releaseCollection()`](https://milvus.io/api-reference/node/v2.4.x/Management/releaseCollection.md) method, specifying the collection name.
+
+
+
+
+
+To release a collection, you can use the [`POST /v2/vectordb/collections/release`](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/Release.md) and the [`POST /v2/vectordb/collections/get_load_state`](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/GetLoadState.md) API endpoints.
+
+
+
+
+
+```python
+# 8. Release the collection
+client.release_collection(
+ collection_name="customized_setup_2"
+)
+
+res = client.get_load_state(
+ collection_name="customized_setup_2"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "state": "
+
+To create aliases, use the [`create_alias()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_alias.md) method, specifying the collection name and the alias.
+
+
+
+
+
+To create aliases, use the [`createAlias()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createAlias.md) method, specifying the collection name and the alias.
+
+
+
+
+
+To create aliases, use the [`createAlias()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createAlias.md) method, specifying the collection name and the alias.
+
+
+
+
+
+To create aliases for a collection, you can use the [`POST /v2/vectordb/aliases/create`](https://milvus.io/api-reference/restful/v2.4.x/v2/Alias%20(v2)/Create.md) API endpoint.
+
+
+
+
+
+```python
+# 9.1. Create aliases
+client.create_alias(
+ collection_name="customized_setup_2",
+ alias="bob"
+)
+
+client.create_alias(
+ collection_name="customized_setup_2",
+ alias="alice"
+)
+```
+
+```java
+import io.milvus.v2.service.utility.request.CreateAliasReq;
+
+// 9. Manage aliases
+
+// 9.1 Create alias
+CreateAliasReq createAliasReq = CreateAliasReq.builder()
+ .collectionName("customized_setup_2")
+ .alias("bob")
+ .build();
+
+client.createAlias(createAliasReq);
+
+createAliasReq = CreateAliasReq.builder()
+ .collectionName("customized_setup_2")
+ .alias("alice")
+ .build();
+
+client.createAlias(createAliasReq);
+```
+
+```javascript
+// 9. Manage aliases
+// 9.1 Create aliases
+res = await client.createAlias({
+ collection_name: "customized_setup_2",
+ alias: "bob"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+
+res = await client.createAlias({
+ collection_name: "customized_setup_2",
+ alias: "alice"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+```
+
+```shell
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/aliases/create" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_2",
+ "aliasName": "bob"
+}'
+
+# Output
+#
+# {
+# "code": 0,
+# "data": {}
+# }
+
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/aliases/create" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_2",
+ "aliasName": "alice"
+}'
+
+# Output
+#
+# {
+# "code": 0,
+# "data": {}
+# }
+```
+
+| Parameter | +Description | +
|---|---|
collection_name |
+ The name of the collection to create an alias for. | +
alias |
+ The alias of the collection. Before this operation, ensure that the alias does not already exist. If it does, exceptions will occur. | +
| Parameter | +Description | +
|---|---|
collectionName |
+ The name of the collection to create an alias for. | +
alias |
+ The alias of the collection. Before this operation, ensure that the alias does not already exist. If it does, exceptions will occur. | +
| Parameter | +Description | +
|---|---|
collection_name |
+ The name of the collection to create an alias for. | +
alias |
+ The alias of the collection. Before this operation, ensure that the alias does not already exist. If it does, exceptions will occur. | +
| Parameter | +Description | +
|---|---|
collectionName |
+ The name of the collection to create an alias for. | +
aliasName |
+ The alias of the collection. Before this operation, ensure that the alias does not already exist. If it does, exceptions will occur. | +
+
+To list aliases, use the [`list_aliases()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/list_aliases.md) method, specifying the collection name.
+
+
+
+
+
+To list aliases, use the [`listAliases()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/listAliases.md) method, specifying the collection name.
+
+
+
+
+
+To list aliases, use the [`listAliases()`](https://milvus.io/api-reference/node/v2.4.x/Collections/listAliases.md) method, specifying the collection name.
+
+
+
+
+
+To list aliases for a collection, you can use the [`POST /v2/vectordb/aliases/list`](https://milvus.io/api-reference/restful/v2.4.x/v2/Alias%20(v2)/List.md) API endpoint.
+
+
+
+
+
+```python
+# 9.2. List aliases
+res = client.list_aliases(
+ collection_name="customized_setup_2"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "aliases": [
+# "bob",
+# "alice"
+# ],
+# "collection_name": "customized_setup_2",
+# "db_name": "default"
+# }
+```
+
+```java
+import io.milvus.v2.service.utility.request.ListAliasesReq;
+import io.milvus.v2.service.utility.response.ListAliasResp;
+
+// 9.2 List alises
+ListAliasesReq listAliasesReq = ListAliasesReq.builder()
+ .collectionName("customized_setup_2")
+ .build();
+
+ListAliasResp listAliasRes = client.listAliases(listAliasesReq);
+
+System.out.println(listAliasRes.getAlias());
+
+// Output:
+// [
+// "bob",
+// "alice"
+// ]
+```
+
+```javascript
+// 9.2 List aliases
+res = await client.listAliases({
+ collection_name: "customized_setup_2"
+})
+
+console.log(res.aliases)
+
+// Output
+//
+// [ 'bob', 'alice' ]
+//
+```
+
+```shell
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/aliases/list" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_2"
+}'
+
+# {
+# "code": 0,
+# "data": [
+# "bob",
+# "alice"
+# ]
+# }
+```
+
+### Describe aliases
+
+
+
+To describe aliases, use the [`describe_alias()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/describe_alias.md) method, specifying the alias.
+
+
+
+
+
+To describe aliases, use the [`describeAlias()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/describeAlias.md) method, specifying the alias.
+
+
+
+
+
+To describe aliases, use the [`describeAlias()`](https://milvus.io/api-reference/node/v2.4.x/Collections/describeAlias.md) method, specifying the alias.
+
+
+
+
+
+To describe aliases for a collection, you can use the [`POST /v2/vectordb/aliases/describe`](https://milvus.io/api-reference/restful/v2.4.x/v2/Alias%20(v2)/Describe.md) API endpoint.
+
+
+
+
+
+```python
+# 9.3. Describe aliases
+res = client.describe_alias(
+ alias="bob"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "alias": "bob",
+# "collection_name": "customized_setup_2",
+# "db_name": "default"
+# }
+```
+
+```java
+import io.milvus.v2.service.utility.request.DescribeAliasReq;
+import io.milvus.v2.service.utility.response.DescribeAliasResp;
+
+// 9.3 Describe alias
+DescribeAliasReq describeAliasReq = DescribeAliasReq.builder()
+ .alias("bob")
+ .build();
+
+DescribeAliasResp describeAliasRes = client.describeAlias(describeAliasReq);
+
+System.out.println(JSONObject.toJSON(describeAliasRes));
+
+// Output:
+// {
+// "alias": "bob",
+// "collectionName": "customized_setup_2"
+// }
+```
+
+```javascript
+// 9.3 Describe aliases
+res = await client.describeAlias({
+ collection_name: "customized_setup_2",
+ alias: "bob"
+})
+
+console.log(res)
+
+// Output
+//
+// {
+// status: {
+// extra_info: {},
+// error_code: 'Success',
+// reason: '',
+// code: 0,
+// retriable: false,
+// detail: ''
+// },
+// db_name: 'default',
+// alias: 'bob',
+// collection: 'customized_setup_2'
+// }
+//
+```
+
+```shell
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/aliases/describe" \
+-H "Content-Type: application/json" \
+-d '{
+ "aliasName": "bob"
+}'
+
+# {
+# "code": 0,
+# "data": {
+# "aliasName": "bob",
+# "collectionName": "quick_setup",
+# "dbName": "default"
+# }
+# }
+```
+
+### Reassign aliases
+
+
+
+To reassign aliases to other collections, use the [`alter_alias()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/alter_alias.md) method, specifying the collection name and the alias.
+
+
+
+
+
+To reassign aliases to other collections, use the [`alterAlias()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/alterAlias.md) method, specifying the collection name and the alias.
+
+
+
+
+
+To reassign aliases to other collections, use the [`alterAlias()`](https://milvus.io/api-reference/node/v2.4.x/Collections/alterAlias.md) method, specifying the collection name and the alias.
+
+
+
+
+
+To reassign aliases to other collections, you can use the [`POST /v2/vectordb/aliases/alter`](https://milvus.io/api-reference/restful/v2.4.x/v2/Alias%20(v2)/Alter.md) API endpoint.
+
+
+
+
+
+```python
+# 9.4 Reassign aliases to other collections
+client.alter_alias(
+ collection_name="customized_setup_1",
+ alias="alice"
+)
+
+res = client.list_aliases(
+ collection_name="customized_setup_1"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "aliases": [
+# "alice"
+# ],
+# "collection_name": "customized_setup_1",
+# "db_name": "default"
+# }
+
+res = client.list_aliases(
+ collection_name="customized_setup_2"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "aliases": [
+# "bob"
+# ],
+# "collection_name": "customized_setup_2",
+# "db_name": "default"
+# }
+```
+
+```java
+import io.milvus.v2.service.utility.request.AlterAliasReq;
+
+// 9.4 Reassign alias to other collections
+AlterAliasReq alterAliasReq = AlterAliasReq.builder()
+ .collectionName("customized_setup_1")
+ .alias("alice")
+ .build();
+
+client.alterAlias(alterAliasReq);
+
+listAliasesReq = ListAliasesReq.builder()
+ .collectionName("customized_setup_1")
+ .build();
+
+listAliasRes = client.listAliases(listAliasesReq);
+
+System.out.println(listAliasRes.getAlias());
+
+// Output:
+// ["alice"]
+
+listAliasesReq = ListAliasesReq.builder()
+ .collectionName("customized_setup_2")
+ .build();
+
+listAliasRes = client.listAliases(listAliasesReq);
+
+System.out.println(listAliasRes.getAlias());
+
+// Output:
+// ["bob"]
+```
+
+```javascript
+// 9.4 Reassign aliases to other collections
+res = await client.alterAlias({
+ collection_name: "customized_setup_1",
+ alias: "alice"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+
+res = await client.listAliases({
+ collection_name: "customized_setup_1"
+})
+
+console.log(res.aliases)
+
+// Output
+//
+// [ 'alice' ]
+//
+
+res = await client.listAliases({
+ collection_name: "customized_setup_2"
+})
+
+console.log(res.aliases)
+
+// Output
+//
+// [ 'bob' ]
+//
+```
+
+```shell
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/aliases/alter" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_1",
+ "aliasName": "alice"
+}'
+
+# {
+# "code": 0,
+# "data": {}
+# }
+
+
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/aliases/list" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_1"
+}'
+
+
+# {
+# "code": 0,
+# "data": [
+# "alice"
+# ]
+# }
+
+
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/aliases/list" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_2"
+}'
+
+
+# {
+# "code": 0,
+# "data": [
+# "bob"
+# ]
+# }
+```
+
+### Drop aliases
+
+
+
+To drop aliases, use the [`drop_alias()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/drop_alias.md) method, specifying the alias.
+
+
+
+
+
+To drop aliases, use the [`dropAlias()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/dropAlias.md) method, specifying the alias.
+
+
+
+
+
+To drop aliases, use the [`dropAlias()`](https://milvus.io/api-reference/node/v2.4.x/Collections/dropAlias.md) method, specifying the alias.
+
+
+
+
+
+To drop aliases for a collection, you can use the [`POST /v2/vectordb/aliases/drop`](https://milvus.io/api-reference/restful/v2.4.x/v2/Alias%20(v2)/Drop.md) API endpoint.
+
+
+
+
+
+```python
+# 9.5 Drop aliases
+client.drop_alias(
+ alias="bob"
+)
+
+client.drop_alias(
+ alias="alice"
+)
+```
+
+```java
+import io.milvus.v2.service.utility.request.DropAliasReq;
+
+// 9.5 Drop alias
+DropAliasReq dropAliasReq = DropAliasReq.builder()
+ .alias("bob")
+ .build();
+
+client.dropAlias(dropAliasReq);
+
+dropAliasReq = DropAliasReq.builder()
+ .alias("alice")
+ .build();
+
+client.dropAlias(dropAliasReq);
+```
+
+```javascript
+// 9.5 Drop aliases
+res = await client.dropAlias({
+ alias: "bob"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+
+res = await client.dropAlias({
+ alias: "alice"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+```
+```shell
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/aliases/drop" \
+-H "Content-Type: application/json" \
+-d '{
+ "aliasName": "bob"
+}'
+
+# {
+# "code": 0,
+# "data": {}
+# }
+
+
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/aliases/drop" \
+-H "Content-Type: application/json" \
+-d '{
+ "aliasName": "alice"
+}'
+
+
+# {
+# "code": 0,
+# "data": {}
+# }
+```
+
+## Set Properties
+
+You can set properties for a collection, such as `ttl.seconds` and `mmap.enabled`. For more information, refer to [set_properties()](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Collection/set_properties.md).
+
+
+
+The code snippets in this section use the PyMilvus ORM module to interact with Milvus. Code snippets with the new MilvusClient SDK will be available soon.
+
+
+
+### Set TTL
+
+Set the Time-To-Live (TTL) for the data in the collection, which specifies how long the data should be retained before it is automatically deleted.
+
+```python
+from pymilvus import Collection, connections
+
+# Connect to Milvus server
+connections.connect(host="localhost", port="19530") # Change to your Milvus server IP and port
+
+# Get existing collection
+collection = Collection("quick_setup")
+
+# Set the TTL for the data in the collection
+collection.set_properties(
+ properties={
+ "collection.ttl.seconds": 60
+ }
+)
+```
+
+### Set MMAP
+
+Configure the memory mapping (MMAP) property for the collection, which determines whether data is mapped into memory to improve query performance. For more information, refer to [Configure memory mapping
+](https://milvus.io/docs/mmap.md#Configure-memory-mapping).
+
+
+
+Before setting the MMAP property, release the collection first. Otherwise, an error will occur.
+
+
+
+```python
+from pymilvus import Collection, connections
+
+# Connect to Milvus server
+connections.connect(host="localhost", port="19530") # Change to your Milvus server IP and port
+
+# Get existing collection
+collection = Collection("quick_setup")
+
+# Before setting memory mapping property, we need to release the collection first.
+collection.release()
+
+# Set memory mapping property to True or Flase
+collection.set_properties(
+ properties={
+ "mmap.enabled": True
+ }
+)
+```
+
+## Drop a Collection
+
+If a collection is no longer needed, you can drop the collection.
+
+
+
+To drop a collection, use the [`drop_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/drop_collection.md) method, specifying the collection name.
+
+
+
+
+
+To drop a collection, use the [`dropCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/dropCollection.md) method, specifying the collection name.
+
+
+
+
+
+To drop a collection, use the [`dropCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/dropCollection.md) method, specifying the collection name.
+
+
+
+
+
+To drop a collection, you can use the [`POST /v2/vectordb/collections/drop`](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/Drop.md) API endpoint.
+
+
+
+
+
+```python
+# 10. Drop the collections
+client.drop_collection(
+ collection_name="quick_setup"
+)
+
+client.drop_collection(
+ collection_name="customized_setup_1"
+)
+
+client.drop_collection(
+ collection_name="customized_setup_2"
+)
+```
+
+```java
+import io.milvus.v2.service.collection.request.DropCollectionReq;
+
+// 10. Drop collections
+
+DropCollectionReq dropQuickSetupParam = DropCollectionReq.builder()
+ .collectionName("quick_setup")
+ .build();
+
+client.dropCollection(dropQuickSetupParam);
+
+DropCollectionReq dropCustomizedSetupParam = DropCollectionReq.builder()
+ .collectionName("customized_setup_1")
+ .build();
+
+client.dropCollection(dropCustomizedSetupParam);
+
+dropCustomizedSetupParam = DropCollectionReq.builder()
+ .collectionName("customized_setup_2")
+ .build();
+
+client.dropCollection(dropCustomizedSetupParam);
+```
+
+```javascript
+// 10. Drop the collection
+res = await client.dropCollection({
+ collection_name: "customized_setup_2"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+
+res = await client.dropCollection({
+ collection_name: "customized_setup_1"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+
+res = await client.dropCollection({
+ collection_name: "quick_setup"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+```
+
+```shell
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/drop" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "quick_setup"
+}'
+
+# {
+# "code": 0,
+# "data": {}
+# }
+
+
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/drop" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_1"
+}'
+
+
+# {
+# "code": 0,
+# "data": {}
+# }
+
+
+$ curl -X POST "http://${MILVUS_URI}/v2/vectordb/collections/drop" \
+-H "Content-Type: application/json" \
+-d '{
+ "collectionName": "customized_setup_2"
+}'
+
+
+# {
+# "code": 0,
+# "data": {}
+# }
+```
+
diff --git a/preview/site/en/userGuide/manage-indexes/index-scalar-fields.md b/preview/site/en/userGuide/manage-indexes/index-scalar-fields.md
new file mode 100644
index 000000000..6de0c94eb
--- /dev/null
+++ b/preview/site/en/userGuide/manage-indexes/index-scalar-fields.md
@@ -0,0 +1,330 @@
+---
+id: index-scalar-fields.md
+order: 2
+summary: This guide will walk you through creating and configuring scalar indexes for fields such as integers, strings, etc.
+title: Index Scalar Fields
+---
+
+# Index Scalar Fields
+
+In Milvus, a scalar index is used to speed up metafiltering by a specific non-vector field value, similar to a traditional database index. This guide will walk you through creating and configuring scalar indexes for fields such as integers, strings, etc.
+
+## Types of scalar indexing
+
+- __[Auto indexing](https://milvus.io/docs/index-scalar-fields.md#Auto-indexing)__: Milvus automatically decides the index type based on the data type of the scalar field. This is suitable when you do not need to control the specific index type.
+
+- __[Custom indexing](https://milvus.io/docs/index-scalar-fields.md#Custom-indexing)__: You specify the exact index type, such as an inverted index. This provides more control over the index type selection.
+
+## Auto indexing
+
+
+
+To use auto indexing, omit the __index_type__ parameter in [`add_index()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/add_index.md), so that Milvus can infer the index type based on the scalar field type.
+
+
+
+
+
+To use auto indexing, omit the __indexType__ parameter in [`IndexParam`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/IndexParam.md), so that Milvus can infer the index type based on the scalar field type.
+
+
+
+
+
+To use auto indexing, omit the __index_type__ parameter in [`createIndex()`](https://milvus.io/api-reference/node/v2.4.x/Management/createIndex.md), so that Milvus can infer the index type based on the scalar field type.
+
+
+
+For mappings between scalar data types and default indexing algorithms, refer to [Scalar field indexing algorithms](https://milvus.io/docs/scalar_index.md#Scalar-field-indexing-algorithms).
+
+
+
+```python
+# Auto indexing
+client = MilvusClient(
+ uri="http://localhost:19530"
+)
+
+index_params = client.create_index_params() # Prepare an empty IndexParams object, without having to specify any index parameters
+
+index_params.add_index(
+ field_name="scalar_1", # Name of the scalar field to be indexed
+ index_type="", # Type of index to be created. For auto indexing, leave it empty or omit this parameter.
+ index_name="default_index" # Name of the index to be created
+)
+
+client.create_index(
+ collection_name="test_scalar_index", # Specify the collection name
+ index_params=index_params
+)
+```
+
+```java
+import io.milvus.v2.common.IndexParam;
+import io.milvus.v2.service.index.request.CreateIndexReq;
+
+IndexParam indexParamForScalarField = IndexParam.builder()
+ .fieldName("scalar_1") // Name of the scalar field to be indexed
+ .indexName("default_index") // Name of the index to be created
+ .indexType("") // Type of index to be created. For auto indexing, leave it empty or omit this parameter.
+ .build();
+
+List
+
+To use custom indexing, specify a particular index type using the __index_type__ parameter in [`add_index()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/add_index.md).
+
+
+
+
+
+To use custom indexing, specify a particular index type using the __indexType__ parameter in [`IndexParam`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/IndexParam.md).
+
+
+
+
+
+To use custom indexing, specify a particular index type using the __index_type__ parameter in [`createIndex()`](https://milvus.io/api-reference/node/v2.4.x/Management/createIndex.md).
+
+
+
+
+
+```python
+index_params = client.create_index_params() # Prepare an IndexParams object
+
+index_params.add_index(
+ field_name="scalar_2", # Name of the scalar field to be indexed
+ index_type="INVERTED", # Type of index to be created
+ index_name="inverted_index" # Name of the index to be created
+)
+
+client.create_index(
+ collection_name="test_scalar_index", # Specify the collection name
+ index_params=index_params
+)
+```
+
+```java
+import io.milvus.v2.common.IndexParam;
+import io.milvus.v2.service.index.request.CreateIndexReq;
+
+IndexParam indexParamForScalarField = IndexParam.builder()
+ .fieldName("scalar_1") // Name of the scalar field to be indexed
+ .indexName("inverted_index") // Name of the index to be created
+ .indexType("INVERTED") // Type of index to be created
+ .build();
+
+List
+
+__Methods and Parameters__
+
+- __create_index_params()__
+
+ Prepares an __IndexParams__ object.
+
+- __add_index()__
+
+ Adds index configurations to the __IndexParams__ object.
+
+ - __field_name__ (_string_)
+
+ The name of the scalar field to index.
+
+ - __index_type__ (_string_):
+
+ The type of the scalar index to create. For implicit indexing, leave it empty or omit this parameter.
+
+ For custom indexing, valid values are:
+
+ - __INVERTED__: (Recommended) An inverted index consists of a term dictionary containing all tokenized words sorted alphabetically. For details, refer to Scalar Index.
+
+ - __STL_SORT__: Sorts scalar fields using the standard template library sort algorithm. Supports Boolean and numeric fields (e.g., INT8, INT16, INT32, INT64, FLOAT, DOUBLE).
+
+ - __Trie__: A tree data structure for fast prefix searches and retrievals. Supports VARCHAR fields.
+
+ - __index_name__ (_string_)
+
+ The name of the scalar index to create. Each scalar field supports one index.
+
+- __create_index()__
+
+ Creates the index in the specified collection.
+
+ - __collection_name__ (_string_)
+
+ The name of the collection for which the index is created.
+
+ - __index_params__
+
+ The __IndexParams__ object that contains index configurations.
+
+
+
+
+
+__Methods and Parameters__
+
+- __IndexParam__
+ Prepares an IndexParam object.
+ - __fieldName__ (_String_)
+ The name of the scalar field to index.
+ - __indexName__ (_String_)
+ The name of the scalar index to create. Each scalar field supports one index.
+ - __indexType__ (_String_)
+ The type of the scalar index to create. For implicit indexing, leave it empty or omit this parameter.
+ For custom indexing, valid values are:
+ - __INVERTED__: (Recommended) An inverted index consists of a term dictionary containing all tokenized words sorted alphabetically. For details, refer to Scalar Index.
+ - __STL_SORT__: Sorts scalar fields using the standard template library sort algorithm. Supports Boolean and numeric fields (e.g., INT8, INT16, INT32, INT64, FLOAT, DOUBLE).
+ - __Trie__: A tree data structure for fast prefix searches and retrievals. Supports VARCHAR fields.
+- __CreateIndexReq__
+ Creates the index in the specified collection.
+ - __collectionName__ (_String_)
+ The name of the collection for which the index is created.
+ - __indexParams__ (_List_)
+ A list of IndexParam objects that contain index configurations.
+
+
+
+
+
+__Methods and Parameters__
+- __createIndex__
+
+ Creates the index in the specified collection.
+ - __collection_name__ (_string_)
+ The name of the collection for which the index is created.
+ - __field_name__ (_string_)
+ The name of the scalar field to index.
+ - __index_name__ (_string_)
+ The name of the scalar index to create. Each scalar field supports one index.
+ - __index_type__ (_string_)
+ The type of the scalar index to create. For implicit indexing, leave it empty or omit this parameter.
+ For custom indexing, valid values are:
+ - __INVERTED__: (Recommended) An inverted index consists of a term dictionary containing all tokenized words sorted alphabetically. For details, refer to [Scalar Index](scalar_index.md).
+ - __STL_SORT__: Sorts scalar fields using the standard template library sort algorithm. Supports Boolean and numeric fields (e.g., INT8, INT16, INT32, INT64, FLOAT, DOUBLE).
+ - __Trie__: A tree data structure for fast prefix searches and retrievals. Supports VARCHAR fields.
+
+
+
+## Verifying the result
+
+
+
+Use the [`list_indexes()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/list_indexes.md) method to verify the creation of scalar indexes:
+
+
+
+
+
+Use the `listIndexes()` method to verify the creation of scalar indexes:
+
+
+
+
+
+Use the `listIndexes()` method to verify the creation of scalar indexes:
+
+
+
+
+
+```python
+client.list_indexes(
+ collection_name="test_scalar_index" # Specify the collection name
+)
+
+# Output:
+# ['default_index','inverted_index']
+```
+
+```java
+import java.util.List;
+import io.milvus.v2.service.index.request.ListIndexesReq;
+
+ListIndexesReq listIndexesReq = ListIndexesReq.builder()
+ .collectionName("test_scalar_index") // Specify the collection name
+ .build();
+
+List
+
+
+
+
+
+
+
+| Metric Types | +Index Types | +
|---|---|
|
+
|
+
+
+
+
+
+
+
+
+| Metric Types | +Index Types | +
|---|---|
|
+
|
+
+
+
+
+
+
+
+
+It is recommended to create indexes for both the vector field and scalar fields that are frequently accessed.
+
+## Preparations
+
+As explained in [Manage Collections](manage-collections.md), Milvus automatically generates an index and loads it into memory when creating a collection if any of the following conditions are specified in the collection creation request:
+
+- The dimensionality of the vector field and the metric type, or
+
+- The schema and the index parameters.
+
+The code snippet below repurposes the existing code to establish a connection to a Milvus instance and create a collection without specifying its index parameters. In this case, the collection lacks an index and remains unloaded.
+
+| Metric Types | +Index Types | +
|---|---|
| IP | +
|
+
+
+To prepare for indexing, use [`MilvusClient`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) to connect to the Milvus server and set up a collection by using [`create_schema()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md), [`add_field()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/CollectionSchema/add_field.md), and [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md).
+
+
+
+
+
+To prepare for indexing, use [`MilvusClientV2`](https://milvus.io/api-reference/java/v2.4.x/v2/Client/MilvusClientV2.md) to connect to the Milvus server and set up a collection by using [`createSchema()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createSchema.md), [`addField()`](https://milvus.io/api-reference/java/v2.4.x/v2/CollectionSchema/addField.md), and [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md).
+
+
+
+
+
+To prepare for indexing, use [`MilvusClient`](https://milvus.io/api-reference/node/v2.4.x/Client/MilvusClient.md) to connect to the Milvus server and set up a collection by using [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md).
+
+
+
+
+
+```python
+from pymilvus import MilvusClient, DataType
+
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri="http://localhost:19530"
+)
+
+# 2. Create schema
+# 2.1. Create schema
+schema = MilvusClient.create_schema(
+ auto_id=False,
+ enable_dynamic_field=True,
+)
+
+# 2.2. Add fields to schema
+schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
+schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=5)
+
+# 3. Create collection
+client.create_collection(
+ collection_name="customized_setup",
+ schema=schema,
+)
+```
+
+```java
+import io.milvus.v2.client.ConnectConfig;
+import io.milvus.v2.client.MilvusClientV2;
+import io.milvus.v2.common.DataType;
+import io.milvus.v2.service.collection.request.CreateCollectionReq;
+
+String CLUSTER_ENDPOINT = "http://localhost:19530";
+
+// 1. Connect to Milvus server
+ConnectConfig connectConfig = ConnectConfig.builder()
+ .uri(CLUSTER_ENDPOINT)
+ .build();
+
+MilvusClientV2 client = new MilvusClientV2(connectConfig);
+
+// 2. Create a collection
+
+// 2.1 Create schema
+CreateCollectionReq.CollectionSchema schema = client.createSchema();
+
+// 2.2 Add fields to schema
+schema.addField(AddFieldReq.builder().fieldName("id").dataType(DataType.Int64).isPrimaryKey(true).autoID(false).build());
+schema.addField(AddFieldReq.builder().fieldName("vector").dataType(DataType.FloatVector).dimension(5).build());
+
+// 3 Create a collection without schema and index parameters
+CreateCollectionReq customizedSetupReq = CreateCollectionReq.builder()
+.collectionName("customized_setup")
+.collectionSchema(schema)
+.build();
+
+client.createCollection(customizedSetupReq);
+```
+
+```javascript
+// 1. Set up a Milvus Client
+client = new MilvusClient({address, token});
+
+// 2. Define fields for the collection
+const fields = [
+ {
+ name: "id",
+ data_type: DataType.Int64,
+ is_primary_key: true,
+ autoID: false
+ },
+ {
+ name: "vector",
+ data_type: DataType.FloatVector,
+ dim: 5
+ },
+]
+
+// 3. Create a collection
+res = await client.createCollection({
+ collection_name: "customized_setup",
+ fields: fields,
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+```
+
+## Index a Collection
+
+
+
+To create an index for a collection or index a collection, use [`prepare_index_params()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/prepare_index_params.md) to prepare index parameters and [`create_index()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/create_index.md) to create the index.
+
+
+
+
+
+To create an index for a collection or index a collection, use [`IndexParam`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/IndexParam.md) to prepare index parameters and [`createIndex()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/createIndex.md) to create the index.
+
+
+
+
+
+To create an index for a collection or index a collection, use [`createIndex()`](https://milvus.io/api-reference/node/v2.4.x/Management/createIndex.md).
+
+
+
+
+
+```python
+# 4.1. Set up the index parameters
+index_params = MilvusClient.prepare_index_params()
+
+# 4.2. Add an index on the vector field.
+index_params.add_index(
+ field_name="vector",
+ metric_type="COSINE",
+ index_type="IVF_FLAT",
+ index_name="vector_index",
+ params={ "nlist": 128 }
+)
+
+# 4.3. Create an index file
+client.create_index(
+ collection_name="customized_setup",
+ index_params=index_params
+)
+```
+
+```java
+import io.milvus.v2.common.IndexParam;
+import io.milvus.v2.service.index.request.CreateIndexReq;
+
+// 4 Prepare index parameters
+
+// 4.2 Add an index for the vector field "vector"
+IndexParam indexParamForVectorField = IndexParam.builder()
+ .fieldName("vector")
+ .indexName("vector_index")
+ .indexType(IndexParam.IndexType.IVF_FLAT)
+ .metricType(IndexParam.MetricType.COSINE)
+ .extraParams(Map.of("nlist", 128))
+ .build();
+
+List| Parameter | +Description | +
|---|---|
field_name |
+ The name of the target file to apply this object applies. | +
metric_type |
+ The algorithm that is used to measure similarity between vectors. Possible values are IP, L2, COSINE, JACCARD, HAMMING. This is available only when the specified field is a vector field. For more information, refer to Indexes supported in Milvus. | +
index_type |
+ The name of the algorithm used to arrange data in the specific field. For applicable algorithms, refer to In-memory Index and On-disk Index. | +
index_name |
+ The name of the index file generated after this object has been applied. | +
params |
+ The fine-tuning parameters for the specified index type. For details on possible keys and value ranges, refer to In-memory Index. | +
collection_name |
+ The name of an existing collection. | +
index_params |
+ An IndexParams object containing a list of IndexParam objects. | +
| Parameter | +Description | +
|---|---|
fieldName |
+ The name of the target field to apply this IndexParam object applies. | +
indexName |
+ The name of the index file generated after this object has been applied. | +
indexType |
+ The name of the algorithm used to arrange data in the specific field. For applicable algorithms, refer to In-memory Index and On-disk Index. | +
metricType |
+ The distance metric to use for the index. Possible values are IP, L2, COSINE, JACCARD, HAMMING. | +
extraParams |
+ Extra index parameters. For details, refer to In-memory Index and On-disk Index. | +
| Parameter | +Description | +
|---|---|
collection_name |
+ The name of an existing collection. | +
field_name |
+ The name of the field in which to create an index. | +
index_type |
+ The type of the index to create. | +
metric_type |
+ The metric type used to measure vector distance. | +
index_name |
+ The name of the index to create. | +
params |
+ Other index-specific parameters. | +
+
+
+
+## Check Index Details
+
+Once you have created an index, you can check its details.
+
+notes
+ +Currently, you can create only one index file for each field in a collection.
+ +
+
+To check the index details, use [`list_indexes()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/list_indexes.md) to list the index names and [`describe_index()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/describe_index.md) to get the index details.
+
+
+
+
+
+To check the index details, use [`describeIndex()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/describeIndex.md) to get the index details.
+
+
+
+
+
+To check the index details, use [`describeIndex()`](https://milvus.io/api-reference/node/v2.4.x/Management/describeIndex.md) to get the index details.
+
+
+
+
+
+```python
+# 5. Describe index
+res = client.list_indexes(
+ collection_name="customized_setup"
+)
+
+print(res)
+
+# Output
+#
+# [
+# "vector_index",
+# ]
+
+res = client.describe_index(
+ collection_name="customized_setup",
+ index_name="vector_index"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "index_type": ,
+# "metric_type": "COSINE",
+# "field_name": "vector",
+# "index_name": "vector_index"
+# }
+```
+
+```java
+import io.milvus.v2.service.index.request.DescribeIndexReq;
+import io.milvus.v2.service.index.response.DescribeIndexResp;
+
+// 5. Describe index
+// 5.1 List the index names
+ListIndexesReq listIndexesReq = ListIndexesReq.builder()
+ .collectionName("customized_setup")
+ .build();
+
+List
+
+Before dropping an index, make sure it has been released first.
+
+
+
+
+
+To drop an index, use [`drop_index()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/drop_index.md).
+
+
+
+
+
+To drop an index, use [`dropIndex()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/dropIndex.md).
+
+
+
+
+
+To drop an index, use [`dropIndex()`](https://milvus.io/api-reference/node/v2.4.x/Management/dropIndex.md).
+
+
+
+
+
+```python
+# 6. Drop index
+client.drop_index(
+ collection_name="customized_setup",
+ index_name="vector_index"
+)
+```
+
+```java
+// 6. Drop index
+
+DropIndexReq dropIndexReq = DropIndexReq.builder()
+ .collectionName("customized_setup")
+ .indexName("vector_index")
+ .build();
+
+client.dropIndex(dropIndexReq);
+```
+
+```javascript
+// 6. Drop the index
+res = await client.dropIndex({
+ collection_name: "customized_setup",
+ index_name: "vector_index"
+})
+
+console.log(res.error_code)
+
+// Output
+//
+// Success
+//
+```
diff --git a/preview/site/en/userGuide/manage-indexes/index-with-gpu.md b/preview/site/en/userGuide/manage-indexes/index-with-gpu.md
new file mode 100644
index 000000000..917fac01d
--- /dev/null
+++ b/preview/site/en/userGuide/manage-indexes/index-with-gpu.md
@@ -0,0 +1,214 @@
+---
+id: index-with-gpu.md
+order: 3
+summary: This guide explains how to build an index with GPU support in Milvus to enhance search performance.
+title: Index with GPU
+---
+
+# Index with GPU
+
+This guide outlines the steps to build an index with GPU support in Milvus, which can significantly improve search performance in high-throughput and high-recall scenarios. For details on the types of GPU indexes supported by Milvus, refer to [GPU Index](gpu_index.md).
+
+## Configure Milvus settings for GPU memory control
+
+Milvus uses a global graphics memory pool to allocate GPU memory.
+
+It supports two parameters `initMemSize` and `maxMemSize` in [Milvus config file](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml#L767-L769). The pool size is initially set to `initMemSize`, and will be automatically expanded to `maxMemSize` after exceeding this limit.
+
+The default `initMemSize` is 1/2 of the available GPU memory when Milvus starts, and the default `maxMemSize` is equal to all available GPU memory.
+
+Up until Milvus 2.4.1( including version 2.4.1), Milvus used a unified GPU memory pool. For versions prior to 2.4.1( including version 2.4.1), it was recommended to set both of the value to 0.
+
+```yaml
+gpu:
+ initMemSize: 0 #set the initial memory pool size.
+ maxMemSize: 0 #maxMemSize sets the maximum memory usage limit. When the memory usage exceed initMemSize, Milvus will attempt to expand the memory pool.
+```
+
+From Milvus 2.4.1 onwards, the GPU memory pool is only used for temporary GPU data during searches. Therefore, it is recommended to set it to 2048 and 4096.
+
+```yaml
+gpu:
+ initMemSize: 2048 #set the initial memory pool size.
+ maxMemSize: 4096 #maxMemSize sets the maximum memory usage limit. When the memory usage exceed initMemSize, Milvus will attempt to expand the memory pool.
+```
+
+## Build an index
+
+The following examples demonstrate how to build GPU indexes of different types.
+
+### Prepare index parameters
+
+When setting up GPU index parameters, define __index_type__, __metric_type__, and __params__:
+
+- __index_type__ (_string_): The type of index used to accelerate vector search. Valid options include __GPU_CAGRA__, __GPU_IVF_FLAT__, __GPU_IVF_PQ__, and __GPU_BRUTE_FORCE__.
+
+- __metric_type__ (_string_): The type of metrics used to measure the similarity of vectors. Valid options are __IP__ and __L2__.
+
+- __params__(_dict_): The index-specific building parameters. The valid options for this parameter depend on the index type.
+
+Here are example configurations for different index types:
+
+- __GPU_CAGRA__ index
+
+ ```python
+ index_params = {
+ "metric_type": "L2",
+ "index_type": "GPU_CAGRA",
+ "params": {
+ 'intermediate_graph_degree': 64,
+ 'graph_degree': 32
+ }
+ }
+ ```
+
+ Possible options for __params__ include:
+
+ - __intermediate_graph_degree__ (_int_): Affects recall and build time by determining the graph's degree before pruning. Recommended values are __32__ or __64__.
+
+ - __graph_degree__ (_int_): Affects search performance and recall by setting the graph's degree after pruning. Typically, it is half of the __intermediate_graph_degree__. A larger difference between these two degrees results in a longer build time. Its value must be smaller than the value of __intermediate_graph_degree__.
+
+ - __build_algo__ (_string_): Selects the graph generation algorithm before pruning. Possible options:
+
+ - __IVF_PQ__: Offers higher quality but slower build time.
+
+ - __NN_DESCENT__: Provides a quicker build with potentially lower recall.
+
+ - __cache_dataset_on_device__ (_string_, __"true"__ | __"false"__): Decides whether to cache the original dataset in GPU memory. Setting this to __"true"__ enhances recall by refining search results, while setting it to __"false"__ conserves GPU memory.
+
+- __GPU_IVF_FLAT__ or __GPU_IVF_PQ__ index
+
+ ```python
+ index_params = {
+ "metric_type": "L2",
+ "index_type": "GPU_IVF_FLAT", # Or GPU_IVF_PQ
+ "params": {
+ "nlist": 1024
+ }
+ }
+ ```
+
+ The __params__ options are identical to those used in __[IVF_FLAT](https://milvus.io/docs/index.md#IVF_FLAT)__ and __[IVF_PQ](https://milvus.io/docs/index.md#IVF_PQ)__.
+
+- __GPU_BRUTE_FORCE__ index
+
+ ```python
+ index_params = {
+ 'index_type': 'GPU_BRUTE_FORCE',
+ 'metric_type': 'L2',
+ 'params': {}
+ }
+ ```
+
+ No additional __params__ configurations are required.
+
+### Build index
+
+After configuring the index parameters in __index_params__, call the [`create_index()`](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Collection/create_index.md) method to build the index.
+
+```python
+# Get an existing collection
+collection = Collection("YOUR_COLLECTION_NAME")
+
+collection.create_index(
+ field_name="vector", # Name of the vector field on which an index is built
+ index_params=index_params
+)
+```
+
+## Search
+
+Once you have built your GPU index, the next step is to prepare the search parameters before conducting a search.
+
+### Prepare search parameters
+
+Below are example configurations for different index types:
+
+- __GPU_BRUTE_FORCE__ index
+
+ ```python
+ search_params = {
+ "metric_type": "L2",
+ "params": {}
+ }
+ ```
+
+ No additional __params__ configurations are required.
+
+- __GPU_CAGRA__ index
+
+ ```python
+ search_params = {
+ "metric_type": "L2",
+ "params": {
+ "itopk_size": 128,
+ "search_width": 4,
+ "min_iterations": 0,
+ "max_iterations": 0,
+ "team_size": 0
+ }
+ }
+ ```
+
+ Key search parameters include:
+
+ - __itopk_size__: Determines the size of intermediate results kept during the search. A larger value may improve recall at the expense of search performance. It should be at least equal to the final top-k (__limit__) value and is typically a power of 2 (e.g., 16, 32, 64, 128).
+
+ - __search_width__: Specifies the number of entry points into the CAGRA graph during the search. Increasing this value can enhance recall but may impact search performance.
+
+ - __min_iterations__ / __max_iterations__: These parameters control the search iteration process. By default, they are set to __0__, and CAGRA automatically determines the number of iterations based on __itopk_size__ and __search_width__. Adjusting these values manually can help balance performance and accuracy.
+
+ - __team_size__: Specifies the number of CUDA threads used for calculating metric distance on the GPU. Common values are a power of 2 up to 32 (e.g. 2, 4, 8, 16, 32). It has a minor impact on search performance. The default value is __0__, where Milvus automatically selects the __team_size__ based on the vector dimension.
+
+- __GPU_IVF_FLAT__ or __GPU_IVF_PQ__ index
+
+ ```python
+ search_params = {
+ "metric_type": "L2",
+ "params": {"nprobe": 10}
+ }
+ ```
+
+ Search parameters for these two index types are similar to those used in __[IVF_FLAT](https://milvus.io/docs/index.md#IVF_FLAT) and [IVF_PQ](https://milvus.io/docs/index.md#IVF_PQ)__. For more information, refer to [Conduct a Vector Similarity Search](https://milvus.io/docs/search.md#Prepare-search-parameters).
+
+### Conduct a search
+
+Use the [`search()`](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Collection/search.md) method to perform a vector similarity search on the GPU index.
+
+```python
+# Load data into memory
+collection.load()
+
+collection.search(
+ data=[[query_vector]], # Your query vector
+ anns_field="vector", # Name of the vector field
+ param=search_params,
+ limit=100 # Number of the results to return
+)
+```
+
+## Limits
+
+When using GPU indexes, be aware of certain constraints:
+
+- For __GPU_IVF_FLAT__, the maximum value for __limit__ is 256.
+
+- For __GPU_IVF_PQ__ and __GPU_CAGRA__, the maximum value for __limit__ is 1024.
+
+- While there is no set limit for __limit__ on __GPU_BRUTE_FORCE__, it is recommended not to exceed 4096 to avoid potential performance issues.
+
+- Currently, GPU indexes do not support COSINE distance. If COSINE distance is required, data should be normalized first, and then inner product (IP) distance can be used as a substitute.
+
+- Loading OOM protection for GPU indexes is not fully supported, too much data might lead to QueryNode crashes.
+
+- GPU indexes do not support search functions like [range search](https://milvus.io/docs/single-vector-search.md#Range-search) and [grouping search](https://milvus.io/docs/single-vector-search.md#Grouping-searchh).
+
+## FAQ
+
+- __When is it appropriate to utilize a GPU index?__
+
+ A GPU index is particularly beneficial in situations that demand high throughput or high recall. For instance, when dealing with large batches, the throughput of GPU indexing can surpass that of CPU indexing by as much as 100 times. In scenarios with smaller batches, GPU indexes still significantly outshine CPU indexes in terms of performance. Furthermore, if there's a requirement for rapid data insertion, incorporating a GPU can substantially speed up the process of building indexes.
+
+- __In which scenarios are GPU indexes like CAGRA, GPU_IVF_PQ, GPU_IVF_FLAT, and GPU_BRUTE_FORCE most suitable?__
+
+ CAGRA indexes are ideal for scenarios that demand enhanced performance, albeit at the cost of consuming more memory. For environments where memory conservation is a priority, the __GPU_IVF_PQ__ index can help minimize storage requirements, though this comes with a higher loss in precision. The __GPU_IVF_FLAT__ index serves as a balanced option, offering a compromise between performance and memory usage. Lastly, the __GPU_BRUTE_FORCE__ index is designed for exhaustive search operations, guaranteeing a recall rate of 1 by performing traversal searches.
\ No newline at end of file
diff --git a/preview/site/en/userGuide/manage-partitions.md b/preview/site/en/userGuide/manage-partitions.md
new file mode 100644
index 000000000..86bc76c94
--- /dev/null
+++ b/preview/site/en/userGuide/manage-partitions.md
@@ -0,0 +1,1038 @@
+---
+id: manage-partitions.md
+title: Manage Partitions
+---
+
+# Manage Partitions
+
+This guide walks you through how to create and manage partitions in a collection.
+
+## Overview
+
+A partition in Milvus represents a sub-division of a collection. This functionality allows the physical storage of a collection to be divided into multiple parts, contributing to improved query performance by narrowing down the focus to a smaller subset of data rather than the entire collection.
+
+Upon the creation of a collection, at least a default partition named ___default__ is automatically created. You can create a maximum of 4,096 partitions within a collection.
+
+
+
+
+
+## Preparations
+
+The code snippet below repurposes the existing code to establish a connection to Milvus and create a collection in a quick-setup mode, indicating that the collection is loaded upon creation.
+
+notes
+ +Milvus introduces a feature called Partition Key, leveraging the underlying partitions to store entities based on the hashed values of a specific field. This feature facilitates the implementation of multi-tenancy, enhancing search performance. For details, read Use Partition Key.
+If the Partition Key feature is on in a collection, Milvus takes care of managing all the partitions, relieving you of this responsibility.
+ +
+
+For preparations, use [`MiluvsClient`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) to connect to Milvus and [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md) to create a collection in a quick-setup mode.
+
+
+
+
+
+For preparations, use [`MilvusClientV2`](https://milvus.io/api-reference/java/v2.4.x/v2/Client/MilvusClientV2.md) to connect to Milvus and [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md) to create a collection in a quick-setup mode.
+
+
+
+
+
+For preparations, use [`MilvusClient`](https://milvus.io/api-reference/node/v2.4.x/Client/MilvusClient.md) to connect to Milvus and [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md) to create a collection in a quick-setup mode.
+
+
+
+
+
+```python
+from pymilvus import MilvusClient, DataType
+
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri="http://localhost:19530"
+)
+
+# 2. Create a collection
+client.create_collection(
+ collection_name="quick_setup",
+ dimension=5,
+)
+
+```
+
+```java
+import io.milvus.v2.client.ConnectConfig;
+import io.milvus.v2.client.MilvusClientV2;
+import io.milvus.v2.service.collection.request.CreateCollectionReq;
+
+String CLUSTER_ENDPOINT = "http://localhost:19530";
+
+// 1. Connect to Milvus server
+ConnectConfig connectConfig = ConnectConfig.builder()
+ .uri(CLUSTER_ENDPOINT)
+ .build();
+
+MilvusClientV2 client = new MilvusClientV2(connectConfig);
+
+// 2. Create a collection in quick setup mode
+CreateCollectionReq quickSetupReq = CreateCollectionReq.builder()
+ .collectionName("quick_setup")
+ .dimension(5)
+ .build();
+
+client.createCollection(quickSetupReq);
+```
+
+```javascript
+const address = "http://localhost:19530"
+
+// 1. Set up a Milvus Client
+client = new MilvusClient({address});
+
+// 2. Create a collection in quick setup mode
+await client.createCollection({
+ collection_name: "quick_setup",
+ dimension: 5,
+});
+```
+
+
+
+
+
+## List Partitions
+
+Once a collection is ready, you can list its partitions.
+
+notes
+ +In the above code snippet, the index of the collection has been created along with the collection, indicating that the collection is loaded upon creation.
+ +
+
+To list partitions, use [`list_partitions()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Partitions/list_partitions.md).
+
+
+
+
+
+To list partitions, use [`listPartitions()`](https://milvus.io/api-reference/java/v2.4.x/v2/Partitions/listPartitions.md).
+
+
+
+
+
+To list partitions, use [`listPartitions()`](https://milvus.io/api-reference/node/v2.4.x/Partitions/listPartitions.md).
+
+
+
+
+
+```python
+# 3. List partitions
+res = client.list_partitions(collection_name="quick_setup")
+print(res)
+
+# Output
+#
+# ["_default"]
+```
+
+```java
+import io.milvus.v2.service.partition.request.ListPartitionsReq;
+
+// 3. List all partitions in the collection
+ListPartitionsReq listPartitionsReq = ListPartitionsReq.builder()
+ .collectionName("quick_setup")
+ .build();
+
+List
+
+
+
+## Create Partitions
+
+You can add more partitions to the collection. A collection can have up to 64 partitions.
+
+notes
+ +If you have set a field as the partition key in a collection, Milvus creates at least 64 partitions along with the collection. When listing the partitions, the results may differ from the output of the above code snippets.
+For details, refer to Use Partition Key.
+ +
+
+To create partitions, use [`create_partition()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Partitions/create_partition.md).
+
+
+
+
+
+To create partitions, use [`createPartition()`](https://milvus.io/api-reference/java/v2.4.x/v2/Partitions/createPartition.md).
+
+
+
+
+
+To create partitions, use [`createPartition()`](https://milvus.io/api-reference/node/v2.4.x/Partitions/createPartition.md).
+
+
+
+
+
+```python
+# 4. Create more partitions
+client.create_partition(
+ collection_name="quick_setup",
+ partition_name="partitionA"
+)
+
+client.create_partition(
+ collection_name="quick_setup",
+ partition_name="partitionB"
+)
+
+res = client.list_partitions(collection_name="quick_setup")
+print(res)
+
+# Output
+#
+# ["_default", "partitionA", "partitionB"]
+```
+
+```java
+import io.milvus.v2.service.partition.request.CreatePartitionReq;
+
+// 4. Create more partitions
+CreatePartitionReq createPartitionReq = CreatePartitionReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionA")
+ .build();
+
+client.createPartition(createPartitionReq);
+
+createPartitionReq = CreatePartitionReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionB")
+ .build();
+
+client.createPartition(createPartitionReq);
+
+listPartitionsReq = ListPartitionsReq.builder()
+ .collectionName("quick_setup")
+ .build();
+
+partitionNames = client.listPartitions(listPartitionsReq);
+
+System.out.println(partitionNames);
+
+// Output:
+// [
+// "_default",
+// "partitionA",
+// "partitionB"
+// ]
+```
+
+```javascript
+// 4. Create more partitions
+await client.createPartition({
+ collection_name: "quick_setup",
+ partition_name: "partitionA"
+})
+
+await client.createPartition({
+ collection_name: "quick_setup",
+ partition_name: "partitionB"
+})
+
+res = await client.listPartitions({
+ collection_name: "quick_setup"
+})
+
+console.log(res.partition_names)
+
+// Output
+//
+// [ '_default', 'partitionA', 'partitionB' ]
+//
+```
+
+The code snippet above creates a partition in a collection and lists the partitions of the collection.
+
+
+
+
+
+## Check for a Specific Partition
+
+You can also check the existence of a specific partition.
+
+notes
+ +If you have set a field as the partition key in a collection, Milvus takes care of managing the partitions in the collection. Therefore, you may encounter prompted errors when attempting to create partitions.
+For details, refer to Use Partition Key.
+ +
+
+To check for a specific partition, use [`has_partition()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Partitions/has_partition.md).
+
+
+
+
+
+To check for a specific partition, use [`hasPartition()`](https://milvus.io/api-reference/java/v2.4.x/v2/Partitions/hasPartition.md).
+
+
+
+
+
+To check for a specific partition, use [`hasPartition()`](https://milvus.io/api-reference/node/v2.4.x/Partitions/hasPartition.md).
+
+
+
+
+
+```python
+# 5. Check whether a partition exists
+res = client.has_partition(
+ collection_name="quick_setup",
+ partition_name="partitionA"
+)
+print(res)
+
+# Output
+#
+# True
+
+res = client.has_partition(
+ collection_name="quick_setup",
+ partition_name="partitionC"
+)
+print(res)
+
+# Output
+#
+# False
+```
+
+```java
+import io.milvus.v2.service.partition.request.HasPartitionReq;
+
+// 5. Check whether a partition exists
+HasPartitionReq hasPartitionReq = HasPartitionReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionA")
+ .build();
+
+boolean exists = client.hasPartition(hasPartitionReq);
+
+System.out.println(exists);
+
+// Output:
+// true
+
+hasPartitionReq = HasPartitionReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionC")
+ .build();
+
+exists = client.hasPartition(hasPartitionReq);
+
+System.out.println(exists);
+
+// Output:
+// false
+```
+
+```javascript
+// 5. Check whether a partition exists
+res = await client.hasPartition({
+ collection_name: "quick_setup",
+ partition_name: "partitionA"
+})
+
+console.log(res.value)
+
+// Output
+//
+// true
+//
+
+res = await client.hasPartition({
+ collection_name: "quick_setup",
+ partition_name: "partitionC"
+})
+
+console.log(res.value)
+
+// Output
+//
+// false
+//
+```
+
+The code snippet above checks whether the collection has a partition named `partitionA` and `partitionC`.
+
+## Load & Release Partitions
+
+You can load and release specific partitions to make them available or unavailable for searches and queries.
+
+### Get Load Status
+
+
+
+To check the load status of a collection and its partitions, use [`get_load_state()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/get_load_state.md).
+
+
+
+
+
+To check the load status of a collection and its partitions, use [`getLoadState()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/getLoadState.md).
+
+
+
+
+
+To check the load status of a collection and its partitions, use [`getLoadState()`](https://milvus.io/api-reference/node/v2.4.x/Management/getLoadState.md).
+
+
+
+
+
+```python
+# Release the collection
+client.release_collection(collection_name="quick_setup")
+
+# Check the load status
+res = client.get_load_state(collection_name="quick_setup")
+print(res)
+
+# Output
+#
+# {
+# "state": "
+
+To load all partitions of a collection, you can just call [`load_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/load_collection.md). To load specific partitions of a collection, use [`load_partitions()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Partitions/load_partitions.md).
+
+
+
+
+
+To load all partitions of a collection, you can just call [`loadCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/loadCollection.md). To load specific partitions of a collection, use [`loadPartitions()`](https://milvus.io/api-reference/java/v2.4.x/v2/Partitions/loadPartitions.md).
+
+
+
+
+
+To load all partitions of a collection, you can just call [`loadCollection()`](https://milvus.io/api-reference/node/v2.4.x/Management/loadCollection.md). To load specific partitions of a collection, use [`loadPartitions()`](https://milvus.io/api-reference/node/v2.4.x/Partitions/loadPartitions.md).
+
+
+
+
+
+```python
+client.load_partitions(
+ collection_name="quick_setup",
+ partition_names=["partitionA"]
+)
+
+res = client.get_load_state(collection_name="quick_setup")
+print(res)
+
+# Output
+#
+# {
+# "state": "
+
+To release all partitions of a collection, you can just call [`release_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/release_collection.md). To release specific partitions of a collection, use [`release_partitions()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Partitions/release_partitions.md).
+
+
+
+
+
+To release all partitions of a collection, you can just call [`releaseCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/releaseCollection.md). To release specific partitions of a collection, use [`releasePartitions()`](https://milvus.io/api-reference/java/v2.4.x/v2/Partitions/releasePartitions.md).
+
+
+
+
+
+To release all partitions of a collection, you can just call [`releaseCollection()`](https://milvus.io/api-reference/node/v2.4.x/Management/releaseCollection.md). To release specific partitions of a collection, use [`releasePartitions()`](https://milvus.io/api-reference/node/v2.4.x/Partitions/releasePartitions.md).
+
+
+
+
+
+```python
+# 7. Release a partition
+client.release_partitions(
+ collection_name="quick_setup",
+ partition_names=["partitionA"]
+)
+
+res = client.get_load_state(
+ collection_name="quick_setup",
+ partition_name="partitionA"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "state": "
+
+To drop a partition, use [`drop_partition()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Partitions/drop_partition.md).
+
+
+
+
+
+To drop a partition, use [`dropPartition()`](https://milvus.io/api-reference/java/v2.4.x/v2/Partitions/dropPartition.md).
+
+
+
+
+
+To drop a partition, use [`dropPartition()`](https://milvus.io/api-reference/node/v2.4.x/Partitions/dropPartition.md).
+
+
+
+
+
+```python
+# 8. Drop a partition
+client.drop_partition(
+ collection_name="quick_setup",
+ partition_name="partitionB"
+)
+
+res = client.list_partitions(collection_name="quick_setup")
+print(res)
+
+# Output
+#
+# ["_default", "partitionA"]
+```
+
+```java
+import io.milvus.v2.service.partition.request.ReleasePartitionsReq;
+
+ReleasePartitionsReq releasePartitionsReq = ReleasePartitionsReq.builder()
+ .collectionName("quick_setup")
+ .partitionNames(List.of("_default", "partitionA", "partitionB"))
+ .build();
+
+client.releasePartitions(releasePartitionsReq);
+
+getLoadStateReq = GetLoadStateReq.builder()
+ .collectionName("quick_setup")
+ .build();
+
+state = client.getLoadState(getLoadStateReq);
+
+System.out.println(state);
+
+// Output:
+// false
+```
+
+```javascript
+
+await client.releasePartitions({
+ collection_name: "quick_setup",
+ partition_names: ["_default", "partitionA", "partitionB"]
+})
+
+res = await client.getLoadState({
+ collection_name: "quick_setup"
+})
+
+console.log(res)
+
+// Output
+//
+// {
+// status: {
+// error_code: 'Success',
+// reason: '',
+// code: 0,
+// retriable: false,
+// detail: ''
+// },
+// state: 'LoadStateNotLoad'
+// }
+//
+```
+
+
+
+
+
+## FAQ
+
+- __How much data can be stored in a partition?__
+
+ It is recommended to store less than 1B of data in a partition.
+
+- __What is the maximum number of partitions that can be created?__
+
+ By default, Milvus allows a maximum of 4,096 partitions to be created. You can adjust the maximum number of partitions by configuring `rootCoord.maxPartitionNum`. For details, refer to [System Configurations](https://milvus.io/docs/configure_rootcoord.md#rootCoordmaxPartitionNum).
+
+- __How can I differentiate between partitions and partition keys?__
+
+ Partitions are physical storage units, whereas partition keys are logical concepts that automatically assign data to specific partitions based on a designated column.
+
+ For instance, in Milvus, if you have a collection with a partition key defined as the `color` field, the system automatically assigns data to partitions based on the hashed values of the `color` field for each entity. This automated process relieves the user of the responsibility to manually specify the partition when inserting or searching data.
+
+ On the other hand, when manually creating partitions, you need to assign data to each partition based on the criteria of the partition key. If you have a collection with a `color` field, you would manually assign entities with a `color` value of `red` to `partition A`, and entities with a `color` value of `blue` to `partition B`. This manual management requires more effort.
+
+ In summary, both partitions and partition keys are utilized to optimize data computation and enhance query efficiency. It is essential to recognize that enabling a partition key means surrendering control over the manual management of partition data insertion and loading, as these processes are fully automated and handled by Milvus.
diff --git a/preview/site/en/userGuide/manage_collections.md b/preview/site/en/userGuide/manage_collections.md
deleted file mode 100644
index 51071bf4f..000000000
--- a/preview/site/en/userGuide/manage_collections.md
+++ /dev/null
@@ -1,20 +0,0 @@
----
-id: manage_collections.md
-title: Manage Collections
----
-
-# Manage Collections
-
-A Milvus collection is similar to a table in traditional databases and is used to store and manage data. Learn about the collection-level operations.
-
-- [Create a Collection](create_collection.md): Create a collection with the book collection example.
-
-- [Check Collection Information](check_collection.md): Check the basic information of a collection in Milvus including its name, number of entities, and more.
-
-- [Drop a Collection](drop_collection.md): Caution is needed as the delete operation irreversibly drops the collection and all data within it.
-
-- [Manage Collection Alias](collection_alias.md): Milvus supports specifying a unique alias for a collection.
-
-- [Load a Collection](load_collection.md): Load the collection to memory before a search or a query. Milvus 2.1 now supports loading a collection as multiple replicas.
-
-- [Release a Collection](release_collection.md): Release a collection from memory after a search or a query to reduce memory usage.
diff --git a/preview/site/en/userGuide/manage_connection.md b/preview/site/en/userGuide/manage_connection.md
deleted file mode 100644
index 47fe41e08..000000000
--- a/preview/site/en/userGuide/manage_connection.md
+++ /dev/null
@@ -1,268 +0,0 @@
----
-id: manage_connection.md
-related_key: connect Milvus
-summary: Learn how to connect to a Milvus server.
----
-
-# Manage Milvus Connections
-
-This topic describes how to connect to and disconnect from a Milvus server.
-
-notes
+ +Before dropping a partition, you need to release it from memory.
+ +
- Ensure to connect to a Milvus server before any operations.
-
-
-Milvus supports two ports, port `19530` and port `9091`:
-
-- Port `19530` is for gRPC. It is the default port when you connect to a Milvus server with different Milvus SDKs.
-
-- Port `9091` is for RESTful API. It is used when you connect to a Milvus server with an HTTP client.
-
-The example below connects to the Milvus server with host as `localhost` and port as `19530` or `9091`, and disconncets from it. If the connection is refused, try unblocking the corresponding port.
-
-## Connect to a Milvus server
-
-Construct a Milvus connection. Ensure to connect to Milvus server before any operations.
-
-
-
-
-```python
-# Run `python3` in your terminal to operate in the Python interactive mode.
-from pymilvus import connections
-connections.connect(
- alias="default",
- host='localhost',
- port='19530'
-)
-```
-
-```javascript
-import { MilvusClient } from "@zilliz/milvus2-sdk-node";
-const address = "localhost:19530";
-const milvusClient = new MilvusClient(address);
-```
-
-```go
-milvusClient, err := client.NewGrpcClient(
- context.Background(), // ctx
- "localhost:19530", // addr
-)
-if err != nil {
- log.Fatal("failed to connect to Milvus:", err.Error())
-}
-```
-
-```java
-final MilvusServiceClient milvusClient = new MilvusServiceClient(
- ConnectParam.newBuilder()
- .withHost("localhost")
- .withPort(19530)
- .build()
-);
-```
-
-```shell
-connect -h localhost -p 19530 -a default
-```
-
-```curl
-curl localhost:9091/api/v1/health
-{"status":"ok"}
-```
-
-| Parameter | -Description | -
|---|---|
alias |
- Alias of the Milvus connection to construct. | -
host |
- IP address of the Milvus server. | -
port |
- Port of the Milvus server. | -
| Parameter | -Description | -
|---|---|
address |
- Address of the Milvus connection to construct. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
addr |
- Address of the Milvus connection to construct. | -
| Parameter | -Description | -
|---|---|
Host |
- IP address of the Milvus server. | -
Port |
- Port of the Milvus server. | -
| Option | -Description | -
|---|---|
| -h (Optional) | -The host name. The default is "127.0.0.1". | -
| -p (Optional) | -The port number. The default is "19530". | -
| -a (Optional) | -The alias name of the Milvus link. The default is "default". | -
| -D (Optional) | -Flag to disconnect from the Milvus server specified by an alias. The default alias is "default". | -
-
-### Return
-
-A Milvus connection created by the passed parameters.
-
-### Raises
-
-
-
-
-## Disconnect from a Milvus server
-
-Disconnect from a Milvus server.
-
-
-
-
-```python
-connections.disconnect("default")
-```
-
-
-```javascript
-await milvusClient.closeConnection();
-```
-
-```go
-milvusClient.Close()
-```
-
-```java
-milvusClient.close()
-```
-
-```shell
-connect -D
-```
-
-```curl
-# Close your HTTP client connection.
-```
-
--
-
- NotImplementedError: If handler in connection parameters is not GRPC. -
- ParamError: If pool in connection parameters is not supported. -
- Exception: If server specified in parameters is not ready, we cannot connect to server. -
| Parameter | -Description | -
|---|---|
alias |
- Alias of the Milvus server to disconnect from. | -
+
+The code snippets on this page use the PyMilvus ORM module to interact with Milvus. Code snippets with the new MilvusClient SDK will be available soon.
+
+
+
+## Create database
+
+
+
+Use [connect()](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Connections/connect.md) to connect to the Milvus server and [create_database()](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/db/create_database.md) to create a new database:
+
+
+
+
+
+Use [MilvusClient](https://milvus.io/api-reference/java/v2.4.x/v1/Connections/MilvusClient.md) to connect to the Milvus server and [createDatabase()](https://milvus.io/api-reference/java/v2.4.x/v1/Database/createDatabase.md) to create a new database:
+
+
+
+
+
+Use [MilvusClient](https://milvus.io/api-reference/node/v2.4.x/Client/MilvusClient.md) to connect to the Milvus server and [createDatabase()](https://milvus.io/api-reference/node/v2.4.x/Database/createDatabase.md) to create a new database:
+
+
+
+
+
+```python
+from pymilvus import connections, db
+
+conn = connections.connect(host="127.0.0.1", port=19530)
+
+database = db.create_database("book")
+```
+
+```java
+import io.milvus.client.MilvusServiceClient;
+import io.milvus.param.ConnectParam;
+import io.milvus.param.collection.CreateDatabaseParam;
+
+// 1. Connect to Milvus server
+ConnectParam connectParam = ConnectParam.newBuilder()
+ .withUri(CLUSTER_ENDPOINT)
+ .withToken(TOKEN)
+ .build();
+
+MilvusServiceClient client = new MilvusServiceClient(connectParam);
+
+// 3. Create a new database
+CreateDatabaseParam createDatabaseParam = CreateDatabaseParam.newBuilder()
+ .withDatabaseName("my_database")
+ .build();
+
+R
+
+To find all existing databases in your Milvus cluster, use the [list_database()](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/db/list_database.md) method:
+
+
+
+
+
+To find all existing databases in your Milvus cluster, use the [listDatabases()](https://milvus.io/api-reference/java/v2.4.x/v1/Database/listDatabases.md) method:
+
+
+
+
+
+To find all existing databases in your Milvus cluster, use the [listDatabases()](https://milvus.io/api-reference/node/v2.4.x/Database/listDatabases.md) method:
+
+
+
+
+
+```python
+db.list_database()
+
+# Output
+['default', 'my_database']
+```
+
+```java
+import io.milvus.grpc.ListDatabasesResponse;
+import io.milvus.param.R;
+
+// 2. List all databases
+R
+
+To drop a database, use the [drop_database()](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/db/drop_database.md) method:
+
+
+
+
+
+To drop a database, use the [dropDatabase()](https://milvus.io/api-reference/java/v2.4.x/v1/Database/dropDatabase.md) method:
+
+
+
+
+
+To drop a database, use the [dropDatabase()](https://milvus.io/api-reference/node/v2.4.x/Database/dropDatabase.md) method:
+
+
+
+
+
+```python
+db.drop_database("book")
+
+db.list_database()
+
+# Output
+['default']
+```
+
+```java
+import io.milvus.param.collection.DropDatabaseParam;
+
+DropDatabaseParam dropDatabaseParam = DropDatabaseParam.newBuilder()
+ .withDatabaseName("my_database")
+ .build();
+
+response = client.dropDatabase(dropDatabaseParam);
+```
+
+```javascript
+res = await client.dropDatabase({
+ db_name: "my_db"
+})
+```
+
+## Use RBAC with database
+
+RBAC also covers database operations and ensures forward compatibility. The word **database** in the Permission APIs (Grant / Revoke / List Grant) has the following meanings:
+
+- If neither a Milvus connection nor a Permission API call specifies a `db_name`, **database** refers to the default database.
+- If a Milvus connection specifies a `db_name`, but a Permission API call afterward does not, **database** refers to the database whose name was specified in the Milvus connection.
+- If a Permission API call is made upon a Milvus connection, with or without `db_name` specified, **database** refers to the database whose name was specified in the Permission API call.
+
+The following code snippet is shared among the listed blocks below.
+
+
+
+```python
+from pymilvus import connections, Role
+
+_URI = "http://localhost:19530"
+_TOKEN = "root:Milvus"
+_DB_NAME = "default"
+
+
+def connect_to_milvus(db_name="default"):
+ print(f"connect to milvus\n")
+ connections.connect(
+ uri=_URI,
+ token=_TOKEN,
+ db_name=db_name
+ )
+```
+
+```java
+String URI = "http://localhost:19530";
+String TOKEN = "root:Milvus";
+
+public class ConnectToMilvus {
+ private String _dbName = "default";
+
+ public newBuilder() {}
+
+ public MilvusServiceClient build() {
+ ConnectParam connectParam = ConnectParam.newBuilder()
+ .withUri(URI)
+ .withToken(TOKEN)
+ .withDatabaseName(_dbNAME)
+ .build();
+
+ return new MilvusServiceClient(connectParam);
+ }
+
+ public newBuilder withDbName(String dbName) {
+ this._dbName = dbName;
+ return this;
+ }
+}
+```
+
+```javascript
+const address = "http://localhost:19530";
+const token = "root:Milvus";
+
+function connectToMilvus(dbName="default") {
+ const client = new MilvusClient({
+ address,
+ token,
+ dbName
+ });
+
+ return client;
+}
+```
+
+- If neither a Milvus connection nor a Permission API call specifies a `db_name`, **database** refers to the default database.
+
+
+
+ ```python
+ _ROLE_NAME = "test_role"
+ _PRIVILEGE_INSERT = "Insert"
+
+ connect_to_milvus()
+ role = Role(_ROLE_NAME)
+ role.create()
+
+ connect_to_milvus()
+ role.grant("Collection", "*", _PRIVILEGE_INSERT)
+ print(role.list_grants())
+ print(role.list_grant("Collection", "*"))
+ role.revoke("Global", "*", _PRIVILEGE_INSERT)
+ ```
+
+ ```java
+ String ROLE_NAME = "test_role";
+ String PRIVILEGE_INSERT = "Insert";
+
+ MilvusServiceClient client = new ConnectToMilvus().build();
+ R
-Output:
-
-```json
-{}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
partition_name (optional) |
- Name of the partition to release. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to release. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to release. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to release. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to release. | -
| -p (Optional/Multiple) | -The name of the partition to release. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to release. | -
| Parameter | -Description | -
|---|---|
partition_name |
- Name of the partition. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to release partitions. | -
partition_names |
- List of names of the partitions to release. | -
| Parameter | -Description | -
|---|---|
ctx |
- Context to control API invocation process. | -
CollectionName |
- Name of the collection to release partitions. | -
partitionNames |
- List of names of the partitions to release. | -
| Parameter | -Description | -
|---|---|
CollectionName |
- Name of the collection to release partition. | -
PartitionNames |
- List of names of the partitions to release. | -
| Option | -Description | -
|---|---|
| -c | -Name of the collection to release partition. | -
| -p (Multiple) | -The name of the partition to release. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to release partitions. | -
partition_names |
- List of names of the partitions to release. | -
+
+Use [`MilvusClient`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) to connect to the Milvus server and [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md) to create a collection.
+
+
+
+
+
+Use [`MilvusClientV2`](https://milvus.io/api-reference/java/v2.4.x/v2/Client/MilvusClientV2.md) to connect to the Milvus server and [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md) to create a collection.
+
+
+
+
+
+Use [`MilvusClient`](https://milvus.io/api-reference/node/v2.4.x/Client/MilvusClient.md) to connect to the Milvus server and [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md) to create a collection.
+
+
+
+
+
+```python
+from pymilvus import MilvusClient
+
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri="http://localhost:19530"
+)
+
+# 2. Create a collection
+client.create_collection(
+ collection_name="quick_setup",
+ dimension=5,
+)
+```
+
+```java
+String CLUSTER_ENDPOINT = "http://localhost:19530";
+
+// 1. Connect to Milvus server
+ConnectConfig connectConfig = ConnectConfig.builder()
+ .uri(CLUSTER_ENDPOINT)
+ .build();
+
+MilvusClientV2 client = new MilvusClientV2(connectConfig);
+
+// 2. Create a collection in quick setup mode
+CreateCollectionReq quickSetupReq = CreateCollectionReq.builder()
+ .collectionName("quick_setup")
+ .dimension(5)
+ .metricType("IP")
+ .build();
+
+client.createCollection(quickSetupReq);
+```
+
+```javascript
+const { MilvusClient, DataType, sleep } = require("@zilliz/milvus2-sdk-node")
+
+const address = "http://localhost:19530"
+
+// 1. Set up a Milvus Client
+client = new MilvusClient({address});
+
+// 2. Create a collection in quick setup mode
+await client.createCollection({
+ collection_name: "quick_setup",
+ dimension: 5,
+});
+```
+
+### Step 2: Insert randomly generated entities
+
+
+
+Use [`insert()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/insert.md) to insert entities into the collection.
+
+
+
+
+
+Use [`insert()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/insert.md) to insert entities into the collection.
+
+
+
+
+
+Use [`insert()`](https://milvus.io/api-reference/node/v2.4.x/Vector/insert.md) to insert entities into the collection.
+
+
+
+
+
+```python
+# 3. Insert randomly generated vectors
+colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
+data = []
+
+for i in range(1000):
+ current_color = random.choice(colors)
+ current_tag = random.randint(1000, 9999)
+ data.append({
+ "id": i,
+ "vector": [ random.uniform(-1, 1) for _ in range(5) ],
+ "color": current_color,
+ "tag": current_tag,
+ "color_tag": f"{current_color}_{str(current_tag)}"
+ })
+
+print(data[0])
+
+# Output
+#
+# {
+# "id": 0,
+# "vector": [
+# 0.7371107800002366,
+# -0.7290389773227746,
+# 0.38367002049157417,
+# 0.36996000494220627,
+# -0.3641898951462792
+# ],
+# "color": "yellow",
+# "tag": 6781,
+# "color_tag": "yellow_6781"
+# }
+
+res = client.insert(
+ collection_name="quick_setup",
+ data=data
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 1000,
+# "ids": [
+# 0,
+# 1,
+# 2,
+# 3,
+# 4,
+# 5,
+# 6,
+# 7,
+# 8,
+# 9,
+# "(990 more items hidden)"
+# ]
+# }
+```
+
+```java
+// 3. Insert randomly generated vectors into the collection
+List
+
+Use [`create_partition()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Partitions/create_partition.md) to create partitions and [`insert()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/insert.md) to insert more entities into the collection.
+
+
+
+
+
+Use [`createPartition()`](https://milvus.io/api-reference/java/v2.4.x/v2/Partitions/createPartition.md) to create partitions and [`insert()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/insert.md) to insert more entities into the collection.
+
+
+
+
+
+Use [`createPartition()`](https://milvus.io/api-reference/node/v2.4.x/Partitions/createPartition.md) to create partitions and [`insert()`](https://milvus.io/api-reference/node/v2.4.x/Vector/insert.md) to insert more entities into the collection.
+
+
+
+
+
+```python
+# 4. Create partitions and insert more entities
+client.create_partition(
+ collection_name="quick_setup",
+ partition_name="partitionA"
+)
+
+client.create_partition(
+ collection_name="quick_setup",
+ partition_name="partitionB"
+)
+
+data = []
+
+for i in range(1000, 1500):
+ current_color = random.choice(colors)
+ data.append({
+ "id": i,
+ "vector": [ random.uniform(-1, 1) for _ in range(5) ],
+ "color": current_color,
+ "tag": current_tag,
+ "color_tag": f"{current_color}_{str(current_tag)}"
+ })
+
+res = client.insert(
+ collection_name="quick_setup",
+ data=data,
+ partition_name="partitionA"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 500,
+# "ids": [
+# 1000,
+# 1001,
+# 1002,
+# 1003,
+# 1004,
+# 1005,
+# 1006,
+# 1007,
+# 1008,
+# 1009,
+# "(490 more items hidden)"
+# ]
+# }
+
+data = []
+
+for i in range(1500, 2000):
+ current_color = random.choice(colors)
+ data.append({
+ "id": i,
+ "vector": [ random.uniform(-1, 1) for _ in range(5) ],
+ "color": current_color,
+ "tag": current_tag,
+ "color_tag": f"{current_color}_{str(current_tag)}"
+ })
+
+res = client.insert(
+ collection_name="quick_setup",
+ data=data,
+ partition_name="partitionB"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 500,
+# "ids": [
+# 1500,
+# 1501,
+# 1502,
+# 1503,
+# 1504,
+# 1505,
+# 1506,
+# 1507,
+# 1508,
+# 1509,
+# "(490 more items hidden)"
+# ]
+# }
+```
+
+```java
+// 4. Create partitions and insert some more data
+CreatePartitionReq createPartitionReq = CreatePartitionReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionA")
+ .build();
+
+client.createPartition(createPartitionReq);
+
+createPartitionReq = CreatePartitionReq.builder()
+ .collectionName("quick_setup")
+ .partitionName("partitionB")
+ .build();
+
+client.createPartition(createPartitionReq);
+
+data.clear();
+
+for (int i=1000; i<1500; i++) {
+ Random rand = new Random();
+ String current_color = colors.get(rand.nextInt(colors.size()-1));
+ int current_tag = rand.nextInt(8999) + 1000;
+ JSONObject row = new JSONObject();
+ row.put("id", Long.valueOf(i));
+ row.put("vector", Arrays.asList(rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat()));
+ row.put("color", current_color);
+ row.put("tag", current_tag);
+ data.add(row);
+}
+
+insertReq = InsertReq.builder()
+ .collectionName("quick_setup")
+ .data(data)
+ .partitionName("partitionA")
+ .build();
+
+insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 500}
+
+data.clear();
+
+for (int i=1500; i<2000; i++) {
+ Random rand = new Random();
+ String current_color = colors.get(rand.nextInt(colors.size()-1));
+ int current_tag = rand.nextInt(8999) + 1000;
+ JSONObject row = new JSONObject();
+ row.put("id", Long.valueOf(i));
+ row.put("vector", Arrays.asList(rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat(), rand.nextFloat()));
+ row.put("color", current_color);
+ row.put("tag", current_tag);
+ data.add(row);
+}
+
+insertReq = InsertReq.builder()
+ .collectionName("quick_setup")
+ .data(data)
+ .partitionName("partitionB")
+ .build();
+
+insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 500}
+```
+
+```javascript
+// 4. Create partitions and insert more entities
+await client.createPartition({
+ collection_name: "quick_setup",
+ partition_name: "partitionA"
+})
+
+await client.createPartition({
+ collection_name: "quick_setup",
+ partition_name: "partitionB"
+})
+
+data = []
+
+for (let i = 1000; i < 1500; i++) {
+ current_color = colors[Math.floor(Math.random() * colors.length)]
+ current_tag = Math.floor(Math.random() * 8999 + 1000)
+ data.push({
+ "id": i,
+ "vector": [Math.random(), Math.random(), Math.random(), Math.random(), Math.random()],
+ "color": current_color,
+ "tag": current_tag,
+ "color_tag": `${current_color}_${current_tag}`
+ })
+}
+
+res = await client.insert({
+ collection_name: "quick_setup",
+ data: data,
+ partition_name: "partitionA"
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 500
+//
+
+await sleep(5000)
+
+data = []
+
+for (let i = 1500; i < 2000; i++) {
+ current_color = colors[Math.floor(Math.random() * colors.length)]
+ current_tag = Math.floor(Math.random() * 8999 + 1000)
+ data.push({
+ "id": i,
+ "vector": [Math.random(), Math.random(), Math.random(), Math.random(), Math.random()],
+ "color": current_color,
+ "tag": current_tag,
+ "color_tag": `${current_color}_${current_tag}`
+ })
+}
+
+res = await client.insert({
+ collection_name: "quick_setup",
+ data: data,
+ partition_name: "partitionB"
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 500
+//
+```
+
+## Get Entities by ID
+
+
+
+If you know the IDs of the entities of your interests, you can use the [`get()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/get.md) method.
+
+
+
+
+
+If you know the IDs of the entities of your interests, you can use the [`get()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/get.md) method.
+
+
+
+
+
+If you know the IDs of the entities of your interests, you can use the [`get()`](https://milvus.io/api-reference/node/v2.4.x/Vector/get.md) method.
+
+
+
+
+
+```python
+# 4. Get entities by ID
+res = client.get(
+ collection_name="quick_setup",
+ ids=[0, 1, 2]
+)
+
+print(res)
+
+# Output
+#
+# [
+# {
+# "id": 0,
+# "vector": [
+# 0.68824464,
+# 0.6552274,
+# 0.33593303,
+# -0.7099536,
+# -0.07070546
+# ],
+# "color_tag": "green_2006",
+# "color": "green"
+# },
+# {
+# "id": 1,
+# "vector": [
+# -0.98531723,
+# 0.33456197,
+# 0.2844234,
+# 0.42886782,
+# 0.32753858
+# ],
+# "color_tag": "white_9298",
+# "color": "white"
+# },
+# {
+# "id": 2,
+# "vector": [
+# -0.9886812,
+# -0.44129863,
+# -0.29859528,
+# 0.06059075,
+# -0.43817034
+# ],
+# "color_tag": "grey_5312",
+# "color": "grey"
+# }
+# ]
+```
+
+```java
+// 5. Get entities by ID
+GetReq getReq = GetReq.builder()
+ .collectionName("quick_setup")
+ .ids(Arrays.asList(0L, 1L, 2L))
+ .build();
+
+GetResp entities = client.get(getReq);
+
+System.out.println(JSONObject.toJSON(entities));
+
+// Output:
+// {"getResults": [
+// {"entity": {
+// "color": "white",
+// "color_tag": "white_4597",
+// "vector": [
+// 0.09665024,
+// 0.1163497,
+// 0.0701347,
+// 0.32577968,
+// 0.40943468
+// ],
+// "tag": 8946,
+// "id": 0
+// }},
+// {"entity": {
+// "color": "green",
+// "color_tag": "green_3039",
+// "vector": [
+// 0.90689456,
+// 0.4377399,
+// 0.75387514,
+// 0.36454988,
+// 0.8702918
+// ],
+// "tag": 2341,
+// "id": 1
+// }},
+// {"entity": {
+// "color": "white",
+// "color_tag": "white_8708",
+// "vector": [
+// 0.9757728,
+// 0.13974023,
+// 0.8023141,
+// 0.61947155,
+// 0.8290197
+// ],
+// "tag": 9913,
+// "id": 2
+// }}
+// ]}
+```
+
+```javascript
+// 5. Get entities by id
+res = await client.get({
+ collection_name: "quick_setup",
+ ids: [0, 1, 2],
+ output_fields: ["vector", "color_tag"]
+})
+
+console.log(res.data)
+
+// Output
+//
+// [
+// {
+// vector: [
+// 0.16022394597530365,
+// 0.6514875292778015,
+// 0.18294484913349152,
+// 0.30227693915367126,
+// 0.47553086280822754
+// ],
+// '$meta': { color: 'blue', tag: 8907, color_tag: 'blue_8907' },
+// id: '0'
+// },
+// {
+// vector: [
+// 0.2459285855293274,
+// 0.4974019527435303,
+// 0.2154673933982849,
+// 0.03719571232795715,
+// 0.8348019123077393
+// ],
+// '$meta': { color: 'grey', tag: 3710, color_tag: 'grey_3710' },
+// id: '1'
+// },
+// {
+// vector: [
+// 0.9404329061508179,
+// 0.49662265181541443,
+// 0.8088793158531189,
+// 0.9337621331214905,
+// 0.8269071578979492
+// ],
+// '$meta': { color: 'blue', tag: 2993, color_tag: 'blue_2993' },
+// id: '2'
+// }
+// ]
+//
+```
+
+### Get entities from partitions
+
+You can also get entities from specific partitions.
+
+
+
+```python
+# 5. Get entities from partitions
+res = client.get(
+ collection_name="quick_setup",
+ ids=[1000, 1001, 1002],
+ partition_names=["partitionA"]
+)
+
+print(res)
+
+# Output
+#
+# [
+# {
+# "color": "green",
+# "tag": 1995,
+# "color_tag": "green_1995",
+# "id": 1000,
+# "vector": [
+# 0.7807706,
+# 0.8083741,
+# 0.17276904,
+# -0.8580777,
+# 0.024156934
+# ]
+# },
+# {
+# "color": "red",
+# "tag": 1995,
+# "color_tag": "red_1995",
+# "id": 1001,
+# "vector": [
+# 0.065074645,
+# -0.44882354,
+# -0.29479212,
+# -0.19798489,
+# -0.77542555
+# ]
+# },
+# {
+# "color": "green",
+# "tag": 1995,
+# "color_tag": "green_1995",
+# "id": 1002,
+# "vector": [
+# 0.027934508,
+# -0.44199976,
+# -0.40262738,
+# -0.041511405,
+# 0.024782438
+# ]
+# }
+# ]
+```
+
+```java
+// 5. Get entities by ID in a partition
+getReq = GetReq.builder()
+ .collectionName("quick_setup")
+ .ids(Arrays.asList(1001L, 1002L, 1003L))
+ .partitionName("partitionA")
+ .build();
+
+entities = client.get(getReq);
+
+System.out.println(JSONObject.toJSON(entities));
+
+// Output:
+// {"getResults": [
+// {"entity": {
+// "color": "yellow",
+// "vector": [
+// 0.4300114,
+// 0.599917,
+// 0.799163,
+// 0.75395125,
+// 0.89947814
+// ],
+// "id": 1001,
+// "tag": 5803
+// }},
+// {"entity": {
+// "color": "blue",
+// "vector": [
+// 0.009218454,
+// 0.64637834,
+// 0.19815737,
+// 0.30519038,
+// 0.8218663
+// ],
+// "id": 1002,
+// "tag": 7212
+// }},
+// {"entity": {
+// "color": "black",
+// "vector": [
+// 0.76521933,
+// 0.7818409,
+// 0.16976339,
+// 0.8719652,
+// 0.1434964
+// ],
+// "id": 1003,
+// "tag": 1710
+// }}
+// ]}
+```
+
+```javascript
+// 5.1 Get entities by id in a partition
+res = await client.get({
+ collection_name: "quick_setup",
+ ids: [1000, 1001, 1002],
+ partition_names: ["partitionA"],
+ output_fields: ["vector", "color_tag"]
+})
+
+console.log(res.data)
+
+// Output
+//
+// [
+// {
+// id: '1000',
+// vector: [
+// 0.014254206791520119,
+// 0.5817716121673584,
+// 0.19793470203876495,
+// 0.8064294457435608,
+// 0.7745839357376099
+// ],
+// '$meta': { color: 'white', tag: 5996, color_tag: 'white_5996' }
+// },
+// {
+// id: '1001',
+// vector: [
+// 0.6073881983757019,
+// 0.05214758217334747,
+// 0.730999231338501,
+// 0.20900958776474,
+// 0.03665429726243019
+// ],
+// '$meta': { color: 'grey', tag: 2834, color_tag: 'grey_2834' }
+// },
+// {
+// id: '1002',
+// vector: [
+// 0.48877206444740295,
+// 0.34028753638267517,
+// 0.6527213454246521,
+// 0.9763909578323364,
+// 0.8031482100486755
+// ],
+// '$meta': { color: 'pink', tag: 9107, color_tag: 'pink_9107' }
+// }
+// ]
+//
+```
+
+
+## Use Basic Operators
+
+In this section, you will find examples of how to use basic operators in scalar filtering. You can apply these filters to [vector searches](https://milvus.io/docs/single-vector-search.md#Filtered-search) and [data deletions](https://milvus.io/docs/insert-update-delete.md#Delete-entities) too.
+
+
+
+For more information, refer to [`query()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/query.md) in the SDK reference.
+
+
+
+
+
+For more information, refer to [`query()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/query.md) in the SDK reference.
+
+
+
+
+
+For more information, refer to [`query()`](https://milvus.io/api-reference/node/v2.4.x/Vector/query.md) in the SDK reference.
+
+
+
+- Filter entities with their tag values falling between 1,000 to 1,500.
+
+
+
+ ```python
+ # 6. Use basic operators
+
+ res = client.query(
+ collection_name="quick_setup",
+ # highlight-start
+ filter="1000 < tag < 1500",
+ output_fields=["color_tag"],
+ # highlight-end
+ limit=3
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "id": 1,
+ # "color_tag": "pink_1023"
+ # },
+ # {
+ # "id": 41,
+ # "color_tag": "red_1483"
+ # },
+ # {
+ # "id": 44,
+ # "color_tag": "grey_1146"
+ # }
+ # ]
+ ```
+
+ ```java
+ // 6. Use basic operators
+
+ QueryReq queryReq = QueryReq.builder()
+ .collectionName("quick_setup")
+ .filter("1000 < tag < 1500")
+ .outputFields(Arrays.asList("color_tag"))
+ .limit(3)
+ .build();
+
+ QueryResp queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [
+ // {"entity": {
+ // "color_tag": "white_7588",
+ // "id": 34
+ // }},
+ // {"entity": {
+ // "color_tag": "orange_4989",
+ // "id": 64
+ // }},
+ // {"entity": {
+ // "color_tag": "white_3415",
+ // "id": 73
+ // }}
+ // ]}
+ ```
+
+ ```javascript
+ // 6. Use basic operators
+ res = await client.query({
+ collection_name: "quick_setup",
+ filter: "1000 < tag < 1500",
+ output_fields: ["color_tag"],
+ limit: 3
+ })
+
+ console.log(res.data)
+
+ // Output
+ //
+ // [
+ // {
+ // '$meta': { color: 'pink', tag: 1050, color_tag: 'pink_1050' },
+ // id: '6'
+ // },
+ // {
+ // '$meta': { color: 'purple', tag: 1174, color_tag: 'purple_1174' },
+ // id: '24'
+ // },
+ // {
+ // '$meta': { color: 'orange', tag: 1023, color_tag: 'orange_1023' },
+ // id: '40'
+ // }
+ // ]
+ //
+ ```
+
+- Filter entities with their __color__ values set to __brown__.
+
+
+
+ ```python
+ res = client.query(
+ collection_name="quick_setup",
+ # highlight-start
+ filter='color == "brown"',
+ output_fields=["color_tag"],
+ # highlight-end
+ limit=3
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "color_tag": "brown_5343",
+ # "id": 15
+ # },
+ # {
+ # "color_tag": "brown_3167",
+ # "id": 27
+ # },
+ # {
+ # "color_tag": "brown_3100",
+ # "id": 30
+ # }
+ # ]
+ ```
+
+ ```java
+ queryReq = QueryReq.builder()
+ .collectionName("quick_setup")
+ .filter("color == \"brown\"")
+ .outputFields(Arrays.asList("color_tag"))
+ .limit(3)
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [
+ // {"entity": {
+ // "color_tag": "brown_7792",
+ // "id": 3
+ // }},
+ // {"entity": {
+ // "color_tag": "brown_9695",
+ // "id": 7
+ // }},
+ // {"entity": {
+ // "color_tag": "brown_2551",
+ // "id": 15
+ // }}
+ // ]}
+ ```
+
+ ```javascript
+ res = await client.query({
+ collection_name: "quick_setup",
+ filter: 'color == "brown"',
+ output_fields: ["color_tag"],
+ limit: 3
+ })
+
+ console.log(res.data)
+
+ // Output
+ //
+ // [
+ // {
+ // '$meta': { color: 'brown', tag: 6839, color_tag: 'brown_6839' },
+ // id: '22'
+ // },
+ // {
+ // '$meta': { color: 'brown', tag: 7849, color_tag: 'brown_7849' },
+ // id: '32'
+ // },
+ // {
+ // '$meta': { color: 'brown', tag: 7855, color_tag: 'brown_7855' },
+ // id: '33'
+ // }
+ // ]
+ //
+ ```
+
+- Filter entities with their __color__ values not set to __green__ and __purple__.
+
+
+
+ ```python
+ res = client.query(
+ collection_name="quick_setup",
+ # highlight-start
+ filter='color not in ["green", "purple"]',
+ output_fields=["color_tag"],
+ # highlight-end
+ limit=3
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "color_tag": "yellow_6781",
+ # "id": 0
+ # },
+ # {
+ # "color_tag": "pink_1023",
+ # "id": 1
+ # },
+ # {
+ # "color_tag": "blue_3972",
+ # "id": 2
+ # }
+ # ]
+ ```
+
+ ```java
+ queryReq = QueryReq.builder()
+ .collectionName("quick_setup")
+ .filter("color not in [\"green\", \"purple\"]")
+ .outputFields(Arrays.asList("color_tag"))
+ .limit(3)
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [
+ // {"entity": {
+ // "color_tag": "white_4597",
+ // "id": 0
+ // }},
+ // {"entity": {
+ // "color_tag": "white_8708",
+ // "id": 2
+ // }},
+ // {"entity": {
+ // "color_tag": "brown_7792",
+ // "id": 3
+ // }}
+ // ]}
+ ```
+
+ ```javascript
+ res = await client.query({
+ collection_name: "quick_setup",
+ filter: 'color not in ["green", "purple"]',
+ output_fields: ["color_tag"],
+ limit: 3
+ })
+
+ console.log(res.data)
+
+ // Output
+ //
+ // [
+ // {
+ // '$meta': { color: 'blue', tag: 8907, color_tag: 'blue_8907' },
+ // id: '0'
+ // },
+ // {
+ // '$meta': { color: 'grey', tag: 3710, color_tag: 'grey_3710' },
+ // id: '1'
+ // },
+ // {
+ // '$meta': { color: 'blue', tag: 2993, color_tag: 'blue_2993' },
+ // id: '2'
+ // }
+ // ]
+ //
+ ```
+
+- Filter articles whose color tags start with __red__.
+
+
+
+ ```python
+ res = client.query(
+ collection_name="quick_setup",
+ # highlight-start
+ filter='color_tag like "red%"',
+ output_fields=["color_tag"],
+ # highlight-end
+ limit=3
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "color_tag": "red_6443",
+ # "id": 17
+ # },
+ # {
+ # "color_tag": "red_1483",
+ # "id": 41
+ # },
+ # {
+ # "color_tag": "red_4348",
+ # "id": 47
+ # }
+ # ]
+ ```
+
+ ```java
+ queryReq = QueryReq.builder()
+ .collectionName("quick_setup")
+ .filter("color_tag like \"red%\"")
+ .outputFields(Arrays.asList("color_tag"))
+ .limit(3)
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [
+ // {"entity": {
+ // "color_tag": "red_4929",
+ // "id": 9
+ // }},
+ // {"entity": {
+ // "color_tag": "red_8284",
+ // "id": 13
+ // }},
+ // {"entity": {
+ // "color_tag": "red_3021",
+ // "id": 44
+ // }}
+ // ]}
+ ```
+
+ ```javascript
+ res = await client.query({
+ collection_name: "quick_setup",
+ filter: 'color_tag like "red%"',
+ output_fields: ["color_tag"],
+ limit: 3
+ })
+
+ console.log(res.data)
+
+ // Output
+ //
+ // [
+ // {
+ // '$meta': { color: 'red', tag: 8773, color_tag: 'red_8773' },
+ // id: '17'
+ // },
+ // {
+ // '$meta': { color: 'red', tag: 9197, color_tag: 'red_9197' },
+ // id: '34'
+ // },
+ // {
+ // '$meta': { color: 'red', tag: 7914, color_tag: 'red_7914' },
+ // id: '46'
+ // }
+ // ]
+ //
+ ```
+
+- Filter entities with their colors set to red and tag values within the range from 1,000 to 1,500.
+
+
+
+ ```python
+ res = client.query(
+ collection_name="quick_setup",
+ # highlight-start
+ filter='(color == "red") and (1000 < tag < 1500)',
+ output_fields=["color_tag"],
+ # highlight-end
+ limit=3
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "color_tag": "red_1483",
+ # "id": 41
+ # },
+ # {
+ # "color_tag": "red_1100",
+ # "id": 94
+ # },
+ # {
+ # "color_tag": "red_1343",
+ # "id": 526
+ # }
+ # ]
+ ```
+
+ ```java
+ queryReq = QueryReq.builder()
+ .collectionName("quick_setup")
+ .filter("(color == \"red\") and (1000 < tag < 1500)")
+ .outputFields(Arrays.asList("color_tag"))
+ .limit(3)
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [
+ // {"entity": {
+ // "color_tag": "red_8124",
+ // "id": 83
+ // }},
+ // {"entity": {
+ // "color_tag": "red_5358",
+ // "id": 501
+ // }},
+ // {"entity": {
+ // "color_tag": "red_3564",
+ // "id": 638
+ // }}
+ // ]}
+ ```
+
+ ```javascript
+ res = await client.query({
+ collection_name: "quick_setup",
+ filter: '(color == "red") and (1000 < tag < 1500)',
+ output_fields: ["color_tag"],
+ limit: 3
+ })
+
+ console.log(res.data)
+
+ // Output
+ //
+ // [
+ // {
+ // '$meta': { color: 'red', tag: 1436, color_tag: 'red_1436' },
+ // id: '67'
+ // },
+ // {
+ // '$meta': { color: 'red', tag: 1463, color_tag: 'red_1463' },
+ // id: '160'
+ // },
+ // {
+ // '$meta': { color: 'red', tag: 1073, color_tag: 'red_1073' },
+ // id: '291'
+ // }
+ // ]
+ //
+ ```
+
+## Use Advanced Operators
+
+In this section, you will find examples of how to use advanced operators in scalar filtering. You can apply these filters to [vector searches](https://milvus.io/docs/single-vector-search.md#Filtered-search) and [data deletions](https://milvus.io/docs/insert-update-delete.md#Delete-entities) too.
+
+### Count entities
+
+- Counts the total number of entities in a collection.
+
+
+
+ ```python
+ # 7. Use advanced operators
+
+ # Count the total number of entities in a collection
+ res = client.query(
+ collection_name="quick_setup",
+ # highlight-start
+ output_fields=["count(*)"]
+ # highlight-end
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "count(*)": 2000
+ # }
+ # ]
+ ```
+
+ ```java
+ // 7. Use advanced operators
+ // Count the total number of entities in the collection
+ queryReq = QueryReq.builder()
+ .collectionName("quick_setup")
+ .filter("")
+ .outputFields(Arrays.asList("count(*)"))
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [{"entity": {"count(*)": 2000}}]}
+ ```
+
+ ```javascript
+ // 7. Use advanced operators
+ // Count the total number of entities in a collection
+ res = await client.query({
+ collection_name: "quick_setup",
+ output_fields: ["count(*)"]
+ })
+
+ console.log(res.data)
+
+ // Output
+ //
+ // [ { 'count(*)': '2000' } ]
+ //
+ ```
+
+- Counts the total number of entities in specific partitions.
+
+
+
+ ```python
+ # Count the number of entities in a partition
+ res = client.query(
+ collection_name="quick_setup",
+ # highlight-start
+ output_fields=["count(*)"],
+ partition_names=["partitionA"]
+ # highlight-end
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "count(*)": 500
+ # }
+ # ]
+ ```
+
+ ```java
+ // Count the number of entities in a partition
+ queryReq = QueryReq.builder()
+ .collectionName("quick_setup")
+ .partitionNames(Arrays.asList("partitionA"))
+ .filter("")
+ .outputFields(Arrays.asList("count(*)"))
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [{"entity": {"count(*)": 500}}]}
+ ```
+
+ ```javascript
+ // Count the number of entities in a partition
+ res = await client.query({
+ collection_name: "quick_setup",
+ output_fields: ["count(*)"],
+ partition_names: ["partitionA"]
+ })
+
+ console.log(res.data)
+
+ // Output
+ //
+ // [ { 'count(*)': '500' } ]
+ //
+ ```
+
+- Counts the number of entities that match a filtering condition
+
+
+
+ ```python
+ # Count the number of entities that match a specific filter
+ res = client.query(
+ collection_name="quick_setup",
+ # highlight-start
+ filter='(color == "red") and (1000 < tag < 1500)',
+ output_fields=["count(*)"],
+ # highlight-end
+ )
+
+ print(res)
+
+ # Output
+ #
+ # [
+ # {
+ # "count(*)": 3
+ # }
+ # ]
+ ```
+
+ ```java
+ // Count the number of entities that match a specific filter
+ queryReq = QueryReq.builder()
+ .collectionName("quick_setup")
+ .filter("(color == \"red\") and (1000 < tag < 1500)")
+ .outputFields(Arrays.asList("count(*)"))
+ .build();
+
+ queryResp = client.query(queryReq);
+
+ System.out.println(JSONObject.toJSON(queryResp));
+
+ // Output:
+ // {"queryResults": [{"entity": {"count(*)": 7}}]}
+ ```
+
+ ```javascript
+ // Count the number of entities that match a specific filter
+ res = await client.query({
+ collection_name: "quick_setup",
+ filter: '(color == "red") and (1000 < tag < 1500)',
+ output_fields: ["count(*)"]
+ })
+
+ console.log(res.data)
+
+ // Output
+ //
+ // [ { 'count(*)': '10' } ]
+ //
+ ```
+
+## Reference on scalar filters
+
+### Basic Operators
+
+A __boolean expression__ is always __a string comprising field names joined by operators__. In this section, you will learn more about basic operators.
+
+| __Operator__ | __Description__ |
+| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
+| __and (&&)__ | True if both operands are true |
+| __or (||)__ | True if either operand is true |
+| __+, -, *, /__ | Addition, subtraction, multiplication, and division |
+| __**__ | Exponent |
+| __%__ | Modulus |
+| __<, >__ | Less than, greater than |
+| __==, !=__ | Equal to, not equal to |
+| __<=, >=__ | Less than or equal to, greater than or equal to |
+| __not__ | Reverses the result of a given condition. |
+| __like__ | Compares a value to similar values using wildcard operators.For example, like "prefix%" matches strings that begin with "prefix". | +| __in__ | Tests if an expression matches any value in a list of values. | + +### Advanced operators + +- `count(*)` + + Counts the exact number of entities in the collection. Use this as an output field to get the exact number of entities in a collection or partition. + +
+
+
+
+
diff --git a/preview/site/en/userGuide/search-query-get/multi-vector-search.md b/preview/site/en/userGuide/search-query-get/multi-vector-search.md
new file mode 100644
index 000000000..25e9f0ca0
--- /dev/null
+++ b/preview/site/en/userGuide/search-query-get/multi-vector-search.md
@@ -0,0 +1,247 @@
+---
+id: multi-vector-search.md
+order: 2
+summary: This guide demonstrates how to perform hybrid search in Milvus and understand the reranking of results.
+title: Hybrid Search
+---
+
+# Hybrid Search
+
+Since Milvus 2.4, we introduced multi-vector support and a hybrid search framework, which means users can bring in several vector fields (up to 10) into a single collection. These vectors in different columns represent diverse facets of data, originating from different embedding models or undergoing distinct processing methods. The results of hybrid searches are integrated using reranking strategies, such as Reciprocal Rank Fusion (RRF) and Weighted Scoring. To learn more about reranking strategies, refer to [Reranking](reranking.md).
+
+This feature is particularly useful in comprehensive search scenarios, such as identifying the most similar person in a vector library based on various attributes like pictures, voice, fingerprints, etc.
+
+In this tutorial, you will learn how to:
+
+- Create multiple `AnnSearchRequest` instances for similarity searches on different vector fields;
+
+- Configure a reranking strategy to combine and rerank search results from multiple `AnnSearchRequest` instances;
+
+- Use the [`hybrid_search()`](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Collection/hybrid_search.md) method to perform a hybrid search.
+
+notes
+ +This applies to loaded collections. You should use it as the only output field.
+ +
+
+The code snippets on this page use the [PyMilvus ORM module](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Connections/connect.md) to interact with Milvus. Code snippets with the new [MilvusClient SDK](https://milvus.io/api-reference/pymilvus/v2.4.x/About.md) will be available soon.
+
+
+
+## Preparations
+
+Before starting a hybrid search, ensure you have a collection with multiple vector fields. Currently, Milvus introduces a default of four vector fields per collection, which can be extended to a maximum of ten by modifying the [proxy.maxVectorFieldNum](https://milvus.io/docs/configure_proxy.md#proxymaxVectorFieldNum) configuration.
+
+Below is an example of creating a collection named `test_collection` with two vector fields, `filmVector` and `posterVector`, and inserting random entities into it.
+
+```python
+from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
+import random
+
+# Connect to Milvus
+connections.connect(
+ host="10.102.7.3", # Replace with your Milvus server IP
+ port="19530"
+)
+
+# Create schema
+fields = [
+ FieldSchema(name="film_id", dtype=DataType.INT64, is_primary=True),
+ FieldSchema(name="filmVector", dtype=DataType.FLOAT_VECTOR, dim=5), # Vector field for film vectors
+ FieldSchema(name="posterVector", dtype=DataType.FLOAT_VECTOR, dim=5)] # Vector field for poster vectors
+
+schema = CollectionSchema(fields=fields,enable_dynamic_field=False)
+
+# Create collection
+collection = Collection(name="test_collection", schema=schema)
+
+# Create index for each vector field
+index_params = {
+ "metric_type": "L2",
+ "index_type": "IVF_FLAT",
+ "params": {"nlist": 128},
+}
+
+collection.create_index("filmVector", index_params)
+collection.create_index("posterVector", index_params)
+
+# Generate random entities to insert
+entities = []
+
+for _ in range(1000):
+ # generate random values for each field in the schema
+ film_id = random.randint(1, 1000)
+ film_vector = [ random.random() for _ in range(5) ]
+ poster_vector = [ random.random() for _ in range(5) ]
+
+ # create a dictionary for each entity
+ entity = {
+ "film_id": film_id,
+ "filmVector": film_vector,
+ "posterVector": poster_vector
+ }
+
+ # add the entity to the list
+ entities.append(entity)
+
+collection.insert(entities)
+```
+
+## Step 1: Create Multiple AnnSearchRequest Instances
+
+A hybrid search uses the `hybrid_search()` API to perform multiple ANN search requests in a single call. Each `AnnSearchRequest` represents a single search request on a specific vector field.
+
+The following example creates two `AnnSearchRequest` instances to perform individual similarity searches on two vector fields.
+
+```python
+from pymilvus import AnnSearchRequest
+
+# Create ANN search request 1 for filmVector
+query_filmVector = [[0.8896863042430693, 0.370613100114602, 0.23779315077113428, 0.38227915951132996, 0.5997064603128835]]
+
+search_param_1 = {
+ "data": query_filmVector, # Query vector
+ "anns_field": "filmVector", # Vector field name
+ "param": {
+ "metric_type": "L2", # This parameter value must be identical to the one used in the collection schema
+ "params": {"nprobe": 10}
+ },
+ "limit": 2 # Number of search results to return in this AnnSearchRequest
+}
+request_1 = AnnSearchRequest(**search_param_1)
+
+# Create ANN search request 2 for posterVector
+query_posterVector = [[0.02550758562349764, 0.006085637357292062, 0.5325251250159071, 0.7676432650114147, 0.5521074424751443]]
+search_param_2 = {
+ "data": query_posterVector, # Query vector
+ "anns_field": "posterVector", # Vector field name
+ "param": {
+ "metric_type": "L2", # This parameter value must be identical to the one used in the collection schema
+ "params": {"nprobe": 10}
+ },
+ "limit": 2 # Number of search results to return in this AnnSearchRequest
+}
+request_2 = AnnSearchRequest(**search_param_2)
+
+# Store these two requests as a list in `reqs`
+reqs = [request_1, request_2]
+```
+
+Parameters:
+
+- `AnnSearchRequest` (_object_)
+
+ A class representing an ANN search request. Each hybrid search can contain 1 to 1,024 `ANNSearchRequest` objects at a time.
+
+- `data` (_list_)
+
+ The query vector to search in a single `AnnSearchRequest`. Currently, this parameter accepts a list containing only a single query vector, for example, `[[0.5791814851218929, 0.5792985702614121, 0.8480776460143558, 0.16098005945243, 0.2842979317256803]]`. In the future, this parameter will be expanded to accept multiple query vectors.
+
+- `anns_field` (_string_)
+
+ The name of the vector field to use in a single `AnnSearchRequest`.
+
+- `param` (_dict_)
+
+ A dictionary of search parameters for a single `AnnSearchRequest`. These search parameters are identical to those for a single-vector search. For more information, refer to [Search parameters](https://milvus.io/docs/single-vector-search.md#Search-parameters).
+
+- `limit` (_int_)
+
+ The maximum number of search results to include in a single `ANNSearchRequest`.
+
+ This parameter only affects the number of search results to return within an individual `ANNSearchRequest`, and it does not decide the final results to return for a `hybrid_search` call. In a hybrid search, the final results are determined by combining and reranking the results from multiple `ANNSearchRequest` instances.
+
+## Step 2: Configure a Reranking Strategy
+
+After creating `AnnSearchRequest` instances, configure a reranking strategy to combine and rerank the results. Currently, there are two options: `WeightedRanker` and `RRFRanker`. For more information about reranking strategies, refer to [Reranking](reranking.md).
+
+- Use weighted scoring
+
+ The `WeightedRanker` is used to assign importance to the results from each vector field search with specified weights. If you prioritize some vector fields over others, `WeightedRanker(value1, value2, ..., valueN)` can reflect this in the combined search results.
+
+ ```python
+ from pymilvus import WeightedRanker
+ # Use WeightedRanker to combine results with specified weights
+ # Assign weights of 0.8 to text search and 0.2 to image search
+ rerank = WeightedRanker(0.8, 0.2)
+ ```
+
+ When using `WeightedRanker`, note that:
+
+ - Each weight value ranges from 0 (least important) to 1 (most important), influencing the final aggregated score.
+ - The total number of weight values provided in `WeightedRanker` should equal the number of `AnnSearchRequest` instances you have created.
+
+- Use Reciprocal Rank Fusion (RFF)
+
+ ```python
+ # Alternatively, use RRFRanker for reciprocal rank fusion reranking
+ from pymilvus import RRFRanker
+
+ rerank = RRFRanker()
+ ```
+
+## Step 3: Perform a Hybrid Search
+
+With the `AnnSearchRequest` instances and reranking strategy set, use the `hybrid_search()` method to perform the hybrid search.
+
+```python
+# Before conducting hybrid search, load the collection into memory.
+collection.load()
+
+res = collection.hybrid_search(
+ reqs, # List of AnnSearchRequests created in step 1
+ rerank, # Reranking strategy specified in step 2
+ limit=2 # Number of final search results to return
+)
+
+print(res)
+```
+
+Parameters:
+
+- `reqs` (_list_)
+
+ A list of search requests, where each request is an `ANNSearchRequest` object. Each request can correspond to a different vector field and a different set of search parameters.
+
+- `rerank` (_object_)
+
+ The reranking strategy to use for hybrid search. Possible values: `WeightedRanker(value1, value2, ..., valueN)` and `RRFRanker()`.
+
+ For more information about reranking strategies, refer to [Reranking](reranking.md).
+
+- `limit` (_int_)
+
+ The maximum number of final results to return in the hybrid search.
+
+The output is similar to the following:
+
+```python
+["['id: 844, distance: 0.006047376897186041, entity: {}', 'id: 876, distance: 0.006422005593776703, entity: {}']"]
+```
+
+## Limits
+
+- Typically, each collection has a default allowance of up to 4 vector fields. However, you have the option to adjust the `proxy.maxVectorFieldNum` configuration to expand the maximum number of vector fields in a collection, with a maximum limit of 10 vector fields per collection. See [Proxy-related Configurations](https://milvus.io/docs/configure_proxy.md#Proxy-related-Configurations) for more.
+
+- Partially indexed or loaded vector fields in a collection will result in an error.
+
+- Currently, each `AnnSearchRequest` in a hybrid search can carry one query vector only.
+
+## FAQ
+
+- **In which scenario is hybrid search recommended?**
+
+ Hybrid search is ideal for complex situations demanding high accuracy, especially when an entity can be represented by multiple, diverse vectors. This applies to cases where the same data, such as a sentence, is processed through different embedding models or when multimodal information (like images, fingerprints, and voiceprints of an individual) is converted into various vector formats. By assigning weights to these vectors, their combined influence can significantly enrich recall and improve the effectiveness of search results.
+
+- **How does a weighted ranker normalize distances between different vector fields?**
+
+ A weighted ranker normalizes the distances between vector fields using assigned weights to each field. It calculates the importance of each vector field according to its weight, prioritizing those with higher weights. It's advised to use the same metric type across ANN search requests to ensure consistency. This method ensures that vectors deemed more significant have a greater influence on the overall ranking.
+
+- **Is it possible to use alternative rankers like Cohere Ranker or BGE Ranker?**
+
+ Currently, only the provided rankers are supported. Plans to include additional rankers are underway for future updates.
+
+- **Is it possible to conduct multiple hybrid search operations at the same time?**
+
+ Yes, simultaneous execution of multiple hybrid search operations is supported.
+
+- **Can I use the same vector field in multiple AnnSearchRequest objects to perform hybrid searches?**
+
+ Technically, it is possible to use the same vector field in multiple AnnSearchRequest objects for hybrid searches. It is not necessary to have multiple vector fields for a hybrid search.
diff --git a/preview/site/en/userGuide/search-query-get/single-vector-search.md b/preview/site/en/userGuide/search-query-get/single-vector-search.md
new file mode 100644
index 000000000..7dc2625a6
--- /dev/null
+++ b/preview/site/en/userGuide/search-query-get/single-vector-search.md
@@ -0,0 +1,1976 @@
+---
+id: single-vector-search.md
+order: 1
+summary: This article describes how to search for vectors in a Milvus collection using a single query vector.
+title: Single-Vector Search
+---
+
+# Single-Vector Search
+
+Once you have inserted your data, the next step is to perform similarity searches on your collection in Milvus.
+
+Milvus allows you to conduct two types of searches, depending on the number of vector fields in your collection:
+
+- **Single-vector search**: If your collection has only one vector field, use the [`search()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/search.md) method to find the most similar entities. This method compares your query vector with the existing vectors in your collection and returns the IDs of the closest matches along with the distances between them. Optionally, it can also return the vector values and metadata of the results.
+- **Hybrid search**: For collections with two or more vector fields, use the [`hybrid_search()`](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Collection/hybrid_search.md) method. This method performs multiple Approximate Nearest Neighbor (ANN) search requests and combines the results to return the most relevant matches after reranking.
+
+This guide focuses on how to perform a single-vector search in Milvus. For details on hybrid search, refer to [Hybrid search](https://milvus.io/docs/multi-vector-search.md).
+
+## Overview
+
+There are a variety of search types to meet different requirements:
+
+- [Basic search](https://milvus.io/docs/single-vector-search.md#Basic-search): Includes single-vector search, bulk-vector search, partition search, and search with specified output fields.
+
+- [Filtered search](https://milvus.io/docs/single-vector-search.md#Filtered-search): Applies filtering criteria based on scalar fields to refine search results.
+
+- [Range search](https://milvus.io/docs/single-vector-search.md#Range-search): Finds vectors within a specific distance range from the query vector.
+
+- [Grouping search](https://milvus.io/docs/single-vector-search.md#Grouping-search): Groups search results based on a specific field to ensure diversity in the results.
+
+## Preparations
+
+The code snippet below repurposes the existing code to establish a connection to Milvus and quickly set up a collection.
+
+
+
+```python
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri=CLUSTER_ENDPOINT,
+ token=TOKEN
+)
+
+# 2. Create a collection
+client.create_collection(
+ collection_name="quick_setup",
+ dimension=5,
+ metric_type="IP"
+)
+
+# 3. Insert randomly generated vectors
+colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
+data = []
+
+for i in range(1000):
+ current_color = random.choice(colors)
+ data.append({
+ "id": i,
+ "vector": [ random.uniform(-1, 1) for _ in range(5) ],
+ "color": current_color,
+ "color_tag": f"{current_color}_{str(random.randint(1000, 9999))}"
+ })
+
+res = client.insert(
+ collection_name="quick_setup",
+ data=data
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 1000,
+# "ids": [
+# 0,
+# 1,
+# 2,
+# 3,
+# 4,
+# 5,
+# 6,
+# 7,
+# 8,
+# 9,
+# "(990 more items hidden)"
+# ]
+# }
+
+# 6.1 Create partitions
+client.create_partition(
+ collection_name="quick_setup",
+ partition_name="red"
+)
+
+client.create_partition(
+ collection_name="quick_setup",
+ partition_name="blue"
+)
+
+# 6.1 Insert data into partitions
+red_data = [ {"id": i, "vector": [ random.uniform(-1, 1) for _ in range(5) ], "color": "red", "color_tag": f"red_{str(random.randint(1000, 9999))}" } for i in range(500) ]
+blue_data = [ {"id": i, "vector": [ random.uniform(-1, 1) for _ in range(5) ], "color": "blue", "color_tag": f"blue_{str(random.randint(1000, 9999))}" } for i in range(500) ]
+
+res = client.insert(
+ collection_name="quick_setup",
+ data=red_data,
+ partition_name="red"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 500,
+# "ids": [
+# 0,
+# 1,
+# 2,
+# 3,
+# 4,
+# 5,
+# 6,
+# 7,
+# 8,
+# 9,
+# "(490 more items hidden)"
+# ]
+# }
+
+res = client.insert(
+ collection_name="quick_setup",
+ data=blue_data,
+ partition_name="blue"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 500,
+# "ids": [
+# 0,
+# 1,
+# 2,
+# 3,
+# 4,
+# 5,
+# 6,
+# 7,
+# 8,
+# 9,
+# "(490 more items hidden)"
+# ]
+# }
+```
+
+```java
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+import com.alibaba.fastjson.JSONObject;
+
+import io.milvus.v2.client.ConnectConfig;
+import io.milvus.v2.client.MilvusClientV2;
+import io.milvus.v2.service.collection.request.CreateCollectionReq;
+import io.milvus.v2.service.collection.request.GetLoadStateReq;
+import io.milvus.v2.service.vector.request.InsertReq;
+import io.milvus.v2.service.vector.response.InsertResp;
+
+String CLUSTER_ENDPOINT = "http://localhost:19530";
+
+// 1. Connect to Milvus server
+ConnectConfig connectConfig = ConnectConfig.builder()
+ .uri(CLUSTER_ENDPOINT)
+ .build();
+
+MilvusClientV2 client = new MilvusClientV2(connectConfig);
+
+// 2. Create a collection in quick setup mode
+CreateCollectionReq quickSetupReq = CreateCollectionReq.builder()
+ .collectionName("quick_setup")
+ .dimension(5)
+ .metricType("IP")
+ .build();
+
+client.createCollection(quickSetupReq);
+
+GetLoadStateReq loadStateReq = GetLoadStateReq.builder()
+ .collectionName("quick_setup")
+ .build();
+
+boolean state = client.getLoadState(loadStateReq);
+
+System.out.println(state);
+
+// Output:
+// true
+
+// 3. Insert randomly generated vectors into the collection
+List| Parameter | +Description | +
|---|---|
collection_name |
+ The name of an existing collection. | +
data |
+ A list of vector embeddings. Milvus searches for the most similar vector embeddings to the specified ones. |
+
limit |
+ The total number of entities to return. You can use this parameter in combination with offset in param to enable pagination. The sum of this value and offset in param should be less than 16,384. |
+
search_params |
+ The parameter settings specific to this operation.
|
+
| Parameter | +Description | +
|---|---|
collectionName |
+ The name of an existing collection. | +
data |
+ A list of vector embeddings. Milvus searches for the most similar vector embeddings to the specified ones. |
+
topK |
+ The number of records to return in the search result. This parameter uses the same syntax as the limit parameter, so you should only set one of them. You can use this parameter in combination with offset in param to enable pagination. The sum of this value and offset in param should be less than 16,384. |
+
| Parameter | +Description | +
|---|---|
collection_name |
+ The name of an existing collection. | +
data |
+ A list of vector embeddings. Milvus searches for the most similar vector embeddings to the specified ones. |
+
limit |
+ The total number of entities to return. You can use this parameter in combination with offset in param to enable pagination. The sum of this value and offset in param should be less than 16,384. |
+
`range_filter` <= distance < `radius` | +| `IP` | Larger IP distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
`radius` < distance <= `range_filter` | +| `COSINE` | Larger cosine value indicates higher similarity. | To exclude the closest vectors from results, ensure that:
`radius` < distance <= `range_filter` | +| `JACCARD` | Smaller Jaccard distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
`range_filter` <= distance < `radius` | +| `HAMMING` | Smaller Hamming distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
`range_filter` <= distance < `radius` | + +To learn more about distance metric types, refer to [Similarity Metrics](metric.md). + +## Grouping search + +In Milvus, grouping search by a specific field can avoid redundancy of the same field item in the results. You can get a varied set of results for the specific field. + +Consider a collection of documents, each document splits into various passages. Each passage is represented by one vector embedding and belongs to one document. To find relevant documents instead of similar passages, you can include the `group_by_field` argument in the `search()` opeartion to group results by the document ID. This helps return the most relevant and unique documents, rather than separate passages from the same document. + +Here is the example code to group search results by field: + +```python +# Connect to Milvus +client = MilvusClient(uri='http://localhost:19530') # Milvus server address + +# Load data into collection +client.load_collection("group_search") # Collection name + +# Group search results +res = client.search( + collection_name="group_search", # Collection name + data=[[0.14529211512077012, 0.9147257273453546, 0.7965055218724449, 0.7009258593102812, 0.5605206522382088]], # Query vector + search_params={ + "metric_type": "L2", + "params": {"nprobe": 10}, + }, # Search parameters + limit=10, # Max. number of search results to return + group_by_field="doc_id", # Group results by document ID + output_fields=["doc_id", "passage_id"] +) + +# Retrieve the values in the `doc_id` column +doc_ids = [result['entity']['doc_id'] for result in res[0]] + +print(doc_ids) +``` + +The output is similar to the following: + +```python +[5, 10, 1, 7, 9, 6, 3, 4, 8, 2] +``` + +In the given output, it can be observed that the returned entities do not contain any duplicate `doc_id` values. + +For comparison, let's comment out the `group_by_field` and conduct a regular search: + +```python +# Connect to Milvus +client = MilvusClient(uri='http://localhost:19530') # Milvus server address + +# Load data into collection +client.load_collection("group_search") # Collection name + +# Search without `group_by_field` +res = client.search( + collection_name="group_search", # Collection name + data=query_passage_vector, # Replace with your query vector + search_params={ + "metric_type": "L2", + "params": {"nprobe": 10}, + }, # Search parameters + limit=10, # Max. number of search results to return + # group_by_field="doc_id", # Group results by document ID + output_fields=["doc_id", "passage_id"] +) + +# Retrieve the values in the `doc_id` column +doc_ids = [result['entity']['doc_id'] for result in res[0]] + +print(doc_ids) +``` + +The output is similar to the following: + +```python +[1, 10, 3, 10, 1, 9, 4, 4, 8, 6] +``` + +In the given output, it can be observed that the returned entities contain duplicate `doc_id` values. + +__Limitations__ + +- __Indexing__: This grouping feature works only for collections that are indexed with the __HNSW__, __IVF_FLAT__, or __FLAT__ type. For more information, refer to [In-memory Index](https://milvus.io/docs/index.md#HNSW). + +- __Vector__: Currently, grouping search does not support a vector field of the __BINARY_VECTOR__ type. For more information on data types, refer to [Supported data types](https://milvus.io/docs/schema.md#Supported-data-types). + +- __Field__: Currently, grouping search allows only for a single column. You cannot specify multiple field names in the `group_by_field` config. Additionally, grouping search is incompatible with data types of JSON, FLOAT, DOUBLE, ARRAY, or vector fields. + +- __Performance Impact__: Be mindful that performance degrades with increasing query vector counts. Using a cluster with 2 CPU cores and 8 GB of memory as an example, the execution time for grouping search increases proportionally with the number of input query vectors. + +- __Functionality__: Currently, grouping search is not supported by [range search](https://milvus.io/docs/single-vector-search.md#Range-search), [search iterators](https://milvus.io/docs/with-iterators.md#Search-with-iterator), or [hybrid search](multi-vector-search.md). + +## Search parameters + +In the above searches except the range search, the default search parameters apply. In normal cases, you do not need to manually set search parameters. + +```python +# In normal cases, you do not need to set search parameters manually +# Except for range searches. +search_parameters = { + 'metric_type': 'L2', + 'params': { + 'nprobe': 10, + 'level': 1， + 'radius': 1.0 + 'range_filter': 0.8 + } +} +``` + +The following table lists all possible settings in the search parameters. + +| __Parameter Name__ | __Parameter Description__ | +| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `metric_type` | How to measure similarity between vector embeddings.
Possible values are `IP`, `L2`, and `COSINE`, and defaults to that of the loaded index file. | +| `params.nprobe` | Number of units to query during the search.
The value falls in the range [1, nlist[1]]. | +| `params.level` | Search precision level.
Possible values are `1`, `2`, and `3`, and defaults to `1`. Higher values yield more accurate results but slower performance. | +| `params.radius` | Minimum similarity between the query vector and candidate vectors.
The value falls in the range [1, nlist[1]]. | +| `params.range_filter` | A similarity range, optionally refining the search for vectors that fall in the range.
The value falls in the range [top-K[2], ∞]. | + +
+
+
+
diff --git a/preview/site/en/userGuide/search-query-get/with-iterators.md b/preview/site/en/userGuide/search-query-get/with-iterators.md
new file mode 100644
index 000000000..e9b9d8254
--- /dev/null
+++ b/preview/site/en/userGuide/search-query-get/with-iterators.md
@@ -0,0 +1,596 @@
+---
+id: with-iterators.md
+order: 4
+summary: Milvus provides search and query iterators for iterating results with a large volume of entities.
+title: With Iterators
+---
+
+# With Iterators
+
+Milvus provides search and query iterators for iterating results with a large volume of entities.
+
+## Overview
+
+Iterators are powerful tools that help you iterate through a large volume of or all data within a collection using primary key values and Boolean expressions. This can significantly improve the way you retrieve data. Unlike the traditional use of __offset__ and __limit__ parameters, which may become less efficient over time, iterators offer a more scalable solution.
+
+### Benefits of using iterators
+
+- __Simplicity__: Eliminates the complex __offset__ and __limit__ settings.
+
+- __Efficiency__: Provides scalable data retrieval by fetching only the data in need.
+
+- __Consistency__: Ensures a consistent dataset size with boolean filters.
+
+notes
+ +[1] Number of cluster units after indexing. When indexing a collection, Milvus sub-divides the vector data into multiple cluster units, the number of which varies with the actual index settings.
+[2] Number of entities to return in a search.
+ +
+
+
+
+## Preparations
+
+The following steps repurpose the code to connect to Milvus, quickly set up a collection, and insert over 10,000 randomly generated entities into the collection.
+
+### Step 1: Create a collection
+
+notes
+ +-
+
+
- This feature is available for Milvus 2.3.x or later. + +
+
+Use [`MilvusClient`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) to connect to the Milvus server and [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md) to create a collection.
+
+
+
+
+
+Use [`MilvusClientV2`](https://milvus.io/api-reference/java/v2.4.x/v2/Client/MilvusClientV2.md) to connect to the Milvus server and [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md) to create a collection.
+
+
+
+
+
+```python
+from pymilvus import MilvusClient
+
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri="http://localhost:19530"
+)
+
+# 2. Create a collection
+client.create_collection(
+ collection_name="quick_setup",
+ dimension=5,
+)
+```
+
+```java
+import io.milvus.client.MilvusServiceClient;
+import io.milvus.param.ConnectParam;
+import io.milvus.param.highlevel.collection.CreateSimpleCollectionParam;
+
+String CLUSTER_ENDPOINT = "http://localhost:19530";
+
+// 1. Connect to Milvus server
+ConnectParam connectParam = ConnectParam.newBuilder()
+ .withUri(CLUSTER_ENDPOINT)
+ .build();
+
+MilvusServiceClient client = new MilvusServiceClient(connectParam);
+
+// 2. Create a collection
+CreateSimpleCollectionParam createCollectionParam = CreateSimpleCollectionParam.newBuilder()
+ .withCollectionName("quick_setup")
+ .withDimension(5)
+ .build();
+
+client.createCollection(createCollectionParam);
+```
+
+### Step 2: Insert randomly generated entities
+
+
+
+Use [`insert()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/insert.md) to insert entities into the collection.
+
+
+
+
+
+Use [`insert()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/insert.md) to insert entities into the collection.
+
+
+
+
+
+```python
+# 3. Insert randomly generated vectors
+colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
+data = []
+
+for i in range(10000):
+ current_color = random.choice(colors)
+ current_tag = random.randint(1000, 9999)
+ data.append({
+ "id": i,
+ "vector": [ random.uniform(-1, 1) for _ in range(5) ],
+ "color": current_color,
+ "tag": current_tag,
+ "color_tag": f"{current_color}_{str(current_tag)}"
+ })
+
+print(data[0])
+
+# Output
+#
+# {
+# "id": 0,
+# "vector": [
+# -0.5705990742218152,
+# 0.39844925120642083,
+# -0.8791287928610869,
+# 0.024163154953680932,
+# 0.6837669917169638
+# ],
+# "color": "purple",
+# "tag": 7774,
+# "color_tag": "purple_7774"
+# }
+
+res = client.insert(
+ collection_name="quick_setup",
+ data=data,
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 10000,
+# "ids": [
+# 0,
+# 1,
+# 2,
+# 3,
+# 4,
+# 5,
+# 6,
+# 7,
+# 8,
+# 9,
+# "(9990 more items hidden)"
+# ]
+# }
+```
+
+```java
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+
+import com.alibaba.fastjson.JSONObject;
+
+import io.milvus.param.R;
+import io.milvus.param.dml.InsertParam;
+import io.milvus.response.MutationResultWrapper;
+import io.milvus.grpc.MutationResult;
+
+
+// 3. Insert randomly generated vectors into the collection
+List
+
+To search with an iterator, call the [search_iterator()](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Collection/search_iterator.md) method:
+
+
+
+
+
+To search with an iterator, call the [searchIterator()](https://milvus.io/api-reference/java/v2.4.x/v1/QuerySearch/searchIterator.md) method:
+
+
+
+1. Initialize the search iterator to define the search parameters and output fields.
+
+1. Use the __next()__ method within a loop to paginate through the search results.
+
+ - If the method returns an empty array, the loop ends, and no more pages are available.
+
+ - All results carry the specified output fields.
+
+1. Manually call the __close()__ method to close the iterator once all data has been retrieved.
+
+
+
+```python
+from pymilvus import Collection
+
+# 4. Search with iterator
+connections.connect(host="127.0.0.1", port=19530)
+collection = Collection("quick_setup")
+
+query_vectors = [[0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]]
+search_params = {
+ "metric_type": "IP",
+ "params": {"nprobe": 10}
+}
+
+iterator = collection.search_iterator(
+ data=query_vectors,
+ anns_field="vector",
+ batch_size=10,
+ param=search_params,
+ output_fields=["color_tag"],
+ limit=3
+)
+
+results = []
+
+while True:
+ result = iterator.next()
+ if not result:
+ iterator.close()
+ break
+
+ results.extend(result)
+
+ for hit in result:
+ results.append(hit.to_dict())
+
+print(results)
+
+# Output
+#
+# [
+# {
+# "id": 1756,
+# "distance": 2.0642056465148926,
+# "entity": {
+# "color_tag": "black_9109"
+# }
+# },
+# {
+# "id": 6488,
+# "distance": 1.9437453746795654,
+# "entity": {
+# "color_tag": "purple_8164"
+# }
+# },
+# {
+# "id": 3338,
+# "distance": 1.9107104539871216,
+# "entity": {
+# "color_tag": "brown_8121"
+# }
+# }
+# ]
+```
+
+```java
+import io.milvus.param.dml.QueryIteratorParam;
+import io.milvus.param.dml.SearchIteratorParam;
+import io.milvus.response.QueryResultsWrapper;
+import io.milvus.orm.iterator.SearchIterator;
+
+// 4. Search with iterators
+SearchIteratorParam iteratorParam = SearchIteratorParam.newBuilder()
+ .withCollectionName("quick_setup")
+ .withVectorFieldName("vector")
+ // Use withFloatVectors() in clusters compatible with Milvus 2.4.x
+ .withVectors(Arrays.asList(0.3580376395471989f, -0.6023495712049978f, 0.18414012509913835f, -0.26286205330961354f, 0.9029438446296592f))
+ .withBatchSize(10L)
+ .withParams("{\"metric_type\": \"COSINE\", \"params\": {\"level\": 1}}")
+ .build();
+
+
+R| Parameter | +Description | +
|---|---|
data |
+ A list of vector embeddings. Milvus searches for the most similar vector embeddings to the specified ones. |
+
anns_field |
+ The name of the vector field in the current collection. | +
batch_size |
+ The number of entities to return each time you call next() on the current iterator.The value defaults to 1000. Set it to a proper value to control the number of entities to return per iteration. |
+
param |
+ The parameter settings specific to this operation.
|
+
output_fields |
+ A list of field names to include in each entity in return. The value defaults to None. If left unspecified, only the primary field is included. |
+
limit |
+ The total number of entities to return. The value defaults to -1, indicating all matching entities will be in return. |
+
| Parameter | +Description | +
|---|---|
withCollectionName |
+ Set the collection name. Collection name cannot be empty or null. | +
withVectorFieldName |
+ Set target vector field by name. Field name cannot be empty or null. | +
withVectors |
+ Set the target vectors. Up to 16384 vectors allowed. | +
withBatchSize |
+ The number of entities to return each time you call next() on the current iterator.The value defaults to 1000. Set it to a proper value to control the number of entities to return per iteration. |
+
withParams |
+ Specifies the parameters of search in JSON format. For more information, refer to searchIterator(). | +
+
+To query with an iterator, call the [query_iterator()](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/Collection/query_iterator.md) method:
+
+
+
+
+
+To search with an iterator, call the [queryIterator()](https://milvus.io/api-reference/java/v2.4.x/v1/QuerySearch/queryIterator.md) method:
+
+
+
+
+
+```python
+# 6. Query with iterator
+iterator = collection.query_iterator(
+ batch_size=10, # Controls the size of the return each time you call next()
+ expr="color_tag like \"brown_8\"",
+ output_fields=["color_tag"]
+)
+
+results = []
+
+while True:
+ result = iterator.next()
+ if not result:
+ iterator.close()
+ break
+
+ results.extend(result)
+
+# 8. Check the search results
+print(len(results))
+
+print(results[:3])
+
+# Output
+#
+# [
+# {
+# "color_tag": "brown_8785",
+# "id": 94
+# },
+# {
+# "color_tag": "brown_8568",
+# "id": 176
+# },
+# {
+# "color_tag": "brown_8721",
+# "id": 289
+# }
+# ]
+```
+
+```java
+import io.milvus.param.dml.QueryIteratorParam;
+import io.milvus.orm.iterator.QueryIterator;
+
+// 5. Query with iterators
+
+try {
+ Files.write(Path.of("results.json"), JSON.toJSONString(new ArrayList<>()).getBytes(), StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING);
+} catch (Exception e) {
+ // TODO: handle exception
+ e.printStackTrace();
+}
+
+QueryIteratorParam queryIteratorParam = QueryIteratorParam.newBuilder()
+ .withCollectionName("quick_setup")
+ .withExpr("color_tag like \"brown_8%\"")
+ .withBatchSize(50L)
+ .addOutField("vector")
+ .addOutField("color_tag")
+ .build();
+
+R| Parameter | +Description | +
|---|---|
batch_size |
+ The number of entities to return each time you call next() on the current iterator.The value defaults to 1000. Set it to a proper value to control the number of entities to return per iteration. |
+
expr |
+ A scalar filtering condition to filter matching entities. The value defaults to None, indicating that scalar filtering is ignored. To build a scalar filtering condition, refer to Boolean Expression Rules. |
+
output_fields |
+ A list of field names to include in each entity in return. The value defaults to None. If left unspecified, only the primary field is included. |
+
limit |
+ The total number of entities to return. The value defaults to -1, indicating all matching entities will be in return. |
+
| Parameter | +Description | +
|---|---|
withCollectionName |
+ Set the collection name. Collection name cannot be empty or null. | +
withExpr |
+ Set the expression to query entities. To build a scalar filtering condition, refer to Boolean Expression Rules. | +
withBatchSize |
+ The number of entities to return each time you call next() on the current iterator.The value defaults to 1000. Set it to a proper value to control the number of entities to return per iteration. |
+
addOutField |
+ Specifies an output scalar field (Optional). | +
-Output:
-
-```json
-{
- "status":{},
- "results":{
- "num_queries":1,
- "top_k":2,
- "fields_data":[
- {
- "type":5,
- "field_name":"book_id",
- "Field":{"Scalars":{"Data":{"LongData":{"data":[11,12]}}}},
- "field_id":100
- }
- ],
- "scores":[119.44999,142.24998],
- "ids":{"IdField":{"IntId":{"data":[11,12]}}},"topks":[2]
- },
- "collection_name":"book"
-}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
data |
- Vectors to search with. | -
anns_field |
- Name of the field to search on. | -
params |
- Search parameter(s) specific to the index. See Vector Index for more information. | -
offset |
- Number of results to skip in the returned set. The sum of this value and `limit` should be less than 65535. | -
limit |
- Number of the most similar results to return. | -
expr |
- Boolean expression used to filter attribute. See Boolean Expression Rules for more information. | -
partition_names (optional) |
- List of names of the partition to search in. The sum of this value and `offset` should be less than 65535. | -
output_fields (optional) |
- Name of the field to return. Vector field is not supported in current release. | -
timeout (optional) |
- A duration of time in seconds to allow for RPC. Clients wait until server responds or error occurs when it is set to None. | -
round_decimal (optional) |
- Number of decimal places of returned distance. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to search in. | -
search_params |
- Parameters (as an object) used for search. | -
vectors |
- Vectors to search with. | -
vector_type |
- Pre-check of binary or float vectors. 100 for binary vectors and 101 for float vectors. |
-
partition_names (optional) |
- List of names of the partition to search in. | -
expr (optional) |
- Boolean expression used to filter attribute. See Boolean Expression Rules for more information. | -
output_fields (optional) |
- Name of the field to return. Vector field not support in current release. | -
| Parameter | -Description | -Options | -
|---|---|---|
NewIndex*SearchParam func |
- Function to create entity.SearchParam according to different index types. | -For floating point vectors:
-
|
-
searchParam |
- Search parameter(s) specific to the index. | -See Vector Index for more information. | -
ctx |
- Context to control API invocation process. | -N/A | -
CollectionName |
- Name of the collection to load. | -N/A | -
partitionNames |
- List of names of the partitions to load. All partitions will be searched if it is left empty. | -N/A | -
expr |
- Boolean expression used to filter attribute. | -See Boolean Expression Rules for more information. | -
output_fields |
- Name of the field to return. | -Vector field is not supported in current release. | -
vectors |
- Vectors to search with. | -N/A | -
vectorField |
- Name of the field to search on. | -N/A | -
metricType |
- Metric type used for search. | -This parameter must be set identical to the metric type used for index building. | -
topK |
- Number of the most similar results to return. | -N/A | -
sp |
- entity.SearchParam specific to the index. | -N/A | -
| Parameter | -Description | -Options | -
|---|---|---|
CollectionName |
- Name of the collection to load. | -N/A | -
MetricType |
- Metric type used for search. | -This parameter must be set identical to the metric type used for index building. | -
OutFields |
- Name of the field to return. | -Vector field is not supported in current release. | -
TopK |
- Number of the most similar results to return. | -N/A | -
Vectors |
- Vectors to search with. | -N/A | -
VectorFieldName |
- Name of the field to search on. | -N/A | -
Expr |
- Boolean expression used to filter attribute. | -See Boolean Expression Rules for more information. | -
Params |
- Search parameter(s) specific to the index. | -See Vector Index for more information. | -
| Option | -Full name | -Description | -
|---|---|---|
| --help | -n/a | -Displays help for using the command. | -
| Parameter | -Description | -
|---|---|
output_fields(optional) |
- Name of the field to return. Vector field is not supported in current release. | -
anns_field |
- Name of the field to search on. | -
topk |
- Number of the most similar results to return. | -
params |
- Search parameter(s) specific to the index. See Vector Index for more information. | -
metric_type |
- Metric type used for search. This parameter must be set identical to the metric type used for index building. | -
round_decimal (optional) |
- Number of decimal places of returned distance. | -
Vectors |
- Vectors to search with. | -
dsl |
- Boolean expression used to filter attribute. Find more expression details in Boolean Expression Rules. | -
dsl_type |
- Type of dsl (Data Search Language) field:
- 0: "Dsl" - 1: "BoolExprV1" - |
-
-Output:
-
-```json
-{
- "status":{},
- "fields_data":[
- {
- "type":5,
- "field_name":"book_id",
- "Field":{"Scalars":{"Data":{"LongData":{"data":[6,8,2,4]}}}},
- "field_id":100
- },
- {
- "type":101,
- "field_name":"book_intro",
- "Field":{"Vectors":{"dim":2,"Data":{"FloatVector":{"data":[6,1,8,1,2,1,4,1]}}}},
- "field_id":102
- }
- ]
-}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
expr |
- Boolean expression used to filter attribute. Find more expression details in Boolean Expression Rules. | -
offset |
- Number of results to skip in the returned set. The sum of this value and `limit` should be less than 65535. | -
limit |
- Number of the most similar results to return. The sum of this value and `offset` should be less than 65535. | -
output_fields (optional) |
- List of names of the field to return. | -
partition_names (optional) |
- List of names of the partitions to query on. | -
consistency_level (optional) |
- Consistency level of the query. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to query. | -
expr |
- Boolean expression used to filter attribute. Find more expression details in Boolean Expression Rules. | -
output_fields (optional) |
- List of names of the field to return. | -
partition_names (optional) |
- List of names of the partitions to query on. | -
| Parameter | -Description | -Options | -
|---|---|---|
ctx |
- Context to control API invocation process. | -N/A | -
CollectionName |
- Name of the collection to query. | -N/A | -
partitionName |
- List of names of the partitions to load. All partitions will be queried if it is left empty. | -N/A | -
expr |
- Boolean expression used to filter attribute. | -See Boolean Expression Rules for more information. | -
OutputFields |
- Name of the field to return. | -Vector field is not supported in current release. | -
| Parameter | -Description | -Options | -
|---|---|---|
CollectionName |
- Name of the collection to load. | -N/A | -
OutFields |
- Name of the field to return. | -Vector field is not supported in current release. | -
Expr |
- Boolean expression used to filter attribute. | -See Boolean Expression Rules for more information. | -
ConsistencyLevel |
- The consistency level used in the query. | -STRONG, BOUNDED, andEVENTUALLY. |
-
| Option | -Full name | -Description | -
|---|---|---|
| --help | -n/a | -Displays help for using the command. | -
| Parameter | -Description | -
|---|---|
output_fields (optional) |
- List of names of the fields to return. | -
vectors |
- Vectors to query. | -
expr |
- Boolean expression used to filter attribute. Find more expression details in Boolean Expression Rules. | -
-Output:
-
-```json
-{
- "status":{},
- "results":{
- "num_queries":1,
- "top_k":2,
- "fields_data":[
- {
- "type":5,
- "field_name":"book_id",
- "Field":{"Scalars":{"Data":{"LongData":{"data":[1,2]}}}},
- "field_id":100
- }
- ],
- "scores":[1.45,4.25],
- "ids":{"IdField":{"IntId":{"data":[1,2]}}},
- "topks":[2]},
- "collection_name":"book"
-}
-```
-
-
-
-| Parameter | -Description | -
|---|---|
metric_type |
- Metrics used to measure similarity of vectors. See Simlarity Metrics for more information. | -
params |
- Search parameter(s) specific to the index. See Vector Index for more information. | -
| Parameter | -Description | -
|---|---|
anns_field |
- Name of the field to search on. | -
topk |
- Number of the most similar results to return. | -
metric_type |
- Metrics used to measure similarity of vectors. See Simlarity Metrics for more information. | -
params |
- Search parameter(s) specific to the index. See Vector Index for more information. | -
| Parameter | -Description | -Options | -
|---|---|---|
NewIndex*SearchParam func |
- Function to create entity.SearchParam according to different index types. | -For floating point vectors:
-
|
-
searchParam |
- Search parameter(s) specific to the index. | -See Vector Index for more information. | -
| Parameter | -Description | -Options | -
|---|---|---|
TopK |
- Number of the most similar results to return. | -N/A | -
Params |
- Search parameter(s) specific to the index. | -See Vector Index for more information. | -
| Option | -Full name | -Description | -
|---|---|---|
| --help | -n/a | -Displays help for using the command. | -
| Parameter | -Description | -
|---|---|
output_fields(optional) |
- Name of the field to return. Vector field is not supported in current release. | -
anns_field |
- Name of the field to search on. | -
topk |
- Number of the most similar results to return. | -
params |
- Search parameter(s) specific to the index. See Vector Index for more information. | -
metric_type |
- Metrics used to measure similarity of vectors. See Simlarity Metrics for more information. | -
round_decimal (optional) |
- Number of decimal places of returned distance. | -
Vectors |
- Vectors to search with. | -
dsl_type |
- Type of dsl (Data Search Language) field:
- 0: "Dsl" - 1: "BoolExprV1" - |
-
| Parameter | -Description | -
|---|---|
data |
- Vectors to search with. | -
anns_field |
- Name of the field to search on. | -
param |
- Search parameter(s) specific to the index. See Vector Index for more information. | -
offset |
- Number of results to skip in the returned set. The sum of this value and `limit` should be less than 16384. | -
limit |
- Number of the most similar results to return. The sum of this value and `offset` should be less than 16384. | -
expr |
- Boolean expression used to filter attribute. See Boolean Expression Rules for more information. | -
partition_names (optional) |
- List of names of the partition to search in. | -
output_fields (optional) |
- Name of the field to return. Vector field is not supported in current release. | -
timeout (optional) |
- A duration of time in seconds to allow for RPC. Clients wait until server responds or error occurs when it is set to None. | -
round_decimal (optional) |
- Number of decimal places of returned distance. | -
consistency_level (optional) |
- Consistency level of the search. | -
| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to search in. | -
search_params |
- Parameters (as an object) used for search. | -
vectors |
- Vectors to search with. | -
vector_type |
- Pre-check of binary or float vectors. 100 for binary vectors and 101 for float vectors. |
-
partition_names (optional) |
- List of names of the partition to search in. | -
expr (optional) |
- Boolean expression used to filter attribute. See Boolean Expression Rules for more information. | -
output_fields (optional) |
- Name of the field to return. Vector field is not supported in current release. | -
| Parameter | -Description | -Options | -
|---|---|---|
ctx |
- Context to control API invocation process. | -N/A | -
CollectionName |
- Name of the collection to load. | -N/A | -
partitionNames |
- List of names of the partitions to load. All partitions will be searched if it is left empty. | -N/A | -
expr |
- Boolean expression used to filter attribute. | -See Boolean Expression Rules for more information. | -
output_fields |
- Name of the field to return. | -Vector field is not supported in current release. | -
vectors |
- Vectors to search with. | -N/A | -
vectorField |
- Name of the field to search on. | -N/A | -
metricType |
- Metric type used for search. | -This parameter must be set identical to the metric type used for index building. | -
topK |
- Number of the most similar results to return. | -N/A | -
sp |
- entity.SearchParam specific to the index. | -N/A | -
| Parameter | -Description | -Options | -
|---|---|---|
CollectionName |
- Name of the collection to load. | -N/A | -
MetricType |
- Metric type used for search. | -This parameter must be set identical to the metric type used for index building. | -
OutFields |
- Name of the field to return. | -Vector field is not supported in current release. | -
Vectors |
- Vectors to search with. | -N/A | -
VectorFieldName |
- Name of the field to search on. | -N/A | -
Expr |
- Boolean expression used to filter attribute. | -See Boolean Expression Rules for more information. | -
ConsistencyLevel |
- The consistency level used in the query. | -STRONG, BOUNDED, andEVENTUALLY. |
-
-By default, Milvus allows Time Travel span of 432,000 seconds (120h0m0s). You can configure this parameter in
-
-## Preparations
-
-The following example code demonstrates the steps prior to inserting data.
-
-If you work with your own dataset in an existing Milvus instance, you can move forward to the next step.
-
-
-
-
-```python
-from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType
-connections.connect("default", host='localhost', port='19530')
-collection_name = "test_time_travel"
-schema = CollectionSchema([
- FieldSchema("pk", DataType.INT64, is_primary=True),
- FieldSchema("example_field", dtype=DataType.FLOAT_VECTOR, dim=2)
-])
-collection = Collection(collection_name, schema)
-```
-
-```javascript
-const { MilvusClient } =require("@zilliz/milvus2-sdk-node");
-const milvusClient = new MilvusClient("localhost:19530");
-const params = {
- collection_name: "test_time_travel",
- fields: [{
- name: "example_field",
- description: "",
- data_type: 101, // DataType.FloatVector
- type_params: {
- dim: "2",
- },
- },
- {
- name: "pk",
- data_type: 5, //DataType.Int64
- is_primary_key: true,
- description: "",
- },
- ],
-};
-await milvusClient.collectionManager.createCollection(params);
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-connect -h localhost -p 19530 -a default
-create collection -c test_time_travel -f pk:INT64:primary_field -f example_field:FLOAT_VECTOR:2 -p pk
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/collection' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "test_time_travel",
- "consistency_level": 1,
- "schema": {
- "name": "test_time_travel",
- "autoID": false,
- "fields": [
- {
- "name": "pk",
- "is_primary_key": true,
- "data_type": 5
- },
- {
- "name": "example_field",
- "data_type": 101,
- "type_params": [
- {
- "key": "dim",
- "value": "2"
- }
- ]
- }
- ]
- }
- }'
-```
-
-## Insert the first batch of data
-
-Insert random data to simulate the original data (Milvus CLI example uses a pre-built, remote CSV file containing similar data).
-
-
-
-
-```python
-import random
-data = [
- [i for i in range(10)],
- [[random.random() for _ in range(2)] for _ in range(10)],
-]
-batch1 = collection.insert(data)
-```
-
-```javascript
-const entities1 = Array.from({ length: 10 }, (v, k) => ({
- "example_field": Array.from({ length: 2 }, () => Math.random()),
- "pk": k,
-}));
-const batch1 = milvusClient.dataManager.insert({
- collection_name: "test_time_travel",
- fields_data: entities1,
-});
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-import -c test_time_travel https://raw.githubusercontent.com/zilliztech/milvus_cli/main/examples/user_guide/search_with_timetravel_1.csv
-Reading file from remote URL.
-Reading csv rows... [####################################] 100%
-Column names are ['pk', 'example_field']
-Processed 11 lines.
-
-Inserted successfully.
-
--------------------------- ------------------
-Total insert entities: 10
-Total collection entities: 10
-Milvus timestamp: 430390410783752199
--------------------------- ------------------
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/entities' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "test_time_travel",
- "fields_data": [
- {
- "field_name": "pk",
- "type": 5,
- "field": [
- 0,1,2,3,4,5,6,7,8,9
- ]
- },
- {
- "field_name": "example_field",
- "type": 101,
- "field": [
- [1,1],[2,1],[3,1],[4,1],[5,1],[6,1],[7,1],[8,1],[9,1],[10,1]
- ]
- }
- ],
- "num_rows": 10
-}'
-```
-
-## Check the timestamp of the first data batch
-
-Check the timestamp of the first data batch for search with Time Travel. Data inserted within the same batch share an identical timestamp.
-
-```python
-batch1.timestamp
-428828271234252802
-```
-
-```javascript
-batch1.timestamp
-428828271234252802
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-# Milvus CLI automatically returns the timestamp as shown in the previous step.
-```
-
-```curl
-# Output:
-{
- "status":{},
- "IDs":{"IdField":{"IntId":{"data":[1,2,3,4,5,6,7,8,9,10]}}},
- "succ_index":[0,1,2,3,4,5,6,7,8,9],
- "insert_cnt":10,
- "timestamp":434575831766925313
-```
-
-common.retentionDuration.
-
- Milvus adopts a combination of physical clock and logic counter as a hybrid timestamp. The 64-bit timestamp consists of a 46-bit physical part (high-order bits) and an 18-bit logic part (low-order bits). The physical part is the number of milliseconds that have elapsed since January 1, 1970 (midnight UTC/GMT).
-
-
-## Insert the second batch of data
-
-Insert the second batch of data to simulate the dirty data, among which a piece of data with primary key value `19` and vector value `[1.0,1.0]` is appended as the target data to search with in the following step (Milvus CLI example uses a pre-built, remote CSV file containing similar data).
-
-
-
-
-```python
-data = [
- [i for i in range(10, 20)],
- [[random.random() for _ in range(2)] for _ in range(9)],
-]
-data[1].append([1.0,1.0])
-batch2 = collection.insert(data)
-```
-
-```javascript
-const entities2 = Array.from({
- length: 9
-}, (v, k) => ({
- "example_field": Array.from({
- length: 2
- }, () => Math.random()),
- "pk": k + 10,
-}));
-entities2.push({
- "pk": 19,
- "example_field": [1.0, 1.0],
-});
-const batch2 = await milvusClient.dataManager.insert({
- collection_name: "test_time_travel",
- fields_data: entities2,
-});
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-import -c test_time_travel https://raw.githubusercontent.com/zilliztech/milvus_cli/main/examples/user_guide/search_with_timetravel_2.csv
-Reading file from remote URL.
-Reading csv rows... [####################################] 100%
-Column names are ['pk', 'example_field']
-Processed 11 lines.
-
-Inserted successfully.
-
--------------------------- ------------------
-Total insert entities: 10
-Total collection entities: 20
-Milvus timestamp: 430390435713122310
--------------------------- ------------------
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/entities' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "test_time_travel",
- "fields_data": [
- {
- "field_name": "pk",
- "type": 5,
- "field": [
- 10,11,12,13,14,15,16,17,18,19
- ]
- },
- {
- "field_name": "example_field",
- "type": 101,
- "field": [
- [11,12],[12,12],[13,12],[14,12],[15,12],[16,12],[17,12],[18,12],[19,12],[1,1]
- ]
- }
- ],
- "num_rows": 10
- }'
-```
-
-
-Output:
-
-```json
-{
- "status":{},
- "IDs":{"IdField":{"IntId":{"data":[10,11,12,13,14,15,16,17,18,19]}}},
- "succ_index":[0,1,2,3,4,5,6,7,8,9],
- "insert_cnt":10,
- "timestamp":434575834238943233
-}
-```
-
-
-
-## Search with a specified timestamp
-
-Load the collection and search the target data with the timestamp of the first data batch. With the timestamp specified, Milvus only retrieves the data view at the point of time the timestamp indicates.
-
-
-
-
-```python
-collection.load()
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "example_field",
- "param": {"metric_type": "L2"},
- "limit": 10,
- "travel_timestamp": batch1.timestamp,
-}
-res = collection.search(**search_param)
-res[0].ids
-```
-
-```javascript
-await milvusClient.collectionManager.loadCollection({
- collection_name: "test_time_travel",
-});
-const res = await milvusClient.dataManager.search({
- collection_name: "test_time_travel",
- vectors: [
- [1.0, 1.0]
- ],
- travel_timestamp: batch1.timestamp,
- search_params: {
- anns_field: "example_field",
- topk: "10",
- metric_type: "L2",
- params: JSON.stringify({
- nprobe: 10
- }),
- },
- vector_type: 101, // DataType.FloatVector,
-});
-console.log(res1.results)
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-search
-Collection name (test_collection_query, test_time_travel): test_time_travel
-The vectors of search data (the length of data is number of query (nq), the dim of every vector in data must be equal to vector field’s of collection. You can also import a CSV file without headers): [[1.0, 1.0]]
-The vector field used to search of collection (example_field): example_field
-The specified number of decimal places of returned distance [-1]:
-The max number of returned record, also known as topk: 10
-The boolean expression used to filter attribute []:
-The names of partitions to search (split by "," if multiple) ['_default'] []:
-Timeout []:
-Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]: 430390410783752199
-```
-
-```curl
-# Load the collection:
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/collection/load' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "test_time_travel"
- }'
-
-# Conduct a vector search:
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/search' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "test_time_travel",
- "output_fields": ["pk"],
- "search_params": [
- {"key": "anns_field", "value": "example_field"},
- {"key": "topk", "value": "10"},
- {"key": "params", "value": "{\"nprobe\": 10}"},
- {"key": "metric_type", "value": "L2"}
- ],
- "travel_timestamp": 434575831766925313,
- "vectors": [ [10,10] ],
- "dsl_type": 1
- }'
-```
-
-As shown below, the target data itself and other data inserted later are not returned as results.
-
-```python
-[8, 7, 4, 2, 5, 6, 9, 3, 0, 1]
-```
-
-```javascript
-[8, 7, 4, 2, 5, 6, 9, 3, 0, 1]
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-Search results:
-
-No.1:
-+---------+------+------------+-----------+
-| Index | ID | Distance | Score |
-+=========+======+============+===========+
-| 0 | 2 | 0.0563737 | 0.0563737 |
-+---------+------+------------+-----------+
-| 1 | 5 | 0.122474 | 0.122474 |
-+---------+------+------------+-----------+
-| 2 | 3 | 0.141737 | 0.141737 |
-+---------+------+------------+-----------+
-| 3 | 8 | 0.331008 | 0.331008 |
-+---------+------+------------+-----------+
-| 4 | 0 | 0.618705 | 0.618705 |
-+---------+------+------------+-----------+
-| 5 | 1 | 0.676788 | 0.676788 |
-+---------+------+------------+-----------+
-| 6 | 9 | 0.69871 | 0.69871 |
-+---------+------+------------+-----------+
-| 7 | 6 | 0.706456 | 0.706456 |
-+---------+------+------------+-----------+
-| 8 | 4 | 0.956929 | 0.956929 |
-+---------+------+------------+-----------+
-| 9 | 7 | 1.19445 | 1.19445 |
-+---------+------+------------+-----------+
-```
-
-
-Output:
-
-```json
-{
- "status":{},
- "results":{
- "num_queries":1,
- "top_k":10,
- "fields_data":[
- {
- "type":5,
- "field_name":"pk",
- "Field":{"Scalars":{"Data":{"LongData":{"data":[9,8,7,6,5,4,3,2,1,0]}}}},
- "field_id":100
- }
- ],
- "scores":[81,82,85,90,97,106,117,130,145,162],
- "ids":{"IdField":{"IntId":{"data":[9,8,7,6,5,4,3,2,1,0]}}},
- "topks":[10]
- },
- "collection_name":"test_time_travel"
-}
-```
-
-
-
-If you do not specify the timestamp or specify it with the timestamp of the second data batch, Milvus will return the results from both batches.
-
-
-
-
-```python
-batch2.timestamp
-428828283406123011
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "example_field",
- "param": {"metric_type": "L2"},
- "limit": 10,
- "travel_timestamp": batch2.timestamp,
-}
-res = collection.search(**search_param)
-res[0].ids
-[19, 10, 8, 7, 4, 17, 2, 5, 13, 15]
-```
-
-```javascript
-batch2.timestamp
-428828283406123011
-const res2 = await milvusClient.dataManager.search({
- collection_name: "test_time_travel",
- vectors: [
- [1.0, 1.0]
- ],
- travel_timestamp: batch2.timestamp,
- search_params: {
- anns_field: "example_field",
- topk: "10",
- metric_type: "L2",
- params: JSON.stringify({
- nprobe: 10
- }),
- },
- vector_type: 101, // DataType.FloatVector,
-});
-console.log(res2.results)
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-search
-Collection name (test_collection_query, test_time_travel): test_time_travel
-The vectors of search data (the length of data is number of query (nq), the dim of every vector in data must be equal to vector field’s of collection. You can also import a CSV file without headers): [[1.0, 1.0]]
-The vector field used to search of collection (example_field): example_field
-The specified number of decimal places of returned distance [-1]:
-The max number of returned record, also known as topk: 10
-The boolean expression used to filter attribute []:
-The names of partitions to search (split by "," if multiple) ['_default'] []:
-Timeout []:
-Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:
-Search results:
-
-No.1:
-+---------+------+------------+------------+
-| Index | ID | Distance | Score |
-+=========+======+============+============+
-| 0 | 19 | 0 | 0 |
-+---------+------+------------+------------+
-| 1 | 12 | 0.00321393 | 0.00321393 |
-+---------+------+------------+------------+
-| 2 | 2 | 0.0563737 | 0.0563737 |
-+---------+------+------------+------------+
-| 3 | 5 | 0.122474 | 0.122474 |
-+---------+------+------------+------------+
-| 4 | 3 | 0.141737 | 0.141737 |
-+---------+------+------------+------------+
-| 5 | 10 | 0.238646 | 0.238646 |
-+---------+------+------------+------------+
-| 6 | 8 | 0.331008 | 0.331008 |
-+---------+------+------------+------------+
-| 7 | 18 | 0.403166 | 0.403166 |
-+---------+------+------------+------------+
-| 8 | 13 | 0.508617 | 0.508617 |
-+---------+------+------------+------------+
-| 9 | 11 | 0.531529 | 0.531529 |
-+---------+------+------------+------------+
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/search' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "test_time_travel",
- "output_fields": ["pk"],
- "search_params": [
- {"key": "anns_field", "value": "example_field"},
- {"key": "topk", "value": "10"},
- {"key": "params", "value": "{\"nprobe\": 10}"},
- {"key": "metric_type", "value": "L2"}
- ],
- "vectors": [ [11,11] ],
- "dsl_type": 1
- }'
-```
-
-
-Output:
-
-```json
-{
- "status":{},
- "results":{
- "num_queries":1,
- "top_k":10,
- "fields_data":[
- {
- "type":5,
- "field_name":"pk",
- "Field":{"Scalars":{"Data":{"LongData":{"data":[10,11,12,13,14,15,16,17,18,9]}}}},
- "field_id":100
- }
- ],
- "scores":[1,2,5,10,17,26,37,50,65,101],
- "ids":{"IdField":{"IntId":{"data":[10,11,12,13,14,15,16,17,18,9]}}},
- "topks":[10]
- },
- "collection_name":"test_time_travel"
-}
-```
-
-
-
-## Generate a timestamp for search
-
-In the case that the previous timestamp is not recorded, Milvus allows you to generate a timestamp using an existing timestamp, Unix Epoch time, or date time.
-
-The following example simulates an unwanted deletion operation and shows how to generate a timestamp prior to the deletion and search with it.
-
-Generate a timestamp based on the date time or Unix Epoch time prior to the deletion.
-
-```python
-import datetime
-datetime = datetime.datetime.now()
-from pymilvus import utility
-pre_del_timestamp = utility.mkts_from_datetime(datetime)
-```
-
-```javascript
-const { datetimeToHybrids } = require("@zilliz/milvus2-sdk-node/milvus/utils/Format");
-const datetime = new Date().getTime()
-const pre_del_timestamp = datetimeToHybrids(datetime)
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-calc mkts_from_unixtime -e 1641809375
-430390476800000000
-```
-
-```curl
-# This function is not supported. It is suggested to use Milvus_CLI.
-```
-
-Delete part of the data to simulate an accidental deletion operation.
-
-```python
-expr = "pk in [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]"
-collection.delete(expr)
-```
-
-```javascript
-const expr = "pk in [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]"
-await milvusClient.dataManager.deleteEntities({
- collection_name: "test_time_travel",
- expr: expr,
-});
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-delete entities -c test_time_travel
-The expression to specify entities to be deleted, such as "film_id in [ 0, 1 ]": pk in [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
-You are trying to delete the entities of collection. This action cannot be undone!
-
-Do you want to continue? [y/N]: y
-(insert count: 0, delete count: 10, upsert count: 0, timestamp: 430390494161534983)
-```
-
-```curl
-curl -X 'DELETE' \
- 'http://localhost:9091/api/v1/entities' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "test_time_travel",
- "expr": "pk in [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]"
- }'
-```
-
-
-Output:
-
-```json
-{
- "status":{},
- "IDs":{"IdField":{"IntId":{"data":[0,2,4,6,8,10,12,14,16,18]}}},
- "delete_cnt":10,
- "timestamp": 434575874068316161
-}
-```
-
-
-
-As shown below, the deleted entities are not returned in the results if you search without specifying the timestamp.
-
-```python
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "example_field",
- "param": {"metric_type": "L2"},
- "limit": 10,
-}
-res = collection.search(**search_param)
-res[0].ids
-```
-
-```javascript
-const res3 = await milvusClient.dataManager.search({
- collection_name: "test_time_travel",
- vectors: [
- [1.0, 1.0]
- ],
- search_params: {
- anns_field: "example_field",
- topk: "10",
- metric_type: "L2",
- params: JSON.stringify({
- nprobe: 10
- }),
- },
- vector_type: 101, // DataType.FloatVector,
-});
-console.log(res3.results)
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-search
-Collection name (test_collection_query, test_time_travel): test_time_travel
-The vectors of search data (the length of data is number of query (nq), the dim of every vector in data must be equal to vector field’s of collection. You can also import a CSV file without headers): [[1.0, 1.0]]
-The vector field used to search of collection (example_field): example_field
-The specified number of decimal places of returned distance [-1]:
-The max number of returned record, also known as topk: 10
-The boolean expression used to filter attribute []:
-The names of partitions to search (split by "," if multiple) ['_default'] []:
-Timeout []:
-Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:
-Search results:
-
-No.1:
-+---------+------+------------+----------+
-| Index | ID | Distance | Score |
-+=========+======+============+==========+
-| 0 | 19 | 0 | 0 |
-+---------+------+------------+----------+
-| 1 | 5 | 0.122474 | 0.122474 |
-+---------+------+------------+----------+
-| 2 | 3 | 0.141737 | 0.141737 |
-+---------+------+------------+----------+
-| 3 | 13 | 0.508617 | 0.508617 |
-+---------+------+------------+----------+
-| 4 | 11 | 0.531529 | 0.531529 |
-+---------+------+------------+----------+
-| 5 | 17 | 0.593702 | 0.593702 |
-+---------+------+------------+----------+
-| 6 | 1 | 0.676788 | 0.676788 |
-+---------+------+------------+----------+
-| 7 | 9 | 0.69871 | 0.69871 |
-+---------+------+------------+----------+
-| 8 | 7 | 1.19445 | 1.19445 |
-+---------+------+------------+----------+
-| 9 | 15 | 1.53964 | 1.53964 |
-+---------+------+------------+----------+
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/search' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "test_time_travel",
- "output_fields": ["pk"],
- "search_params": [
- {"key": "anns_field", "value": "example_field"},
- {"key": "topk", "value": "10"},
- {"key": "params", "value": "{\"nprobe\": 10}"},
- {"key": "metric_type", "value": "L2"}
- ],
- "vectors": [ [11,11] ],
- "dsl_type": 1
- }'
-```
-
-
-Output:
-
-```json
-{
- "status":{},
- "results":{
- "num_queries":1,
- "top_k":10,
- "fields_data":[
- {
- "type":5,
- "field_name":"pk",
- "Field":{"Scalars":{"Data":{"LongData":{"data":[11,13,15,17,9,7,5,3,1,19]}}}},
- "field_id":100
- }
- ],
- "scores":[2,10,26,50,101,109,125,149,181,200],
- "ids":{"IdField":{"IntId":{"data":[11,13,15,17,9,7,5,3,1,19]}}},
- "topks":[10]
- },
- "collection_name":"test_time_travel"
-}
-```
-
-
-
-Search with the prior-to-deletion timestamp. Milvus retrieves entities from the data before the deletion.
-
-```python
-search_param = {
- "data": [[1.0, 1.0]],
- "anns_field": "example_field",
- "param": {"metric_type": "L2"},
- "limit": 10,
- "travel_timestamp": pre_del_timestamp,
-}
-res = collection.search(**search_param)
-res[0].ids
-```
-
-```javascript
-const res4 = await milvusClient.dataManager.search({
- collection_name: "test_time_travel",
- vectors: [
- [1.0, 1.0]
- ],
- travel_timestamp: pre_del_timestamp,
- search_params: {
- anns_field: "example_field",
- topk: "10",
- metric_type: "L2",
- params: JSON.stringify({
- nprobe: 10
- }),
- },
- vector_type: 101, // DataType.FloatVector,
-});
-console.log(res4.results)
-```
-
-```go
-// This function is under active development on the GO client.
-```
-
-```java
-// Java User Guide will be ready soon.
-```
-
-```shell
-search
-Collection name (test_collection_query, test_time_travel): test_time_travel
-The vectors of search data (the length of data is number of query (nq), the dim of every vector in data must be equal to vector field’s of collection. You can also import a CSV file without headers): [[1.0, 1.0]]
-The vector field used to search of collection (example_field): example_field
-The specified number of decimal places of returned distance [-1]:
-The max number of returned record, also known as topk: 10
-The boolean expression used to filter attribute []:
-The names of partitions to search (split by "," if multiple) ['_default'] []:
-Timeout []:
-Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]: 430390476800000000
-Search results:
-
-No.1:
-+---------+------+------------+------------+
-| Index | ID | Distance | Score |
-+=========+======+============+============+
-| 0 | 19 | 0 | 0 |
-+---------+------+------------+------------+
-| 1 | 12 | 0.00321393 | 0.00321393 |
-+---------+------+------------+------------+
-| 2 | 2 | 0.0563737 | 0.0563737 |
-+---------+------+------------+------------+
-| 3 | 5 | 0.122474 | 0.122474 |
-+---------+------+------------+------------+
-| 4 | 3 | 0.141737 | 0.141737 |
-+---------+------+------------+------------+
-| 5 | 10 | 0.238646 | 0.238646 |
-+---------+------+------------+------------+
-| 6 | 8 | 0.331008 | 0.331008 |
-+---------+------+------------+------------+
-| 7 | 18 | 0.403166 | 0.403166 |
-+---------+------+------------+------------+
-| 8 | 13 | 0.508617 | 0.508617 |
-+---------+------+------------+------------+
-| 9 | 11 | 0.531529 | 0.531529 |
-+---------+------+------------+------------+
-```
-
-```curl
-curl -X 'POST' \
- 'http://localhost:9091/api/v1/search' \
- -H 'accept: application/json' \
- -H 'Content-Type: application/json' \
- -d '{
- "collection_name": "test_time_travel",
- "output_fields": ["pk"],
- "search_params": [
- {"key": "anns_field", "value": "example_field"},
- {"key": "topk", "value": "10"},
- {"key": "params", "value": "{\"nprobe\": 10}"},
- {"key": "metric_type", "value": "L2"}
- ],
- "travel_timestamp": 434284782724317186,
- "vectors": [ [10,10] ],
- "dsl_type": 1
- }'
-```
-
-
-Output:
-
-```json
-{
- "status":{},
- "results":{
- "num_queries":1,
- "top_k":10,
- "fields_data":[
- {
- "type":5,
- "field_name":"pk",
- "Field":{"Scalars":{"Data":{"LongData":{"data":[11,12,13,14,15,16,17,18,10,9]}}}},
- "field_id":100}
- ],
- "scores":[5,8,13,20,29,40,53,68,81,82],
- "ids":{"IdField":{"IntId":{"data":[11,12,13,14,15,16,17,18,10,9]}}},
- "topks":[10]
- },
- "collection_name":"test_time_travel"
-}
-```
-
-
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Query vectors](query.md)
- - [Conduct a hybrid search](hybridsearch.md)
-
-- Explore API references for Milvus SDKs:
-
- - [PyMilvus API reference](/api-reference/pymilvus/v2.2.2/About.md)
- - [Node.js API reference](/api-reference/node/v2.2.x/About.md)
- - [Go API reference](/api-reference/go/v2.2.0/About.md)
- - [Java API reference](/api-reference/java/v2.2.1/About.md)
-
-
diff --git a/preview/site/en/userGuide/tools/attu.md b/preview/site/en/userGuide/tools/attu.md
deleted file mode 100644
index 14cad1557..000000000
--- a/preview/site/en/userGuide/tools/attu.md
+++ /dev/null
@@ -1,54 +0,0 @@
----
-id: attu.md
-related_key: attu
-summary: Attu, an intuitive GUI for Milvus service management.
----
-
-# Attu
-
-[Attu](https://github.com/zilliztech/attu) is an efficient open-source management tool for Milvus. It features an intuitive graphical user interface (GUI), allowing you to easily interact with your databases. With just a few clicks, you can visualize your cluster status, manage metadata, perform data queries, and much more.
-
-Attu is an open source project of [Zilliz](https://zilliz.com/).
-
-
-
-## Features
-
-Attu is under rapid development and new features are added on a weekly basis. We will release a new version every time when a new feature is ready.
-
-Below is the features we have to offer:
-
-- Support username, password, and TLS connection
- 
- 
-- View milvus cluster statistics in a glance.
-
-
-
-- Browse, query, and manage collections in a simple and straightforward way.
-
-
-
-- Perform CRUD or bulk operations with just a few clicks.
-
-
-
-- Create vector index instantly.
-
-
-
-- Conduct vector searches in a brand new way.
-
-
-
-- New code-mode provides a better user experience for you.
-
-
-
-Learn more about how to [install Attu](attu_install-docker.md).
-
-## Contribution
-
-Attu is an open-source project. All contributions are welcome. Pleae read our [Contribute guide](https://github.com/zilliztech/attu) before making contributions.
-
-If you find a bug or want to request a new feature, please create a [GitHub Issue](https://github.com/zilliztech/attu), and make sure that the same issue has not been created by someone else.
diff --git a/preview/site/en/userGuide/tools/attu_collection.md b/preview/site/en/userGuide/tools/attu_collection.md
deleted file mode 100644
index bb58f81d0..000000000
--- a/preview/site/en/userGuide/tools/attu_collection.md
+++ /dev/null
@@ -1,79 +0,0 @@
----
-id: attu_collection.md
-related_key: attu
-summary: Learn how to manage collections with Attu, an intuitive GUI tool for Milvus.
----
-
-# Manage Collections with Attu
-
-This topic describes how to manage collections with Attu.
-
-## Create a collection
-
-1. Click the **Collection** icon on the left-side navigation pane and then click **Create Collection**. The **Create Collection** dialog box appears as shown below.
-
-
-
-2. Enter the required information. This example creates a collection named `test` with a primary key field, a vector field, and a scalar field. You can add scalar fields as needed.
-
-
-
-3. Click **Create** to create a collection.
-
-
-
-## Delete a collection
-
-1. Tick the collection you want to delete in the data grid.
-2. Click the **Trash** icon and the **Delete Collection** dialog box appears as shown below.
-3. Type `delete` to confirm the deletion.
-4. Click **Delete** to delete the collection.
-
-
-Deleting a collection is irreversible.
-
-
-
-
-## Load a collection
-
-1. Hover on the collection you want to load, the **Load** icon appears on the right end of the entry.
-
-
-
-2. Click the **Load** icon and the **Load Collection** dialog box appears.
-3. Click **Load** in the **Load Collection** dialog box.
-
-
-
-4. Loading a collection might take a while. If successful, Loaded For Search appears in the Status column.
-
-
-
-## Release a collection
-
-1. Hover on the loaded collection you want to release, the **Release** icon appears on the right end of the entry.
-
-
-
-2. Click the **Release** icon and the **Release Collection** dialog box appears.
-3. Click **Release** in the **Release Collection** dialog box.
-4. If successful, the Status becomes **Unloaded** in the **Status** column.
-
-
-
-## View the schema of a collection
-
-1. Click the name of the collection that you want to view the schema of, and the corresponding detail page appears.
-2. Click **Schema** on the detail page, which lists the information of all fields.
-
-Attributes of a schema include:
-
-- Field Name
-- Field Type
-- Dimension (Only applies to vector fields)
-- Index Type (Only applies to vector fields)
-- Index Parameters (Only applies to vector fields)
-- Collection description
-
-
diff --git a/preview/site/en/userGuide/tools/attu_data.md b/preview/site/en/userGuide/tools/attu_data.md
deleted file mode 100644
index 5bc4898bc..000000000
--- a/preview/site/en/userGuide/tools/attu_data.md
+++ /dev/null
@@ -1,56 +0,0 @@
----
-id: attu_data.md
-related_key: attu
-summary: Learn how to manage data with Attu, an intuitive GUI tool for Milvus.
----
-
-# Manage Data with Attu
-
-This topic describes how to manage data with Attu.
-
-## Import data
-
-This example imports 20,000 rows of data. Importing data appends data instead of overwriting data.
-
-1. Click **Import Data** on the **Collection** page. The **Import Data** dialog box appears as shown below.
-
-
-
-2. Select the collection you want to import data to in the **Collection** dropdown list.
-3. Select the partition you want to import data to in the **Partition** dropdown list.
-4. Click **Choose CSV File** and choose a CSV file.
-
- Ensure that the CSV file meets the following criteria:
-
-
-5. After a legal CSV file is selected, you can then proceed by clicking **Next**.
-
-
-
-6. On the new dialog box, you can match the field names by clicking the corresponding cells in the dropdown lists.
-
--
-
- Column names are the same as the field names specified in the schema; -
- The file is smaller than 150MB and the row count is less than 100,000. -
-We recommend making the headers (column names) as the first row in your CSV file.
-
-
-
-
-7. After confirming the column names corresponding to the field names, click **Import Data** to import the CSV file into Milvus. Importing data might take a while.
-
-
-
-8. If successful, the row count status updates in the Entity Count column of the collection. On the corresponding Partition tab page, the row count status updates in the Entity Count column of the partition your imported data in. It might take a while for the entity count to update.
-
-
-
-## Export Data
-
-1. Click **Data Query** on the **Collection** page. On the **Data Query** tab page, enter query conditions in the field and then click **Query** to retrieve all query results that match your query conditions.
-
-2. Click the **Download** icon to download the query results as a CSV file.
-
-
diff --git a/preview/site/en/userGuide/tools/attu_faq.md b/preview/site/en/userGuide/tools/attu_faq.md
deleted file mode 100644
index fbb9c56ec..000000000
--- a/preview/site/en/userGuide/tools/attu_faq.md
+++ /dev/null
@@ -1,19 +0,0 @@
----
-id: attu_faq.md
-related_key: attu
-summary: Find the most commonly asked questions about Attu, an intuitive GUI tool for Milvus.
----
-
-# Attu FAQ
-
-#### Why is Attu throwing a network error?
-
-A: Check whether you have assigned a correct value to `HOST_URL` in the `docker run` command. Alternatively, you can enter `{HOST_URL}/api/v1/healthy` in the address bar of your browser to check the network status of Attu.
-
-#### Why did Attu fail to connect to Milvus?
-
-A: Ensure that Milvus and Attu are on the same network.
-
-#### How do I use Attu with K8s?
-
-A: You can [install Attu while deploying Milvus with Helm](attu_install-helm.md).
diff --git a/preview/site/en/userGuide/tools/attu_index.md b/preview/site/en/userGuide/tools/attu_index.md
deleted file mode 100644
index c07144cd2..000000000
--- a/preview/site/en/userGuide/tools/attu_index.md
+++ /dev/null
@@ -1,36 +0,0 @@
----
-id: attu_index.md
-related_key: attu
-summary: Learn how to manage indexes with Attu, an intuitive GUI tool for Milvus.
----
-
-# Manage Index with Attu
-
-This topic describes how to manage an index with Attu.
-
-## Create indexes
-
-This example builds an IVF_FLAT index with Euclidean distance as the similarity metrics and an `nlist` value of `1024`.
-
-1. Click **Schema** on the **Collection** page. On the **Schema** tab page, click **CREATE INDEX** and the **Create Index** dialog box appears.
-
-2. In the **Create Index** dialog box, select **IVF_FLAT** from the **Index Type** dropdown list, select **L2** from the **Metric Type** dropdown list, and enter `1024` in the `nlist` field.
-
-3. (Optional) Turn on **View Code** and the **Code View** page appears. You can check the code in Python or Node.js as you want.
-
-4. Click **Create** to create the index.
-
-If successful, the type of the index you created appears in the **Index Type** column for the vector field.
-
-
-
-
-
-## Delete indexes
-
-1. Click the **Trash** icon in the **Index Type** column and the **Delete Index** dialog box appears.
-2. Enter `delete` to confirm the deletion and click **Delete** to delete the indexes.
-
-If successful, **CREATE INDEX** button appears in the Index Type column.
-
-
diff --git a/preview/site/en/userGuide/tools/attu_install-docker.md b/preview/site/en/userGuide/tools/attu_install-docker.md
deleted file mode 100644
index 63eb47456..000000000
--- a/preview/site/en/userGuide/tools/attu_install-docker.md
+++ /dev/null
@@ -1,42 +0,0 @@
----
-id: attu_install-docker.md
-label: Install with Docker Compose
-order: 0
-group: attu_install-docker.md
-related_key: attu
-summary: Learn how to install Attu with Docker Compose to manage your Milvus service.
----
-
-
-
-# Install Attu with Docker Compose
-
-Attu is an efficient open-source management tool for Milvus. This topic describes how to install Attu with Docker Compose, an efficient open-source management tool for Milvus.
-
-## Prerequisites
-
-- Milvus installed on [your local device](install_standalone-docker.md) or [cluster](install_cluster-docker.md).
-- Docker 19.03 or later
-- Milvus 2.1.0 or later
-
-
-See v2.0.x Attu doc if you are using Milvus 2.0.x.
-
-
-## Start an Attu instance
-
-```Apache
-docker run -p 8000:3000 -e MILVUS_URL={your machine IP}:19530 zilliz/attu:latest
-```
-
-Once you start the docker, visit `http://{ your machine IP }:8000` in your browser, and click **Connect** to enter the Attu service.
-And we alsow support TLS connection, username and password.
-
-
-
-
-## Contribution
-
-Attu is an open-source project. All contributions are welcome. Please read our [Contribute guide](https://github.com/zilliztech/attu) before making contributions.
-
-If you find a bug or want to request a new feature, please create a [GitHub Issue](https://github.com/zilliztech/attu), and make sure that the same issue has not been created by someone else.
diff --git a/preview/site/en/userGuide/tools/attu_install-helm.md b/preview/site/en/userGuide/tools/attu_install-helm.md
deleted file mode 100644
index 6c3d42641..000000000
--- a/preview/site/en/userGuide/tools/attu_install-helm.md
+++ /dev/null
@@ -1,69 +0,0 @@
----
-id: attu_install-helm.md
-label: Install with Helm Chart
-order: 1
-group: attu_install-docker.md
-related_key: attu
-summary: Learn how to install Attu with Helm Chart to manage your Milvus service.
----
-
-
-
-# Install Attu with Helm Chart
-
-Attu is an efficient open-source management tool for Milvus. This topic describes how to install Attu with Helm Chart.
-
-## Prerequisites
-
-- Kubernetes 1.16 or later
-- Helm 3.0.0 or later
-
-
-Attu only supports Milvus 2.x.
-
-
-## Install Helm Chart for Milvus
-
-Helm is a Kubernetes package manager that can help you deploy Milvus quickly.
-
-1. Add Milvus Helm repository.
-
-```
-$ helm repo add milvus https://milvus-io.github.io/milvus-helm/
-```
-
-2. Update charts locally.
-
-```
-$ helm repo update
-```
-
-## Install Attu while installing Milvus
-
-Start Milvus and Attu with Helm by specifying the release name, the chart, and the parameters that indicate the installation of Attu. This topic uses `my-release` as the release name. To use a different release name, replace `my-release` in the command.
-
-Kubernetes provides four service modes: ClusterIp, Ingress, LoadBalancer, and NodePort. The default service mode of Attu is ClusterIp. You can choose the service mode that suits your application by configuring the parameters `attu.service.type` and `attu.ingress.enable`.
-
-
-1. Install Milvus and Attu
-
-```
-helm install my-release milvus/milvus --set attu.enabled=true
-```
-
-2. Forward the Attu service to local port `3000`.
-
-```
-kubectl port-forward service/my-release-milvus-attu 3000
-```
-
-3. Visit `http://127.0.0.1:3000/connect` in your browser, and click **Connect** to enter the Attu service.
-
-
-
-
-## Contribution
-
-Attu is an open-source project. All contributions are welcome. Please read our [Contribute guide](https://github.com/zilliztech/attu) before making contributions.
-
-If you find a bug or want to request a new feature, please create a [GitHub Issue](https://github.com/zilliztech/attu), and make sure that the same issue has not been created by someone else.
diff --git a/preview/site/en/userGuide/tools/attu_install-package.md b/preview/site/en/userGuide/tools/attu_install-package.md
deleted file mode 100644
index d72abf640..000000000
--- a/preview/site/en/userGuide/tools/attu_install-package.md
+++ /dev/null
@@ -1,41 +0,0 @@
----
-id: attu_install-package.md
-label: Install with Package
-order: 2
-group: attu_install-docker.md
-related_key: attu
-summary: Learn how to install Attu with package to manage your Milvus service.
----
-
-
-
-# Install Attu with Pre-built Package
-
-Attu is an efficient open-source management tool for Milvus. This topic describes how to install Attu with the pre-built package.
-
-## Download and run installation package
-
-Download and run an installation package for Attu based on your operating system to install Attu on your device.
-
-| Operating System | -Download Link | -
|---|---|
Linux |
- Linux installation package | -
macOS |
- macOS installation package | -
Windows |
- Windows installation package | -
+
+If the image available on DockerHub is not the latest, you can build an image of Birdwatcher using the Dockerfile provided with the source code as follows:
+
+```shell
+git clone https://github.com/milvus-io/birdwatcher.git
+cd birdwatcher
+docker build -t milvusdb/birdwatcher .
+```
+
+To deploy a locally built image, you need to add `imagePullPolicy` to the above specs and set it to `Never`.
+
+```yaml
+...
+ - name: birdwatcher
+ image: milvusdb/birdwatcher
+ imagePullPolicy: Never
+...
+```
+
+
+
+### Apply deployment.yml
+
+Save the above YAML in a file and name it `deployment.yml`, and run the following command
+
+```shell
+kubectl apply -f deployment.yml
+```
diff --git a/preview/site/en/userGuide/tools/birdwatcher_overview.md b/preview/site/en/userGuide/tools/birdwatcher_overview.md
new file mode 100644
index 000000000..a8b4a9767
--- /dev/null
+++ b/preview/site/en/userGuide/tools/birdwatcher_overview.md
@@ -0,0 +1,23 @@
+---
+id: birdwatcher_overview.md
+summary: Birdwatcher is a debug tool for Milvus 2.x. It connects to etcd and inspects the status of the Milvus system.
+title: Birdwatcher
+---
+
+# Birdwatcher
+
+Milvus is a stateless vector database that separates reads and writes and has etcd play the role of the single source of state. All coordinators have to query the state from etcd before they can make any changes to it. Once users need to check or clean the state, they need a tool to communicate with etcd. Here is where Birdwatcher comes to the scene.
+
+Birdwatcher is a debugging tool for Milvus. Using it to connect to etcd, you can check the state of your Milvus system or configure it on the fly.
+
+## Prerequisite
+
+- You have already installed [Go 1.18 or higher](https://go.dev/doc/install).
+
+## Architecture
+
+
+
+## Latest release
+
+[Release v1.0.2](https://github.com/milvus-io/birdwatcher/releases/tag/v1.0.3)
\ No newline at end of file
diff --git a/preview/site/en/userGuide/tools/birdwatcher_usage_guides.md b/preview/site/en/userGuide/tools/birdwatcher_usage_guides.md
new file mode 100644
index 000000000..fa6267d46
--- /dev/null
+++ b/preview/site/en/userGuide/tools/birdwatcher_usage_guides.md
@@ -0,0 +1,436 @@
+---
+id: birdwatcher_usage_guides.md
+summary: Learn how to use Birdwatch to debug Milvus.
+title: Use Birdwatcher
+---
+
+# Use Birdwatcher
+
+This guide walks you through how to use Birdwatcher to check the state of your Milvus and configure it on the fly.
+
+## Start Birdwatcher
+
+Birdwatcher is a command-line tool, you can start it as follows:
+
+```shell
+./birdwatcher
+```
+
+Then you will be greeted with the following prompt:
+
+```shell
+Offline >
+```
+
+## Connect to etcd
+
+You need to use Birdwatcher to connect to etcd before any other operations.
+
+- Connect with default settings
+
+ ```shell
+ Offline > connect
+ Milvus(by-dev) >
+ ```
+
+- Connect from Birdwatcher in a pod
+
+ If you choose to run Birdwatcher in a Kubernetes pod, you need first obtain the IP address of etcd as follows:
+
+ ```shell
+ kubectl get pod my-release-etcd-0 -o 'jsonpath={.status.podIP}'
+ ```
+
+ Then access the shell of the pod.
+
+ ```shell
+ kubectl exec --stdin --tty birdwatcher-7f48547ddc-zcbxj -- /bin/sh
+ ```
+
+ Finally, use the returned IP address to connect to etcd as follows:
+
+ ```shell
+ Offline > connect --etcd ${ETCD_IP_ADDR}:2379
+ Milvus(by-dev)
+ ```
+
+- Connect with a different root path
+
+ If the root path of your Milvus is different from `by-dev` and you are prompted with an error reporting about an incorrect root path, you can connect to etcd as follows:
+
+ ```shell
+ Offline > connect --rootPath my-release
+ Milvus(my-release) >
+ ```
+
+ If you do not know the root path of your Milvus, connect to etcd as follows:
+
+ ```shell
+ Offline > connect --dry
+ using dry mode, ignore rootPath and metaPath
+ Etcd(127.0.0.1:2379) > find-milvus
+ 1 candidates found:
+ my-release
+ Etcd(127.0.0.1:2379) > use my-release
+ Milvus(my-release) >
+ ```
+
+## Check Milvus status
+
+You can use the `show` commands to check Milvus status.
+
+```shell
+Milvus(my-release) > show -h
+Usage:
+ show [command]
+
+Available Commands:
+ alias list alias meta info
+ channel-watch display channel watching info from data coord meta store
+ checkpoint list checkpoint collection vchannels
+ collection-history display collection change history
+ collection-loaded display information of loaded collection from querycoord
+ collections list current available collection from RootCoord
+ config-etcd list configuations set by etcd source
+ configurations iterate all online components and inspect configuration
+ current-version
+ database display Database info from rootcoord meta
+ index
+ partition list partitions of provided collection
+ querycoord-channel display querynode information from querycoord cluster
+ querycoord-cluster display querynode information from querycoord cluster
+ querycoord-task display task information from querycoord
+ replica list current replica information from QueryCoord
+ segment display segment information from data coord meta store
+ segment-index display segment index information
+ segment-loaded display segment information from querycoordv1 meta
+ segment-loaded-grpc list segments loaded information
+ session list online milvus components
+
+Flags:
+ -h, --help help for show
+
+Use " show [command] --help" for more information about a command.
+```
+
+### List sessions
+
+You can list all etcd sessions as follows:
+
+```shell
+Milvus(by-dev) > show session
+Session:datacoord, ServerID: 3, Version: 2.2.11, Address: 10.244.0.8:13333
+Session:datanode, ServerID: 6, Version: 2.2.11, Address: 10.244.0.8:21124
+Session:indexcoord, ServerID: 4, Version: 2.2.11, Address: 10.244.0.8:31000
+Session:indexnode, ServerID: 5, Version: 2.2.11, Address: 10.244.0.8:21121
+Session:proxy, ServerID: 8, Version: 2.2.11, Address: 10.244.0.8:19529
+Session:querycoord, ServerID: 7, Version: 2.2.11, Address: 10.244.0.8:19531
+Session:querynode, ServerID: 2, Version: 2.2.11, Address: 10.244.0.8:21123
+Session:rootcoord, ServerID: 1, Version: 2.2.11, Address: 10.244.0.8:53100
+```
+
+In the command output, you can find sessions from all Milvus components to etcd.
+
+### Check databases and collections
+
+You can list all databases and collections.
+
+- List databases
+
+ In the command output, you can find information about every database.
+
+ ```shell
+ Milvus(by-dev) > show database
+ =============================
+ ID: 1 Name: default
+ TenantID: State: DatabaseCreated
+ --- Total Database(s): 1
+ ```
+
+- List collections
+
+ In the command output, you can find detailed information about every collection.
+
+ ```shell
+ Milvus(by-dev) > show collections
+ ================================================================================
+ DBID: 1
+ Collection ID: 443407225551410746 Collection Name: medium_articles_2020
+ Collection State: CollectionCreated Create Time: 2023-08-08 09:27:08
+ Fields:
+ - Field ID: 0 Field Name: RowID Field Type: Int64
+ - Field ID: 1 Field Name: Timestamp Field Type: Int64
+ - Field ID: 100 Field Name: id Field Type: Int64
+ - Primary Key: true, AutoID: false
+ - Field ID: 101 Field Name: title Field Type: VarChar
+ - Type Param max_length: 512
+ - Field ID: 102 Field Name: title_vector Field Type: FloatVector
+ - Type Param dim: 768
+ - Field ID: 103 Field Name: link Field Type: VarChar
+ - Type Param max_length: 512
+ - Field ID: 104 Field Name: reading_time Field Type: Int64
+ - Field ID: 105 Field Name: publication Field Type: VarChar
+ - Type Param max_length: 512
+ - Field ID: 106 Field Name: claps Field Type: Int64
+ - Field ID: 107 Field Name: responses Field Type: Int64
+ Enable Dynamic Schema: false
+ Consistency Level: Bounded
+ Start position for channel by-dev-rootcoord-dml_0(by-dev-rootcoord-dml_0_443407225551410746v0): [1 0 28 175 133 76 39 6]
+ --- Total collections: 1 Matched collections: 1
+ --- Total channel: 1 Healthy collections: 1
+ ================================================================================
+ ```
+
+- View a specific collection
+
+ You can view a specific collection by specifying its ID.
+
+ ```shell
+ Milvus(by-dev) > show collection-history --id 443407225551410746
+ ================================================================================
+ DBID: 1
+ Collection ID: 443407225551410746 Collection Name: medium_articles_2020
+ Collection State: CollectionCreated Create Time: 2023-08-08 09:27:08
+ Fields:
+ - Field ID: 0 Field Name: RowID Field Type: Int64
+ - Field ID: 1 Field Name: Timestamp Field Type: Int64
+ - Field ID: 100 Field Name: id Field Type: Int64
+ - Primary Key: true, AutoID: false
+ - Field ID: 101 Field Name: title Field Type: VarChar
+ - Type Param max_length: 512
+ - Field ID: 102 Field Name: title_vector Field Type: FloatVector
+ - Type Param dim: 768
+ - Field ID: 103 Field Name: link Field Type: VarChar
+ - Type Param max_length: 512
+ - Field ID: 104 Field Name: reading_time Field Type: Int64
+ - Field ID: 105 Field Name: publication Field Type: VarChar
+ - Type Param max_length: 512
+ - Field ID: 106 Field Name: claps Field Type: Int64
+ - Field ID: 107 Field Name: responses Field Type: Int64
+ Enable Dynamic Schema: false
+ Consistency Level: Bounded
+ Start position for channel by-dev-rootcoord-dml_0(by-dev-rootcoord-dml_0_443407225551410746v0): [1 0 28 175 133 76 39 6]
+ ```
+
+- View all loaded collections
+
+ You can have Birdwatcher filter all loaded collections.
+
+ ```shell
+ Milvus(by-dev) > show collection-loaded
+ Version: [>= 2.2.0] CollectionID: 443407225551410746
+ ReplicaNumber: 1 LoadStatus: Loaded
+ --- Collections Loaded: 1
+ ```
+
+- List all channel checkpoints of a collection
+
+ You can have Birdwatcher list all checkpoints of a specific collection.
+
+ ```shell
+ Milvus(by-dev) > show checkpoint --collection 443407225551410746
+ vchannel by-dev-rootcoord-dml_0_443407225551410746v0 seek to 2023-08-08 09:36:09.54 +0000 UTC, cp channel: by-dev-rootcoord-dml_0_443407225551410746v0, Source: Channel Checkpoint
+ ```
+
+### Check index details
+
+Run the following command to list all index files in detail.
+
+```shell
+Milvus(by-dev) > show index
+*************2.1.x***************
+*************2.2.x***************
+==================================================================
+Index ID: 443407225551410801 Index Name: _default_idx_102 CollectionID:443407225551410746
+Create Time: 2023-08-08 09:27:19.139 +0000 Deleted: false
+Index Type: HNSW Metric Type: L2
+Index Params:
+==================================================================
+```
+
+### List partitions
+
+Run the following command to list all partitions in a specific collection.
+
+```shell
+Milvus(by-dev) > show partition --collection 443407225551410746
+Parition ID: 443407225551410747 Name: _default State: PartitionCreated
+--- Total Database(s): 1
+```
+
+### Check channel status
+
+Run the following command to view channel status
+
+```shell
+Milvus(by-dev) > show channel-watch
+=============================
+key: by-dev/meta/channelwatch/6/by-dev-rootcoord-dml_0_443407225551410746v0
+Channel Name:by-dev-rootcoord-dml_0_443407225551410746v0 WatchState: WatchSuccess
+Channel Watch start from: 2023-08-08 09:27:09 +0000, timeout at: 1970-01-01 00:00:00 +0000
+Start Position ID: [1 0 28 175 133 76 39 6], time: 1970-01-01 00:00:00 +0000
+Unflushed segments: []
+Flushed segments: []
+Dropped segments: []
+--- Total Channels: 1
+```
+
+### List all replicas and segments
+
+- List all replicas
+
+ Run the following command to list all replicas and their corresponding collections.
+
+ ```shell
+ Milvus(by-dev) > show replica
+ ================================================================================
+ ReplicaID: 443407225685278721 CollectionID: 443407225551410746 version:>=2.2.0
+ All Nodes:[2]
+ ```
+
+- List all segments
+
+ Run the following command to list all segments and their status
+
+ ```shell
+ SegmentID: 443407225551610865 State: Flushed, Row Count:5979
+ --- Growing: 0, Sealed: 0, Flushed: 1
+ --- Total Segments: 1, row count: 5979
+ ```
+
+ Run the following command to list all loaded segments in detail. For Milvus 2.1.x, use `show segment-loaded` instead.
+
+ ```shell
+ Milvus(by-dev) > show segment-loaded-grpc
+ ===========
+ ServerID 2
+ Channel by-dev-rootcoord-dml_0_443407225551410746v0, collection: 443407225551410746, version 1691486840680656937
+ Leader view for channel: by-dev-rootcoord-dml_0_443407225551410746v0
+ Growing segments number: 0 , ids: []
+ SegmentID: 443407225551610865 CollectionID: 443407225551410746 Channel: by-dev-rootcoord-dml_0_443407225551410746v0
+ Sealed segments number: 1
+ ```
+
+### List configurations
+
+You can have Birdwatcher list the current configurations of each Milvus component.
+
+```shell
+Milvus(by-dev) > show configurations
+client nil Session:proxy, ServerID: 8, Version: 2.2.11, Address: 10.244.0.8:19529
+Component rootcoord-1
+rootcoord.importtaskexpiration: 900
+rootcoord.enableactivestandby: false
+rootcoord.importtaskretention: 86400
+rootcoord.maxpartitionnum: 4096
+rootcoord.dmlchannelnum: 16
+rootcoord.minsegmentsizetoenableindex: 1024
+rootcoord.port: 53100
+rootcoord.address: localhost
+rootcoord.maxdatabasenum: 64
+Component datacoord-3
+...
+querynode.gracefulstoptimeout: 30
+querynode.cache.enabled: true
+querynode.cache.memorylimit: 2147483648
+querynode.scheduler.maxreadconcurrentratio: 2
+```
+
+As an alternative, you can visit each Milvus component to find its configuration. The following demonstrates how to list the configuration of the QueryCoord with ID 7.
+
+```shell
+Milvus(by-dev) > show session
+Session:datacoord, ServerID: 3, Version: 2.2.11, Address: 10.244.0.8:13333
+Session:datanode, ServerID: 6, Version: 2.2.11, Address: 10.244.0.8:21124
+Session:indexcoord, ServerID: 4, Version: 2.2.11, Address: 10.244.0.8:31000
+Session:indexnode, ServerID: 5, Version: 2.2.11, Address: 10.244.0.8:21121
+Session:proxy, ServerID: 8, Version: 2.2.11, Address: 10.244.0.8:19529
+Session:querycoord, ServerID: 7, Version: 2.2.11, Address: 10.244.0.8:19531
+Session:querynode, ServerID: 2, Version: 2.2.11, Address: 10.244.0.8:21123
+Session:rootcoord, ServerID: 1, Version: 2.2.11, Address: 10.244.0.8:53100
+
+Milvus(by-dev) > visit querycoord 7
+QueryCoord-7(10.244.0.8:19531) > configuration
+Key: querycoord.enableactivestandby, Value: false
+Key: querycoord.channeltasktimeout, Value: 60000
+Key: querycoord.overloadedmemorythresholdpercentage, Value: 90
+Key: querycoord.distpullinterval, Value: 500
+Key: querycoord.checkinterval, Value: 10000
+Key: querycoord.checkhandoffinterval, Value: 5000
+Key: querycoord.taskexecutioncap, Value: 256
+Key: querycoord.taskmergecap, Value: 8
+Key: querycoord.autohandoff, Value: true
+Key: querycoord.address, Value: localhost
+Key: querycoord.port, Value: 19531
+Key: querycoord.memoryusagemaxdifferencepercentage, Value: 30
+Key: querycoord.refreshtargetsintervalseconds, Value: 300
+Key: querycoord.balanceintervalseconds, Value: 60
+Key: querycoord.loadtimeoutseconds, Value: 1800
+Key: querycoord.globalrowcountfactor, Value: 0.1
+Key: querycoord.scoreunbalancetolerationfactor, Value: 0.05
+Key: querycoord.reverseunbalancetolerationfactor, Value: 1.3
+Key: querycoord.balancer, Value: ScoreBasedBalancer
+Key: querycoord.autobalance, Value: true
+Key: querycoord.segmenttasktimeout, Value: 120000
+```
+
+## Backup metrics
+
+You can have Birdwatcher back up metrics of all components
+
+```shell
+Milvus(my-release) > backup
+Backing up ... 100%(2452/2451)
+backup etcd for prefix done
+http://10.244.0.10:9091/metrics
+http://10.244.0.10:9091/metrics
+http://10.244.0.10:9091/metrics
+http://10.244.0.10:9091/metrics
+http://10.244.0.10:9091/metrics
+http://10.244.0.10:9091/metrics
+http://10.244.0.10:9091/metrics
+http://10.244.0.10:9091/metrics
+backup for prefix done, stored in file: bw_etcd_ALL.230810-075211.bak.gz
+```
+
+Then you can check the file in the directory where you start Birdwatcher.
+
+## Probe collections
+
+You can have Birdwatcher probe the status of loaded collections with specified primary keys or mock queries.
+
+### Probe collection with known primary key
+
+In the `probe` command, you should specify the primary key using the `pk` flag, and the collection ID using the `collection` flag.
+
+```shell
+Milvus(by-dev) > probe pk --pk 110 --collection 442844725212299747
+PK 110 found on segment 442844725212299830
+Field id, value: &{long_data:Syntax
+Syntax
```shell -calc distance +clear ``` -Options
+Options
| Option | Full name | Description | | :----- | :-------- | :----------------------------------- | | --help | n/a | Displays help for using the command. | -Example
+## connect + +Connects to Milvus. -To calculate the distance between two vector arrays and be prompted for the required input: +Syntax
```shell -milvus_cli > calc distance +connect [-uri (text)] [-t (text)] +``` -Import left operator vectors from existing collection? [y/N]: n +Options
-The vector's type (float_vectors, bin_vectors): float_vectors +| Option | Full name | Description | +| :----- | :----------- | :---------------------------------------------------------------------------------------------------------- | +| -uri | --uri | (Optional) The uri name. The default is "http://127.0.0.1:19530". | +| -t | --token | (Optional) The zilliz cloud apikey or `username:password`. The default is None. | +| --help | n/a | Displays help for using the command. | -Left vectors: - [[0.083, 0.992, 0.931, 0.433, 0.93, 0.706, 0.668, 0.481, 0.255, 0.088, 0.121, 0.701, 0.935, 0.142, 0.012, 0.197, 0.066, 0.864, 0.263, 0.732, 0.445, 0.672, 0.184, 0.675, 0.361, 0.115, 0.396, 0.206, 0.084, 0.274, 0.523, 0.958, 0.071, 0.646, 0.864, 0.434, 0.212, 0.5, 0.319, 0.608, 0.356, 0.745, 0.672, 0.488, 0.221, 0.485, 0.193, 0.557, 0.546, 0.626, 0.593, 0.526, 0.404, 0.795, 0.076, 0.156, 0.231, 0.1, 0.18, 0.796, 0.716, 0.752, 0.816, 0.363], [0.284, 0.135, 0.172, 0.198, 0.752, 0.174, 0.314, 0.18, 0.672, 0.727, 0.062, 0.611, 0.921, 0.851, 0.238, 0.648, 0.794, 0.177, 0.639, 0.339, 0.402, 0.977, 0.887, 0.528, 0.768, 0.16, 0.698, 0.016, 0.906, 0.261, 0.902, 0.93, 0.547, 0.146, 0.65, 0.072, 0.876, 0.645, 0.303, 0.922, 0.807, 0.093, 0.063, 0.344, 0.667, 0.81, 0.662, 0.147, 0.242, 0.641, 0.903, 0.714, 0.637, 0.365, 0.512, 0.267, 0.577, 0.809, 0.698, 0.62, 0.768, 0.402, 0.922, 0.592]] +Example
-Import right operator vectors from existing collection? [y/N]: n +```shell +milvus_cli > connect -uri http://127.0.0.1:19530 +``` -The vector's type (float_vectors, bin_vectors): float_vectors +## create Database -Right vectors: - [[0.518, 0.034, 0.786, 0.251, 0.04, 0.247, 0.55, 0.595, 0.638, 0.957, 0.303, 0.023, 0.007, 0.712, 0.841, 0.648, 0.807, 0.429, 0.402, 0.904, 0.002, 0.882, 0.69, 0.268, 0.732, 0.511, 0.942, 0.202, 0.749, 0.234, 0.666, 0.517, 0.787, 0.399, 0.565, 0.457, 0.57, 0.937, 0.712, 0.981, 0.928, 0.678, 0.154, 0.775, 0.754, 0.532, 0.074, 0.493, 0.288, 0.229, 0.9, 0.657, 0.936, 0.184, 0.478, 0.587, 0.592, 0.84, 0.793, 0.985, 0.826, 0.595, 0.947, 0.175], [0.704, 0.02, 0.937, 0.249, 0.431, 0.99, 0.779, 0.855, 0.731, 0.665, 0.773, 0.647, 0.135, 0.44, 0.621, 0.329, 0.718, 0.003, 0.927, 0.511, 0.515, 0.359, 0.744, 0.828, 0.31, 0.161, 0.605, 0.539, 0.331, 0.077, 0.503, 0.668, 0.275, 0.72, 0.172, 0.035, 0.88, 0.762, 0.646, 0.727, 0.83, 0.001, 0.085, 0.188, 0.583, 0.709, 0.134, 0.683, 0.246, 0.214, 0.863, 0.109, 0.168, 0.539, 0.451, 0.303, 0.064, 0.575, 0.547, 0.85, 0.75, 0.789, 0.681, 0.735], [0.648, 0.769, 0.525, 0.716, 0.752, 0.199, 0.095, 0.222, 0.767, 0.029, 0.244, 0.527, 0.496, 0.691, 0.487, 0.83, 0.546, 0.102, 0.845, 0.096, 0.744, 0.758, 0.092, 0.289, 0.139, 0.005, 0.204, 0.245, 0.528, 0.607, 0.446, 0.029, 0.686, 0.558, 0.705, 0.451, 0.87, 0.404, 0.824, 0.727, 0.058, 0.283, 0.512, 0.682, 0.027, 0.026, 0.809, 0.669, 0.241, 0.103, 0.101, 0.225, 0.989, 0.662, 0.917, 0.972, 0.93, 0.447, 0.318, 0.434, 0.437, 0.036, 0.009, 0.96], [0.726, 0.418, 0.404, 0.244, 0.618, 0.356, 0.07, 0.842, 0.137, 0.967, 0.465, 0.811, 0.027, 0.704, 0.935, 0.546, 0.92, 0.125, 0.917, 0.089, 0.463, 0.929, 0.289, 0.721, 0.368, 0.837, 0.14, 0.431, 0.495, 0.75, 0.484, 0.083, 0.431, 0.392, 0.177, 0.303, 0.013, 0.317, 0.593, 0.047, 0.695, 0.185, 0.633, 0.825, 0.203, 0.619, 0.597, 0.152, 0.899, 0.061, 0.512, 0.67, 0.82, 0.52, 0.743, 0.07, 0.99, 0.119, 0.949, 0.284, 0.529, 0.65, 0.523, 0.059]] +Create Database in Milvus -Supported metric type. Default is "L2" (L2, IP, HAMMING, TANIMOTO) [L2]: -L2 +Syntax
-sqrt [False]: True +```shell +create database -db (text) +``` + +### Options -Timeout(optional) []: +| Option | Full name | Description | +| :----- | :--------- | :-------------------------------------------------- | +| -db | --database | [Required] The database name in milvus. | +| --help | n/a | Displays help for using the command. | -====== -Return type: -Assume the vectors_left: L_1, L_2, L_3 -Assume the vectors_right: R_a, R_b -Distance between L_n and R_m we called "D_n_m" -The returned distances are arranged like this: -[[D_1_a, D_1_b], -[D_2_a, D_2_b], -[D_3_a, D_3_b]] +### Examples -Note: if some vectors do not exist in collection, the returned distance is "-1.0" -====== +#### Example 1 -Result: +The following example create the databasetestdb in milvus.
-[[3.625464916229248, 3.234992742538452, 3.568333148956299, 3.694913148880005], [2.556027889251709, 2.8901233673095703, 3.385758399963379, 3.3239054679870605]]
+```shell
+milvus_cli > create database -db testdb
```
-## calc mkts_from_hybridts
+## use Database
-Generates a hybrid timestamp based on an existing hybrid timestamp, timedelta, and incremental time interval.
+Use Database in Milvus
-Syntax
+Syntax
```shell -calc mkts_from_hybridts -h (int) -m (float) +use database -db (text) ``` -Options
+### Options -| Option | Full name | Description | -| :----- | :------------- | :---------------------------------------------------------------------------------------------------------------------------------------- | -| -h | --hybridts | The original hybrid timestamp used to generate a new hybrid timestamp. A non-negative integer that ranges from 0 to 18446744073709551615. | -| -m | --milliseconds | The incremental interval in milliseconds. | -| --help | n/a | Displays help for using the command. | +| Option | Full name | Description | +| :----- | :--------- | :-------------------------------------------------- | +| -db | --database | [Required] The database name in milvus. | +| --help | n/a | Displays help for using the command. | -## calc mkts_from_unixtime +### Examples -Generates a hybrid timestamp based on the Unix Epoch time, timedelta, and incremental time interval. +#### Example 1 -Syntax
+The following example use the databasetestdb in milvus.
```shell
-calc mkts_from_unixtime -e (float) -m (float)
+milvus_cli > use database -db testdb
```
-Options
- -| Option | Full name | Description | -| :----- | :------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| -e | --epoch | The known Unix timestamp used to generate a hybrid timestamp. The Unix epoch is the number of seconds that have elapsed since January 1, 1970 (midnight UTC/GMT). | -| -m | --milliseconds | The incremental interval in milliseconds. | -| --help | n/a | Displays help for using the command. | - -## calc hybridts_to_unixtime +## List Databases -Converts a hybrid timestamp to the UNIX timestamp ignoring the logic part. +List Databases in Milvus -Syntax
+Syntax
```shell -calc hybridts_to_unixtime -h (int) +list databases ``` +### Examples -Options
- -| Option | Full name | Description | -| :----- | :--------- | :--------------------------------------------------------------------------------------------------------------------------------- | -| -h | --hybridts | The known hybrid timestamp to be converted to a UNIX timestamp. A non-negative integer that ranges from 0 to 18446744073709551615. | -| --help | n/a | Displays help for using the command. | - -## clear - -Clears the screen. +#### Example 1 -Syntax
+The following example list the databases in milvus. ```shell -clear +milvus_cli > list databases ``` -Options
+## Delete Database -| Option | Full name | Description | -| :----- | :-------- | :----------------------------------- | -| --help | n/a | Displays help for using the command. | - -## connect - -Connects to Milvus. +Delete Database in Milvus -Syntax
+Syntax
```shell -connect [-h (text)] [-p (int)] [-a (text)] [-s (text)] [-u (text)] [-pwd (text)] [-D] +delete database -db (text) ``` -Options
+### Options -| Option | Full name | Description | -| :----- | :----------- | :---------------------------------------------------------------------------------------------------------- | -| -h | --host | (Optional) The host name. The default is "127.0.0.1". | -| -p | --port | (Optional) The port number. The default is "19530". | -| -a | --alias | (Optional) The alias name of the Milvus link. The default is "default". | -| -s | --secure | (Optional) Whether to enable secure connection or not. The default is "False". | -| -pwd | --password | (Optional) The user password in Milvus. The default is "None". | -| -u | --username | (Optional) The username in Milvus. The default is "None". | -| -D | --disconnect | (Optional) Flag to disconnect from the Milvus server specified by an alias. The default alias is "default". | -| --help | n/a | Displays help for using the command. | +| Option | Full name | Description | +| :----- | :--------- | :-------------------------------------------------- | +| -db | --database | [Required] The database name in milvus. | +| --help | n/a | Displays help for using the command. | -Example
+### Examples + +#### Example 1 + +The following example delete the databasetestdb in milvus.
```shell
-milvus_cli > connect -h 127.0.0.1 -p 19530 -a default
+milvus_cli > delete database -db testdb
```
## create user
@@ -206,7 +187,7 @@ Specifies unique aliases for a collection.
Syntax
```shell -create alias -c (text) -a (text) [-A] [-t (float)] +create alias -c (text) -a (text) [-A] ```Options
@@ -216,27 +197,26 @@ create alias -c (text) -a (text) [-A] [-t (float)] | -c | --collection-name | The name of the collection. | | -a | --alias-name | The alias. | | -A | --alter | (Optional) Flag to transfer the alias to a specified collection. | -| -t | --timeout | (Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs. | | --help | n/a | Displays help for using the command. |Examples
-Example 1
+Example 1
The following example creates thecarAlias1 and carAlias2 aliases for the car collection.
```shell
-milvus_cli > create alias -c car -a carAlias1 -a carAlias2
+milvus_cli > create alias -c car -a carAlias1
```
-Example 2
+Example 2
Example 2 is based on Example 1.
-The following example transfers the carAlias1 and carAlias2 aliases from the car collection to the car2 collection.
+The following example transfers the carAlias1 alias from the car collection to the car2 collection.
```shell
-milvus_cli > create alias -c car2 -A -a carAlias -a carAlias2
+milvus_cli > create alias -c car2 -A -a carAlias1
```
## create collection
@@ -257,13 +237,18 @@ create collection -c (text) -f (text) -p (text) [-a] [-d (text)]
| -f | --schema-field | (Multiple) The field schema in the `Example
```shell -milvus_cli > create collection -c car -f id:INT64:primary_field -f vector:FLOAT_VECTOR:128 -f color:INT64:color -f brand:INT64:brand -p id -a -d 'car_collection' +## For array field: --schema-field supportSyntax
```shell -delete alias -a (text) [-t (float)] +delete alias -a (text) ```Options
@@ -369,7 +354,6 @@ delete alias -a (text) [-t (float)] | Option | Full name | Description | | :----- | :----------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -a | --alias-name | The alias. | -| -t | --timeout | (Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs. | | --help | n/a | Displays help for using the command. | | | @@ -380,7 +364,7 @@ Deletes a collection.Syntax
```shell -delete collection -c (text) [-t (float)] +delete collection -c (text) ```Options
@@ -388,7 +372,6 @@ delete collection -c (text) [-t (float)] | Option | Full name | Description | | :----- | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -c | --collection-name | The name of the collection to be deleted. | -| -t | --timeout | (Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs. | | --help | n/a | Displays help for using the command. |Example
@@ -404,7 +387,7 @@ Deletes entities.Syntax
``` -delete entities -c (text) -p (text) [-t (float)] +delete entities -c (text) -p (text) ```Options
@@ -413,7 +396,6 @@ delete entities -c (text) -p (text) [-t (float)] | :----- | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -c | --collection-name | The name of the collection that entities to be deleted belongs to. | | -p | --partition | (Optional) The name of the partition to be deleted. | -| -t | --timeout | (Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs. | | --help | n/a | Displays help for using the command. |Example
@@ -435,7 +417,7 @@ Deletes a partition.Syntax
```shell -delete partition -c (text) -p (text) [-t (float)] +delete partition -c (text) -p (text) ```Options
@@ -443,7 +425,6 @@ delete partition -c (text) -p (text) [-t (float)] | Option | Full name | Description | | :----- | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -c | --collection-name | The name of the collection that the partition to be deleted belongs to. | -| -t | --timeout | (Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs. | | -p | --partition | The name of the partition to be deleted. | | --help | n/a | Displays help for using the command. | @@ -462,7 +443,7 @@ Deletes an index and the corresponding index files.Syntax
```shell -delete index -c (text) -in (text) [-t (float)] +delete index -c (text) -in (text) ```Options
@@ -471,7 +452,6 @@ delete index -c (text) -in (text) [-t (float)] | :----- | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -c | --collection-name | The name of the collection. | | -in | --index-name | The name of the index name. | -| -t | --timeout | (Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs. | | --help | n/a | Displays help for using the command. |Example
@@ -480,40 +460,40 @@ delete index -c (text) -in (text) [-t (float)] milvus_cli > delete index -c car -in indexName ``` -## describe collection +## show collection Shows the detailed information of a collection. -Syntax
+Syntax
```shell -describe collection -c (text) +show collection -c (text) ``` -Options
+Options
| Option | Full name | Description | | :----- | :---------------- | :----------------------------------- | | -c | --collection-name | The name of the collection. | | --help | n/a | Displays help for using the command. | -Example
+Example
```shell -milvus_cli > describe collection -c test_collection_insert +milvus_cli > show collection -c test_collection_insert ``` -## describe partition +## show partition Shows the detailed information of a partition. -Syntax
+Syntax
```shell -describe partition -c (text) -p (text) +show partition -c (text) -p (text) ``` -Options
+Options
| Option | Full name | Description | | :----- | :---------------- | :-------------------------------------------------------- | @@ -521,23 +501,23 @@ describe partition -c (text) -p (text) | -p | --partition | The name of the partition. | | --help | n/a | Displays help for using the command. | -Example
+Example
```shell -milvus_cli > describe partition -c test_collection_insert -p _default +milvus_cli > show partition -c test_collection_insert -p _default ``` -## describe index +## show index Shows the detailed information of an index. -Syntax
+Syntax
```shell -describe index -c (text) -in (text) +show index -c (text) -in (text) ``` -Options
+Options
| Option | Full name | Description | | :----- | :---------------- | :-------------------------- | @@ -546,6 +526,11 @@ describe index -c (text) -in (text) | --help | n/a | Displays help for using the command. | +Example
+ +```shell +milvus_cli > show index -c test_collection -in index_name +``` ## exit Closes the command line window. @@ -576,22 +561,21 @@ helpSyntax
```shell -import -c (text)[-p (text)][-t (float)]Options
@@ -610,7 +594,6 @@ import -c (text)[-p (text)][-t (float)]Example 1
@@ -683,15 +666,13 @@ Lists all collections.Syntax
```shell
-list collections [-t (float)][-l (boolean)]
+list collections
```
Options
| Option | Full name | Description |
| :----- | :------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| -t | --timeout | (Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs. |
-| -l | --show-loaded | (Optional) Shows the loaded collections only. |
| --help | n/a | Displays help for using the command. |
## list indexes
@@ -748,26 +729,6 @@ load -c (text) [-p (text)]
| -p | --partition | (Optional/Multiple) The name of the partition. |
| --help | n/a | Displays help for using the command. |
-## load_balance
-
-Performs load balancing by transferring segments from a source query node to a destination one.
-
-Syntax
-
-```shell
-load_balance -s (int) -d (int) -ss (int) [-t (int)]
-```
-
-Options
-
-| Option | Full name | Description |
-| :----- | :------------------- | :----------------------------------------------------------------------- |
-| -s | --src-node-id | The ID of the source query node to be balanced. |
-| -d | --dst-node-id | (Multiple) The ID of the destination query node to transfer segments to. |
-| -ss | --sealed-segment-ids | (Multiple) The ID of the sealed segment to be transferred. |
-| -t | --timeout | (Optional) The timeout in seconds. |
-| --help | n/a | Displays help for using the command. |
-
## query
Shows query results that match all the criteria that you enter.
@@ -806,7 +767,6 @@ timeout []:
Guarantee timestamp. This instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date. [0]:
Graceful time. Only used in bounded consistency level. If graceful_time is set, PyMilvus will use current timestamp minus the graceful_time as the guarantee_timestamp. This option is 5s by default if not set. [5]:
-Travel timestamp. Users can specify a timestamp in a search to get results based on a data view at a specified point in time. [0]: 428960801420883491
```
Example 2
@@ -830,7 +790,6 @@ timeout []:
Guarantee timestamp. This instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date. [0]:
Graceful time. Only used in bounded consistency level. If graceful_time is set, PyMilvus will use current timestamp minus the graceful_time as the guarantee_timestamp. This option is 5s by default if not set. [5]:
-Travel timestamp. Users can specify a timestamp in a search to get results based on a data view at a specified point in time. [0]: 428960801420883491
```
## release
@@ -882,8 +841,6 @@ out headers): examples/import_csv/search_vectors.csv
The vector field used to search of collection (vector): vector
-Metric type: L2
-
Search parameter nprobe's value: 10
The max number of returned record, also known as topk: 2
@@ -896,7 +853,6 @@ timeout []:
Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:
```
Example 2
@@ -913,8 +869,6 @@ The vectors of search data(the length of data is number of query (nq), the dim o
The vector field used to search of collection (vector): vector
-Metric type: L2
-
Search parameter nprobe's value: 10
The specified number of decimal places of returned distance [-1]: 5
@@ -929,7 +883,6 @@ timeout []:
Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:
```
Example 3
@@ -957,24 +910,22 @@ timeout []:
Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:
```
-## show connection
+## List connection
-Shows the current connection.
+List connections.
Syntax
```shell
-show connection [-a]
+list connections
```
Options
| Option | Full name | Description |
| :----- | :-------- | :---------------------------------------- |
-| -a | --all | （Optional) Flag to show all connections. |
| --help | n/a | Displays help for using the command. |
## show index_progress
@@ -1012,23 +963,6 @@ show loading_progress -c (text) [-p (text)]
|-p|--partition|(Optional/Multiple) The name of the loading partition.|
|--help|n/a|Displays help for using the command.|
-## show query_segment
-
-Shows the segment information of a collection.
-
-Syntax
-
-```shell
-show query_segment -c (text) [-t (float)]
-```
-
-Options
-
-|Option|Full name|Description
-|:---|:---|:---|
-|-c|--collection-name|The name of the collection.|
-|-t|--timeout|(Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs.|
-|--help|n/a|Displays help for using the command.|
## version
@@ -1051,5 +985,5 @@ version
```shell
$ milvus_cli --version
-Milvus_CLI v0.1.7
+Milvus_CLI v0.4.0
```
diff --git a/preview/site/en/userGuide/tools/cli_overview.md b/preview/site/en/userGuide/tools/cli_overview.md
index 0ebbc2794..29321f455 100644
--- a/preview/site/en/userGuide/tools/cli_overview.md
+++ b/preview/site/en/userGuide/tools/cli_overview.md
@@ -1,6 +1,7 @@
---
id: cli_overview.md
summary: Milvus Command-Line Interface (CLI) is a command-line tool that supports database connection, data operations, and import and export of data.
+title: Milvus Command-Line Interface
---
# Milvus Command-Line Interface
@@ -24,11 +25,15 @@ In the following table, you can find the recommended versions of PyMilvus and Mi
| 2.0.0-RC8 | 2.0.0rc8 | 0.1.8 |
| 2.0.0-RC9 | 2.0.0rc9 | 0.1.9 |
| 2.1.0 | 2.1.0 | 0.3.0 |
-| 2.2.x | 2.2.x | 0.3.2 |
+| 2.2.x | 2.2.x | 0.4.0 |
+| 2.3.x | 2.3.x | 0.4.2 |
+| 2.4.x | 2.4.x | 1.0.0 |
+
+
Milvus 2.0.0-RC7 and later are not backward compatible with 2.0.0-RC6 and earlier due to changes made to storage formats.
## Current version
-The current version of Milvus_CLI is 0.3.2.
+The current version of Milvus_CLI is 1.0.0.
To find your installed version and see if you need to update, run `milvus_cli --version`.
diff --git a/preview/site/en/userGuide/tools/install_cli.md b/preview/site/en/userGuide/tools/install_cli.md
index 467178de5..c1d6fdf4b 100644
--- a/preview/site/en/userGuide/tools/install_cli.md
+++ b/preview/site/en/userGuide/tools/install_cli.md
@@ -1,6 +1,7 @@
---
id: install_cli.md
summary: Learn how to install Milvus_CLI.
+title: Install Milvus_CLI
---
# Install Milvus_CLI
diff --git a/preview/site/en/userGuide/tools/milvus-cdc/cdc-monitoring.md b/preview/site/en/userGuide/tools/milvus-cdc/cdc-monitoring.md
new file mode 100644
index 000000000..74f4ffd42
--- /dev/null
+++ b/preview/site/en/userGuide/tools/milvus-cdc/cdc-monitoring.md
@@ -0,0 +1,43 @@
+---
+id: cdc-monitoring.md
+order: 4
+summary: Milvus-CDC provides comprehensive monitoring capabilities through Grafana dashboards.
+title: Monitoring
+---
+
+# Monitoring
+
+Milvus-CDC provides comprehensive monitoring capabilities through Grafana dashboards, allowing you to visualize key metrics and ensure the smooth operation of your Change Data Capture (CDC) tasks and server health.
+
+### Metrics for CDC tasks
+
+To get started, import the [cdc-grafana.json](https://github.com/zilliztech/milvus-cdc/blob/main/server/configs/cdc-grafana.json) file into Grafana. This will add a dashboard specifically designed for monitoring the status of CDC tasks.
+
+__CDC Grafana Dashboard Overview__:
+
+
+
+__Key Metrics Explained:__
+
+- __Task__: Number of CDC tasks in different states, including __Initial__, __Running__, and __Paused__.
+
+- __Request Total__: Total number of requests received by Milvus-CDC.
+
+- __Request Success__: Number of successful requests received by Milvus-CDC.
+
+- __task num__: Number of tasks in __Initial__, __Paused__, and __Running__ states over time.
+
+- __task state__: State of individual tasks.
+
+- __request count__: Number of successful and total requests
+
+- __request latency__: Latency of requests through p99, average and other statistics.
+
+- __replicate data rate__: Replication data rate for read/write operations
+
+- __replicate tt lag__: Replication time lag for read/write operations.
+
+- __api execute count__: Number of times different Milvus-CDC APIs were executed.
+
+- __center ts__: Timestamp for read/write tasks.
+
diff --git a/preview/site/en/userGuide/tools/milvus-cdc/deploy-cdc-server.md b/preview/site/en/userGuide/tools/milvus-cdc/deploy-cdc-server.md
new file mode 100644
index 000000000..74d5d505c
--- /dev/null
+++ b/preview/site/en/userGuide/tools/milvus-cdc/deploy-cdc-server.md
@@ -0,0 +1,151 @@
+---
+id: deploy-cdc-server.md
+order: 2
+summary: This guide provides a step-by-step process for deploying a Milvus-CDC server.
+title: Deploy CDC Server
+---
+
+# Deploy CDC Server
+
+This guide provides a step-by-step process for deploying a Milvus-CDC server.
+
+## Prerequisites
+
+Ensure the following conditions are met before deploying a Milvus-CDC server:
+
+- __Milvus Instances__: Both the source Milvus and at least one target Milvus should be deployed and operational.
+
+ - Both the source and target Milvus versions must be 2.3.2 or higher, preferably 2.4.x. We recommend uisng the same version for the source and target Milvus to ensure compatibility.
+
+ - Set the `common.ttMsgEnabled` configuration of the target Milvus to `false`.
+
+ - Configure the source and target Milvus with distinct meta and message storage settings to prevent conflicts. For instance, avoid using the same etcd and rootPath configurations, as well as identical Pulsar services and `chanNamePrefix` in multiple Milvus instances.
+
+- __Metastore__: Have an etcd or MySQL database ready for the Milvus-CDC metastore.
+
+## Steps
+
+### Obtain the Milvus-CDC config file
+
+Clone the [Milvus-CDC repo](https://github.com/zilliztech/milvus-cdc) and navigate to the `milvus-cdc/server/configs` directory to access the `cdc.yaml` config file.
+
+```bash
+git clone https://github.com/zilliztech/milvus-cdc.git
+
+cd milvus-cdc/server/configs
+```
+
+### Edit the config file
+
+In the `milvus-cdc/server/configs` directory, modify the `cdc.yaml` file to customize configurations related to the Milvus-CDC metastore and connection details of the source Milvus.
+
+- __Metastore Configuration__:
+
+ - `metaStoreConfig.storeType`: Type of metastore for Milvus-CDC. Possible values are `etcd` or `mysql`.
+
+ - `metaStoreConfig.etcdEndpoints`: Address for connecting to the etcd of Milvus-CDC. Required if `storeType` is set to `etcd`.
+
+ - `metaStoreConfig.mysqlSourceUrl`: Connection address of the MySQL database for the Milvus-CDC server. Required if `storeType` is set to `mysql`.
+
+ - `metaStoreConfig.rootPath`: Root path of the Milvus-CDC metastore. This configuration enables multi-tenancy, allowing multiple CDC services to utilize the same etcd or MySQL instance while achieving isolation through different root paths.
+
+ Example configuration:
+
+ ```yaml
+ # cdc meta data config
+ metaStoreConfig:
+ # the metastore type, available value: etcd, mysql
+ storeType: etcd
+ # etcd address
+ etcdEndpoints:
+ - localhost:2379
+ # mysql connection address
+ # mysqlSourceUrl: root:root@tcp(127.0.0.1:3306)/milvus-cdc?charset=utf8
+ # meta data prefix, if multiple cdc services use the same store service, you can set different rootPaths to achieve multi-tenancy
+ rootPath: cdc
+ ```
+
+- __Source Milvus Configuration:__
+
+ Specify the connection details of the source Milvus, including etcd and message storage, to establish a connection between the Milvus-CDC server and the source Milvus.
+
+ - `sourceConfig.etcdAddress`: Address for connecting to the etcd of the source Milvus. For more information, refer to [etcd-related Configurations](https://milvus.io/docs/configure_etcd.md#etcd-related-Configurations).
+
+ - `sourceConfig.etcdRootPath`: Root prefix of the key where the source Milvus stores data in etcd. The value may vary based on the deployment method of the Milvus instance:
+
+ - __Helm__ or __Docker Compose__: Defaults to `by-dev`.
+
+ - __Operator__: Defaults to ``.
+
+ - `sourceConfig.pulsar`: Pulsar configurations for the source Milvus. If the source Milvus uses Kafka for message storage, remove all Pulsar-related configurations. For more information, refer to [Pulsar-related Configurations](https://milvus.io/docs/configure_pulsar.md).
+
+ - `sourceConfig.kafka.address`: Kafka address for the source Milvus. Uncomment this configuration if the source Milvus uses Kafka for message storage. For more information, refer to [Kafka-related Configurations](https://milvus.io/docs/configure_kafka.md).
+
+Example configuration:
+
+```yaml
+# milvus-source config, these settings are basically the same as the corresponding configuration of milvus.yaml in milvus source.
+sourceConfig:
+ # etcd config
+ etcdAddress:
+ - localhost:2379
+ etcdRootPath: by-dev
+ etcdMetaSubPath: meta
+ # default partition name
+ defaultPartitionName: _default
+ # read buffer length, mainly used for buffering if writing data to milvus-target is slow.
+ readChanLen: 10
+ # milvus-source mq config, which is pulsar or kafka
+ pulsar:
+ address: pulsar://localhost:6650
+ webAddress: localhost:80
+ maxMessageSize: 5242880
+ tenant: public
+ namespace: default
+# authPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken
+# authParams: token:xxx
+# kafka:
+# address: 127.0.0.1:9092
+```
+
+### Compile the Milvus-CDC server
+
+After saving the `cdc.yaml` file, navigate to the `milvus-cdc` directory and run one of the following commands to compile the server:
+
+- For a binary file:
+
+ ```bash
+ make build
+ ```
+
+- For a Docker image:
+
+ ```bash
+ bash build_image.sh
+ ```
+
+ For a Docker image, mount the compiled file to `/app/server/configs/cdc.yaml` within the container.
+
+### Start the server
+
+- Using the binary
+
+ Navigate to the directory containing the `milvus-cdc` binary and the `configs` directory with the `cdc.yaml` file, then start the server:
+
+ ```bash
+ # dir tree
+ .
+ ├── milvus-cdc # build from source code or download from release page
+ ├── configs
+ │ └── cdc.yaml # config for cdc and source milvus
+
+ # start milvus cdc
+ ./milvus-cdc server
+ ```
+
+- Using Docker Compose:
+
+ ```bash
+ docker-compose up -d
+ ```
+
diff --git a/preview/site/en/userGuide/tools/milvus-cdc/manage-cdc-tasks.md b/preview/site/en/userGuide/tools/milvus-cdc/manage-cdc-tasks.md
new file mode 100644
index 000000000..bdec15b94
--- /dev/null
+++ b/preview/site/en/userGuide/tools/milvus-cdc/manage-cdc-tasks.md
@@ -0,0 +1,244 @@
+---
+id: manage-cdc-tasks.md
+order: 3
+summary: A Capture Data Change (CDC) task enables the synchronization of data from a source Milvus instance to a target Milvus instance.
+title: Manage CDC Tasks
+---
+
+# Manage CDC Tasks
+
+A Capture Data Change (CDC) task enables the synchronization of data from a source Milvus instance to a target Milvus instance. It monitors operation logs from the source and replicates data changes such as insertions, deletions, and index operations to the target in real-time. This facilitates real-time disaster recovery or active-active load balancing between Milvus deployments.
+
+This guide covers how to manage CDC tasks, including creation, pausing, resuming, retrieving details, listing, and deletion through HTTP requests.
+
+## Create a task
+
+Creating a CDC task allows data change operations in the source Milvus to be synced to the target Milvus.
+
+To create a CDC task:
+
+```bash
+curl -X POST http:_//localhost:8444/cdc \
+-H "Content-Type: application/json" \
+-d '{
+ "request_type": "create",
+ "request_data": {
+ "milvus_connect_param": {
+ "host": "localhost",
+ "port": 19530,
+ "username": "root",
+ "password": "Milvus",
+ "enable_tls": false,
+ "connect_timeout": 10
+ },
+ "collection_infos": [
+ {
+ "name": "*"
+ }
+ ],
+ "rpc_channel_info": {
+ "name": "by-dev-replicate-msg"
+ }
+ }
+}'
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+__Parameters__:
+
+- __milvus_connect_param__: Connection parameters of the target Milvus.
+
+ - __host__: Hostname or IP address of the Milvus server.
+
+ - __port__: Port number the Milvus server listens on.
+
+ - __username__: Username for authenticating with the Milvus server.
+
+ - __password__: Password for authenticating with the Milvus server.
+
+ - __enable_tls__: Whether to use TLS/SSL encryption for the connection.
+
+ - __connect_timeout__: Timeout period in seconds for establishing the connection.
+
+- __collection_infos__: Collections to synchronize. Currently, only an asterisk (__*__) is supported, as Milvus-CDC synchronizes at the cluster level, not individual collections.
+
+- __rpc_channel_info__: RPC channel name for synchronization, constructed by concatenating the values of __common.chanNamePrefix.cluster__ and __common.chanNamePrefix.replicateMsg__ from the source Milvus configuration, separated by a hyphen (__-__).
+
+Expected response:
+
+```json
+{
+ "code": 200,
+ "data": {
+ "task_id":"xxxx"
+ }
+}
+```
+
+## List tasks
+
+To list all created CDC tasks:
+
+```bash
+curl -X POST -H "Content-Type: application/json" -d '{
+ "request_type": "list"
+}' http://localhost:8444/cdc
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+Expected response:
+
+```json
+{
+ "code": 200,
+ "data": {
+ "tasks": [
+ {
+ "task_id": "xxxxx",
+ "milvus_connect_param": {
+ "host": "localhost",
+ "port": 19530,
+ "connect_timeout": 10
+ },
+ "collection_infos": [
+ {
+ "name": "*"
+ }
+ ],
+ "state": "Running"
+ }
+ ]
+ }
+}
+```
+
+## Pause a task
+
+To pause a CDC task:
+
+```bash
+curl -X POST -H "Content-Type: application/json" -d '{
+ "request_type":"pause",
+ "request_data": {
+ "task_id": "xxxx"
+ }
+}' http://localhost:8444/cdc
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+__Parameters__:
+
+- __task_id__: ID of the CDC task to pause.
+
+Expected response:
+
+```bash
+{
+ "code": 200,
+ "data": {}
+}
+```
+
+## Resume a task
+
+To resume a paused CDC task:
+
+```bash
+curl -X POST -H "Content-Type: application/json" -d '{
+ "request_type":"resume",
+ "request_data": {
+ "task_id": "xxxx"
+ }
+}' http://localhost:8444/cdc
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+__Parameters__:
+
+- __task_id__: ID of the CDC task to resume.
+
+Expected response:
+
+```bash
+{
+ "code": 200,
+ "data": {}
+}
+```
+
+## Retrieve task details
+
+To retrieve the details of a specific CDC task:
+
+```bash
+curl -X POST -H "Content-Type: application/json" -d '{
+ "request_type":"get",
+ "request_data": {
+ "task_id": "xxxx"
+ }
+}' http://localhost:8444/cdc
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+__Parameters__:
+
+- __task_id__: ID of the CDC task to query.
+
+Expected response:
+
+```bash
+{
+ "code": 200,
+ "data": {
+ "Task": {
+ "collection_infos": [
+ {
+ "name": "*"
+ }
+ ],
+ "milvus_connect_param": {
+ "connect_timeout": 10,
+ "enable_tls": true,
+ "host": "localhost",
+ "port": 19530
+ },
+ "state": "Running",
+ "task_id": "xxxx"
+ }
+ }
+}
+```
+
+## Delete a task
+
+To delete a CDC task:
+
+```bash
+curl -X POST -H "Content-Type: application/json" -d '{
+ "request_type":"delete",
+ "request_data": {
+ "task_id": "30d1e325df604ebb99e14c2a335a1421"
+ }
+}' http://localhost:8444/cdc
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+__Parameters__:
+
+- __task_id__: ID of the CDC task to delete.
+
+Expected response:
+
+```json
+{
+ "code": 200,
+ "data": {}
+}
+```
+
diff --git a/preview/site/en/userGuide/tools/milvus-cdc/milvus-cdc-overview.md b/preview/site/en/userGuide/tools/milvus-cdc/milvus-cdc-overview.md
new file mode 100644
index 000000000..3b2548125
--- /dev/null
+++ b/preview/site/en/userGuide/tools/milvus-cdc/milvus-cdc-overview.md
@@ -0,0 +1,69 @@
+---
+id: milvus-cdc-overview.md
+order: 1
+summary: Milvus-CDC is a user-friendly tool that can capture and synchronize incremental data in Milvus instances.
+title: CDC Overview
+---
+
+# Overview
+
+Milvus-CDC is a user-friendly tool that can capture and synchronize incremental data in Milvus instances. It ensures the reliability of business data by seamlessly transferring it between source and target instances, allowing for easy incremental backup and disaster recovery.
+
+## Key capabilities
+
+- __Sequential Data Synchronization__: Ensures data integrity and consistency by synchronizing data changes sequentially between Milvus instances.
+
+- __Incremental Data Replication__: Replicates incremental data, including insertions and deletions, from source Milvus to target Milvus, offering persistent storage.
+
+- __CDC Task Management__: Allows for the management of CDC tasks through OpenAPI requests, including creating, querying status, and deleting CDC tasks.
+
+Additionally, we are planning to expand our capabilities to include support for integration with stream processing systems in the future.
+
+## Architecture
+
+Milvus-CDC adopts an architecture with two main components - an HTTP server that manages tasks and metadata, and __corelib__ that synchronizes task execution with a reader that obtains data from the source Milvus instance and a writer that sends processed data to the target Milvus instance.
+
+
+
+In the preceding diagram,
+
+- __HTTP server__: Handles user requests, executes tasks, and maintains metadata. It serves as the control plane for task orchestration within the Milvus-CDC system.
+
+- __Corelib__: Responsible for the actual synchronization of tasks. It includes a reader component that retrieves information from the source Milvus's etcd and message queue (MQ), and a writer component that translates messages from the MQ into API parameters for the Milvus system and sends these requests to the target Milvus to complete the synchronization process.
+
+## Workflow
+
+The Milvus-CDC data processing flow involves the following steps:
+
+1. __Task creation__: Users initiate a CDC task via HTTP requests.
+
+1. __Metadata retrieval__: The system fetches collection-specific metadata from the source Milvus's etcd, including channel and checkpoint information for the collection.
+
+1. __MQ connection__: With the metadata at hand, the system connects to the MQ to begin subscribing to the data stream.
+
+1. __Data processing__: Data from MQ is read, parsed, and either passed on using the Go SDK or processed to replicate operations performed in the source Milvus.
+
+
+
+## Limits
+
+- __Incremental Data Synchronization__: As of now, Milvus-CDC is designed to synchronize only incremental data. If your business requires a full data backup, please [reach out to us](https://milvus.io/community) for assistance.
+
+- __Synchronization Scope__: Currently, Milvus-CDC can synchronize data at the cluster level. We are working on adding support for collection-level data synchronization in upcoming releases.
+
+- __Supported API Requests__: Milvus-CDC currently supports the following API requests. We plan to extend support to additional requests in future releases:
+
+ - Create/Drop Collection
+
+ - Insert/Delete/Upsert
+
+ - Create/Drop Partition
+
+ - Create/Drop Index
+
+ - Load/Release/Flush
+
+ - Load/Release Partition
+
+ - Create/Drop Database
+
diff --git a/preview/site/en/userGuide/tools/milvus_backup-api.md b/preview/site/en/userGuide/tools/milvus_backup_api.md
similarity index 97%
rename from preview/site/en/userGuide/tools/milvus_backup-api.md
rename to preview/site/en/userGuide/tools/milvus_backup_api.md
index 71861c36c..7d9715430 100644
--- a/preview/site/en/userGuide/tools/milvus_backup-api.md
+++ b/preview/site/en/userGuide/tools/milvus_backup_api.md
@@ -1,6 +1,7 @@
---
-id: milvus_backup-api.md
+id: milvus_backup_api.md
summary: Learn how to use Milvus Backup through API
+title: Back up and Restore Data Using APIs
---
# Back up and Restore Data Using APIs
@@ -79,6 +80,8 @@ This step is optional. If you skip this, ensure that you already have some data
## Back up data
+
+
Note that running Milvus Backup against a Milvus instance will not normally affect the running of the instance. Your Milvus instance is fully functional during backup or restore.
Run the following command to create a backup. Change `collection_names` and `backup_name` if necessary.
@@ -115,6 +118,8 @@ Now, you can save the backup files to a safe place for restoration in the future
## Restore data
+
+
You can call the restore API command with a `collection_suffix` option to create a new collection by restoring the data from the backup. Change `collection_names` and `backup_name` if necessary.
```shell
@@ -172,4 +177,3 @@ python example/verify_data.py
Note that the above script assumes that you have run the `restore` command with the `-s` flag and the suffix is set to `-recover`. Feel free to make necessary changes to the script to fit your need.
-
diff --git a/preview/site/en/userGuide/tools/milvus_backup-cli.md b/preview/site/en/userGuide/tools/milvus_backup_cli.md
similarity index 83%
rename from preview/site/en/userGuide/tools/milvus_backup-cli.md
rename to preview/site/en/userGuide/tools/milvus_backup_cli.md
index 3ee182d8b..14fc32e74 100644
--- a/preview/site/en/userGuide/tools/milvus_backup-cli.md
+++ b/preview/site/en/userGuide/tools/milvus_backup_cli.md
@@ -1,6 +1,7 @@
---
-id: milvus_backup-cli.md
+id: milvus_backup_cli.md
summary: Learn how to use Milvus Backup through CLI
+title: Back up and Restore Data Using Commands
---
# Back up and Restore Data Using Commands
@@ -45,7 +46,7 @@ The name of the default Minio bucket varies with the way you install Milvus. Whe
| field | Docker Compose | Helm / Milvus Operator |
| --------------- | -------------- | ---------------------- |
| `bucketName` | a-bucket | milvus-bucket |
-| `rootPath` | files | file |
+| `rootPath` | files | file |
Options
| Option | Full name | Description |
| :----- | :------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| -t | --timeout | (Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs. |
-| -l | --show-loaded | (Optional) Shows the loaded collections only. |
| --help | n/a | Displays help for using the command. |
## list indexes
@@ -748,26 +729,6 @@ load -c (text) [-p (text)]
| -p | --partition | (Optional/Multiple) The name of the partition. |
| --help | n/a | Displays help for using the command. |
-## load_balance
-
-Performs load balancing by transferring segments from a source query node to a destination one.
-
-Syntax
-
-```shell
-load_balance -s (int) -d (int) -ss (int) [-t (int)]
-```
-
-Options
-
-| Option | Full name | Description |
-| :----- | :------------------- | :----------------------------------------------------------------------- |
-| -s | --src-node-id | The ID of the source query node to be balanced. |
-| -d | --dst-node-id | (Multiple) The ID of the destination query node to transfer segments to. |
-| -ss | --sealed-segment-ids | (Multiple) The ID of the sealed segment to be transferred. |
-| -t | --timeout | (Optional) The timeout in seconds. |
-| --help | n/a | Displays help for using the command. |
-
## query
Shows query results that match all the criteria that you enter.
@@ -806,7 +767,6 @@ timeout []:
Guarantee timestamp. This instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date. [0]:
Graceful time. Only used in bounded consistency level. If graceful_time is set, PyMilvus will use current timestamp minus the graceful_time as the guarantee_timestamp. This option is 5s by default if not set. [5]:
-Travel timestamp. Users can specify a timestamp in a search to get results based on a data view at a specified point in time. [0]: 428960801420883491
```
Example 2
@@ -830,7 +790,6 @@ timeout []:
Guarantee timestamp. This instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date. [0]:
Graceful time. Only used in bounded consistency level. If graceful_time is set, PyMilvus will use current timestamp minus the graceful_time as the guarantee_timestamp. This option is 5s by default if not set. [5]:
-Travel timestamp. Users can specify a timestamp in a search to get results based on a data view at a specified point in time. [0]: 428960801420883491
```
## release
@@ -882,8 +841,6 @@ out headers): examples/import_csv/search_vectors.csv
The vector field used to search of collection (vector): vector
-Metric type: L2
-
Search parameter nprobe's value: 10
The max number of returned record, also known as topk: 2
@@ -896,7 +853,6 @@ timeout []:
Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:
```
Example 2
@@ -913,8 +869,6 @@ The vectors of search data(the length of data is number of query (nq), the dim o
The vector field used to search of collection (vector): vector
-Metric type: L2
-
Search parameter nprobe's value: 10
The specified number of decimal places of returned distance [-1]: 5
@@ -929,7 +883,6 @@ timeout []:
Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:
```
Example 3
@@ -957,24 +910,22 @@ timeout []:
Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]:
-Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]:
```
-## show connection
+## List connection
-Shows the current connection.
+List connections.
Syntax
```shell
-show connection [-a]
+list connections
```
Options
| Option | Full name | Description |
| :----- | :-------- | :---------------------------------------- |
-| -a | --all | （Optional) Flag to show all connections. |
| --help | n/a | Displays help for using the command. |
## show index_progress
@@ -1012,23 +963,6 @@ show loading_progress -c (text) [-p (text)]
|-p|--partition|(Optional/Multiple) The name of the loading partition.|
|--help|n/a|Displays help for using the command.|
-## show query_segment
-
-Shows the segment information of a collection.
-
-Syntax
-
-```shell
-show query_segment -c (text) [-t (float)]
-```
-
-Options
-
-|Option|Full name|Description
-|:---|:---|:---|
-|-c|--collection-name|The name of the collection.|
-|-t|--timeout|(Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs.|
-|--help|n/a|Displays help for using the command.|
## version
@@ -1051,5 +985,5 @@ version
```shell
$ milvus_cli --version
-Milvus_CLI v0.1.7
+Milvus_CLI v0.4.0
```
diff --git a/preview/site/en/userGuide/tools/cli_overview.md b/preview/site/en/userGuide/tools/cli_overview.md
index 0ebbc2794..29321f455 100644
--- a/preview/site/en/userGuide/tools/cli_overview.md
+++ b/preview/site/en/userGuide/tools/cli_overview.md
@@ -1,6 +1,7 @@
---
id: cli_overview.md
summary: Milvus Command-Line Interface (CLI) is a command-line tool that supports database connection, data operations, and import and export of data.
+title: Milvus Command-Line Interface
---
# Milvus Command-Line Interface
@@ -24,11 +25,15 @@ In the following table, you can find the recommended versions of PyMilvus and Mi
| 2.0.0-RC8 | 2.0.0rc8 | 0.1.8 |
| 2.0.0-RC9 | 2.0.0rc9 | 0.1.9 |
| 2.1.0 | 2.1.0 | 0.3.0 |
-| 2.2.x | 2.2.x | 0.3.2 |
+| 2.2.x | 2.2.x | 0.4.0 |
+| 2.3.x | 2.3.x | 0.4.2 |
+| 2.4.x | 2.4.x | 1.0.0 |
+
+
Milvus 2.0.0-RC7 and later are not backward compatible with 2.0.0-RC6 and earlier due to changes made to storage formats.
## Current version
-The current version of Milvus_CLI is 0.3.2.
+The current version of Milvus_CLI is 1.0.0.
To find your installed version and see if you need to update, run `milvus_cli --version`.
diff --git a/preview/site/en/userGuide/tools/install_cli.md b/preview/site/en/userGuide/tools/install_cli.md
index 467178de5..c1d6fdf4b 100644
--- a/preview/site/en/userGuide/tools/install_cli.md
+++ b/preview/site/en/userGuide/tools/install_cli.md
@@ -1,6 +1,7 @@
---
id: install_cli.md
summary: Learn how to install Milvus_CLI.
+title: Install Milvus_CLI
---
# Install Milvus_CLI
diff --git a/preview/site/en/userGuide/tools/milvus-cdc/cdc-monitoring.md b/preview/site/en/userGuide/tools/milvus-cdc/cdc-monitoring.md
new file mode 100644
index 000000000..74f4ffd42
--- /dev/null
+++ b/preview/site/en/userGuide/tools/milvus-cdc/cdc-monitoring.md
@@ -0,0 +1,43 @@
+---
+id: cdc-monitoring.md
+order: 4
+summary: Milvus-CDC provides comprehensive monitoring capabilities through Grafana dashboards.
+title: Monitoring
+---
+
+# Monitoring
+
+Milvus-CDC provides comprehensive monitoring capabilities through Grafana dashboards, allowing you to visualize key metrics and ensure the smooth operation of your Change Data Capture (CDC) tasks and server health.
+
+### Metrics for CDC tasks
+
+To get started, import the [cdc-grafana.json](https://github.com/zilliztech/milvus-cdc/blob/main/server/configs/cdc-grafana.json) file into Grafana. This will add a dashboard specifically designed for monitoring the status of CDC tasks.
+
+__CDC Grafana Dashboard Overview__:
+
+
+
+__Key Metrics Explained:__
+
+- __Task__: Number of CDC tasks in different states, including __Initial__, __Running__, and __Paused__.
+
+- __Request Total__: Total number of requests received by Milvus-CDC.
+
+- __Request Success__: Number of successful requests received by Milvus-CDC.
+
+- __task num__: Number of tasks in __Initial__, __Paused__, and __Running__ states over time.
+
+- __task state__: State of individual tasks.
+
+- __request count__: Number of successful and total requests
+
+- __request latency__: Latency of requests through p99, average and other statistics.
+
+- __replicate data rate__: Replication data rate for read/write operations
+
+- __replicate tt lag__: Replication time lag for read/write operations.
+
+- __api execute count__: Number of times different Milvus-CDC APIs were executed.
+
+- __center ts__: Timestamp for read/write tasks.
+
diff --git a/preview/site/en/userGuide/tools/milvus-cdc/deploy-cdc-server.md b/preview/site/en/userGuide/tools/milvus-cdc/deploy-cdc-server.md
new file mode 100644
index 000000000..74d5d505c
--- /dev/null
+++ b/preview/site/en/userGuide/tools/milvus-cdc/deploy-cdc-server.md
@@ -0,0 +1,151 @@
+---
+id: deploy-cdc-server.md
+order: 2
+summary: This guide provides a step-by-step process for deploying a Milvus-CDC server.
+title: Deploy CDC Server
+---
+
+# Deploy CDC Server
+
+This guide provides a step-by-step process for deploying a Milvus-CDC server.
+
+## Prerequisites
+
+Ensure the following conditions are met before deploying a Milvus-CDC server:
+
+- __Milvus Instances__: Both the source Milvus and at least one target Milvus should be deployed and operational.
+
+ - Both the source and target Milvus versions must be 2.3.2 or higher, preferably 2.4.x. We recommend uisng the same version for the source and target Milvus to ensure compatibility.
+
+ - Set the `common.ttMsgEnabled` configuration of the target Milvus to `false`.
+
+ - Configure the source and target Milvus with distinct meta and message storage settings to prevent conflicts. For instance, avoid using the same etcd and rootPath configurations, as well as identical Pulsar services and `chanNamePrefix` in multiple Milvus instances.
+
+- __Metastore__: Have an etcd or MySQL database ready for the Milvus-CDC metastore.
+
+## Steps
+
+### Obtain the Milvus-CDC config file
+
+Clone the [Milvus-CDC repo](https://github.com/zilliztech/milvus-cdc) and navigate to the `milvus-cdc/server/configs` directory to access the `cdc.yaml` config file.
+
+```bash
+git clone https://github.com/zilliztech/milvus-cdc.git
+
+cd milvus-cdc/server/configs
+```
+
+### Edit the config file
+
+In the `milvus-cdc/server/configs` directory, modify the `cdc.yaml` file to customize configurations related to the Milvus-CDC metastore and connection details of the source Milvus.
+
+- __Metastore Configuration__:
+
+ - `metaStoreConfig.storeType`: Type of metastore for Milvus-CDC. Possible values are `etcd` or `mysql`.
+
+ - `metaStoreConfig.etcdEndpoints`: Address for connecting to the etcd of Milvus-CDC. Required if `storeType` is set to `etcd`.
+
+ - `metaStoreConfig.mysqlSourceUrl`: Connection address of the MySQL database for the Milvus-CDC server. Required if `storeType` is set to `mysql`.
+
+ - `metaStoreConfig.rootPath`: Root path of the Milvus-CDC metastore. This configuration enables multi-tenancy, allowing multiple CDC services to utilize the same etcd or MySQL instance while achieving isolation through different root paths.
+
+ Example configuration:
+
+ ```yaml
+ # cdc meta data config
+ metaStoreConfig:
+ # the metastore type, available value: etcd, mysql
+ storeType: etcd
+ # etcd address
+ etcdEndpoints:
+ - localhost:2379
+ # mysql connection address
+ # mysqlSourceUrl: root:root@tcp(127.0.0.1:3306)/milvus-cdc?charset=utf8
+ # meta data prefix, if multiple cdc services use the same store service, you can set different rootPaths to achieve multi-tenancy
+ rootPath: cdc
+ ```
+
+- __Source Milvus Configuration:__
+
+ Specify the connection details of the source Milvus, including etcd and message storage, to establish a connection between the Milvus-CDC server and the source Milvus.
+
+ - `sourceConfig.etcdAddress`: Address for connecting to the etcd of the source Milvus. For more information, refer to [etcd-related Configurations](https://milvus.io/docs/configure_etcd.md#etcd-related-Configurations).
+
+ - `sourceConfig.etcdRootPath`: Root prefix of the key where the source Milvus stores data in etcd. The value may vary based on the deployment method of the Milvus instance:
+
+ - __Helm__ or __Docker Compose__: Defaults to `by-dev`.
+
+ - __Operator__: Defaults to ``.
+
+ - `sourceConfig.pulsar`: Pulsar configurations for the source Milvus. If the source Milvus uses Kafka for message storage, remove all Pulsar-related configurations. For more information, refer to [Pulsar-related Configurations](https://milvus.io/docs/configure_pulsar.md).
+
+ - `sourceConfig.kafka.address`: Kafka address for the source Milvus. Uncomment this configuration if the source Milvus uses Kafka for message storage. For more information, refer to [Kafka-related Configurations](https://milvus.io/docs/configure_kafka.md).
+
+Example configuration:
+
+```yaml
+# milvus-source config, these settings are basically the same as the corresponding configuration of milvus.yaml in milvus source.
+sourceConfig:
+ # etcd config
+ etcdAddress:
+ - localhost:2379
+ etcdRootPath: by-dev
+ etcdMetaSubPath: meta
+ # default partition name
+ defaultPartitionName: _default
+ # read buffer length, mainly used for buffering if writing data to milvus-target is slow.
+ readChanLen: 10
+ # milvus-source mq config, which is pulsar or kafka
+ pulsar:
+ address: pulsar://localhost:6650
+ webAddress: localhost:80
+ maxMessageSize: 5242880
+ tenant: public
+ namespace: default
+# authPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken
+# authParams: token:xxx
+# kafka:
+# address: 127.0.0.1:9092
+```
+
+### Compile the Milvus-CDC server
+
+After saving the `cdc.yaml` file, navigate to the `milvus-cdc` directory and run one of the following commands to compile the server:
+
+- For a binary file:
+
+ ```bash
+ make build
+ ```
+
+- For a Docker image:
+
+ ```bash
+ bash build_image.sh
+ ```
+
+ For a Docker image, mount the compiled file to `/app/server/configs/cdc.yaml` within the container.
+
+### Start the server
+
+- Using the binary
+
+ Navigate to the directory containing the `milvus-cdc` binary and the `configs` directory with the `cdc.yaml` file, then start the server:
+
+ ```bash
+ # dir tree
+ .
+ ├── milvus-cdc # build from source code or download from release page
+ ├── configs
+ │ └── cdc.yaml # config for cdc and source milvus
+
+ # start milvus cdc
+ ./milvus-cdc server
+ ```
+
+- Using Docker Compose:
+
+ ```bash
+ docker-compose up -d
+ ```
+
diff --git a/preview/site/en/userGuide/tools/milvus-cdc/manage-cdc-tasks.md b/preview/site/en/userGuide/tools/milvus-cdc/manage-cdc-tasks.md
new file mode 100644
index 000000000..bdec15b94
--- /dev/null
+++ b/preview/site/en/userGuide/tools/milvus-cdc/manage-cdc-tasks.md
@@ -0,0 +1,244 @@
+---
+id: manage-cdc-tasks.md
+order: 3
+summary: A Capture Data Change (CDC) task enables the synchronization of data from a source Milvus instance to a target Milvus instance.
+title: Manage CDC Tasks
+---
+
+# Manage CDC Tasks
+
+A Capture Data Change (CDC) task enables the synchronization of data from a source Milvus instance to a target Milvus instance. It monitors operation logs from the source and replicates data changes such as insertions, deletions, and index operations to the target in real-time. This facilitates real-time disaster recovery or active-active load balancing between Milvus deployments.
+
+This guide covers how to manage CDC tasks, including creation, pausing, resuming, retrieving details, listing, and deletion through HTTP requests.
+
+## Create a task
+
+Creating a CDC task allows data change operations in the source Milvus to be synced to the target Milvus.
+
+To create a CDC task:
+
+```bash
+curl -X POST http:_//localhost:8444/cdc \
+-H "Content-Type: application/json" \
+-d '{
+ "request_type": "create",
+ "request_data": {
+ "milvus_connect_param": {
+ "host": "localhost",
+ "port": 19530,
+ "username": "root",
+ "password": "Milvus",
+ "enable_tls": false,
+ "connect_timeout": 10
+ },
+ "collection_infos": [
+ {
+ "name": "*"
+ }
+ ],
+ "rpc_channel_info": {
+ "name": "by-dev-replicate-msg"
+ }
+ }
+}'
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+__Parameters__:
+
+- __milvus_connect_param__: Connection parameters of the target Milvus.
+
+ - __host__: Hostname or IP address of the Milvus server.
+
+ - __port__: Port number the Milvus server listens on.
+
+ - __username__: Username for authenticating with the Milvus server.
+
+ - __password__: Password for authenticating with the Milvus server.
+
+ - __enable_tls__: Whether to use TLS/SSL encryption for the connection.
+
+ - __connect_timeout__: Timeout period in seconds for establishing the connection.
+
+- __collection_infos__: Collections to synchronize. Currently, only an asterisk (__*__) is supported, as Milvus-CDC synchronizes at the cluster level, not individual collections.
+
+- __rpc_channel_info__: RPC channel name for synchronization, constructed by concatenating the values of __common.chanNamePrefix.cluster__ and __common.chanNamePrefix.replicateMsg__ from the source Milvus configuration, separated by a hyphen (__-__).
+
+Expected response:
+
+```json
+{
+ "code": 200,
+ "data": {
+ "task_id":"xxxx"
+ }
+}
+```
+
+## List tasks
+
+To list all created CDC tasks:
+
+```bash
+curl -X POST -H "Content-Type: application/json" -d '{
+ "request_type": "list"
+}' http://localhost:8444/cdc
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+Expected response:
+
+```json
+{
+ "code": 200,
+ "data": {
+ "tasks": [
+ {
+ "task_id": "xxxxx",
+ "milvus_connect_param": {
+ "host": "localhost",
+ "port": 19530,
+ "connect_timeout": 10
+ },
+ "collection_infos": [
+ {
+ "name": "*"
+ }
+ ],
+ "state": "Running"
+ }
+ ]
+ }
+}
+```
+
+## Pause a task
+
+To pause a CDC task:
+
+```bash
+curl -X POST -H "Content-Type: application/json" -d '{
+ "request_type":"pause",
+ "request_data": {
+ "task_id": "xxxx"
+ }
+}' http://localhost:8444/cdc
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+__Parameters__:
+
+- __task_id__: ID of the CDC task to pause.
+
+Expected response:
+
+```bash
+{
+ "code": 200,
+ "data": {}
+}
+```
+
+## Resume a task
+
+To resume a paused CDC task:
+
+```bash
+curl -X POST -H "Content-Type: application/json" -d '{
+ "request_type":"resume",
+ "request_data": {
+ "task_id": "xxxx"
+ }
+}' http://localhost:8444/cdc
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+__Parameters__:
+
+- __task_id__: ID of the CDC task to resume.
+
+Expected response:
+
+```bash
+{
+ "code": 200,
+ "data": {}
+}
+```
+
+## Retrieve task details
+
+To retrieve the details of a specific CDC task:
+
+```bash
+curl -X POST -H "Content-Type: application/json" -d '{
+ "request_type":"get",
+ "request_data": {
+ "task_id": "xxxx"
+ }
+}' http://localhost:8444/cdc
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+__Parameters__:
+
+- __task_id__: ID of the CDC task to query.
+
+Expected response:
+
+```bash
+{
+ "code": 200,
+ "data": {
+ "Task": {
+ "collection_infos": [
+ {
+ "name": "*"
+ }
+ ],
+ "milvus_connect_param": {
+ "connect_timeout": 10,
+ "enable_tls": true,
+ "host": "localhost",
+ "port": 19530
+ },
+ "state": "Running",
+ "task_id": "xxxx"
+ }
+ }
+}
+```
+
+## Delete a task
+
+To delete a CDC task:
+
+```bash
+curl -X POST -H "Content-Type: application/json" -d '{
+ "request_type":"delete",
+ "request_data": {
+ "task_id": "30d1e325df604ebb99e14c2a335a1421"
+ }
+}' http://localhost:8444/cdc
+```
+
+Replace __localhost__ with the IP address of the target Milvus server.
+
+__Parameters__:
+
+- __task_id__: ID of the CDC task to delete.
+
+Expected response:
+
+```json
+{
+ "code": 200,
+ "data": {}
+}
+```
+
diff --git a/preview/site/en/userGuide/tools/milvus-cdc/milvus-cdc-overview.md b/preview/site/en/userGuide/tools/milvus-cdc/milvus-cdc-overview.md
new file mode 100644
index 000000000..3b2548125
--- /dev/null
+++ b/preview/site/en/userGuide/tools/milvus-cdc/milvus-cdc-overview.md
@@ -0,0 +1,69 @@
+---
+id: milvus-cdc-overview.md
+order: 1
+summary: Milvus-CDC is a user-friendly tool that can capture and synchronize incremental data in Milvus instances.
+title: CDC Overview
+---
+
+# Overview
+
+Milvus-CDC is a user-friendly tool that can capture and synchronize incremental data in Milvus instances. It ensures the reliability of business data by seamlessly transferring it between source and target instances, allowing for easy incremental backup and disaster recovery.
+
+## Key capabilities
+
+- __Sequential Data Synchronization__: Ensures data integrity and consistency by synchronizing data changes sequentially between Milvus instances.
+
+- __Incremental Data Replication__: Replicates incremental data, including insertions and deletions, from source Milvus to target Milvus, offering persistent storage.
+
+- __CDC Task Management__: Allows for the management of CDC tasks through OpenAPI requests, including creating, querying status, and deleting CDC tasks.
+
+Additionally, we are planning to expand our capabilities to include support for integration with stream processing systems in the future.
+
+## Architecture
+
+Milvus-CDC adopts an architecture with two main components - an HTTP server that manages tasks and metadata, and __corelib__ that synchronizes task execution with a reader that obtains data from the source Milvus instance and a writer that sends processed data to the target Milvus instance.
+
+
+
+In the preceding diagram,
+
+- __HTTP server__: Handles user requests, executes tasks, and maintains metadata. It serves as the control plane for task orchestration within the Milvus-CDC system.
+
+- __Corelib__: Responsible for the actual synchronization of tasks. It includes a reader component that retrieves information from the source Milvus's etcd and message queue (MQ), and a writer component that translates messages from the MQ into API parameters for the Milvus system and sends these requests to the target Milvus to complete the synchronization process.
+
+## Workflow
+
+The Milvus-CDC data processing flow involves the following steps:
+
+1. __Task creation__: Users initiate a CDC task via HTTP requests.
+
+1. __Metadata retrieval__: The system fetches collection-specific metadata from the source Milvus's etcd, including channel and checkpoint information for the collection.
+
+1. __MQ connection__: With the metadata at hand, the system connects to the MQ to begin subscribing to the data stream.
+
+1. __Data processing__: Data from MQ is read, parsed, and either passed on using the Go SDK or processed to replicate operations performed in the source Milvus.
+
+
+
+## Limits
+
+- __Incremental Data Synchronization__: As of now, Milvus-CDC is designed to synchronize only incremental data. If your business requires a full data backup, please [reach out to us](https://milvus.io/community) for assistance.
+
+- __Synchronization Scope__: Currently, Milvus-CDC can synchronize data at the cluster level. We are working on adding support for collection-level data synchronization in upcoming releases.
+
+- __Supported API Requests__: Milvus-CDC currently supports the following API requests. We plan to extend support to additional requests in future releases:
+
+ - Create/Drop Collection
+
+ - Insert/Delete/Upsert
+
+ - Create/Drop Partition
+
+ - Create/Drop Index
+
+ - Load/Release/Flush
+
+ - Load/Release Partition
+
+ - Create/Drop Database
+
diff --git a/preview/site/en/userGuide/tools/milvus_backup-api.md b/preview/site/en/userGuide/tools/milvus_backup_api.md
similarity index 97%
rename from preview/site/en/userGuide/tools/milvus_backup-api.md
rename to preview/site/en/userGuide/tools/milvus_backup_api.md
index 71861c36c..7d9715430 100644
--- a/preview/site/en/userGuide/tools/milvus_backup-api.md
+++ b/preview/site/en/userGuide/tools/milvus_backup_api.md
@@ -1,6 +1,7 @@
---
-id: milvus_backup-api.md
+id: milvus_backup_api.md
summary: Learn how to use Milvus Backup through API
+title: Back up and Restore Data Using APIs
---
# Back up and Restore Data Using APIs
@@ -79,6 +80,8 @@ This step is optional. If you skip this, ensure that you already have some data
## Back up data
+
+
Note that running Milvus Backup against a Milvus instance will not normally affect the running of the instance. Your Milvus instance is fully functional during backup or restore.
Run the following command to create a backup. Change `collection_names` and `backup_name` if necessary.
@@ -115,6 +118,8 @@ Now, you can save the backup files to a safe place for restoration in the future
## Restore data
+
+
You can call the restore API command with a `collection_suffix` option to create a new collection by restoring the data from the backup. Change `collection_names` and `backup_name` if necessary.
```shell
@@ -172,4 +177,3 @@ python example/verify_data.py
Note that the above script assumes that you have run the `restore` command with the `-s` flag and the suffix is set to `-recover`. Feel free to make necessary changes to the script to fit your need.
-
diff --git a/preview/site/en/userGuide/tools/milvus_backup-cli.md b/preview/site/en/userGuide/tools/milvus_backup_cli.md
similarity index 83%
rename from preview/site/en/userGuide/tools/milvus_backup-cli.md
rename to preview/site/en/userGuide/tools/milvus_backup_cli.md
index 3ee182d8b..14fc32e74 100644
--- a/preview/site/en/userGuide/tools/milvus_backup-cli.md
+++ b/preview/site/en/userGuide/tools/milvus_backup_cli.md
@@ -1,6 +1,7 @@
---
-id: milvus_backup-cli.md
+id: milvus_backup_cli.md
summary: Learn how to use Milvus Backup through CLI
+title: Back up and Restore Data Using Commands
---
# Back up and Restore Data Using Commands
@@ -45,7 +46,7 @@ The name of the default Minio bucket varies with the way you install Milvus. Whe
| field | Docker Compose | Helm / Milvus Operator |
| --------------- | -------------- | ---------------------- |
| `bucketName` | a-bucket | milvus-bucket |
-| `rootPath` | files | file |
+| `rootPath` | files | file |
Syntax
- -```shell -load_balance -s (int) -d (int) -ss (int) [-t (int)] -``` - -Options
- -| Option | Full name | Description | -| :----- | :------------------- | :----------------------------------------------------------------------- | -| -s | --src-node-id | The ID of the source query node to be balanced. | -| -d | --dst-node-id | (Multiple) The ID of the destination query node to transfer segments to. | -| -ss | --sealed-segment-ids | (Multiple) The ID of the sealed segment to be transferred. | -| -t | --timeout | (Optional) The timeout in seconds. | -| --help | n/a | Displays help for using the command. | - ## query Shows query results that match all the criteria that you enter. @@ -806,7 +767,6 @@ timeout []: Guarantee timestamp. This instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date. [0]: Graceful time. Only used in bounded consistency level. If graceful_time is set, PyMilvus will use current timestamp minus the graceful_time as the guarantee_timestamp. This option is 5s by default if not set. [5]: -Travel timestamp. Users can specify a timestamp in a search to get results based on a data view at a specified point in time. [0]: 428960801420883491 ```Example 2
@@ -830,7 +790,6 @@ timeout []: Guarantee timestamp. This instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date. [0]: Graceful time. Only used in bounded consistency level. If graceful_time is set, PyMilvus will use current timestamp minus the graceful_time as the guarantee_timestamp. This option is 5s by default if not set. [5]: -Travel timestamp. Users can specify a timestamp in a search to get results based on a data view at a specified point in time. [0]: 428960801420883491 ``` ## release @@ -882,8 +841,6 @@ out headers): examples/import_csv/search_vectors.csv The vector field used to search of collection (vector): vector -Metric type: L2 - Search parameter nprobe's value: 10 The max number of returned record, also known as topk: 2 @@ -896,7 +853,6 @@ timeout []: Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]: -Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]: ```Example 2
@@ -913,8 +869,6 @@ The vectors of search data(the length of data is number of query (nq), the dim o The vector field used to search of collection (vector): vector -Metric type: L2 - Search parameter nprobe's value: 10 The specified number of decimal places of returned distance [-1]: 5 @@ -929,7 +883,6 @@ timeout []: Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]: -Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]: ```Example 3
@@ -957,24 +910,22 @@ timeout []: Guarantee Timestamp(It instructs Milvus to see all operations performed before a provided timestamp. If no such timestamp is provided, then Milvus will search all operations performed to date) [0]: -Travel Timestamp(Specify a timestamp in a search to get results based on a data view) [0]: ``` -## show connection +## List connection -Shows the current connection. +List connections.Syntax
```shell -show connection [-a] +list connections ```Options
| Option | Full name | Description | | :----- | :-------- | :---------------------------------------- | -| -a | --all | （Optional) Flag to show all connections. | | --help | n/a | Displays help for using the command. | ## show index_progress @@ -1012,23 +963,6 @@ show loading_progress -c (text) [-p (text)] |-p|--partition|(Optional/Multiple) The name of the loading partition.| |--help|n/a|Displays help for using the command.| -## show query_segment - -Shows the segment information of a collection. - -Syntax
- -```shell -show query_segment -c (text) [-t (float)] -``` - -Options
- -|Option|Full name|Description -|:---|:---|:---| -|-c|--collection-name|The name of the collection.| -|-t|--timeout|(Optional) The maximum allowed duration in seconds of an RPC call. Not passing this option indicates that the client keeps waiting until the server responds or an error occurs.| -|--help|n/a|Displays help for using the command.| ## version @@ -1051,5 +985,5 @@ version ```shell $ milvus_cli --version -Milvus_CLI v0.1.7 +Milvus_CLI v0.4.0 ``` diff --git a/preview/site/en/userGuide/tools/cli_overview.md b/preview/site/en/userGuide/tools/cli_overview.md index 0ebbc2794..29321f455 100644 --- a/preview/site/en/userGuide/tools/cli_overview.md +++ b/preview/site/en/userGuide/tools/cli_overview.md @@ -1,6 +1,7 @@ --- id: cli_overview.md summary: Milvus Command-Line Interface (CLI) is a command-line tool that supports database connection, data operations, and import and export of data. +title: Milvus Command-Line Interface --- # Milvus Command-Line Interface @@ -24,11 +25,15 @@ In the following table, you can find the recommended versions of PyMilvus and Mi | 2.0.0-RC8 | 2.0.0rc8 | 0.1.8 | | 2.0.0-RC9 | 2.0.0rc9 | 0.1.9 | | 2.1.0 | 2.1.0 | 0.3.0 | -| 2.2.x | 2.2.x | 0.3.2 | +| 2.2.x | 2.2.x | 0.4.0 | +| 2.3.x | 2.3.x | 0.4.2 | +| 2.4.x | 2.4.x | 1.0.0 | + +Milvus 2.0.0-RC7 and later are not backward compatible with 2.0.0-RC6 and earlier due to changes made to storage formats.
## Current version
-The current version of Milvus_CLI is 0.3.2.
+The current version of Milvus_CLI is 1.0.0.
To find your installed version and see if you need to update, run `milvus_cli --version`.
diff --git a/preview/site/en/userGuide/tools/install_cli.md b/preview/site/en/userGuide/tools/install_cli.md
index 467178de5..c1d6fdf4b 100644
--- a/preview/site/en/userGuide/tools/install_cli.md
+++ b/preview/site/en/userGuide/tools/install_cli.md
@@ -1,6 +1,7 @@
---
id: install_cli.md
summary: Learn how to install Milvus_CLI.
+title: Install Milvus_CLI
---
# Install Milvus_CLI
diff --git a/preview/site/en/userGuide/tools/milvus-cdc/cdc-monitoring.md b/preview/site/en/userGuide/tools/milvus-cdc/cdc-monitoring.md
new file mode 100644
index 000000000..74f4ffd42
--- /dev/null
+++ b/preview/site/en/userGuide/tools/milvus-cdc/cdc-monitoring.md
@@ -0,0 +1,43 @@
+---
+id: cdc-monitoring.md
+order: 4
+summary: Milvus-CDC provides comprehensive monitoring capabilities through Grafana dashboards.
+title: Monitoring
+---
+
+# Monitoring
+
+Milvus-CDC provides comprehensive monitoring capabilities through Grafana dashboards, allowing you to visualize key metrics and ensure the smooth operation of your Change Data Capture (CDC) tasks and server health.
+
+### Metrics for CDC tasks
+
+To get started, import the [cdc-grafana.json](https://github.com/zilliztech/milvus-cdc/blob/main/server/configs/cdc-grafana.json) file into Grafana. This will add a dashboard specifically designed for monitoring the status of CDC tasks.
+
+__CDC Grafana Dashboard Overview__:
+
+
+
+__Key Metrics Explained:__
+
+- __Task__: Number of CDC tasks in different states, including __Initial__, __Running__, and __Paused__.
+
+- __Request Total__: Total number of requests received by Milvus-CDC.
+
+- __Request Success__: Number of successful requests received by Milvus-CDC.
+
+- __task num__: Number of tasks in __Initial__, __Paused__, and __Running__ states over time.
+
+- __task state__: State of individual tasks.
+
+- __request count__: Number of successful and total requests
+
+- __request latency__: Latency of requests through p99, average and other statistics.
+
+- __replicate data rate__: Replication data rate for read/write operations
+
+- __replicate tt lag__: Replication time lag for read/write operations.
+
+- __api execute count__: Number of times different Milvus-CDC APIs were executed.
+
+- __center ts__: Timestamp for read/write tasks.
+
diff --git a/preview/site/en/userGuide/tools/milvus-cdc/deploy-cdc-server.md b/preview/site/en/userGuide/tools/milvus-cdc/deploy-cdc-server.md
new file mode 100644
index 000000000..74d5d505c
--- /dev/null
+++ b/preview/site/en/userGuide/tools/milvus-cdc/deploy-cdc-server.md
@@ -0,0 +1,151 @@
+---
+id: deploy-cdc-server.md
+order: 2
+summary: This guide provides a step-by-step process for deploying a Milvus-CDC server.
+title: Deploy CDC Server
+---
+
+# Deploy CDC Server
+
+This guide provides a step-by-step process for deploying a Milvus-CDC server.
+
+## Prerequisites
+
+Ensure the following conditions are met before deploying a Milvus-CDC server:
+
+- __Milvus Instances__: Both the source Milvus and at least one target Milvus should be deployed and operational.
+
+ - Both the source and target Milvus versions must be 2.3.2 or higher, preferably 2.4.x. We recommend uisng the same version for the source and target Milvus to ensure compatibility.
+
+ - Set the `common.ttMsgEnabled` configuration of the target Milvus to `false`.
+
+ - Configure the source and target Milvus with distinct meta and message storage settings to prevent conflicts. For instance, avoid using the same etcd and rootPath configurations, as well as identical Pulsar services and `chanNamePrefix` in multiple Milvus instances.
+
+- __Metastore__: Have an etcd or MySQL database ready for the Milvus-CDC metastore.
+
+## Steps
+
+### Obtain the Milvus-CDC config file
+
+Clone the [Milvus-CDC repo](https://github.com/zilliztech/milvus-cdc) and navigate to the `milvus-cdc/server/configs` directory to access the `cdc.yaml` config file.
+
+```bash
+git clone https://github.com/zilliztech/milvus-cdc.git
+
+cd milvus-cdc/server/configs
+```
+
+### Edit the config file
+
+In the `milvus-cdc/server/configs` directory, modify the `cdc.yaml` file to customize configurations related to the Milvus-CDC metastore and connection details of the source Milvus.
+
+- __Metastore Configuration__:
+
+ - `metaStoreConfig.storeType`: Type of metastore for Milvus-CDC. Possible values are `etcd` or `mysql`.
+
+ - `metaStoreConfig.etcdEndpoints`: Address for connecting to the etcd of Milvus-CDC. Required if `storeType` is set to `etcd`.
+
+ - `metaStoreConfig.mysqlSourceUrl`: Connection address of the MySQL database for the Milvus-CDC server. Required if `storeType` is set to `mysql`.
+
+ - `metaStoreConfig.rootPath`: Root path of the Milvus-CDC metastore. This configuration enables multi-tenancy, allowing multiple CDC services to utilize the same etcd or MySQL instance while achieving isolation through different root paths.
+
+ Example configuration:
+
+ ```yaml
+ # cdc meta data config
+ metaStoreConfig:
+ # the metastore type, available value: etcd, mysql
+ storeType: etcd
+ # etcd address
+ etcdEndpoints:
+ - localhost:2379
+ # mysql connection address
+ # mysqlSourceUrl: root:root@tcp(127.0.0.1:3306)/milvus-cdc?charset=utf8
+ # meta data prefix, if multiple cdc services use the same store service, you can set different rootPaths to achieve multi-tenancy
+ rootPath: cdc
+ ```
+
+- __Source Milvus Configuration:__
+
+ Specify the connection details of the source Milvus, including etcd and message storage, to establish a connection between the Milvus-CDC server and the source Milvus.
+
+ - `sourceConfig.etcdAddress`: Address for connecting to the etcd of the source Milvus. For more information, refer to [etcd-related Configurations](https://milvus.io/docs/configure_etcd.md#etcd-related-Configurations).
+
+ - `sourceConfig.etcdRootPath`: Root prefix of the key where the source Milvus stores data in etcd. The value may vary based on the deployment method of the Milvus instance:
+
+ - __Helm__ or __Docker Compose__: Defaults to `by-dev`.
+
+ - __Operator__: Defaults to `Install with Docker ComposeSQLite-based MilvusInstall with Helm ChartMySQL-based MilvusInstall with PackageFAISS
+
Run the following command to create a backup.
@@ -94,7 +95,7 @@ Now, you can save the backup files to a safe place for restoration in the future
## Restore data
-Install with Docker ComposeSQLite-based MilvusInstall with Helm ChartMySQL-based MilvusInstall with PackageFAISS
+
You can run the `restore` command with the `-s` flag to create a new collection by restoring the data from the backup:
@@ -128,4 +129,3 @@ Note that the above script assumes that you have run the `restore` command with
-
diff --git a/preview/site/en/userGuide/tools/milvus_backup_overview.md b/preview/site/en/userGuide/tools/milvus_backup_overview.md
index c8af7c39c..568786570 100644
--- a/preview/site/en/userGuide/tools/milvus_backup_overview.md
+++ b/preview/site/en/userGuide/tools/milvus_backup_overview.md
@@ -1,6 +1,7 @@
---
id: milvus_backup_overview.md
-summary: Milvus-Backup is a tool that allows users to backup and restore Milvus data.
+summary: Milvus-Backup is a tool that allows users to backup and restore Milvus data.
+title: Milvus Backup
---
# Milvus Backup
@@ -14,6 +15,16 @@ Before start using Milvus Backup, ensure that
- The operating system is CentOS 7.5+ or Ubuntu LTS 18.04+,
- Go version is 1.20.2 or later.
+## Architecture
+
+
+
+Milvus Backup facilitates backup and restore of metadata, segments, and data across Milvus instances. It provides northbound interfaces, such as CLI, API, and gRPC-based Go module, for flexible manipulation of the backup and restore processes.
+
+Milvus Backup reads collection metadata and segments from the source Milvus instance to create a backup. It then copies collection data from the root path of the source Milvus instance and saves the copied data into the backup root path.
+
+To restore from a backup, Milvus Backup creates a new collection in the target Milvus instance based on the collection metadata and segment information in the backup. It then copies the backup data from the backup root path to the root path of the target instance.
+
## Latest release
-- [v0.0.2](https://github.com/zilliztech/milvus-backup/releases/tag/v0.2.0)
\ No newline at end of file
+- [v0.4.12](https://github.com/zilliztech/milvus-backup/releases/tag/v0.4.12)
diff --git a/preview/site/en/userGuide/tools/milvus_migration-faiss.md b/preview/site/en/userGuide/tools/milvus_migration-faiss.md
deleted file mode 100644
index c523e710b..000000000
--- a/preview/site/en/userGuide/tools/milvus_migration-faiss.md
+++ /dev/null
@@ -1,87 +0,0 @@
----
-id: milvus_migration-faiss.md
-label: FAISS
-order: 3
-group: milvus_migration-sqlite.md
-summary: Learn how to use Milvus Migration
----
-
-# Migrate Data from FAISS to Milvus 2.x
-
-Due to architecture changes, Milvus offers a migration tool for you to safely migrate data from Milvus 1.x (including Milvus 0.9.x) to Milvus 2.x.
-
-## Obtain Milvus Migration
-
-You can either download the compiled binary or build from the source.
-
-To download the compiled binary, go to the [release](https://github.com/zilliztech/milvus-migration/releases) page, where you can find all official releases. Remember, always use the binaries in the release marked as **Latest**.
-
-You should choose the binary that fits the architecture of your operating system.
-
-To compile from the source, do as follows:
-
-```shell
-git clone git@github.com:zilliztech/milvus-migration.git
-go build
-```
-
-## Prepare migration
-
-Create a configuration file, name it `migration.yaml`, and place it in the folder where the downloaded or built binary resides.
-
-
-
-```yaml
-dumper:
- worker:
- limit: 20 # number of dumper threads run concurrently
- workMode: faiss
- reader:
- bufferSize: 1024 # in KB
- writer:
- bufferSize: 1024 # in KB
-loader:
- worker:
- limit: 20 # number of loader threads run concurrently
-source:
- mode: local
- local:
- faissFile: /data/faiss.index
-target:
- mode: remote
- remote:
- outputDir: output/ # do not prefix it with a forward slash (/).
- cloud: aws
- endpoint: 127.0.0.1:9000
- region: ap-southeast-1
- bucket: a-bucket
- ak: minioadmin
- sk: minioadmin
- useIAM: false
- useSSL: false
- checkBucket: true
- milvus2x: # Milvus 2.x connection endpoint and its authentication credentials
- endpoint: xxxxxx:19530
- username: xxxxx
- password: xxxxx
- create:
- collection:
- name: test
- shardsNum: 2
- dim: 512
- metricType: L2
-```
-
-For details on configuration items, see [Milvus Migration Configuration Reference](milvus_migration_conf.md).
-
-## Migrate data
-
-Once you prepared the configuration file and placed it alongside the Milvus Migration binary, you can start migrating data from a legacy Milvus instance or FAISS to a Milvus 2.x instance.
-
-```shell
-# Dump the data from a legacy Milvus instance
-./milvus-migration dump
-
-# Load the data to a Milvus 2.x instance
-./milvus-migration load
-```
\ No newline at end of file
diff --git a/preview/site/en/userGuide/tools/milvus_migration-mysql.md b/preview/site/en/userGuide/tools/milvus_migration-mysql.md
deleted file mode 100644
index 55ec38047..000000000
--- a/preview/site/en/userGuide/tools/milvus_migration-mysql.md
+++ /dev/null
@@ -1,84 +0,0 @@
----
-id: milvus_migration-mysql.md
-label: MySQL-based Milvus
-order: 2
-group: milvus_migration-sqlite.md
-summary: Learn how to use Milvus Migration
----
-
-# Migrate Data from MySQL-based Milvus to Milvus 2.x
-
-Due to architecture changes, Milvus offers a migration tool for you to safely migrate data from Milvus 1.x (including Milvus 0.9.x) to Milvus 2.x.
-
-## Obtain Milvus Migration
-
-You can either download the compiled binary or build from the source.
-
-To download the compiled binary, go to the [release](https://github.com/zilliztech/milvus-migration/releases) page, where you can find all official releases. Remember, always use the binaries in the release marked as **Latest**.
-
-You should choose the binary that fits the architecture of your operating system.
-
-To compile from the source, do as follows:
-
-```shell
-git clone git@github.com:zilliztech/milvus-migration.git
-go build
-```
-
-## Prepare migration
-
-Create a configuration file, name it `migration.yaml`, and place it in the folder where the downloaded or built binary resides.
-
-
-
-```yaml
-dumper:
- worker:
- limit: 20 # number of dumper threads run concurrently
- workMode: milvus1x
- reader:
- bufferSize: 1024 # in KB
- writer:
- bufferSize: 1024 # in KB
-loader:
- worker:
- limit: 20 # number of loader threads run concurrently
-meta:
- mode: mysql
- mysqlUrl: "user:password@tcp(localhost:3306)/milvus?charset=utf8mb4&parseTime=True&loc=Local"
-source:
- mode: local
- local:
- tablesDir: /db/tables/
-target:
- mode: remote
- remote:
- outputDir: output/ # do not prefix it with a forward slash (/).
- cloud: aws
- endpoint: 127.0.0.1:9000
- region: ap-southeast-1
- bucket: a-bucket
- ak: minioadmin
- sk: minioadmin
- useIAM: false
- useSSL: false
- checkBucket: true
- milvus2x: # Milvus 2.x connection endpoint and its authentication credentials
- endpoint: xxxxxx:19530
- username: xxxxx
- password: xxxxx
-```
-
-For details on configuration items, see [Milvus Migration Configuration Reference](milvus_migration_conf.md).
-
-## Migrate data
-
-Once you prepared the configuration file and placed it alongside the Milvus Migration binary, you can start migrating data from a legacy Milvus instance or FAISS to a Milvus 2.x instance.
-
-```shell
-# Dump the data from a legacy Milvus instance
-./milvus-migration dump
-
-# Load the data to a Milvus 2.x instance
-./milvus-migration load
-```
\ No newline at end of file
diff --git a/preview/site/en/userGuide/tools/milvus_migration-sqlite.md b/preview/site/en/userGuide/tools/milvus_migration-sqlite.md
deleted file mode 100644
index 94880c08e..000000000
--- a/preview/site/en/userGuide/tools/milvus_migration-sqlite.md
+++ /dev/null
@@ -1,84 +0,0 @@
----
-id: milvus_migration-sqlite.md
-label: SQLite-based Milvus
-order: 1
-group: milvus_migration-sqlite.md
-summary: Learn how to use Milvus Migration
----
-
-# Migrate Data from SQLite-based Milvus to Milvus 2.x
-
-Due to architecture changes, Milvus offers a migration tool for you to safely migrate data from Milvus 1.x (including Milvus 0.9.x) to Milvus 2.x.
-
-## Obtain Milvus Migration
-
-You can either download the compiled binary or build from the source.
-
-To download the compiled binary, go to the [release](https://github.com/zilliztech/milvus-migration/releases) page, where you can find all official releases. Remember, always use the binaries in the release marked as **Latest**.
-
-You should choose the binary that fits the architecture of your operating system.
-
-To compile from the source, do as follows:
-
-```shell
-git clone git@github.com:zilliztech/milvus-migration.git
-go build
-```
-
-## Prepare migration
-
-Create a configuration file, name it `migration.yaml`, and place it in the folder where the downloaded or built binary resides.
-
-
-
-```yaml
-dumper:
- worker:
- limit: 20 # number of dumper threads run concurrently
- workMode: milvus1x
- reader:
- bufferSize: 1024 # in KB
- writer:
- ufferSize: 1024 # in KB
-loader:
- worker:
- limit: 20 # number of loader threads run concurrently
-meta:
- mode: sqlite
- sqliteFile: /db/meta.sqlite
-source:
- mode: local
- local:
- tablesDir: /db/tables/
-target:
- mode: remote
- remote:
- outputDir: output/ # do not prefix it with a forward slash (/).
- cloud: aws
- endpoint: 127.0.0.1:9000
- region: ap-southeast-1
- bucket: a-bucket
- ak: minioadmin
- sk: minioadmin
- useIAM: false
- useSSL: false
- checkBucket: true
- milvus2x: # Milvus 2.x connection endpoint and its authentication credentials
- endpoint: xxxxxx:19530
- username: xxxxx
- password: xxxxx
-```
-
-For details on configuration items, see [Milvus Migration Configuration Reference](milvus_migration_conf.md).
-
-## Migrate data
-
-Once you prepared the configuration file and placed it alongside the Milvus Migration binary, you can start migrating data from a legacy Milvus instance or FAISS to a Milvus 2.x instance.
-
-```shell
-# Dump the data from a legacy Milvus instance
-./milvus-migration dump
-
-# Load the data to a Milvus 2.x instance
-./milvus-migration load
-```
\ No newline at end of file
diff --git a/preview/site/en/userGuide/tools/milvus_migration_conf.md b/preview/site/en/userGuide/tools/milvus_migration_conf.md
deleted file mode 100644
index 49d54f942..000000000
--- a/preview/site/en/userGuide/tools/milvus_migration_conf.md
+++ /dev/null
@@ -1,61 +0,0 @@
----
-id: milvus_migration_conf.md
-summary: Milvus Migration is a migration tool that allows you to migrate data from Milvus 0.9.x through 1.x or FAISS to Milvus 2.x.
----
-
-# Milvus Migration Configuration Reference
-
-### `dumper`
-
-| Parameter | Description | Example |
-|---------------------------------|-----------------------------------------------------|-----------------------------------------------------------------|
-| dumper.worker.workMode | Work mode for milvus-migration dumper | milvus1x: dump data from Milvus1.x; faiss: dump data from Faiss |
-| dumper.worker.limit | The number of dumper threads to run concurrently | 20: means to dump 20 segment files simultaneously |
-| dumper.worker.reader.bufferSize | The buffer size for each segment file reader, in KB | 1024 |
-| dumper.worker.writer.bufferSize | The buffer size for each segment file writer, in KB | 1024 |
-
-### `loader`
-
-| Parameter | Description | Example |
-|---------------------|-------------------------------|---------------------------------------------------|
-| loader.worker.limit | Concurrency of loader threads | 20: means load 20 segments files at the same time |
-
-### `meta`
-
-| parameter | description | example |
-|-----------------|-----------------------------------------------------------------------|-------------------------------------------------------------------------------------|
-| meta.mode | Where to read the source meta information from | mock/mysql/sqlite/remote |
-| meta.mockFile | When meta.mode is mock, read milvus1.x meta info from local meta.json | |
-| meta.sqliteFile | When meta.mode is mysql, read milvus1.x meta info from meta.sqlite | /home/root/milvus/db/meta.sqlite |
-| meta.mysqlUrl | When meta.mode is sqlite, read milvus1.x meta info from mysql | "user:password@tcp(localhost:3306)/milvus?charset=utf8mb4&parseTime=True&loc=Local" |
-
-### `source`
-
-| parameter | description | example |
-|------------------------|---------------------------------------------------|---------------------------------------------------------------|
-| source.mode | Where the source files are read from | local: read files from local disk, remote: read files from S3 |
-| source.local.faissFile | faissFile position | /db/faiss.index |
-| source.local.tablesDir | Position of the Milvus 0.9.x~1.x tables directory | /home/${user}/milvus/db/tables |
-
-### `target`
-
-| parameter | description | example |
-|-------------------------------------|------------------------------------------------------|---------------------------------------------------------------------------|
-| target.mode | Where to store the dumped files | local: store dumped files on local disk; remote: store dumped files on S3 |
-| target.remote.outputDir | Directory path in bucket where to store files | output/ |
-| target.remote.cloud | Storage in Milvus 2.x | aws (if using Minio, use aws), GCP, or Azure |
-| target.remote.endpoint | Endpoint of the Milvus 2.x storage | 127.0.0.1:9000 |
-| target.remote.region | Region of the Milvus 2.x storage | If using local Minio, can use any value |
-| target.remote.bucket | Bucket of the Milvus 2.x storage | Must use the same bucket as configured in milvus.yaml for Milvus 2.x |
-| target.remote.ak | Access Key of the Milvus 2.x storage | minioadmin |
-| target.remote.sk | Secret Key of the Milvus 2.x storage | minioadmin |
-| target.remote.useIAM | Whether to use IAM Role to connect to Milvus 2.x | false |
-| target.remote.useSSL | Whether to use SSL when connecting to Milvus 2.x | For local Minio, use false; for remote S3, use true |
-| target.remote.checkBucket | Whether to check if the bucket exists in the storage | True to check if you can connect to the Milvus 2.x storage |
-| target.milvus2x.endpoint | Endpoint of Milvus 2.x | xxxxxx:19530 |
-| target.milvus2x.username | Username of Milvus 2.x | root |
-| target.milvus2x.password | Password of Milvus 2.x | xxxxxxx |
-| target.create.collection.name | milvus2.x createCollection param name | collection_name |
-| target.create.collection.shardsNum | milvus2.x createCollection param shardsNum | default is 2 |
-| target.create.collection.dim | milvus2.x createCollection param dim | must same with faiss.index data's dim |
-| target.create.collection.metricType | milvus2.x createCollection param metricType | metricType: L2 or IP now
\ No newline at end of file
diff --git a/preview/site/en/userGuide/tools/milvus_migration_overview.md b/preview/site/en/userGuide/tools/milvus_migration_overview.md
deleted file mode 100644
index 946de8bdf..000000000
--- a/preview/site/en/userGuide/tools/milvus_migration_overview.md
+++ /dev/null
@@ -1,29 +0,0 @@
----
-id: milvus_migration_overview.md
-summary: Milvus Migration is a migration tool that allows you to migrate data from Milvus 0.9.x through 1.x or FAISS to Milvus 2.x.
----
-
-# Milvus Migration
-
-Milvus Migration is a data migration tool that allows you to migrate data from Milvus 0.9.x through 1.x or FAISS to Milvus 2.x.
-
-## Prerequisites
-
-Before start using Milvus Migration, ensure that
-
-- The operating system is CentOS 7.5+ or Ubuntu LTS 18.04+,
-- Milvus version is 0.9.x, 1.x, or 2.x, and
-- Go version is 1.20.2 or later.
-
-And acknowledge that Milvus Migration now offers
-
-- Official support for data migration from Milvus 0.9.x and Milvus 1.x to Milvus 2.x, and
-- Beta support for data migration from FAISS to Milvus 2.x.
-
-## Architecture
-
-
-
-## Latest release
-
-- [v0.1.0](https://github.com/zilliztech/milvus-migration/releases/tag/v0.1.0)
\ No newline at end of file
diff --git a/preview/site/en/userGuide/tools/tools.md b/preview/site/en/userGuide/tools/tools.md
deleted file mode 100644
index 15db08b70..000000000
--- a/preview/site/en/userGuide/tools/tools.md
+++ /dev/null
@@ -1,20 +0,0 @@
----
-id: tools.md
-title: Tools
----
-
-# Tools
-
-Check out the useful developer tools in the Milvus ecosystem which make working with Milvus much easier.
-
-- [Milvus CLI](cli_overview.md): A command-line tool that supports database connection, data operations, and import and export of data.
-
-- [Milvus Migration](milvus_migration): Milvus Migration is a data migration tool that allows you to migrate data from Milvus 0.9.x through 1.x or FAISS to Milvus 2.x.
-
-- [Milvus Backup](milvus_backup): Milvus Backup is a tool that allows users to back up and restore Milvus data. It provides both CLI and API to fit itself into different application scenarios.
-
-- [Attu](attu.md): An intuitive and efficient graphical user interface (GUI) tool that allows you to interact with your databases and manage your data with just a few clicks.
-
-- [Milvus DM](migrate_overview.md): An open-source tool designed for importing and exporting data with Milvus.
-
-- [Milvus Sizing Tool](https://milvus.io/tools/sizing/): A tool that helps you estimate the raw file size, memory size, and stable disk size needed for a specified number of vectors with various index types.
diff --git a/preview/site/en/userGuide/tune_consistency.md b/preview/site/en/userGuide/tune_consistency.md
index a8d8f0a2a..3a5d6af1f 100644
--- a/preview/site/en/userGuide/tune_consistency.md
+++ b/preview/site/en/userGuide/tune_consistency.md
@@ -3,6 +3,7 @@ id: tune_consistency.md
related_key: tune consistency
title: Tune Consistency
summary: Learn how to tune consistency level in Milvus.
+deprecate: true
---
# Tune consistency
diff --git a/preview/site/en/userGuide/use-json-fields.md b/preview/site/en/userGuide/use-json-fields.md
new file mode 100644
index 000000000..5ac039651
--- /dev/null
+++ b/preview/site/en/userGuide/use-json-fields.md
@@ -0,0 +1,1804 @@
+---
+id: use-json-fields.md
+title: Use JSON Fields
+---
+
+# Use JSON Fields
+
+This guide explains how to use the JSON fields, such as inserting JSON values as well as searching and querying in JSON fields with basic and advanced operators.
+
+## Overview
+
+JSON stands for Javascript Object Notation, which is a lightweight and simple text-based data format. Data in JSON is structured in key-value pairs, where every key is a string that maps to a value of a number, string, boolean, list, or array. With Milvus clusters, it's possible to store dictionaries as a field value in collections.
+
+For example, the following code randomly generates key-value pairs, each containing a JSON field with the key __color__.
+
+
+
+```python
+# 3. Insert randomly generated vectors
+colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
+data = []
+
+for i in range(1000):
+ current_color = random.choice(colors)
+ current_tag = random.randint(1000, 9999)
+ current_coord = [ random.randint(0, 40) for _ in range(3) ]
+ current_ref = [ [ random.choice(colors) for _ in range(3) ] for _ in range(3) ]
+ data.append({
+ "id": i,
+ "vector": [ random.uniform(-1, 1) for _ in range(5) ],
+ "color": {
+ "label": current_color,
+ "tag": current_tag,
+ "coord": current_coord,
+ "ref": current_ref
+ }
+ })
+
+print(data[0])
+```
+
+```java
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+import com.alibaba.fastjson.JSONObject;
+
+// 3. Insert randomly generated vectors and JSON data into the collection
+List
+
+
+
+## Define JSON field
+
+To define a JSON field, simply follow the same procedure as defining fields of other types.
+
+notes
+ +-
+
Ensure that all values in a list or array are of the same data type.
+Any nested dictionaries in a JSON field value will be considered strings.
+Use only alphanumeric characters and underscores to name JSON keys, as other characters may cause problems with filtering or searching.
+- Currently, indexing JSON fields is not available, which can make filtering time-consuming. However, this limitation will be addressed in upcoming releases. +
+
+For more information on parameters, refer to [`MilvusClient`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md), [`create_schema()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md), [`add_field()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/CollectionSchema/add_field.md), [`add_index()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/add_index.md), [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md), and [`get_load_state()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/get_load_state.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`MilvusClientV2`](https://milvus.io/api-reference/java/v2.4.x/v2/Client/MilvusClientV2.md), [`createSchema()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createSchema.md), [`addField()`](https://milvus.io/api-reference/java/v2.4.x/v2/CollectionSchema/addField.md), [`IndexParam`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/IndexParam.md), [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md), and [`getLoadState()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/getLoadState.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`MilvusClient`](https://milvus.io/api-reference/node/v2.4.x/Client/MilvusClient.md) and [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md) and [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md) in the SDK reference.
+
+
+
+
+
+```python
+import random, time
+from pymilvus import connections, MilvusClient, DataType
+
+CLUSTER_ENDPOINT = "http://localhost:19530"
+
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri=CLUSTER_ENDPOINT
+)
+
+# 2. Create a collection
+schema = MilvusClient.create_schema(
+ auto_id=False,
+ enable_dynamic_field=False,
+)
+
+schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
+schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=5)
+# highlight-next-line
+schema.add_field(field_name="color", datatype=DataType.JSON)
+
+index_params = MilvusClient.prepare_index_params()
+
+index_params.add_index(
+ field_name="id",
+ index_type="STL_SORT"
+)
+
+index_params.add_index(
+ field_name="vector",
+ index_type="IVF_FLAT",
+ metric_type="L2",
+ params={"nlist": 1024}
+)
+
+client.create_collection(
+ collection_name="test_collection",
+ schema=schema,
+ index_params=index_params
+)
+
+res = client.get_load_state(
+ collection_name="test_collection"
+)
+
+print(res)
+
+# Output
+#
+# {
+# "state": "
+
+For more information on parameters, refer to [`MilvusClient`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md), [`create_schema()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md), [`add_field()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/CollectionSchema/add_field.md), [`add_index()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/add_index.md), [`create_collection()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_collection.md), and [`get_load_state()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Management/get_load_state.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`MilvusClientV2`](https://milvus.io/api-reference/java/v2.4.x/v2/Client/MilvusClientV2.md), [`createSchema()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createSchema.md), [`addField()`](https://milvus.io/api-reference/java/v2.4.x/v2/CollectionSchema/addField.md), [`IndexParam`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/IndexParam.md), [`createCollection()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createCollection.md), and [`getLoadState()`](https://milvus.io/api-reference/java/v2.4.x/v2/Management/getLoadState.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`MilvusClient`](https://milvus.io/api-reference/node/v2.4.x/Client/MilvusClient.md), [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md), and [`getLoadState()`](https://milvus.io/api-reference/node/v2.4.x/Management/getLoadState.md) in the SDK reference.
+
+
+
+## Insert field values
+
+After creating a collection from the `CollectionSchema` object, dictionaries such as the one above can be inserted into it.
+
+
+
+Use the [`insert()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/insert.md) method to insert the data into the collection.
+
+
+
+
+
+Use the [`insert()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/insert.md) method to insert the data into the collection.
+
+
+
+
+
+Use the [`insert()`](https://milvus.io/api-reference/node/v2.4.x/Vector/insert.md) method to insert the data into the collection.
+
+
+
+
+
+```python
+res = client.insert(
+ collection_name="test_collection",
+ data=data
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 1000,
+# "ids": [
+# 0,
+# 1,
+# 2,
+# 3,
+# 4,
+# 5,
+# 6,
+# 7,
+# 8,
+# 9,
+# "(990 more items hidden)"
+# ]
+# }
+```
+
+```java
+// 3.1 Insert data into the collection
+InsertReq insertReq = InsertReq.builder()
+ .collectionName("test_collection")
+ .data(data)
+ .build();
+
+InsertResp insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 1000}
+```
+
+```javascript
+// 3. Insert randomly generated vectors
+const colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
+var data = []
+
+for (let i = 0; i < 1000; i++) {
+ const current_color = colors[Math.floor(Math.random() * colors.length)]
+ const current_tag = Math.floor(Math.random() * 8999 + 1000)
+ const current_coord = Array(3).fill(0).map(() => Math.floor(Math.random() * 40))
+ const current_ref = [ Array(3).fill(0).map(() => colors[Math.floor(Math.random() * colors.length)]) ]
+
+ data.push({
+ id: i,
+ vector: [Math.random(), Math.random(), Math.random(), Math.random(), Math.random()],
+ color: {
+ label: current_color,
+ tag: current_tag,
+ coord: current_coord,
+ ref: current_ref
+ }
+ })
+}
+
+console.log(data[0])
+
+// Output
+//
+// {
+// id: 0,
+// vector: [
+// 0.11455530974226114,
+// 0.21704086958595314,
+// 0.9430119822312437,
+// 0.7802712923612023,
+// 0.9106927960926137
+// ],
+// color: { label: 'grey', tag: 7393, coord: [ 22, 1, 22 ], ref: [ [Array] ] }
+// }
+//
+
+res = await client.insert({
+ collection_name: "test_collection",
+ data: data,
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 1000
+//
+```
+
+## Basic scalar filtering
+
+Once all of your data has been added, you can conduct searches and queries using the keys in the JSON field in the same manner as you would with a standard scalar field.
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/search.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/search.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/node/v2.4.x/Vector/search.md) in the SDK reference.
+
+
+
+
+
+```python
+# 4. Basic search with a JSON field
+query_vectors = [ [ random.uniform(-1, 1) for _ in range(5) ]]
+
+res = client.search(
+ collection_name="test_collection",
+ data=query_vectors,
+ # highlight-next-line
+ filter='color["label"] in ["red"]',
+ search_params={
+ "metric_type": "L2",
+ "params": {"nprobe": 16}
+ },
+ output_fields=["id", "color"],
+ limit=3
+)
+
+print(res)
+
+# Output
+#
+# [
+# [
+# {
+# "id": 460,
+# "distance": 0.4016231596469879,
+# "entity": {
+# "id": 460,
+# "color": {
+# "label": "red",
+# "tag": 5030,
+# "coord": [14, 32, 40],
+# "ref": [
+# [ "pink", "green", "brown" ],
+# [ "red", "grey", "black"],
+# [ "red", "yellow", "orange"]
+# ]
+# }
+# }
+# },
+# {
+# "id": 785,
+# "distance": 0.451080858707428,
+# "entity": {
+# "id": 785,
+# "color": {
+# "label": "red",
+# "tag": 5290,
+# "coord": [31, 13, 23],
+# "ref": [
+# ["yellow", "pink", "pink"],
+# ["purple", "grey", "orange"],
+# ["grey", "purple", "pink"]
+# ]
+# }
+# }
+# },
+# {
+# "id": 355,
+# "distance": 0.5839247703552246,
+# "entity": {
+# "id": 355,
+# "color": {
+# "label": "red",
+# "tag": 8725,
+# "coord": [5, 10, 22],
+# "ref": [
+# ["white", "purple", "yellow"],
+# ["white", "purple", "white"],
+# ["orange", "white", "pink"]
+# ]
+# }
+# }
+# }
+# ]
+# ]
+```
+
+```java
+// 4. Basic search with a JSON field
+List
+
+
+
+### Basic Operators in JSON Fields
+
+The following table assumes that the value of a JSON field named `json_key` has a key named `A`. Use it as a reference when constructing boolean expressions using JSON field keys.
+
+| __Operator__ | __Examples__ | __Remarks__ |
+| -------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| __<__ | `'json_field["A"] < 3'` | This expression evaluates to true if the value of `json_field["A"]` is less than `3`. |
+| __>__ | `'json_field["A"] > 1'` | This expression evaluates to true if the value of `json_field["A"]` is greater than `1`. |
+| __==__ | `'json_field["A"] == 1'` | This expression evaluates to true if the value of `json_field["A"]` is equal to `1`. |
+| __!=__ | `'json_field["A"][0]' != "abc"'` | This expression evaluates to true ifnotes
+ +-
+
- Milvus stores string values in the JSON field as is without performing semantic escape or conversion. +
For instance, 'a"b', "a'b", 'a\\\\'b', and "a\\\\"b" will be saved as is, while 'a'b' and "a"b" will be treated as invalid values.
-
+
To build filter expressions using a JSON field, you can utilize the keys within the field.
+If a key's value is an integer or a float, you can compare it with another integer or float key or an INT32/64 or FLOAT32/64 field.
+If a key's value is a string, you can compare it only with another string key or a VARCHAR field.
+
- `json_field` does not have a key named `A`.
- `json_field` has a key named `A` but `json_field["A"]` is not an array.
- `json_field["A"]` is an empty array.
- `json_field["A"]` is an array but the first element is not `abc`.
| +| __<=__ | `'json_field["A"] <= 5'` | This expression evaluates to true if the value of `json_field["A"]` is less than or equal to `5`. | +| __>=__ | `'json_field["A"] >= 1'` | This expression evaluates to true if the value of `json_field["A"]` is greater than or equal to `1`. | +| __not__ | `'not json_field["A"] == 1'` | This expression evaluates to true if
- `json_field` does not have a key named `A`.
- `json_field["A"]` is not equal to `1`.
| +| __in__ | `'json_field["A"] in [1, 2, 3]'` | This expression evaluates to true if the value of `json_field["A"]` is `1`, `2`, or `3`. | +| __and (&&)__ | `'json_field["A"] > 1 && json_field["A"] < 3'` | This expression evaluates to true if the value of `json_field["A"]` is greater than 1 and less than `3`. | +| __or (||)__ |
'json_field["A"] > 1 || json_field["A"] < 3' | This expression evaluates to true if the value of `json_field["A"]` is greater than `1` or less than `3`. |
+| __exists__ | `'exists json_field["A"]'` | This expression evaluates to true if `json_field` has a key named `A`. |
+
+### Advanced Operators
+
+The following operators are specific to JSON fields:
+
+- `json_contains(identifier, jsonExpr)`
+
+ This operator filters entities whose identifier contains the specified JSON expression.
+
+ - Example 1: `{"x": [1,2,3]}`
+
+ ```python
+ json_contains(x, 1) # => True (x contains 1.)
+ json_contains(x, "a") # => False (x does not contain a member "a".)
+ ```
+
+ - Example 2: `{"x", [[1,2,3], [4,5,6], [7,8,9]]}`
+
+ ```python
+ json_contains(x, [1,2,3]) # => True (x contains [1,2,3].)
+ json_contains(x, [3,2,1]) # => False (x does contain a member [3,2,1].)
+ ```
+
+- `json_contains_all(identifier, jsonExpr)`
+
+ This operator filters entities whose identifier contains all the members of the JSON expression.
+
+ Example: `{"x": [1,2,3,4,5,7,8]}`
+
+ ```python
+ json_contains_all(x, [1,2,8]) # => True (x contains 1, 2, and 8.)
+ json_contains_all(x, [4,5,6]) # => False (x does not has a member 6.)
+ ```
+
+- `json_contains_any(identifier, jsonExpr)`
+
+ This operator filters entities whose identifier contains any members of the JSON expression.
+
+ Example: `{"x": [1,2,3,4,5,7,8]}`
+
+ ```python
+ json_contains_any(x, [1,2,8]) # => True (x contains 1, 2, and 8.)
+ json_contains_any(x, [4,5,6]) # => True (x contains 4 and 5.)
+ json_contains_any(x, [6,9]) # => False (x contains none of 6 and 9.)
+ ```
diff --git a/preview/site/en/userGuide/use-partition-key.md b/preview/site/en/userGuide/use-partition-key.md
new file mode 100644
index 000000000..f8f3c0b4d
--- /dev/null
+++ b/preview/site/en/userGuide/use-partition-key.md
@@ -0,0 +1,579 @@
+---
+id: use-partition-key.md
+title: Use Partition Key
+---
+
+# Use Partition Key
+
+This guide walks you through using the partition key to accelerate data retrieval from your collection.
+
+## Overview
+
+You can set a particular field in a collection as the partition key so that Milvus distributes incoming entities into different partitions according to their respective partition values in this field. This allows entities with the same key value to be grouped in a partition, accelerating search performance by avoiding the need to scan irrelevant partitions when filtering by the key field. When compared to traditional filtering methods, the partition key can greatly enhance query performance.
+
+You can use the partition key to implement multi-tenancy. For details on multi-tenancy, read [Multi-tenancy](https://milvus.io/docs/multi_tenancy.md) for more.
+
+## Enable partition key
+
+To set a field as the partition key, specify `partition_key_field` when creating a collection schema.
+
+In the example code below, `num_partitions` determines the number of partitions that will be created. By default, it is set to `16`. We recommend you retain the default value.
+
+
+
+For more information on parameters, refer to [`MilvusClient`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md), [`create_schema()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/create_schema.md), and [`add_field()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/CollectionSchema/add_field.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`MilvusClientV2`](https://milvus.io/api-reference/java/v2.4.x/v2/Client/MilvusClientV2.md), [`createSchema()`](https://milvus.io/api-reference/java/v2.4.x/v2/Collections/createSchema.md), and [`addField()`](https://milvus.io/api-reference/java/v2.4.x/v2/CollectionSchema/addField.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`MilvusClient`](https://milvus.io/api-reference/node/v2.4.x/Client/MilvusClient.md) and [`createCollection()`](https://milvus.io/api-reference/node/v2.4.x/Collections/createCollection.md) in the SDK reference.
+
+
+
+
+
+```python
+import random, time
+from pymilvus import connections, MilvusClient, DataType
+
+SERVER_ADDR = "http://localhost:19530"
+
+# 1. Set up a Milvus client
+client = MilvusClient(
+ uri=SERVER_ADDR
+)
+
+# 2. Create a collection
+schema = MilvusClient.create_schema(
+ auto_id=False,
+ enable_dynamic_field=True,
+ # highlight-next-line
+ partition_key_field="color",
+ num_partitions=16 # Number of partitions. Defaults to 16.
+)
+
+schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
+schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=5)
+schema.add_field(field_name="color", datatype=DataType.VARCHAR, max_length=512)
+```
+
+```java
+import io.milvus.v2.client.ConnectConfig;
+import io.milvus.v2.client.MilvusClientV2;
+import io.milvus.v2.common.DataType;
+import io.milvus.v2.common.IndexParam;
+import io.milvus.v2.service.collection.request.AddFieldReq;
+import io.milvus.v2.service.collection.request.CreateCollectionReq;
+
+String CLUSTER_ENDPOINT = "http://localhost:19530";
+
+// 1. Connect to Milvus server
+ConnectConfig connectConfig = ConnectConfig.builder()
+ .uri(CLUSTER_ENDPOINT)
+ .build();
+
+MilvusClientV2 client = new MilvusClientV2(connectConfig);
+
+// 2. Create a collection in customized setup mode
+
+// 2.1 Create schema
+CreateCollectionReq.CollectionSchema schema = client.createSchema();
+
+// 2.2 Add fields to schema
+schema.addField(AddFieldReq.builder()
+ .fieldName("id")
+ .dataType(DataType.Int64)
+ .isPrimaryKey(true)
+ .autoID(false)
+ .build());
+
+schema.addField(AddFieldReq.builder()
+ .fieldName("vector")
+ .dataType(DataType.FloatVector)
+ .dimension(5)
+ .build());
+
+schema.addField(AddFieldReq.builder()
+ .fieldName("color")
+ .dataType(DataType.VarChar)
+ .maxLength(512)
+ // highlight-next-line
+ .isPartitionKey(true)
+ .build());
+```
+
+```javascript
+const { MilvusClient, DataType, sleep } = require("@zilliz/milvus2-sdk-node")
+
+const address = "http://localhost:19530"
+
+async function main() {
+// 1. Set up a Milvus Client
+client = new MilvusClient({address});
+
+// 2. Create a collection
+// 2.1 Define fields
+const fields = [
+ {
+ name: "id",
+ data_type: DataType.Int64,
+ is_primary_key: true,
+ auto_id: false
+ },
+ {
+ name: "vector",
+ data_type: DataType.FloatVector,
+ dim: 5
+ },
+ {
+ name: "color",
+ data_type: DataType.VarChar,
+ max_length: 512,
+ // highlight-next-line
+ is_partition_key: true
+ }
+]
+```
+
+After you have defined the fields, set up the index parameters.
+
+
+
+```python
+index_params = MilvusClient.prepare_index_params()
+
+index_params.add_index(
+ field_name="id",
+ index_type="STL_SORT"
+)
+
+index_params.add_index(
+ field_name="color",
+ index_type="Trie"
+)
+
+index_params.add_index(
+ field_name="vector",
+ index_type="IVF_FLAT",
+ metric_type="L2",
+ params={"nlist": 1024}
+)
+```
+
+```java
+// 2.3 Prepare index parameters
+IndexParam indexParamForVectorField = IndexParam.builder()
+ .fieldName("vector")
+ .indexType(IndexParam.IndexType.IVF_FLAT)
+ .metricType(IndexParam.MetricType.IP)
+ .extraParams(Map.of("nlist", 1024))
+ .build();
+
+List
+
+Use the [`insert()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/insert.md) method to insert the data into the collection.
+
+
+
+
+
+Use the [`insert()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/insert.md) method to insert the data into the collection.
+
+
+
+
+
+Use the [`insert()`](https://milvus.io/api-reference/node/v2.4.x/Vector/insert.md) method to insert the data into the collection.
+
+
+
+
+
+```python
+res = client.insert(
+ collection_name="test_collection",
+ data=data
+)
+
+print(res)
+
+# Output
+#
+# {
+# "insert_count": 1000,
+# "ids": [
+# 0,
+# 1,
+# 2,
+# 3,
+# 4,
+# 5,
+# 6,
+# 7,
+# 8,
+# 9,
+# "(990 more items hidden)"
+# ]
+# }
+```
+
+```java
+// 3.1 Insert data into the collection
+InsertReq insertReq = InsertReq.builder()
+ .collectionName("test_collection")
+ .data(data)
+ .build();
+
+InsertResp insertResp = client.insert(insertReq);
+
+System.out.println(JSONObject.toJSON(insertResp));
+
+// Output:
+// {"insertCnt": 1000}
+```
+
+```javascript
+res = await client.insert({
+ collection_name: "test_collection",
+ data: data,
+})
+
+console.log(res.insert_cnt)
+
+// Output
+//
+// 1000
+//
+```
+
+## Use partition key
+
+Once you have indexed and loaded the collection as well as inserted data, you can conduct a similarity search using the partition key.
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Vector/search.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/java/v2.4.x/v2/Vector/search.md) in the SDK reference.
+
+
+
+
+
+For more information on parameters, refer to [`search()`](https://milvus.io/api-reference/node/v2.4.x/Vector/search.md) in the SDK reference.
+
+
+
+
+
+
+
+
+
+```python
+# 4. Search with partition key
+query_vectors = [[0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]]
+
+res = client.search(
+ collection_name="test_collection",
+ data=query_vectors,
+ filter="color == 'green'",
+ search_params={"metric_type": "L2", "params": {"nprobe": 10}},
+ output_fields=["id", "color_tag"],
+ limit=3
+)
+
+print(res)
+
+# Output
+#
+# [
+# [
+# {
+# "id": 970,
+# "distance": 0.5770174264907837,
+# "entity": {
+# "id": 970,
+# "color_tag": "green_9828"
+# }
+# },
+# {
+# "id": 115,
+# "distance": 0.6898155808448792,
+# "entity": {
+# "id": 115,
+# "color_tag": "green_4073"
+# }
+# },
+# {
+# "id": 899,
+# "distance": 0.7028976678848267,
+# "entity": {
+# "id": 899,
+# "color_tag": "green_9897"
+# }
+# }
+# ]
+# ]
+```
+
+```java
+// 4. Search with partition key
+Listnotes
+ +To conduct a similarity search using the partition key, you should include either of the following in the boolean expression of the search request:
+-
+
expr='<partition_key>=="xxxx"'
+expr='<partition_key> in ["xxx", "xxx"]'
+
Do replace <partition_key> with the name of the field that is designated as the partition key.
+ Ensure that Milvus is running.
+
+
+The MilvusClient supports a single unified way of connecting to the service through the use of a URI. A few examples of valid URIs are:
+1. "http://localhost:19530"
+2. "https://user:password@mysite:19530"
+2. "https://username:password@in01-12a.aws-us-west-2.vectordb.zillizcloud.com:19538"
+
+When using an HTTPS connection we expect a username and password.
+
+Now lets go over a quick example using the MilvusClient
+
+## Basics
+
+### Create the Client
+
+Most of the information needed to use the Client is provided in the construction call. There are two main use cases for the client, creating a new Milvus collection or using a previously made collection.
+
+If creating a new collection, you must specify the vector_field name as this cannot be parsed from the inserted data. If you want to manually handle the primary field keys for this collection then you must also specify pk_field, otherwise an autogenerated int field will be used. If a collection with the same name exists in the Milvus instance then you must set overwrite to `True` to remove the previous collection.
+
+If you want to connect to a previously created collection then the only values that need to be provided are the uri and the collection_name, the rest of the information will be inferred from the collection itself.
+
+
+```python
+from pymilvus import MilvusClient
+
+client = MilvusClient(
+ collection_name="qux",
+ uri="http://localhost:19530",
+ vector_field="float_vector",
+ # pk_field= "id", # If you wanted to provide your own PK
+ overwrite=True,
+)
+```
+
+### Insert Data
+
+With the MilvusClient created we can begin to insert data. Data is inserted in the form of a list of dictionaries where each dict corresponds to a row in the collection. Each dict must include values for all the columns in the collection, otherwise the insert will throw an exception.
+
+If the client was created on a collection that doesnt exist or overwrite is set to True, the first entry in the list of dicts will be used to construct the schema of the collection. All subsequent inserts will need to contain the same fields as the first dict. If no index parameteres were supplied at construction time, then a default HNSW index will be used to index the data.
+
+```python
+data = [
+ {
+ "float_vector": [1,2,3],
+ "id": 1,
+ "text": "foo"
+ },
+ {
+ "float_vector": [4,5,6],
+ "id": 2,
+ "text": "bar"
+ },
+ {
+ "float_vector": [7,8,9],
+ "id": 3,
+ "text": "baz"
+ }
+]
+client.insert_data(data)
+```
+
+### Search the Data
+
+Once the data has been inserted into Milvus we can proceed to search the collection. The search takes in the search vector/s and how many search results we want (top_k). In addition to this, if you want you can also supply search parameters. Ths search parameters should correspond to the index_parameters if you supplied them at construction time. If not supplied, MilvusClient will use default search parameters.
+
+```python
+res = client.search_data(
+ data = [[1,3,5], [7,8,9]],
+ top_k = 2,
+)
+# [[
+# {'data': {'id': 1, 'internal_pk_4537': 441340318978146436, 'text': 'foo'}, 'score': 5.0},
+# {'data': {'id': 2, 'internal_pk_4537': 441340318978146437, 'text': 'bar'}, 'score': 14.0}
+# ],
+# [
+# {'data': {'id': 3, 'internal_pk_4537': 441340318978146438, 'text': 'baz'}, 'score': 0.0},
+# {'data': {'id': 2, 'internal_pk_4537': 441340318978146437, 'text': 'bar'}, 'score': 27.0}
+# ]]
+```
+
+The search results will come in the form of a list of lists. For each search vector, you will recieve a list of dicts, with each dict containing the distance and the corresponding result data. If not all of the data is needed you can adjust what data is returned using the return_fields argument.
+
+## Advanced
+
+### Partitions
+
+The MilvusClient supports partitions in its current release. Partitions can be specified both at MilvusClient construction and later on. Here is a quick example on using the partitions functionality.
+
+```python
+from pymilvus import MilvusClient
+
+
+client = MilvusClient(
+ collection_name="qux",
+ uri="http://localhost:19530",
+ vector_field="float_vector",
+ partitions = ["zaz"],
+ overwrite=True,
+)
+
+data = [
+ {
+ "float_vector": [1,2,3],
+ "id": 1,
+ "text": "foo"
+ },
+]
+client.insert_data(data, partition="zaz")
+
+client.add_partitions(["zoo"])
+
+data = [
+ {
+ "float_vector": [4,5,6],
+ "id": 2,
+ "text": "bar"
+ },
+]
+client.insert_data(data, partition="zoo")
+
+res = client.search_data(
+ data = [1,3,5],
+ top_k = 2,
+)
+
+# [[
+# {'data': {'id': 1, 'internal_pk_3bd4': 441363276234227849, 'text': 'foo'}, 'score': 5.0},
+# {'data': {'id': 2, 'internal_pk_3bd4': 441363276234227866, 'text': 'bar'}, 'score': 14.0}
+# ]]
+
+
+res = client.search_data(
+ data = [1,3,5],
+ top_k = 2,
+ partitions=["zaz"]
+)
+
+# [[
+# {'data': {'id': 1, 'internal_pk_3bd4': 441363276234227849, 'text': 'foo'}, 'score': 5.0}
+# ]]
+
+res = client.search_data(
+ data = [1,3,5],
+ top_k = 2,
+ partitions=["zoo"]
+)
+
+# [[
+# {'data': {'id': 2, 'internal_pk_3bd4': 441363276234227866, 'text': 'bar'}, 'score': 14.0}
+# ]]
+```
+
+
+### Filtering
+
+Filtering can be used to narrow down results to match metadata or to query data based on metadata.
+
+```python
+from pymilvus import MilvusClient
+
+client = MilvusClient(
+ collection_name="qux",
+ uri="http://localhost:19530",
+ vector_field="float_vector",
+ # pk_field= "id", # If you wanted to provide your own PK
+ overwrite=True,
+)
+
+data = [
+ {
+ "float_vector": [1,2,3],
+ "id": 1,
+ "text": "foo"
+ },
+ {
+ "float_vector": [4,5,6],
+ "id": 2,
+ "text": "bar"
+ },
+ {
+ "float_vector": [7,8,9],
+ "id": 3,
+ "text": "baz"
+ }
+]
+client.insert_data(data)
+
+res = client.search_data(
+ data = [1,3,5],
+ top_k = 2,
+ filter_expression = "id > 1"
+)
+
+# [[
+# {'score': 14.0, 'data': {'id': 2, 'text': 'bar', 'internal_pk_5465': 441363276234227922}},
+# {'score': 77.0, 'data': {'id': 3, 'text': 'baz', 'internal_pk_5465': 441363276234227923}}
+# ]]
+
+res = client.query_data(
+ filter_expression = "id == 1"
+)
+
+# [
+# {'id': 1, 'text': 'foo', 'internal_pk_5465': 441363276234227921}
+# ]
+```
+
+### Vector Retrieval and Deletion
+
+As a vector database we have the ability to return the actual vectors and delete their entries. In order to do these two functions we need to first get the pks corresponding to the entry we are trying to act on. Here is an example below.
+
+```python
+from pymilvus import MilvusClient
+
+client = MilvusClient(
+ collection_name="qux",
+ uri="http://localhost:19530",
+ vector_field="float_vector",
+ pk_field= "text",
+ overwrite=True,
+)
+
+data = [
+ {
+ "float_vector": [1,2,3],
+ "id": 1,
+ "text": "foo"
+ },
+ {
+ "float_vector": [4,5,6],
+ "id": 2,
+ "text": "bar"
+ },
+ {
+ "float_vector": [7,8,9],
+ "id": 3,
+ "text": "baz"
+ }
+]
+
+client.insert_data(data)
+
+res = client.query_data(
+ filter_expression = "id == 1"
+)
+
+# [
+# {'id': 1, 'text': 'foo'}
+# ]
+
+res = client.get_vectors_by_pk(pks = res[0]["text"])
+
+# [
+# {'float_vector': [1.0, 2.0, 3.0], 'text': 'foo'}
+# ]
+
+client.delete_by_pk(pks = res[0]["text"])
+
+res = client.query_data(
+ filter_expression = "id == 1"
+)
+
+# []
+```
\ No newline at end of file
diff --git a/preview/version.json b/preview/version.json
index f9f0ff4f3..8085d97f0 100644
--- a/preview/version.json
+++ b/preview/version.json
@@ -1,4 +1,4 @@
{
- "version": "v2.2.x",
+ "version": "v2.4.x",
"released": "yes"
}
diff --git a/v2.0.x/site/en/reference/sys_config/configure_knowhere.md b/v2.0.x/site/en/reference/sys_config/configure_knowhere.md
index c58fb5a83..922fcf41e 100644
--- a/v2.0.x/site/en/reference/sys_config/configure_knowhere.md
+++ b/v2.0.x/site/en/reference/sys_config/configure_knowhere.md
@@ -9,7 +9,7 @@ summary: Learn how to configure Knowhere-related parameters of Milvus.
This topic introduces the knowhere-related configurations of Milvus.
-[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus.
+Knowhere is the search engine of Milvus.
Under this section, you can configure the default SIMD instruction set type of the system.
diff --git a/v2.0.x/site/en/reference/sys_config/system_configuration.md b/v2.0.x/site/en/reference/sys_config/system_configuration.md
index 6b3576292..8e6e1a019 100644
--- a/v2.0.x/site/en/reference/sys_config/system_configuration.md
+++ b/v2.0.x/site/en/reference/sys_config/system_configuration.md
@@ -146,7 +146,7 @@ See [Common Configurations](configure_common.md) for detailed description for ea
### `knowhere`
-[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus.
+Knowhere is the search engine of Milvus.
Under this section, you can configure the default SIMD instruction set type of the system.
diff --git a/v2.0.x/site/en/reference/timetravel_ref.md b/v2.0.x/site/en/reference/timetravel_ref.md
index 1520f0713..cf8edccdb 100644
--- a/v2.0.x/site/en/reference/timetravel_ref.md
+++ b/v2.0.x/site/en/reference/timetravel_ref.md
@@ -20,7 +20,7 @@ During a search, if the search request received by the proxy contains the parame
## Search implementation
-Searches with filtering in [knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is achieved by bitset. Bitset can be applied in the following three aspects:
+Searches with filtering in knowhere is achieved by bitset. Bitset can be applied in the following three aspects:
- Delete data
- Timestamp
diff --git a/v2.0.x/site/en/tutorials/audio_similarity_search.md b/v2.0.x/site/en/tutorials/audio_similarity_search.md
index 656a89c60..4e41c35eb 100644
--- a/v2.0.x/site/en/tutorials/audio_similarity_search.md
+++ b/v2.0.x/site/en/tutorials/audio_similarity_search.md
@@ -6,7 +6,6 @@ summary: Build an audio similarity search system with Milvus.
# Audio Similarity Search
This tutorial demonstrates how to use Milvus, the open-source vector database to build an audio similarity search system.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/audio/audio_similarity_search/audio_similarity_search.ipynb)
- [Quick deploy](https://github.com/milvus-io/bootcamp/tree/master/solutions/audio/audio_similarity_search/quick_deploy)
The ML model and third-party software used include:
- PANNs (Large-Scale Pretrained Audio Neural Networks)
diff --git a/v2.0.x/site/en/tutorials/dna_sequence_classification.md b/v2.0.x/site/en/tutorials/dna_sequence_classification.md
index fc0d325e4..7472fb7b9 100644
--- a/v2.0.x/site/en/tutorials/dna_sequence_classification.md
+++ b/v2.0.x/site/en/tutorials/dna_sequence_classification.md
@@ -6,8 +6,7 @@ summary: Build a DNA sequence classification system with Milvus.
# DNA Sequence Classification
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a DNA sequence classification model.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/medical/dna_sequence_classification/dna_sequence_classification.ipynb)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/medical/dna_sequence_classification/quick_deploy)
+
The ML model and third-party software used include:
- CountVectorizer
- MySQL
diff --git a/v2.0.x/site/en/tutorials/image_similarity_search.md b/v2.0.x/site/en/tutorials/image_similarity_search.md
index d32b47c46..0cac4148c 100644
--- a/v2.0.x/site/en/tutorials/image_similarity_search.md
+++ b/v2.0.x/site/en/tutorials/image_similarity_search.md
@@ -7,7 +7,6 @@ summary: Build an image similarity search system with Milvus.
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a reverse image search system.
- [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/image/reverse_image_search)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/image/reverse_image_search/quick_deploy)
- [Try online demo](https://milvus.io/milvus-demos/)
The ML models and third-party software used include:
- YOLOv3
diff --git a/v2.0.x/site/en/tutorials/question_answering_system.md b/v2.0.x/site/en/tutorials/question_answering_system.md
index 4b99466cd..dab824cfd 100644
--- a/v2.0.x/site/en/tutorials/question_answering_system.md
+++ b/v2.0.x/site/en/tutorials/question_answering_system.md
@@ -7,7 +7,6 @@ summary: Build a question answering system with Milvus.
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a question answering (QA) system.
- [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/nlp/question_answering)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/question_answering_system/quick_deploy)
- [Try online demo](https://milvus.io/milvus-demos/)
The ML model and third-party software used include:
diff --git a/v2.0.x/site/en/tutorials/recommendation_system.md b/v2.0.x/site/en/tutorials/recommendation_system.md
index 07a9dfcb8..839953d1a 100644
--- a/v2.0.x/site/en/tutorials/recommendation_system.md
+++ b/v2.0.x/site/en/tutorials/recommendation_system.md
@@ -6,7 +6,7 @@ summary: Build a personalized recommender system with Milvus.
# Recommender System
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a recommendation system.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/recommender_system.ipynb)
+
- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/quick_deploy)
The ML Model and third-party software used include:
diff --git a/v2.0.x/site/en/tutorials/video_similarity_search.md b/v2.0.x/site/en/tutorials/video_similarity_search.md
index f304ddbcb..99d99df32 100644
--- a/v2.0.x/site/en/tutorials/video_similarity_search.md
+++ b/v2.0.x/site/en/tutorials/video_similarity_search.md
@@ -7,7 +7,6 @@ summary: Build a video similarity search system with Milvus.
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a video similarity search system.
- [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/video/reverse_video_search)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/video/video_similarity_search/quick_deploy)
The ML models and third-party software used include:
- OpenCV
- ResNet-50
diff --git a/v2.0.x/site/zh-CN/reference/sys_config/system_configuration.md b/v2.0.x/site/zh-CN/reference/sys_config/system_configuration.md
index cbcc10f2f..b946b608e 100644
--- a/v2.0.x/site/zh-CN/reference/sys_config/system_configuration.md
+++ b/v2.0.x/site/zh-CN/reference/sys_config/system_configuration.md
@@ -153,7 +153,7 @@ See [Common Configurations](configure_common.md) for detailed description for ea
### `knowhere`
-[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus.
+Knowhere is the search engine of Milvus.
Under this section, you can configure the default SIMD instruction set type of the system.
diff --git a/v2.0.x/site/zh-CN/reference/timetravel_ref.md b/v2.0.x/site/zh-CN/reference/timetravel_ref.md
index 2f6130b36..51c49eb99 100644
--- a/v2.0.x/site/zh-CN/reference/timetravel_ref.md
+++ b/v2.0.x/site/zh-CN/reference/timetravel_ref.md
@@ -27,7 +27,7 @@ During a search, if the search request received by the proxy contains the parame
## Search implementation
-Searches with filtering in [knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is achieved by bitset. Bitset can be applied in the following three aspects:
+Searches with filtering in knowhere is achieved by bitset. Bitset can be applied in the following three aspects:
- Delete data
- Timestamp
diff --git a/v2.1.x/site/en/adminGuide/monitor/monitor_overview.md b/v2.1.x/site/en/adminGuide/monitor/monitor_overview.md
index ee1fa03a2..0d03d786a 100644
--- a/v2.1.x/site/en/adminGuide/monitor/monitor_overview.md
+++ b/v2.1.x/site/en/adminGuide/monitor/monitor_overview.md
@@ -1,6 +1,6 @@
---
id: monitor_overview.md
-title: Monitoring Overview
+title: Monitor Overview
related_key: monitor, alert
summary: Learn how Prometheus and Grafana are used in Milvus for montoring and alerting services.
---
diff --git a/v2.1.x/site/en/reference/consistency.md b/v2.1.x/site/en/reference/consistency.md
index ce981e249..be237ae6d 100644
--- a/v2.1.x/site/en/reference/consistency.md
+++ b/v2.1.x/site/en/reference/consistency.md
@@ -64,7 +64,7 @@ A GuaranteeTs serves to inform query nodes that a search or query request will n
- **Eventually**: GuaranteeTs is set to a very small value to skip the consistency check. Query nodes search immediately on the existing data view.
-See [How GuaranteeTs Works](https://github.com/milvus-io/milvus/blob/f3f46d3bb2dcae2de0bdb7bc0f7b20a72efceaab/docs/developer_guides/how-guarantee-ts-works.md) and [All The Things You Should Know About Time Synchronization](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/milvus_timesync_en.md?from=from_parent_mindnote) for more information about the mechanism behind ensuring different levels of consistency in Milvus.
+See [How GuaranteeTs Works](https://github.com/milvus-io/milvus/blob/f3f46d3bb2dcae2de0bdb7bc0f7b20a72efceaab/docs/developer_guides/how-guarantee-ts-works.md) for more information about the mechanism behind ensuring different levels of consistency in Milvus.
## What's next
diff --git a/v2.1.x/site/en/reference/sys_config/configure_knowhere.md b/v2.1.x/site/en/reference/sys_config/configure_knowhere.md
index c58fb5a83..922fcf41e 100644
--- a/v2.1.x/site/en/reference/sys_config/configure_knowhere.md
+++ b/v2.1.x/site/en/reference/sys_config/configure_knowhere.md
@@ -9,7 +9,7 @@ summary: Learn how to configure Knowhere-related parameters of Milvus.
This topic introduces the knowhere-related configurations of Milvus.
-[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus.
+Knowhere is the search engine of Milvus.
Under this section, you can configure the default SIMD instruction set type of the system.
diff --git a/v2.1.x/site/en/reference/sys_config/system_configuration.md b/v2.1.x/site/en/reference/sys_config/system_configuration.md
index 6b3576292..8e6e1a019 100644
--- a/v2.1.x/site/en/reference/sys_config/system_configuration.md
+++ b/v2.1.x/site/en/reference/sys_config/system_configuration.md
@@ -146,7 +146,7 @@ See [Common Configurations](configure_common.md) for detailed description for ea
### `knowhere`
-[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus.
+Knowhere is the search engine of Milvus.
Under this section, you can configure the default SIMD instruction set type of the system.
diff --git a/v2.1.x/site/en/reference/timestamp.md b/v2.1.x/site/en/reference/timestamp.md
index 78ae33ca8..434982976 100644
--- a/v2.1.x/site/en/reference/timestamp.md
+++ b/v2.1.x/site/en/reference/timestamp.md
@@ -30,7 +30,7 @@ Several timestamp-related parameters are involved when you conduct a vector simi
If the `Guarantee_timestamp` is not configured, Milvus automatically takes the point in time when the search request is made. Therefore, the search is conducted on a data view with all data updates by DML operations before the search.
-To save you the trouble of understanding the [TSO](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/milvus_hybrid_ts_en.md?from=from_parent_mindnote) inside Milvus, as a user, you do not have to directly configure the `Guarantee_timestamp` parameter. You only need to choose the [consistency level](https://milvus.io/docs/v2.1.x/consistency.md), and Milvus automatically handles the `Guarantee_timestamp` parameter for you. Each consistency level corresponds to a certain `Guarantee_timestamp` value.
+To save you the trouble of understanding the TSO inside Milvus, as a user, you do not have to directly configure the `Guarantee_timestamp` parameter. You only need to choose the [consistency level](https://milvus.io/docs/v2.1.x/consistency.md), and Milvus automatically handles the `Guarantee_timestamp` parameter for you. Each consistency level corresponds to a certain `Guarantee_timestamp` value.
.
diff --git a/v2.1.x/site/en/reference/timetravel_ref.md b/v2.1.x/site/en/reference/timetravel_ref.md
index 05a6dcbdc..199a09107 100644
--- a/v2.1.x/site/en/reference/timetravel_ref.md
+++ b/v2.1.x/site/en/reference/timetravel_ref.md
@@ -38,7 +38,7 @@ During a search, if the search request received by the proxy contains the parame
## Bitset for timestamp
-To go into details, searches and queries with filtering in [knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) are facilitated by bitset. And the underlying mechanism behind Time Travel is enabled by bitset.
+To go into details, searches and queries with filtering in knowhere are facilitated by bitset. And the underlying mechanism behind Time Travel is enabled by bitset.
When a search is conducted, the segcore obtains a bitset indicating if the timestamp meets the condition. Then Milvus judges the range of data to query or search based on this bitset.
diff --git a/v2.1.x/site/en/tutorials/audio_similarity_search.md b/v2.1.x/site/en/tutorials/audio_similarity_search.md
index 656a89c60..4e41c35eb 100644
--- a/v2.1.x/site/en/tutorials/audio_similarity_search.md
+++ b/v2.1.x/site/en/tutorials/audio_similarity_search.md
@@ -6,7 +6,6 @@ summary: Build an audio similarity search system with Milvus.
# Audio Similarity Search
This tutorial demonstrates how to use Milvus, the open-source vector database to build an audio similarity search system.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/audio/audio_similarity_search/audio_similarity_search.ipynb)
- [Quick deploy](https://github.com/milvus-io/bootcamp/tree/master/solutions/audio/audio_similarity_search/quick_deploy)
The ML model and third-party software used include:
- PANNs (Large-Scale Pretrained Audio Neural Networks)
diff --git a/v2.1.x/site/en/tutorials/dna_sequence_classification.md b/v2.1.x/site/en/tutorials/dna_sequence_classification.md
index fc0d325e4..f7c77b4ef 100644
--- a/v2.1.x/site/en/tutorials/dna_sequence_classification.md
+++ b/v2.1.x/site/en/tutorials/dna_sequence_classification.md
@@ -6,8 +6,6 @@ summary: Build a DNA sequence classification system with Milvus.
# DNA Sequence Classification
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a DNA sequence classification model.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/medical/dna_sequence_classification/dna_sequence_classification.ipynb)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/medical/dna_sequence_classification/quick_deploy)
The ML model and third-party software used include:
- CountVectorizer
- MySQL
diff --git a/v2.1.x/site/en/tutorials/image_similarity_search.md b/v2.1.x/site/en/tutorials/image_similarity_search.md
index d32b47c46..0cac4148c 100644
--- a/v2.1.x/site/en/tutorials/image_similarity_search.md
+++ b/v2.1.x/site/en/tutorials/image_similarity_search.md
@@ -7,7 +7,6 @@ summary: Build an image similarity search system with Milvus.
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a reverse image search system.
- [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/image/reverse_image_search)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/image/reverse_image_search/quick_deploy)
- [Try online demo](https://milvus.io/milvus-demos/)
The ML models and third-party software used include:
- YOLOv3
diff --git a/v2.1.x/site/en/tutorials/question_answering_system.md b/v2.1.x/site/en/tutorials/question_answering_system.md
index 4b99466cd..dab824cfd 100644
--- a/v2.1.x/site/en/tutorials/question_answering_system.md
+++ b/v2.1.x/site/en/tutorials/question_answering_system.md
@@ -7,7 +7,6 @@ summary: Build a question answering system with Milvus.
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a question answering (QA) system.
- [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/nlp/question_answering)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/question_answering_system/quick_deploy)
- [Try online demo](https://milvus.io/milvus-demos/)
The ML model and third-party software used include:
diff --git a/v2.1.x/site/en/tutorials/recommendation_system.md b/v2.1.x/site/en/tutorials/recommendation_system.md
index 07a9dfcb8..08a61bcc7 100644
--- a/v2.1.x/site/en/tutorials/recommendation_system.md
+++ b/v2.1.x/site/en/tutorials/recommendation_system.md
@@ -6,8 +6,6 @@ summary: Build a personalized recommender system with Milvus.
# Recommender System
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a recommendation system.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/recommender_system.ipynb)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/quick_deploy)
The ML Model and third-party software used include:
- PaddlePaddle
diff --git a/v2.1.x/site/en/tutorials/video_similarity_search.md b/v2.1.x/site/en/tutorials/video_similarity_search.md
index f304ddbcb..99d99df32 100644
--- a/v2.1.x/site/en/tutorials/video_similarity_search.md
+++ b/v2.1.x/site/en/tutorials/video_similarity_search.md
@@ -7,7 +7,6 @@ summary: Build a video similarity search system with Milvus.
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a video similarity search system.
- [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/video/reverse_video_search)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/video/video_similarity_search/quick_deploy)
The ML models and third-party software used include:
- OpenCV
- ResNet-50
diff --git a/v2.2.x/site/en/adminGuide/clouds/gcp/gcp.md b/v2.2.x/site/en/adminGuide/clouds/gcp/gcp.md
index 0a0f11232..c4c857901 100644
--- a/v2.2.x/site/en/adminGuide/clouds/gcp/gcp.md
+++ b/v2.2.x/site/en/adminGuide/clouds/gcp/gcp.md
@@ -5,7 +5,7 @@ related_key: cluster
summary: Learn how to deploy a Milvus cluster on GCP.
---
-## Deploy a Milvus Cluster on GCP
+# Deploy a Milvus Cluster on GCP
Milvus is a cloud-native vector database and can be deployed on various cloud environments. This guide walks you through every detail about setting up Milvus on Google Cloud Platform (GCP).
@@ -151,4 +151,4 @@ kubectl get services
The result is similar to the following:
-
\ No newline at end of file
+
diff --git a/v2.2.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md b/v2.2.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md
index d7e33a7c3..c1fbea405 100644
--- a/v2.2.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md
+++ b/v2.2.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md
@@ -5,7 +5,7 @@ related_key: cluster
summary: Learn how to deploy a Milvus cluster behind a Layer-7 load balancer on GCP.
---
-## Set up a Layer-7 Load Balancer for Milvus on GCP
+# Set up a Layer-7 Load Balancer for Milvus on GCP
When compared to a Layer-4 load balancer, a Layer-7 load balancer offers smart load balancing and caching capabilities and is a great choice for cloud-native services.
@@ -259,4 +259,4 @@ connections.connect("default", host="34.111.144.65", port="443", secure=True, se
- The IP address and port number in **host** and **port** should match those listed at the end of [Create an Ingress to generate a Layer-7 Load Balancer](#create-an-ingress-to-generate-a-layer-7-load-balancer).
- If you have set up a DNS record to map domain name to the host IP address, replace the IP address in **host** with the domain name and omit **server_name**.
-
\ No newline at end of file
+
diff --git a/v2.2.x/site/en/adminGuide/resource_group.md b/v2.2.x/site/en/adminGuide/resource_group.md
index a040abe57..4a96a5acf 100644
--- a/v2.2.x/site/en/adminGuide/resource_group.md
+++ b/v2.2.x/site/en/adminGuide/resource_group.md
@@ -4,7 +4,7 @@ related_key: Manage Resource Groups
summary: Learn how to manage resource groups.
---
-## Manage Resource Groups
+# Manage Resource Groups
In Milvus, you can use a resource group to physically isolate certain query nodes from others. This guide walks you through how to create and manage custom resource groups as well as transfer nodes between them.
@@ -165,4 +165,4 @@ All code samples on this page are in PyMilvus 2.2.x. Upgrade your PyMilvus insta
To deploy a multi-tenant Milvus instance, read the following:
- [Enable RBAC](rbac.md)
-- [Users and roles](users_and_roles.md)
\ No newline at end of file
+- [Users and roles](users_and_roles.md)
diff --git a/v2.2.x/site/en/getstarted/milvus_lite.md b/v2.2.x/site/en/getstarted/milvus_lite.md
index cc16f55e2..add3d7d05 100644
--- a/v2.2.x/site/en/getstarted/milvus_lite.md
+++ b/v2.2.x/site/en/getstarted/milvus_lite.md
@@ -185,7 +185,7 @@ You can configure Milvus Lite in the same way as you do Milvus instances through
$ milvus-server --extra-config dataCoord.segment.maxSize=1024
```
-You can find all configurable configuration items in `config.yaml` template shipped with the Milvus package. You can also find [the template in our project repository](https://github.com/milvus-io/milvus-lite/blob/main/src/milvus/data/config.yaml.template).
+You can find all configurable configuration items in `config.yaml` template shipped with the Milvus package.
## What's next
diff --git a/v2.2.x/site/en/reference/benchmark.md b/v2.2.x/site/en/reference/benchmark.md
index 53990c9c4..a2303a62e 100644
--- a/v2.2.x/site/en/reference/benchmark.md
+++ b/v2.2.x/site/en/reference/benchmark.md
@@ -10,7 +10,7 @@ This report shows the major test results of Milvus 2.2.0. It aims to provide a p
-
+
+
We have recently run a benchmark against Milvus 2.2.3 and have the following key findings:
@@ -20,7 +20,7 @@ This report shows the major test results of Milvus 2.2.0. It aims to provide a pFor details, welcome referring to this whitepaper and related benchmark test code.
+For details, welcome referring to this whitepaper.
-
- If You have set `service.beta.kubernetes.io/aws-load-balancer-scheme` to `internal` in `milvus.yaml`. You should run the example code within the EKS VPC.
-
-
```python
...
@@ -478,7 +472,7 @@ In case you need to restore the environment by uninstalling Milvus, destroying t
2. Destroy the EKS cluster.
```shell
- eksctl delete cluster --name milvus-eks-cluster
+ eksctl delete cluster --name milvus-eks-cluster --region us-east-2
```
3. Delete the AWS S3 bucket and related IAM policies.
@@ -486,6 +480,8 @@ In case you need to restore the environment by uninstalling Milvus, destroying t
You should replace the bucket name and policy ARN with your own.
```shell
+ aws s3 rm s3://milvus-bucket-039dd013c0712f085d60e21f --recursive
+
aws s3api delete-bucket --bucket milvus-bucket-039dd013c0712f085d60e21f --region us-east-2
aws iam delete-policy --policy-arn 'arn:aws:iam::12345678901:policy/MilvusS3ReadWrite'
diff --git a/v2.3.x/site/en/adminGuide/clouds/azure/ingress.md b/v2.3.x/site/en/adminGuide/clouds/azure/ingress.md
index 2aa83f9d9..aed0f660b 100644
--- a/v2.3.x/site/en/adminGuide/clouds/azure/ingress.md
+++ b/v2.3.x/site/en/adminGuide/clouds/azure/ingress.md
@@ -5,7 +5,7 @@ related_key: ingress nginx
summary: Learn how to configure ingress nginx with Milvus.
---
-## Configure ingress nginx with Milvus
+# Configure ingress nginx with Milvus
This topic introduces how to configure ingress nginx with Milvus.
For more details, refer to [ingress-nginx](https://learn.microsoft.com/en-us/azure/aks/ingress-tls?tabs=azure-cli).
diff --git a/v2.3.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md b/v2.3.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md
index d7e33a7c3..c1fbea405 100644
--- a/v2.3.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md
+++ b/v2.3.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md
@@ -5,7 +5,7 @@ related_key: cluster
summary: Learn how to deploy a Milvus cluster behind a Layer-7 load balancer on GCP.
---
-## Set up a Layer-7 Load Balancer for Milvus on GCP
+# Set up a Layer-7 Load Balancer for Milvus on GCP
When compared to a Layer-4 load balancer, a Layer-7 load balancer offers smart load balancing and caching capabilities and is a great choice for cloud-native services.
@@ -259,4 +259,4 @@ connections.connect("default", host="34.111.144.65", port="443", secure=True, se
- The IP address and port number in **host** and **port** should match those listed at the end of [Create an Ingress to generate a Layer-7 Load Balancer](#create-an-ingress-to-generate-a-layer-7-load-balancer).
- If you have set up a DNS record to map domain name to the host IP address, replace the IP address in **host** with the domain name and omit **server_name**.
-
\ No newline at end of file
+
diff --git a/v2.3.x/site/en/getstarted/milvus_lite.md b/v2.3.x/site/en/getstarted/milvus_lite.md
index cc16f55e2..add3d7d05 100644
--- a/v2.3.x/site/en/getstarted/milvus_lite.md
+++ b/v2.3.x/site/en/getstarted/milvus_lite.md
@@ -185,7 +185,7 @@ You can configure Milvus Lite in the same way as you do Milvus instances through
$ milvus-server --extra-config dataCoord.segment.maxSize=1024
```
-You can find all configurable configuration items in `config.yaml` template shipped with the Milvus package. You can also find [the template in our project repository](https://github.com/milvus-io/milvus-lite/blob/main/src/milvus/data/config.yaml.template).
+You can find all configurable configuration items in `config.yaml` template shipped with the Milvus package.
## What's next
diff --git a/v2.3.x/site/en/migrate/migrate_overview.md b/v2.3.x/site/en/migrate/migrate_overview.md
index 775c03a83..1dff41927 100644
--- a/v2.3.x/site/en/migrate/migrate_overview.md
+++ b/v2.3.x/site/en/migrate/migrate_overview.md
@@ -2,7 +2,6 @@
id: migrate_overview.md
title: Migrate Overview
summary: MilvusDM allows data migration between Milvus and many other sources of data.
-title: Overview
---
# Overview
diff --git a/v2.3.x/site/en/reference/benchmark.md b/v2.3.x/site/en/reference/benchmark.md
index b4e61db42..a356098c6 100644
--- a/v2.3.x/site/en/reference/benchmark.md
+++ b/v2.3.x/site/en/reference/benchmark.md
@@ -11,7 +11,7 @@ This report shows the major test results of Milvus 2.2.0. It aims to provide a p
-
+
+
We have recently run a benchmark against Milvus 2.2.3 and have the following key findings:
diff --git a/v2.3.x/site/en/reference/sys_config/configure_knowhere.md b/v2.3.x/site/en/reference/sys_config/configure_knowhere.md index 12baead16..9ab1798b0 100644 --- a/v2.3.x/site/en/reference/sys_config/configure_knowhere.md +++ b/v2.3.x/site/en/reference/sys_config/configure_knowhere.md @@ -10,7 +10,7 @@ title: Knowhere-related Configurations This topic introduces the knowhere-related configurations of Milvus. -[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus. +Knowhere is the search engine of Milvus. Under this section, you can configure the default SIMD instruction set type of the system. diff --git a/v2.3.x/site/en/reference/sys_config/system_configuration.md b/v2.3.x/site/en/reference/sys_config/system_configuration.md index 7e271f422..61b543af4 100644 --- a/v2.3.x/site/en/reference/sys_config/system_configuration.md +++ b/v2.3.x/site/en/reference/sys_config/system_configuration.md @@ -160,7 +160,7 @@ See [Common Configurations](configure_common.md) for detailed description for ea ### `knowhere` -[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus. +Knowhere is the search engine of Milvus. Under this section, you can configure the default SIMD instruction set type of the system. diff --git a/v2.3.x/site/en/tutorials/audio_similarity_search.md b/v2.3.x/site/en/tutorials/audio_similarity_search.md index 4a8468f28..3c3550fff 100644 --- a/v2.3.x/site/en/tutorials/audio_similarity_search.md +++ b/v2.3.x/site/en/tutorials/audio_similarity_search.md @@ -7,7 +7,6 @@ title: Audio Similarity Search # Audio Similarity Search This tutorial demonstrates how to use Milvus, the open-source vector database to build an audio similarity search system. -- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/audio/audio_similarity_search/audio_similarity_search.ipynb) The ML model and third-party software used include: - PANNs (Large-Scale Pretrained Audio Neural Networks) diff --git a/v2.3.x/site/en/tutorials/dna_sequence_classification.md b/v2.3.x/site/en/tutorials/dna_sequence_classification.md index 39a792c48..e84371264 100644 --- a/v2.3.x/site/en/tutorials/dna_sequence_classification.md +++ b/v2.3.x/site/en/tutorials/dna_sequence_classification.md @@ -7,8 +7,6 @@ title: DNA Sequence Classification # DNA Sequence Classification This tutorial demonstrates how to use Milvus, the open-source vector database, to build a DNA sequence classification model. -- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/medical/dna_sequence_classification/dna_sequence_classification.ipynb) -- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/medical/dna_sequence_classification/quick_deploy) The ML model and third-party software used include: - CountVectorizer diff --git a/v2.3.x/site/en/tutorials/image_similarity_search.md b/v2.3.x/site/en/tutorials/image_similarity_search.md index d795e5e31..d4bf0e0e6 100644 --- a/v2.3.x/site/en/tutorials/image_similarity_search.md +++ b/v2.3.x/site/en/tutorials/image_similarity_search.md @@ -8,7 +8,6 @@ title: Image Similarity Search This tutorial demonstrates how to use Milvus, the open-source vector database, to build a reverse image search system. - [Open Jupyter notebook](https://github.com/towhee-io/examples/tree/main/image/reverse_image_search) -- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/image/reverse_image_search/quick_deploy) - [Try online demo](https://milvus.io/milvus-demos/) The ML models and third-party software used include: diff --git a/v2.3.x/site/en/tutorials/recommendation_system.md b/v2.3.x/site/en/tutorials/recommendation_system.md index 3bdebe99f..cee4bf2db 100644 --- a/v2.3.x/site/en/tutorials/recommendation_system.md +++ b/v2.3.x/site/en/tutorials/recommendation_system.md @@ -7,7 +7,6 @@ title: Recommender System # Recommender System This tutorial demonstrates how to use Milvus, the open-source vector database, to build a recommendation system. -- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/recommender_system.ipynb) The ML model and third-party software used include: - PaddlePaddle diff --git a/v2.3.x/site/en/userGuide/tools/birdwatcher_usage_guides.md b/v2.3.x/site/en/userGuide/tools/birdwatcher_usage_guides.md index 7b2e391ae..fa6267d46 100644 --- a/v2.3.x/site/en/userGuide/tools/birdwatcher_usage_guides.md +++ b/v2.3.x/site/en/userGuide/tools/birdwatcher_usage_guides.md @@ -1,6 +1,6 @@ --- id: birdwatcher_usage_guides.md -summary: Learn how to install Birdwatch to debug Milvus. +summary: Learn how to use Birdwatch to debug Milvus. title: Use Birdwatcher --- diff --git a/v2.4.x/site/en/adminGuide/backup/h2m.md b/v2.4.x/site/en/adminGuide/backup/h2m.md index 56e3be787..4faec6718 100644 --- a/v2.4.x/site/en/adminGuide/backup/h2m.md +++ b/v2.4.x/site/en/adminGuide/backup/h2m.md @@ -3,6 +3,7 @@ id: h2m.md title: HDF5 to Milvus related_key: HDF5, migrate, import summary: Import HDF5 files to Milvus. +deprecate: true --- # Migrate Data from HDF5 to Milvus diff --git a/v2.4.x/site/en/adminGuide/backup/m2h.md b/v2.4.x/site/en/adminGuide/backup/m2h.md index f43fae527..87f7f1f9e 100644 --- a/v2.4.x/site/en/adminGuide/backup/m2h.md +++ b/v2.4.x/site/en/adminGuide/backup/m2h.md @@ -3,6 +3,7 @@ id: m2h.md title: Migrate from Milvus to HDF5 related_key: Milvus to HDF5 summary: Save Milvus data as HDF5 files. +deprecate: true --- # Migrate Data from Milvus to HDF5 diff --git a/v2.4.x/site/en/adminGuide/chunk_cache.md b/v2.4.x/site/en/adminGuide/chunk_cache.md index a49a551b5..d75b55a32 100644 --- a/v2.4.x/site/en/adminGuide/chunk_cache.md +++ b/v2.4.x/site/en/adminGuide/chunk_cache.md @@ -37,11 +37,11 @@ queryNode: warmup: async ``` -The `warmup` parameter determines whether Milvus pre-loads data from the object storage into the cache on the local hard disk of the query nodes before it is needed. This parameter defaults to `async`. Possible options are as follows: +The `warmup` parameter determines whether Milvus pre-loads data from the object storage into the cache on the local hard disk of the query nodes before it is needed. This parameter defaults to `disable`. Possible options are as follows: - `async`: Milvus pre-loads data asynchronously in the background, which does not affect the time it takes to load a collection. However, users may experience a delay when retrieving vectors for a short period of time after the load process is complete. This is the default option. - `sync`: Milvus pre-loads data synchronously, which may affect the time it takes to load a collection. However, users can perform queries immediately after the load process is complete without any delay. -- `off`: Milvus does not pre-load data into the memory cache. +- `disable`: Milvus does not pre-load data into the memory cache. Note that the chunk cache settings also apply when new data is inserted into collections or the collection indexes are rebuilt. diff --git a/v2.4.x/site/en/adminGuide/clouds/aws/aws.md b/v2.4.x/site/en/adminGuide/clouds/aws/aws.md index 96f5fadb6..e5920a724 100644 --- a/v2.4.x/site/en/adminGuide/clouds/aws/aws.md +++ b/v2.4.x/site/en/adminGuide/clouds/aws/aws.md @@ -5,10 +5,16 @@ related_key: cluster summary: Learn how to deploy a Milvus cluster on AWS EC2. --- -# Deploy a Milvus Cluster on EC2 +# (Deprecated) Deploy a Milvus Cluster on EC2 This topic describes how to deploy a Milvus cluster on [Amazon EC2](https://docs.aws.amazon.com/ec2/) with Terraform and Ansible. +
+
+This topic is outdated and will be removed soon. You are advised to refer to [Deploy a Milvus Cluster on EKS](eks.md) instead.
+
+
+
## Provision a Milvus cluster
This section describes how to use Terraform to provision a Milvus cluster.
diff --git a/v2.4.x/site/en/adminGuide/clouds/aws/eks.md b/v2.4.x/site/en/adminGuide/clouds/aws/eks.md
index dcd99d31a..9fd15df97 100644
--- a/v2.4.x/site/en/adminGuide/clouds/aws/eks.md
+++ b/v2.4.x/site/en/adminGuide/clouds/aws/eks.md
@@ -158,7 +158,7 @@ You can set up the required AWS resources, including an AWS S3 bucket and an EKS
namespace: milvus
labels: {aws-usage: "milvus"}
attachPolicyARNs:
- - "MilvusS3ReadWrite_Policy_ARN"
+ - "MilvusS3ReadWrite_Policy_ARN" # arn:aws:iam::12345678901:policy/MilvusS3ReadWrite
managedNodeGroups:
- name: milvus-node-group
@@ -268,7 +268,6 @@ In this guide, we will use Milvus Helm Charts to deploy a Milvus cluster. You ca
- To configure HA for your Milvus, refer to [this calculator](https://milvus.io/tools/sizing/) for more information. You can download the related configurations directly from the calculator, and you should remove MinIO-related configurations.
- To implement multi-replica deployments of coordinators, set `xxCoordinator.activeStandby.enabled` to `true`.
- - To access your Milvus from the Internet, change `service.beta.kubernetes.io/aws-load-balancer-scheme` from `internal` to `internet-facing`.
-
- If You have set `service.beta.kubernetes.io/aws-load-balancer-scheme` to `internal` in `milvus.yaml`. You should run the example code within the EKS VPC.
-
-
```python
...
@@ -478,7 +472,7 @@ In case you need to restore the environment by uninstalling Milvus, destroying t
2. Destroy the EKS cluster.
```shell
- eksctl delete cluster --name milvus-eks-cluster
+ eksctl delete cluster --name milvus-eks-cluster --region us-east-2
```
3. Delete the AWS S3 bucket and related IAM policies.
@@ -486,6 +480,8 @@ In case you need to restore the environment by uninstalling Milvus, destroying t
You should replace the bucket name and policy ARN with your own.
```shell
+ aws s3 rm s3://milvus-bucket-039dd013c0712f085d60e21f --recursive
+
aws s3api delete-bucket --bucket milvus-bucket-039dd013c0712f085d60e21f --region us-east-2
aws iam delete-policy --policy-arn 'arn:aws:iam::12345678901:policy/MilvusS3ReadWrite'
diff --git a/v2.4.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md b/v2.4.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md
index d7e33a7c3..c1fbea405 100644
--- a/v2.4.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md
+++ b/v2.4.x/site/en/adminGuide/clouds/gcp/gcp_layer7.md
@@ -5,7 +5,7 @@ related_key: cluster
summary: Learn how to deploy a Milvus cluster behind a Layer-7 load balancer on GCP.
---
-## Set up a Layer-7 Load Balancer for Milvus on GCP
+# Set up a Layer-7 Load Balancer for Milvus on GCP
When compared to a Layer-4 load balancer, a Layer-7 load balancer offers smart load balancing and caching capabilities and is a great choice for cloud-native services.
@@ -259,4 +259,4 @@ connections.connect("default", host="34.111.144.65", port="443", secure=True, se
- The IP address and port number in **host** and **port** should match those listed at the end of [Create an Ingress to generate a Layer-7 Load Balancer](#create-an-ingress-to-generate-a-layer-7-load-balancer).
- If you have set up a DNS record to map domain name to the host IP address, replace the IP address in **host** with the domain name and omit **server_name**.
-
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
Or with old connections.connect API (not recommended):
diff --git a/v2.4.x/site/en/integrations/integrate_with_fastgpt.md b/v2.4.x/site/en/integrations/integrate_with_fastgpt.md
new file mode 100644
index 000000000..84d25592f
--- /dev/null
+++ b/v2.4.x/site/en/integrations/integrate_with_fastgpt.md
@@ -0,0 +1,54 @@
+---
+id: integrate_with_fastgpt.md
+summary: This tutorial will guide you on how to swiftly deploy your own exclusive FastGPT application using [Milvus](https://milvus.io/).
+title: Deploying FastGPT with Milvus
+---
+
+# Deploying FastGPT with Milvus
+
+[FastGPT](https://fastgpt.in/) is a knowledge-based question and answer system built on the LLM large language model, offering ready-to-use capabilities for data processing and model invocation. Furthermore, it enables workflow orchestration through Flow visualization, thus facilitating complex question and answer scenarios. This tutorial will guide you on how to swiftly deploy your own exclusive FastGPT application using [Milvus](https://milvus.io/).
+
+## Download docker-compose.yml
+Ensure that you have already installed [Docker Compose](https://docs.docker.com/compose/).
+Execute the command below to download the docker-compose.yml file.
+```shell
+$ mkdir fastgpt
+$ cd fastgpt
+$ curl -O https://raw.githubusercontent.com/labring/FastGPT/main/projects/app/data/config.json
+
+# milvus version
+$ curl -o docker-compose.yml https://raw.githubusercontent.com/labring/FastGPT/main/files/docker/docker-compose-milvus.yml
+# zilliz version
+# curl -o docker-compose.yml https://raw.githubusercontent.com/labring/FastGPT/main/files/docker/docker-compose-zilliz.yml
+```
+> If you're using the Zilliz version, adjust the `MILVUS_ADDRESS` and `MILVUS_TOKEN` link parameters in the docker-compose.yml file, which corresponds to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in [Zilliz Cloud](https://zilliz.com/cloud).
+
+## Launch the Container
+Execute in the same directory as docker-compose.yml. Ensure that the docker-compose version is ideally above 2.17, as some automation commands may not function otherwise.
+```shell
+# Launch the container
+$ docker-compose up -d
+# Wait for 10s, OneAPI typically needs to restart a few times to initially connect to Mysql
+$ sleep 10
+# Restart oneapi (Due to certain issues with the default Key of OneAPI, it will display 'channel not found' if not restarted, this can be temporarily resolved by manually restarting once, while waiting for the author's fix)
+$ docker restart oneapi
+```
+
+## Access OneAPI to Add Models
+OneAPI can be accessed at `ip:3001`. The default username is root, and the password is 123456. You can alter the password after logging in.
+Using OpenAI's model as an example, click on the "Channel" tab, and select your chat model and embedding model under "Models".
+Input your [OpenAI API Key](https://platform.openai.com/docs/quickstart) in the "Secrets" section.
+For the use of models beyond OpenAI, and further information, please consult [One API](https://doc.fastgpt.in/docs/development/one-api/).
+
+## Setting Tokens
+Click on the "Tokens" tab. By default, there is a token `Initial Root Token`. You can also create a new token and set a quota on your own.
+Click "Copy" on your token, ensuring that the value of this token matches the `CHAT_API_KEY` value set in the docker-compose.yml file.
+
+## Accessing FastGPT
+At present, FastGPT can be directly accessed at `ip:3000` (please mind the firewall). The login username is root, with the password set to `DEFAULT_ROOT_PSW` within the docker-compose.yml environment variable. Should you require domain name access, you would need to install and configure [Nginx](https://nginx.org/en/) on your own.
+
+## Stop the Container
+Run the following command to stop the container.
+```shell
+$ docker-compose down
+```
\ No newline at end of file
diff --git a/v2.4.x/site/en/integrations/integrate_with_haystack.md b/v2.4.x/site/en/integrations/integrate_with_haystack.md
index 296623702..1e7082f6d 100644
--- a/v2.4.x/site/en/integrations/integrate_with_haystack.md
+++ b/v2.4.x/site/en/integrations/integrate_with_haystack.md
@@ -10,7 +10,7 @@ title: Retrieval-Augmented Generation (RAG) with Milvus and Haystack
This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using Haystack and Milvus.
-The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using a vector similarity search engine like Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
[Haystack](https://haystack.deepset.ai/) is the open source Python framework by deepset for building custom apps with large language models (LLMs). [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
@@ -22,12 +22,12 @@ Before running this notebook, make sure you have the following dependencies inst
```python
-$ pip install --upgrade --quiet pymilvus milvus-haystack markdown-it-py mdit_plain
+! pip install --upgrade --quiet pymilvus milvus-haystack markdown-it-py mdit_plain
```
-If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**.
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime** (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
@@ -69,14 +69,31 @@ from haystack.components.converters import MarkdownToDocument
from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
+from haystack.utils import Secret
from milvus_haystack import MilvusDocumentStore
from milvus_haystack.milvus_embedding_retriever import MilvusEmbeddingRetriever
+
document_store = MilvusDocumentStore(
connection_args={"uri": "./milvus.db"},
+ # connection_args={"uri": "http://localhost:19530"},
+ # connection_args={"uri": YOUR_ZILLIZ_CLOUD_URI, "token": Secret.from_env_var("ZILLIZ_CLOUD_API_KEY")},
drop_old=True,
)
+```
+
+
+
+For the connection_args:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+
+```python
indexing_pipeline = Pipeline()
indexing_pipeline.add_component("converter", MarkdownToDocument())
indexing_pipeline.add_component(
@@ -196,3 +213,5 @@ print("RAG answer:", results["generator"]["replies"][0])
RAG answer: The painting "Warrior" is currently stored in the Malcolm Collection in the British Museum.
+
+For more information about how to use milvus-haystack, please refer to the [milvus-haystack Readme](https://github.com/milvus-io/milvus-haystack).
diff --git a/v2.4.x/site/en/integrations/integrate_with_hugging-face.md b/v2.4.x/site/en/integrations/integrate_with_hugging-face.md
index b589232af..c5c956904 100644
--- a/v2.4.x/site/en/integrations/integrate_with_hugging-face.md
+++ b/v2.4.x/site/en/integrations/integrate_with_hugging-face.md
@@ -114,7 +114,7 @@ data_list = data.to_list()
Now we have question-answer pairs ready with question embeddings. The next step is to insert them into the vector database.
-We will first need to connect to Milvus service and create a Milvus collection. This section will use [Milvus Lite](https://milvus.io/docs/milvus_lite.md) as example. If you want to use other types of Milvus or [Zilliz Cloud](https://zilliz.com), please make sure you have started the service and connect with your own URI & credentials. You are also able to change parameters to customize your collection.
+We will first need to connect to Milvus service and create a Milvus collection.
```python
@@ -138,6 +138,15 @@ milvus_client.create_collection(
)
```
+
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
Insert all data into the collection:
diff --git a/v2.4.x/site/en/integrations/integrate_with_jina.md b/v2.4.x/site/en/integrations/integrate_with_jina.md
index 0d6ed6467..8d9b67ed7 100644
--- a/v2.4.x/site/en/integrations/integrate_with_jina.md
+++ b/v2.4.x/site/en/integrations/integrate_with_jina.md
@@ -148,6 +148,15 @@ res = milvus_client.insert(collection_name=COLLECTION_NAME, data=data)
print(res["insert_count"])
```
+
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
With all data in Milvus vector database, we can now perform semantic search by generating vector embedding for the query and conduct vector search.
@@ -199,5 +208,3 @@ rf(query, documents)
RerankResult(text='The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.', score=0.35420963168144226, index=3),
RerankResult(text="In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.", score=0.3498658835887909, index=0),
RerankResult(text='In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.', score=0.2728956639766693, index=2)]
-
-
diff --git a/v2.4.x/site/en/integrations/integrate_with_langchain.md b/v2.4.x/site/en/integrations/integrate_with_langchain.md
index f6e85e4f2..ad6547357 100644
--- a/v2.4.x/site/en/integrations/integrate_with_langchain.md
+++ b/v2.4.x/site/en/integrations/integrate_with_langchain.md
@@ -10,7 +10,7 @@ title: Retrieval-Augmented Generation (RAG) with Milvus and LangChain
This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LangChain and Milvus.
-The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using a vector similarity search engine like Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
[LangChain](https://www.langchain.com/) is a framework for developing applications powered by large language models (LLMs). [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
@@ -54,7 +54,10 @@ from langchain_text_splitters import RecursiveCharacterTextSplitter
# Create a WebBaseLoader instance to load documents from web sources
loader = WebBaseLoader(
- web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
+ web_paths=(
+ "https://lilianweng.github.io/posts/2023-06-23-agent/",
+ "https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/",
+ ),
bs_kwargs=dict(
parse_only=bs4.SoupStrainer(
class_=("post-content", "post-title", "post-header")
@@ -88,12 +91,12 @@ We will initialize a Milvus vector store with the documents, which load the docu
```python
-from langchain_milvus import Milvus
+from langchain_milvus import Milvus, Zilliz
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
-vectorstore = Milvus.from_documents(
+vectorstore = Milvus.from_documents( # or Zilliz.from_documents
documents=docs,
embedding=embeddings,
connection_args={
@@ -103,24 +106,27 @@ vectorstore = Milvus.from_documents(
)
```
- /Users/zilliz/miniforge3/envs/py39/lib/python3.9/site-packages/scipy/__init__.py:155: UserWarning: A NumPy version >=1.18.5 and <1.26.0 is required for this version of SciPy (detected version 1.26.4
- warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
+
+For the `connection_args`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, replace `Milvus.from_documents` with `Zilliz.from_documents`, and adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
-Search the documents in the Milvus vector store using a test query question. We will get the top 3 documents.
+
+
+Search the documents in the Milvus vector store using a test query question. Let's take a look at the top 1 document.
```python
query = "What is self-reflection of an AI Agent?"
-vectorstore.similarity_search(query, k=3)
+vectorstore.similarity_search(query, k=1)
```
- [Document(page_content='Self-Reflection#\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\nThought: ...\nAction: ...\nObservation: ...\n... (Repeated many times)', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'pk': 449757808726900738}),
- Document(page_content='Fig. 2. Examples of reasoning trajectories for knowledge-intensive tasks (e.g. HotpotQA, FEVER) and decision-making tasks (e.g. AlfWorld Env, WebShop). (Image source: Yao et al. 2023).\nIn both experiments on knowledge-intensive tasks and decision-making tasks, ReAct works better than the Act-only baseline where Thought: … step is removed.\nReflexion (Shinn & Labash 2023) is a framework to equips agents with dynamic memory and self-reflection capabilities to improve reasoning skills. Reflexion has a standard RL setup, in which the reward model provides a simple binary reward and the action space follows the setup in ReAct where the task-specific action space is augmented with language to enable complex reasoning steps. After each action $a_t$, the agent computes a heuristic $h_t$ and optionally may decide to reset the environment to start a new trial depending on the self-reflection results.\n\nFig. 3. Illustration of the Reflexion framework. (Image source: Shinn & Labash, 2023)\nThe heuristic function determines when the trajectory is inefficient or contains hallucination and should be stopped. Inefficient planning refers to trajectories that take too long without success. Hallucination is defined as encountering a sequence of consecutive identical actions that lead to the same observation in the environment.\nSelf-reflection is created by showing two-shot examples to LLM and each example is a pair of (failed trajectory, ideal reflection for guiding future changes in the plan). Then reflections are added into the agent’s working memory, up to three, to be used as context for querying LLM.', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'pk': 449757808726900739}),
- Document(page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\nSelf-Reflection#', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'pk': 449757808726900737})]
+ [Document(page_content='Self-Reflection#\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\nThought: ...\nAction: ...\nObservation: ...\n... (Repeated many times)', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'pk': 449281835035555859})]
@@ -186,8 +192,120 @@ res
- "Self-reflection of an AI agent involves the process of synthesizing memories into higher-level inferences over time to guide the agent's future behavior. It includes recency, importance, and relevance factors to determine the salient high-level questions and optimize believability. This mechanism helps the agent improve iteratively by refining past action decisions and correcting previous mistakes."
+ "Self-reflection of an AI agent involves the process of synthesizing memories into higher-level inferences over time to guide the agent's future behavior. It serves as a mechanism to create higher-level summaries of past events. One approach to self-reflection involves prompting the language model with the 100 most recent observations and asking it to generate the 3 most salient high-level questions based on those observations. This process helps the AI agent optimize believability in the current moment and over time."
+
+
+
+Congratulations! You have built a basic RAG chain powered by Milvus and LangChain.
+
+## Metadata filtering
+
+We can use the [Milvus Scalar Filtering Rules](https://milvus.io/docs/boolean.md) to filter the documents based on metadata. We have loaded the documents from two different sources, and we can filter the documents by the metadata `source`.
+
+
+```python
+vectorstore.similarity_search(
+ "What is CoT?",
+ k=1,
+ expr="source == 'https://lilianweng.github.io/posts/2023-06-23-agent/'",
+)
+
+# The same as:
+# vectorstore.as_retriever(search_kwargs=dict(
+# k=1,
+# expr="source == 'https://lilianweng.github.io/posts/2023-06-23-agent/'",
+# )).invoke("What is CoT?")
+```
+
+
+
+
+ [Document(page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\nSelf-Reflection#', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'pk': 449281835035555858})]
+
+
+
+If we want to dynamically change the search parameters without rebuilding the chain, we can [configure the runtime chain internals](https://python.langchain.com/v0.2/docs/how_to/configure/) . Let's define a new retriever with this dynamically configure and use it to build a new RAG chain.
+
+
+```python
+from langchain_core.runnables import ConfigurableField
+
+# Define a new retriever with a configurable field for search_kwargs
+retriever2 = vectorstore.as_retriever().configurable_fields(
+ search_kwargs=ConfigurableField(
+ id="retriever_search_kwargs",
+ )
+)
+
+# Invoke the retriever with a specific search_kwargs which filter the documents by source
+retriever2.with_config(
+ configurable={
+ "retriever_search_kwargs": dict(
+ expr="source == 'https://lilianweng.github.io/posts/2023-06-23-agent/'",
+ k=1,
+ )
+ }
+).invoke(query)
+```
+
+
+
+
+ [Document(page_content='Self-Reflection#\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\nThought: ...\nAction: ...\nObservation: ...\n... (Repeated many times)', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'pk': 449281835035555859})]
+
+
+
+
+```python
+# Define a new RAG chain with this dynamically configurable retriever
+rag_chain2 = (
+ {"context": retriever2 | format_docs, "question": RunnablePassthrough()}
+ | prompt
+ | llm
+ | StrOutputParser()
+)
+```
+
+Let's try this dynamically configurable RAG chain with different filter conditions.
+
+
+```python
+# Invoke this RAG chain with a specific question and config
+rag_chain2.with_config(
+ configurable={
+ "retriever_search_kwargs": dict(
+ expr="source == 'https://lilianweng.github.io/posts/2023-06-23-agent/'",
+ )
+ }
+).invoke(query)
+```
+
+
+
+
+ "Self-reflection of an AI agent involves the process of synthesizing memories into higher-level inferences over time to guide the agent's future behavior. It serves as a mechanism to create higher-level summaries of past events. One approach to self-reflection involves prompting the language model with the 100 most recent observations and asking it to generate the 3 most salient high-level questions based on those observations. This process helps the AI agent optimize believability in the current moment and over time."
+
+
+
+When we change the search condition to filter the documents by the second source, as the content of this blog source has nothing todo with the query question, we get an answer with no relevant information.
+
+
+```python
+rag_chain2.with_config(
+ configurable={
+ "retriever_search_kwargs": dict(
+ expr="source == 'https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/'",
+ )
+ }
+).invoke(query)
+```
+
+
+
+
+ "I'm sorry, but based on the provided context, there is no specific information or statistical data available regarding the self-reflection of an AI agent."
-This tutorial build a vanilla RAG chain powered by Milvus. For more advanced RAG techniques, please refer to the [advanced rag bootcamp](https://github.com/milvus-io/bootcamp/tree/master/bootcamp/RAG/advanced_rag).
\ No newline at end of file
+----
+This tutorial focus the basic usage of Milvus LangChain integration and simple RAG approach. For more advanced RAG techniques, please refer to the [advanced rag bootcamp](https://github.com/milvus-io/bootcamp/tree/master/bootcamp/RAG/advanced_rag).
diff --git a/v2.4.x/site/en/integrations/integrate_with_llamaindex.md b/v2.4.x/site/en/integrations/integrate_with_llamaindex.md
index d2ab177e8..dcc8aa07f 100644
--- a/v2.4.x/site/en/integrations/integrate_with_llamaindex.md
+++ b/v2.4.x/site/en/integrations/integrate_with_llamaindex.md
@@ -10,7 +10,7 @@ title: Retrieval-Augmented Generation (RAG) with Milvus and LlamaIndex
This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LlamaIndex and Milvus.
-The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using a vector similarity search engine like Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
[LlamaIndex](https://www.llamaindex.ai/) is a simple, flexible data framework for connecting custom data sources to large language models (LLMs). [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
@@ -59,54 +59,40 @@ You can download sample data with the following commands:
```python
-$ mkdir -p 'data/paul_graham/'
-$ wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'
+! mkdir -p 'data/'
+! wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham_essay.txt'
+! wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O 'data/uber_2021.pdf'
```
## Getting Started
### Generate our data
-As a first example, lets generate a document from the file found in the `data/paul_graham/` folder. In this folder there is a single essay from Paul Graham titled `What I Worked On`. To generate the documents we will use the SimpleDirectoryReader.
+As a first example, lets generate a document from the file `paul_graham_essay.txt`. It is a single essay from Paul Graham titled `What I Worked On`. To generate the documents we will use the SimpleDirectoryReader.
```python
from llama_index.core import SimpleDirectoryReader
# load documents
-documents = SimpleDirectoryReader("./data/paul_graham/").load_data()
+documents = SimpleDirectoryReader(
+ input_files=["./data/paul_graham_essay.txt"]
+).load_data()
print("Document ID:", documents[0].doc_id)
```
- Document ID: 11c3a6fe-799e-4e40-8122-2339936c2722
+ Document ID: 95f25e4d-f270-4650-87ce-006d69d82033
### Create an index across the data
-Now that we have a document, we can can create an index and insert the document. For the index we will use a GPTMilvusIndex. GPTMilvusIndex takes in a few arguments:
-
-- `uri (str, optional)`: The URI to connect to, comes in the form of "https://address:port" if using Milvus or Zilliz Cloud service, or "path/to/local/milvus.db" if using a lite local Milvus. Defaults to "./milvus_llamaindex.db".
-- `token (str, optional)`: The token for log in. Empty if not using rbac, if using rbac it will most likely be "username:password". Defaults to "".
-- `collection_name (str, optional)`: The name of the collection where data will be stored. Defaults to "llamalection".
-- `dim (int, optional)`: The dimension of the embeddings. If it is not provided, collection creation will be done on first insert. Defaults to None.
-- `embedding_field (str, optional)`: The name of the embedding field for the collection, defaults to DEFAULT_EMBEDDING_KEY.
-- `doc_id_field (str, optional)`: The name of the doc_id field for the collection, defaults to DEFAULT_DOC_ID_KEY.
-- `similarity_metric (str, optional)`: The similarity metric to use, currently supports IP and L2. Defaults to "IP".
-- `consistency_level (str, optional)`: Which consistency level to use for a newly created collection. Defaults to "Strong".
-- `overwrite (bool, optional)`: Whether to overwrite existing collection with same name. Defaults to False.
-- `text_key (str, optional)`: What key text is stored in in the passed collection. Used when bringing your own collection. Defaults to None.
-- `index_config (dict, optional)`: The configuration used for building the Milvus index. Defaults to None.
-- `search_config (dict, optional)`: The configuration used for searching the Milvus index. Note that this must be compatible with the index type specified by index_config. Defaults to None.
+Now that we have a document, we can can create an index and insert the document.
-
-
-Please note that **Milvus Lite** requires `pymilvus>=2.4.2`.
-
-
+> Please note that **Milvus Lite** requires `pymilvus>=2.4.2`.
```python
-# Create an index over the documnts
+# Create an index over the documents
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.milvus import MilvusVectorStore
@@ -116,35 +102,35 @@ storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)
```
+
+
+For the parameters of `MilvusVectorStore`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
### Query the data
Now that we have our document stored in the index, we can ask questions against the index. The index will use the data stored in itself as the knowledge base for chatgpt.
```python
-import textwrap
-
-
query_engine = index.as_query_engine()
-response = query_engine.query("What did the author learn?")
-print(textwrap.fill(str(response), 100))
+res = query_engine.query("What did the author learn?")
+print(res)
```
- The author learned about programming on early computers like the IBM 1401 using Fortran, the
- limitations of early computing technology, the transition to microcomputers, and the excitement of
- having a personal computer like the TRS-80. Additionally, the author explored different academic
- paths, initially planning to study philosophy but eventually switching to AI due to a lack of
- interest in philosophy courses. Later on, the author pursued art education, attending RISD and the
- Accademia di Belli Arti in Florence, where they encountered a different approach to teaching art.
+ The author learned that philosophy courses in college were boring to him, leading him to switch his focus to studying AI.
```python
-response = query_engine.query("What was a hard moment for the author?")
-print(textwrap.fill(str(response), 100))
+res = query_engine.query("What challenges did the disease pose for the author?")
+print(res)
```
- Dealing with the stress and challenges related to managing Hacker News was a difficult moment for
- the author.
+ The disease posed challenges for the author as it affected his mother's health, leading to a stroke caused by colon cancer. This resulted in her losing her balance and needing to be placed in a nursing home. The author and his sister were determined to help their mother get out of the nursing home and back to her house.
This next test shows that overwriting removes the previous data.
@@ -162,10 +148,10 @@ index = VectorStoreIndex.from_documents(
)
query_engine = index.as_query_engine()
res = query_engine.query("Who is the author?")
-print("Res:", res)
+print(res)
```
- Res: The author is the individual who created the content or work in question.
+ The author is the individual who created the context information.
The next test shows adding additional data to an already existing index.
@@ -179,17 +165,65 @@ storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)
query_engine = index.as_query_engine()
res = query_engine.query("What is the number?")
-print("Res:", res)
+print(res)
```
- Res: The number is ten.
+ The number is ten.
```python
res = query_engine.query("Who is the author?")
-print("Res:", res)
+print(res)
+```
+
+ Paul Graham
+
+
+## Metadata filtering
+
+We can generate results by filtering specific sources. The following example illustrates loading all documents from the directory and subsequently filtering them based on metadata.
+
+
+```python
+from llama_index.core.vector_stores import ExactMatchFilter, MetadataFilters
+
+# Load all the two documents loaded before
+documents_all = SimpleDirectoryReader("./data/").load_data()
+
+vector_store = MilvusVectorStore(uri="./milvus_demo.db", dim=1536, overwrite=True)
+storage_context = StorageContext.from_defaults(vector_store=vector_store)
+index = VectorStoreIndex.from_documents(documents_all, storage_context)
+```
+
+We want to only retrieve documents from the file `uber_2021.pdf`.
+
+
+```python
+filters = MetadataFilters(
+ filters=[ExactMatchFilter(key="file_name", value="uber_2021.pdf")]
+)
+query_engine = index.as_query_engine(filters=filters)
+res = query_engine.query("What challenges did the disease pose for the author?")
+
+print(res)
+```
+
+ The disease posed challenges related to the adverse impact on the business and operations, including reduced demand for Mobility offerings globally, affecting travel behavior and demand. Additionally, the pandemic led to driver supply constraints, impacted by concerns regarding COVID-19, with uncertainties about when supply levels would return to normal. The rise of the Omicron variant further affected travel, resulting in advisories and restrictions that could adversely impact both driver supply and consumer demand for Mobility offerings.
+
+
+We get a different result this time when retrieve from the file `paul_graham_essay.txt`.
+
+
+```python
+filters = MetadataFilters(
+ filters=[ExactMatchFilter(key="file_name", value="paul_graham_essay.txt")]
+)
+query_engine = index.as_query_engine(filters=filters)
+res = query_engine.query("What challenges did the disease pose for the author?")
+
+print(res)
```
- Res: Paul Graham
+ The disease posed challenges for the author as it affected his mother's health, leading to a stroke caused by colon cancer. This resulted in his mother losing her balance and needing to be placed in a nursing home. The author and his sister were determined to help their mother get out of the nursing home and back to her house.
diff --git a/v2.4.x/site/en/integrations/integrate_with_memgpt.md b/v2.4.x/site/en/integrations/integrate_with_memgpt.md
index 440c74df7..ea5f54c1d 100644
--- a/v2.4.x/site/en/integrations/integrate_with_memgpt.md
+++ b/v2.4.x/site/en/integrations/integrate_with_memgpt.md
@@ -12,6 +12,8 @@ In this example, we're going to use MemGPT to chat with a custom data source whi
## Configuration
+To run MemGPT, you should make sure the Python version >= 3.10.
+
To enable the Milvus backend, make sure to install the required dependencies with:
```shell
diff --git a/v2.4.x/site/en/integrations/integrate_with_openai.md b/v2.4.x/site/en/integrations/integrate_with_openai.md
index 7ab968957..246992162 100644
--- a/v2.4.x/site/en/integrations/integrate_with_openai.md
+++ b/v2.4.x/site/en/integrations/integrate_with_openai.md
@@ -66,7 +66,7 @@ data = [
# Connect to Milvus, all data is stored in a local file named "milvus_openai_demo.db"
# in current directory. You can also connect to a remote Milvus server following this
# instruction: https://milvus.io/docs/install_standalone-docker.md.
-milvus_client = MilvusClient("milvus_openai_demo.db")
+milvus_client = MilvusClient(uri="milvus_openai_demo.db")
COLLECTION_NAME = "demo_collection" # Milvus collection name
# Create a collection to store the vectors and text.
if milvus_client.has_collection(collection_name=COLLECTION_NAME):
@@ -79,6 +79,15 @@ res = milvus_client.insert(collection_name="demo_collection", data=data)
print(res["insert_count"])
```
+
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
With all data in Milvus vector database, we can now perform semantic search by generating vector embedding for the query and conduct vector search.
diff --git a/v2.4.x/site/en/integrations/integrate_with_ragas.md b/v2.4.x/site/en/integrations/integrate_with_ragas.md
index 3373327d9..e7f911ad3 100644
--- a/v2.4.x/site/en/integrations/integrate_with_ragas.md
+++ b/v2.4.x/site/en/integrations/integrate_with_ragas.md
@@ -10,7 +10,7 @@ title: Evaluation with Ragas
This guide demonstrates how to use Ragas to evaluate a Retrieval-Augmented Generation (RAG) pipeline built upon [Milvus](https://milvus.io/).
-The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using a vector similarity search engine like Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
[Ragas](https://docs.ragas.io/en/latest/index.html#) is a framework that helps you evaluate your RAG pipelines. There are existing tools and frameworks that help you build these pipelines but evaluating it and quantifying your pipeline performance can be hard. This is where Ragas (RAG Assessment) comes in.
@@ -28,7 +28,7 @@ $ pip install --upgrade pymilvus openai requests tqdm pandas ragas
-If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime**.
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime** (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
@@ -162,11 +162,20 @@ Let's initialize the RAG class with OpenAI and Milvus clients.
```python
openai_client = OpenAI()
-milvus_client = MilvusClient("./milvus_demo.db")
+milvus_client = MilvusClient(uri="./milvus_demo.db")
my_rag = RAG(openai_client=openai_client, milvus_client=milvus_client)
```
+
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
## Run the RAG pipeline and get results
We use the [Milvus development guide](https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md) to be as the private knowledge in our RAG, which is a good data source for a simple RAG pipeline.
diff --git a/v2.4.x/site/en/integrations/integrate_with_vanna.md b/v2.4.x/site/en/integrations/integrate_with_vanna.md
new file mode 100644
index 000000000..e08a757f7
--- /dev/null
+++ b/v2.4.x/site/en/integrations/integrate_with_vanna.md
@@ -0,0 +1,376 @@
+---
+id: integrate_with_vanna.md
+summary: This guide demonstrates how to use Vanna to generate and execute SQL queries based on your data stored in a database.
+title: Write SQL with Vanna and Milvus
+---
+
+# Write SQL with Vanna and Milvus
+
+[Vanna](https://vanna.ai/) is an open-source Python RAG (Retrieval-Augmented Generation) framework for SQL generation and related functionality. [Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.
+
+Vanna works in two easy steps - train a RAG "model" on your data, and then ask questions which will return SQL queries that can be set up to run on your database. This guide demonstrates how to use Vanna to generate and execute SQL queries based on your data stored in a database.
+
+
+
+## Prerequisites
+
+Before running this notebook, make sure you have the following dependencies installed:
+
+
+```python
+$ pip install "vanna[milvus,openai]"
+```
+
+
+
+If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime** (Click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).
+
+
+
+And you need set the `OPENAI_API_KEY` in your environment variables. You can get the API key from [OpenAI](https://platform.openai.com/docs/guides/production-best-practices/api-keys).
+
+
+```python
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+```
+
+## Data preparation
+
+First, we need to inherit from the `Milvus_VectorStore` and `OpenAI_Chat` classes from Vanna and define a new class `VannaMilvus` that combines capabilities from both.
+
+
+```python
+from pymilvus import MilvusClient, model
+from vanna.milvus import Milvus_VectorStore
+from vanna.openai import OpenAI_Chat
+
+
+class VannaMilvus(Milvus_VectorStore, OpenAI_Chat):
+ def __init__(self, config=None):
+ Milvus_VectorStore.__init__(self, config=config)
+ OpenAI_Chat.__init__(self, config=config)
+```
+
+We initialize the `VannaMilvus` class with the necessary configuration parameters. We use a `milvus_client` instance to store embeddings and the `model.DefaultEmbeddingFunction()` initialized from [milvus_model](https://milvus.io/docs/embeddings.md) to generate embeddings.C
+
+
+
+As for the argument of `MilvusClient`:
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
+
+```python
+milvus_uri = "./milvus_vanna.db"
+
+milvus_client = MilvusClient(uri=milvus_uri)
+
+vn_milvus = VannaMilvus(
+ config={
+ "api_key": os.getenv("OPENAI_API_KEY"),
+ "model": "gpt-3.5-turbo",
+ "milvus_client": milvus_client,
+ "embedding_function": model.DefaultEmbeddingFunction(),
+ "n_results": 2, # The number of results to return from Milvus semantic search.
+ }
+)
+```
+
+This is a simple example with only a few sample of data, so we set `n_results` to 2 to make sure we search for the top 2 most similar results.
+In practice, you should set `n_results` to a higher value when dealing with larger training dataset.
+
+We will use a sample SQLite database with few tables containing some sample data.
+
+
+```python
+import sqlite3
+
+sqlite_path = "./my-database.sqlite"
+sql_connect = sqlite3.connect(sqlite_path)
+c = sql_connect.cursor()
+
+init_sqls = """
+CREATE TABLE IF NOT EXISTS Customer (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Name TEXT NOT NULL,
+ Company TEXT NOT NULL,
+ City TEXT NOT NULL,
+ Phone TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS Company (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Name TEXT NOT NULL,
+ Industry TEXT NOT NULL,
+ Location TEXT NOT NULL,
+ EmployeeCount INTEGER NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS User (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Username TEXT NOT NULL UNIQUE,
+ Email TEXT NOT NULL UNIQUE
+);
+
+INSERT INTO Customer (Name, Company, City, Phone)
+VALUES ('John Doe', 'ABC Corp', 'New York', '123-456-7890');
+
+INSERT INTO Customer (Name, Company, City, Phone)
+VALUES ('Jane Smith', 'XYZ Inc', 'Los Angeles', '098-765-4321');
+
+INSERT INTO Company (Name, Industry, Location, EmployeeCount)
+VALUES ('ABC Corp', 'cutting-edge technology', 'New York', 100);
+
+INSERT INTO User (Username, Email)
+VALUES ('johndoe123', 'johndoe123@example.com');
+"""
+
+for sql in init_sqls.split(";"):
+ c.execute(sql)
+
+sql_connect.commit()
+
+# Connect to the SQLite database
+vn_milvus.connect_to_sqlite(sqlite_path)
+```
+
+## Train with data
+We can train the model on the DDL data of the SQLite database. We get the DDL data and feed it to the `train` function.
+
+
+```python
+# If there exists training data, we should remove it before training.
+existing_training_data = vn_milvus.get_training_data()
+if len(existing_training_data) > 0:
+ for _, training_data in existing_training_data.iterrows():
+ vn_milvus.remove_training_data(training_data["id"])
+
+# Get the DDL of the SQLite database
+df_ddl = vn_milvus.run_sql("SELECT type, sql FROM sqlite_master WHERE sql is not null")
+
+# Train the model on the DDL data
+for ddl in df_ddl["sql"].to_list():
+ vn_milvus.train(ddl=ddl)
+```
+
+ Adding ddl: CREATE TABLE Customer (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Name TEXT NOT NULL,
+ Company TEXT NOT NULL,
+ City TEXT NOT NULL,
+ Phone TEXT NOT NULL
+ )
+ Adding ddl: CREATE TABLE sqlite_sequence(name,seq)
+ Adding ddl: CREATE TABLE Company (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Name TEXT NOT NULL,
+ Industry TEXT NOT NULL,
+ Location TEXT NOT NULL,
+ EmployeeCount INTEGER NOT NULL
+ )
+ Adding ddl: CREATE TABLE User (
+ ID INTEGER PRIMARY KEY AUTOINCREMENT,
+ Username TEXT NOT NULL UNIQUE,
+ Email TEXT NOT NULL UNIQUE
+ )
+
+
+Besides training on the DDL data, we can also train on the documentation and SQL queries of the database.
+
+
+```python
+# Add documentation about your business terminology or definitions.
+vn_milvus.train(
+ documentation="ABC Corp specializes in cutting-edge technology solutions and innovation."
+)
+vn_milvus.train(
+ documentation="XYZ Inc is a global leader in manufacturing and supply chain management."
+)
+
+# You can also add SQL queries to your training data.
+vn_milvus.train(sql="SELECT * FROM Customer WHERE Name = 'John Doe'")
+```
+
+ Adding documentation....
+ Adding documentation....
+ Using model gpt-3.5-turbo for 65.0 tokens (approx)
+ Question generated with sql: What are the details of the customer named John Doe?
+ Adding SQL...
+
+
+
+
+
+ '595b185c-e6ad-47b0-98fd-0e93ef9b6a0a-sql'
+
+
+
+Let's take a look at the training data.
+
+
+```python
+training_data = vn_milvus.get_training_data()
+training_data
+```
+
+
+
+
+
+
+
+
+
+
+
+
+## Generate SQLs and execute them
+As we have trained with the DDL data, the table structure is now available for generating SQL queries.
+
+Let's try a simple question.
+
+
+```python
+sql = vn_milvus.generate_sql("what is the phone number of John Doe?")
+vn_milvus.run_sql(sql)
+```
+
+ SQL Prompt: [{'role': 'system', 'content': "You are a SQLite expert. Please help to generate a SQL query to answer the question. Your response should ONLY be based on the given context and follow the response guidelines and format instructions. \n===Tables \nCREATE TABLE Customer (\n ID INTEGER PRIMARY KEY AUTOINCREMENT,\n Name TEXT NOT NULL,\n Company TEXT NOT NULL,\n City TEXT NOT NULL,\n Phone TEXT NOT NULL\n)\n\nCREATE TABLE User (\n ID INTEGER PRIMARY KEY AUTOINCREMENT,\n Username TEXT NOT NULL UNIQUE,\n Email TEXT NOT NULL UNIQUE\n)\n\n\n===Additional Context \n\nABC Corp specializes in cutting-edge technology solutions and innovation.\n\nXYZ Inc is a global leader in manufacturing and supply chain management.\n\n===Response Guidelines \n1. If the provided context is sufficient, please generate a valid SQL query without any explanations for the question. \n2. If the provided context is almost sufficient but requires knowledge of a specific string in a particular column, please generate an intermediate SQL query to find the distinct strings in that column. Prepend the query with a comment saying intermediate_sql \n3. If the provided context is insufficient, please explain why it can't be generated. \n4. Please use the most relevant table(s). \n5. If the question has been asked and answered before, please repeat the answer exactly as it was given before. \n"}, {'role': 'user', 'content': 'What are the details of the customer named John Doe?'}, {'role': 'assistant', 'content': "SELECT * FROM Customer WHERE Name = 'John Doe'"}, {'role': 'user', 'content': 'what is the phone number of John Doe?'}]
+ Using model gpt-3.5-turbo for 367.25 tokens (approx)
+ LLM Response: SELECT Phone FROM Customer WHERE Name = 'John Doe'
+
+
+
+
+
+| # | +id | +question | +content | +
|---|---|---|---|
| 0 | +595b185c-e6ad-47b0-98fd-0e93ef9b6a0a-sql | +What are the details of the customer named Joh... | +SELECT * FROM Customer WHERE Name = 'John Doe' | +
| 0 | +25f4956c-e370-4097-994f-996f22d145fa-ddl | +None | +CREATE TABLE Company (\n ID INTEGER PRIMARY... | +
| 1 | +b95ecc66-f65b-49dc-a9f1-c1842ad230ff-ddl | +None | +CREATE TABLE Customer (\n ID INTEGER PRIMAR... | +
| 2 | +fcc73d15-30a5-4421-9d73-b8c3b0ed5305-ddl | +None | +CREATE TABLE sqlite_sequence(name,seq) | +
| 3 | +feae618c-5910-4f6f-8b4b-6cc3e03aec06-ddl | +None | +CREATE TABLE User (\n ID INTEGER PRIMARY KE... | +
| 0 | +79a48db1-ba1f-4fd5-be99-74f2ca2eaeeb-doc | +None | +XYZ Inc is a global leader in manufacturing an... | +
| 1 | +9f9df1b8-ae62-4823-ad28-d7e0f2d1f4c0-doc | +None | +ABC Corp specializes in cutting-edge technolog... | +
+
+
+
+
+
+
+
+Here is a more complex question. The manufacturing corporation name information is in the document data, which is background information. The generated SQL query will retrieve the customer information based on the specific manufacturing corporation name.
+
+
+```python
+sql = vn_milvus.generate_sql("which customer works for a manufacturing corporation?")
+vn_milvus.run_sql(sql)
+```
+
+ SQL Prompt: [{'role': 'system', 'content': "You are a SQLite expert. Please help to generate a SQL query to answer the question. Your response should ONLY be based on the given context and follow the response guidelines and format instructions. \n===Tables \nCREATE TABLE Company (\n ID INTEGER PRIMARY KEY AUTOINCREMENT,\n Name TEXT NOT NULL,\n Industry TEXT NOT NULL,\n Location TEXT NOT NULL,\n EmployeeCount INTEGER NOT NULL\n)\n\nCREATE TABLE Customer (\n ID INTEGER PRIMARY KEY AUTOINCREMENT,\n Name TEXT NOT NULL,\n Company TEXT NOT NULL,\n City TEXT NOT NULL,\n Phone TEXT NOT NULL\n)\n\n\n===Additional Context \n\nXYZ Inc is a global leader in manufacturing and supply chain management.\n\nABC Corp specializes in cutting-edge technology solutions and innovation.\n\n===Response Guidelines \n1. If the provided context is sufficient, please generate a valid SQL query without any explanations for the question. \n2. If the provided context is almost sufficient but requires knowledge of a specific string in a particular column, please generate an intermediate SQL query to find the distinct strings in that column. Prepend the query with a comment saying intermediate_sql \n3. If the provided context is insufficient, please explain why it can't be generated. \n4. Please use the most relevant table(s). \n5. If the question has been asked and answered before, please repeat the answer exactly as it was given before. \n"}, {'role': 'user', 'content': 'What are the details of the customer named John Doe?'}, {'role': 'assistant', 'content': "SELECT * FROM Customer WHERE Name = 'John Doe'"}, {'role': 'user', 'content': 'which customer works for a manufacturing corporation?'}]
+ Using model gpt-3.5-turbo for 384.25 tokens (approx)
+ LLM Response: SELECT *
+ FROM Customer
+ WHERE Company = 'XYZ Inc'
+
+
+
+
+
+| # | +Phone | +
|---|---|
| 0 | +123-456-7890 | +
+
+
+
+
+
+
+
+Disconnect from the SQLite and Milvus and remove them to free up resources.
+
+
+```python
+sql_connect.close()
+milvus_client.close()
+
+os.remove(sqlite_path)
+if os.path.exists(milvus_uri):
+ os.remove(milvus_uri)
+```
diff --git a/v2.4.x/site/en/integrations/integrate_with_voyageai.md b/v2.4.x/site/en/integrations/integrate_with_voyageai.md
index 46d5a4162..80a026cc0 100644
--- a/v2.4.x/site/en/integrations/integrate_with_voyageai.md
+++ b/v2.4.x/site/en/integrations/integrate_with_voyageai.md
@@ -64,7 +64,7 @@ data = [
# Connect to Milvus, all data is stored in a local file named "milvus_voyage_demo.db"
# in current directory. You can also connect to a remote Milvus server following this
# instruction: https://milvus.io/docs/install_standalone-docker.md.
-milvus_client = MilvusClient("milvus_voyage_demo.db")
+milvus_client = MilvusClient(uri="milvus_voyage_demo.db")
COLLECTION_NAME = "demo_collection" # Milvus collection name
# Create a collection to store the vectors and text.
if milvus_client.has_collection(collection_name=COLLECTION_NAME):
@@ -77,6 +77,16 @@ res = milvus_client.insert(collection_name="demo_collection", data=data)
print(res["insert_count"])
```
+| # | +ID | +Name | +Company | +City | +Phone | +
|---|---|---|---|---|---|
| 0 | +2 | +Jane Smith | +XYZ Inc | +Los Angeles | +098-765-4321 | +
+
+As for the argument of `MilvusClient`:
+
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
With all data in Milvus vector database, we can now perform semantic search by generating vector embedding for the query and conduct vector search.
diff --git a/v2.4.x/site/en/menuStructure/en.json b/v2.4.x/site/en/menuStructure/en.json
index 756d4d91c..936a05219 100644
--- a/v2.4.x/site/en/menuStructure/en.json
+++ b/v2.4.x/site/en/menuStructure/en.json
@@ -639,6 +639,12 @@
"id": "m2m.md",
"order": 4,
"children": []
+ },
+ {
+ "label": "From Milvus 2.3.x",
+ "id": "from-m2x.md",
+ "order": 5,
+ "children": []
}
]
},
@@ -1241,6 +1247,18 @@
"id": "integrate_with_memgpt.md",
"order": 16,
"children": []
+ },
+ {
+ "label": "FastGPT",
+ "id": "integrate_with_fastgpt.md",
+ "order": 17,
+ "children": []
+ },
+ {
+ "label": "Vanna",
+ "id": "integrate_with_vanna.md",
+ "order": 18,
+ "children": []
}
]
},
diff --git a/v2.4.x/site/en/migrate/from-m2x.md b/v2.4.x/site/en/migrate/from-m2x.md
new file mode 100644
index 000000000..d9ac26bd7
--- /dev/null
+++ b/v2.4.x/site/en/migrate/from-m2x.md
@@ -0,0 +1,148 @@
+---
+id: from-m2x.md
+summary: This guide provides a comprehensive, step-by-step process for migrating data from Milvus 2.3.x to Milvus 2.3.x or above.
+title: From Milvus 2.3.x
+---
+
+# From Milvus 2.3.x
+
+This guide provides a comprehensive, step-by-step process for migrating data from Milvus 2.3.x to Milvus 2.3.x or above.
+
+## Prerequisites
+
+- **Software versions**:
+ - Source Milvus: 2.3.0+ (The tool uses the iterator to fetch source collection data, requiring source Milvus to be version 2.3.0 or above.)
+ - Target Milvus: 2.3.0+
+- **Required tools**:
+ - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md).
+- **Data preparation**:
+ - Ensure that the source Milvus collection is loaded and ready for data export.
+ - If the target Milvus does not contain a collection corresponding to the source collection, the [milvus-migration](https://github.com/zilliztech/milvus-migration) tool will automatically create it. Note that after migration, the target collection will not be indexed, and you must manually index the collection afterward.
+
+## Configure the migration file
+
+Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory.
+
+```yaml
+dumper:
+ worker:
+ workMode: milvus2x
+ reader:
+ bufferSize: 500
+
+meta:
+ mode: config
+ version: 2.3.0
+ collection: src_table_name
+
+source:
+ milvus2x:
+ endpoint: {milvus2x_domain}:{milvus2x_port}
+ username: xxxx
+ password: xxxxx
+
+target:
+ milvus2x:
+ endpoint: {milvus2x_domain}:{milvus2x_port}
+ username: xxxx
+ password: xxxxx
+```
+
+The following table describes the parameters in the example config file. For more information, refer to [Milvus Migration: Milvus2.x to Milvus2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_2X.md#milvus-migration-milvus2x-to-milvus2x).
+
+- `dumper`
+
+ | Parameter | Description |
+ | --- | --- |
+ | `dumper.worker.workMode` | The operational mode of the migration job. Set to milvus2x when migrating from Milvus 2.x. |
+ | `dumper.worker.reader.bufferSize` | Buffer size to read from Milvus 2.x in each batch. |
+
+- `meta`
+
+ | Parameter | Description |
+ | --- | --- |
+ | `meta.mode` | Specifies where the meta file is read from. Set to config, indicating that the meta config can be obtained from this migration.yaml file. |
+ | `meta.version` | Source Milvus version. Set to 2.3.0 or above. |
+ | `meta.collection` | Source collection name. |
+
+- `source`
+
+ | Parameter | Description |
+ | --- | --- |
+ | `source.milvus2x.endpoint` | Address of the source Milvus server. |
+ | `source.milvus2x.username` | Username for the source Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). |
+ | `source.milvus2x.password` | Password for the source Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). |
+
+- `target`
+
+ | Parameter | Description |
+ | --- | --- |
+ | `target.milvus2x.endpoint` | Address of the target Milvus server. |
+ | `target.milvus2x.username` | Username for the target Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). |
+ | `target.milvus2x.password` | Password for the target Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). |
+
+## Start the migration task
+
+You have two options to start the migration task - using CLI or making API requests. Choose the one that best fits your needs.
+
+### Option 1: Using CLI
+
+Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides.
+
+```bash
+./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml
+```
+
+Monitor the logs for progress updates. Successful migration logs should include entries like:
+
+```bash
+[INFO] [migration/milvus2x_starter.go:79] ["=================>JobProcess!"] [Percent=100]
+[INFO] [migration/milvus2x_starter.go:27] ["[Starter] migration Milvus2x to Milvus2x finish!!!"] [Cost=94.877717375]
+[INFO] [starter/starter.go:109] ["[Starter] Migration Success!"] [Cost=94.878243583]
+```
+
+### Option 2: Making API requests
+
+You can also use the Restful API to execute the migration. Start the API server with:
+
+```bash
+./milvus-migration server run -p 8080
+```
+
+Once the server starts successfully, place the `migration.yaml` file in the `configs/` directory of the project and start the migration using:
+
+```bash
+curl -XPOST http://localhost:8080/api/v1/start
+```
+
+## Verify the result
+
+After the migration task is completed, use Attu to view the number of entities migrated. Additionally, you can create indexes and load collections in Attu. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md).
+
+## Additional configuration options
+
+In addition to the basic configurations mentioned above, you can also add additional settings based on your specific requirements.
+
+- **Selective field migration**: If you need to migrate only specific fields in a collection rather than all fields, specify the fields to be migrated in the `meta` section of the `migration.yaml` file.
+
+ ```yaml
+ meta:
+ fields:
+ - name: id
+ - name: title_vector
+ - name: reading_time
+ ```
+
+- **Custom target collection**: To customize the properties of the target collection, add the related configurations in the `meta` section of the `migration.yaml` file.
+
+ ```yaml
+ meta:
+ milvus:
+ collection: target_collection_name
+ shardNum: 2
+ closeDynamicField: false
+ consistencyLevel: Customized
+ ```
+
+
+For detailed information, refer to [Milvus Migration: Milvus2.x to Milvus2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_2X.md#milvus-migration-milvus2x-to-milvus2x).
diff --git a/v2.4.x/site/en/migrate/migrate_overview.md b/v2.4.x/site/en/migrate/migrate_overview.md
index ac12e358f..65a2d9be1 100644
--- a/v2.4.x/site/en/migrate/migrate_overview.md
+++ b/v2.4.x/site/en/migrate/migrate_overview.md
@@ -1,10 +1,10 @@
---
id: migrate_overview.md
summary: This article provides an overview of the Milvus-migration tool, including supported migrations, features, and architecture.
-title: Migrate Overview
+title: Milvus Migration Overview
---
-# Overview
+# Milvus Migration Overview
Recognizing the diverse needs of user base, Milvus has expanded its migration tools to not only facilitate upgrades from earlier Milvus 1.x versions but also to enable seamless integration of data from other systems like [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html) and [Faiss](https://github.com/facebookresearch/faiss). The [Milvus-migration](https://github.com/zilliztech/milvus-migration) project is designed to bridge the gap between these varied data environments and the latest advancements in Milvus technology, ensuring you can harness improved features and performance seamlessly.
@@ -15,6 +15,7 @@ The [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool supp
- [Elasticsearch to Milvus 2.x](es2m.md): Enabling users to migrate data from Elasticsearch environments to take advantage of Milvus's optimized vector search capabilities.
- [Faiss to Milvus 2.x](f2m.md): Providing experimental support for transferring data from Faiss, a popular library for efficient similarity search.
- [Milvus 1.x to Milvus 2.x](m2m.md): Ensuring data from earlier versions is transitioned smoothly to the latest framework.
+- [Milvus 2.3.x to Milvus 2.3.x or above](from-m2x.md): Providing a one-time migration path for users who have already migrated to 2.3.x.
## Features
@@ -47,4 +48,4 @@ The development team is committed to enhancing [Milvus-migration](https://github
- **Support for more data sources**: Plans to extend support to additional databases and file systems, such as Pinecone, Chroma, Qdrant. If you need support for a specific data source, please submit your request through this [GitHub issue link](https://github.com/zilliztech/milvus-migration/issues).
- **Command simplification**: Efforts to streamline the command process for easier execution.
- **SPI parser** / **convert**: The architecture expects to include Service Provider Interface (SPI) tools for both parsing and converting. These tools allow for custom implementations that users can plug into the migration process to handle specific data formats or conversion rules.
-- **Checkpoint resumption**: Enabling migrations to resume from the last checkpoint to enhance reliability and efficiency in case of interruptions. Save points will be created to ensure data integrity and are stored in databases such as SQLite or MySQL to track the progress of the migration process.
\ No newline at end of file
+- **Checkpoint resumption**: Enabling migrations to resume from the last checkpoint to enhance reliability and efficiency in case of interruptions. Save points will be created to ensure data integrity and are stored in databases such as SQLite or MySQL to track the progress of the migration process.
diff --git a/v2.4.x/site/en/reference/architecture/architecture.md b/v2.4.x/site/en/reference/architecture/architecture.md
index e72ed0c48..bfb3039ca 100644
--- a/v2.4.x/site/en/reference/architecture/architecture.md
+++ b/v2.4.x/site/en/reference/architecture/architecture.md
@@ -1,6 +1,7 @@
---
id: architecture.md
title: Architecture
+deprecate: true
---
# Architecture
diff --git a/v2.4.x/site/en/reference/benchmark.md b/v2.4.x/site/en/reference/benchmark.md
index 7455efc40..e381ae55c 100644
--- a/v2.4.x/site/en/reference/benchmark.md
+++ b/v2.4.x/site/en/reference/benchmark.md
@@ -11,7 +11,7 @@ This report shows the major test results of Milvus 2.2.0. It aims to provide a p
-
+
+
diff --git a/v2.4.x/site/en/reference/boolean.md b/v2.4.x/site/en/reference/boolean.md
index 3c9807fdf..5f309d301 100644
--- a/v2.4.x/site/en/reference/boolean.md
+++ b/v2.4.x/site/en/reference/boolean.md
@@ -8,7 +8,7 @@ title: Scalar Filtering Rules
## Overview
-A predicate expression outputs a boolean value. Milvus conducts scalar filtering by searching with predicates. A predicate expression, when evaluated, returns either TRUE or FALSE. View [Python SDK API Reference](/api-reference/pymilvus/v2.4.x/Collection/query().md) for instruction on using predicate expressions.
+A predicate expression outputs a boolean value. Milvus conducts scalar filtering by searching with predicates. A predicate expression, when evaluated, returns either TRUE or FALSE. View [Python SDK API Reference](/api-reference/pymilvus/v2.4.x/About.md) for instruction on using predicate expressions.
[EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) grammar rules describe boolean expressions rules:
diff --git a/v2.4.x/site/en/reference/index.md b/v2.4.x/site/en/reference/index.md
index 426e56ffe..5261c0b0c 100644
--- a/v2.4.x/site/en/reference/index.md
+++ b/v2.4.x/site/en/reference/index.md
@@ -295,7 +295,7 @@ Index building parameters and search parameters vary with Milvus distribution. S
| --------- | ----------------------------------------- | ------------------- |
| `nlist` | Number of cluster units | [1, 65536] |
| `m` | Number of factors of product quantization | `dim mod m == 0` |
- | `nbits` | [Optional] Number of bits in which each low-dimensional vector is stored. | [1, 16] (8 by default) |
+ | `nbits` | [Optional] Number of bits in which each low-dimensional vector is stored. | [1, 64] (8 by default) |
- Search parameters
diff --git a/v2.4.x/site/en/reference/sys_config/configure_knowhere.md b/v2.4.x/site/en/reference/sys_config/configure_knowhere.md
index 12baead16..1d2c57258 100644
--- a/v2.4.x/site/en/reference/sys_config/configure_knowhere.md
+++ b/v2.4.x/site/en/reference/sys_config/configure_knowhere.md
@@ -10,8 +10,6 @@ title: Knowhere-related Configurations
This topic introduces the knowhere-related configurations of Milvus.
-[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus.
-
Under this section, you can configure the default SIMD instruction set type of the system.
## `knowhere.simdType`
diff --git a/v2.4.x/site/en/reference/sys_config/system_configuration.md b/v2.4.x/site/en/reference/sys_config/system_configuration.md
index 305e147db..8597db584 100644
--- a/v2.4.x/site/en/reference/sys_config/system_configuration.md
+++ b/v2.4.x/site/en/reference/sys_config/system_configuration.md
@@ -152,8 +152,6 @@ See [Common Configurations](configure_common.md) for detailed description for ea
### `knowhere`
-[Knowhere](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/knowhere_design.md) is the search engine of Milvus.
-
Under this section, you can configure the default SIMD instruction set type of the system.
See [Knowhere-related Configurations](configure_knowhere.md) for detailed description for each parameter under this section.
diff --git a/v2.4.x/site/en/rerankers/rerankers-jina.md b/v2.4.x/site/en/rerankers/rerankers-jina.md
index 3eea8a3cb..9646228ee 100644
--- a/v2.4.x/site/en/rerankers/rerankers-jina.md
+++ b/v2.4.x/site/en/rerankers/rerankers-jina.md
@@ -22,7 +22,7 @@ Then, instantiate the `JinaRerankFunction`:
from pymilvus.model.reranker import JinaRerankFunction
jina_rf = JinaRerankFunction(
- model_name="jina-reranker-v1-base-en", # Defaults to `jina-reranker-v1-base-en`
+ model_name="jina-reranker-v2-base-multilingual", # Defaults to `jina-reranker-v2-base-multilingual`
api_key=JINAAI_API_KEY
)
```
@@ -31,7 +31,7 @@ jina_rf = JinaRerankFunction(
- `model_name` (*string*)
- The name of the Jina AI reranker model to use for encoding. If you leave this parameter unspecified, `jina-reranker-v1-base-en` will be used. For a list of available models, refer to [Jina AI Rerankers](https://jina.ai/reranker/#apiform).
+ The name of the Jina AI reranker model to use for encoding. If you leave this parameter unspecified, `jina-reranker-v2-base-multilingual` will be used. For a list of available models, refer to [Jina AI Rerankers](https://jina.ai/reranker/#apiform).
- `api_key` (*string*)
diff --git a/v2.4.x/site/en/rerankers/rerankers-overview.md b/v2.4.x/site/en/rerankers/rerankers-overview.md
index 11b230f37..7ccdd4349 100644
--- a/v2.4.x/site/en/rerankers/rerankers-overview.md
+++ b/v2.4.x/site/en/rerankers/rerankers-overview.md
@@ -5,7 +5,7 @@ summary: PyMilvus model library integrates rerank functions to optimize the orde
title: Rerankers Overview
---
-# Overview
+# Rerankers Overview
In the realm of information retrieval and generative AI, a reranker is an essential tool that optimizes the order of results from initial searches. Rerankers differ from traditional [embedding models](embeddings.md) by taking a query and document as input and directly returning a similarity score instead of embeddings. This score indicates the relevance between the input query and document.
diff --git a/v2.4.x/site/en/tutorials/audio_similarity_search.md b/v2.4.x/site/en/tutorials/audio_similarity_search.md
index 4a8468f28..3c3550fff 100644
--- a/v2.4.x/site/en/tutorials/audio_similarity_search.md
+++ b/v2.4.x/site/en/tutorials/audio_similarity_search.md
@@ -7,7 +7,6 @@ title: Audio Similarity Search
# Audio Similarity Search
This tutorial demonstrates how to use Milvus, the open-source vector database to build an audio similarity search system.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/audio/audio_similarity_search/audio_similarity_search.ipynb)
The ML model and third-party software used include:
- PANNs (Large-Scale Pretrained Audio Neural Networks)
diff --git a/v2.4.x/site/en/tutorials/build-rag-with-milvus.md b/v2.4.x/site/en/tutorials/build-rag-with-milvus.md
index 5ca669e89..fa281696d 100644
--- a/v2.4.x/site/en/tutorials/build-rag-with-milvus.md
+++ b/v2.4.x/site/en/tutorials/build-rag-with-milvus.md
@@ -10,7 +10,7 @@ title: Build RAG with Milvus
In this tutorial, we will show you how to build a RAG(Retrieval-Augmented Generation) pipeline with Milvus.
-The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using a vector similarity search engine like Milvus, and then uses a generative model to generate new text based on the retrieved documents.
+The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.
## Preparation
@@ -109,11 +109,21 @@ print(test_embedding[:10])
```python
from pymilvus import MilvusClient
-milvus_client = MilvusClient("./milvus_demo.db")
+milvus_client = MilvusClient(uri="./milvus_demo.db")
collection_name = "my_rag_collection"
```
+
+
diff --git a/v2.4.x/site/en/tutorials/recommendation_system.md b/v2.4.x/site/en/tutorials/recommendation_system.md
index 3bdebe99f..cee4bf2db 100644
--- a/v2.4.x/site/en/tutorials/recommendation_system.md
+++ b/v2.4.x/site/en/tutorials/recommendation_system.md
@@ -7,7 +7,6 @@ title: Recommender System
# Recommender System
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a recommendation system.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/recommender_system.ipynb)
The ML model and third-party software used include:
- PaddlePaddle
diff --git a/v2.4.x/site/en/userGuide/import_data.md b/v2.4.x/site/en/userGuide/import_data.md
deleted file mode 100644
index f2485a0ef..000000000
--- a/v2.4.x/site/en/userGuide/import_data.md
+++ /dev/null
@@ -1,198 +0,0 @@
----
-id: import_data.md
-related_key: bulk load
-summary: Learn how to bulk load data in Milvus.
-title: Import Data
----
-
-# Import Data
-
-This topic describes how to import data in Milvus via bulk load.
-
-Regular method to insert a large batch of entities to Milvus usually leads to a massive network transmission across client, proxy, Pulsar and data nodes. To avoid such situation, Milvus 2.1 supports loading data from files via bulk load. You can import large amounts of data into a collection by just a few lines of code, and endow atomicity to a whole batch of entities.
-
-You can also migrate data to Milvus with [MilvusDM](migrate_overview.md), an open-source tool designed specifically for importing and exporting data with Milvus.
-
-
-
-
-
-## Check the import task state
-
-Check the state of the import task.
-
-```python
-state = utility.get_bulk_load_state(tasks[0])
-print(state.state_name())
-print(state.ids())
-print(state.infos())
-```
-The state codes and their corresponding descriptions.
-
-| State code | State | Description |
-| ---------- | ----------------------- | -------------------------------------------------------------- |
-| 0 | BulkLoadPending | Task is in pending list |
-| 1 | BulkLoadFailed | Task failed, get the failed reason with `state.infos["failed_reason"]` |
-| 2 | BulkLoadStarted | Task is dispatched to data node, gonna to be executed |
-| 3 | BulkLoadDownloaded | Data file has been downloaded from MinIO to local |
-| 4 | BulkLoadParsed | Data file has been validated and parsed |
-| 5 | BulkLoadPersisted | New segments have been generated and persisted |
-| 6 | BulkLoadCompleted | Task completed |
-
-
-## Limits
-
-|Feature|Maximum limit|
-|---|---|
-|Max size of task pending list|32|
-|Max size of a data file|4GB|
-
-## What's next
-
-- Learn more basic operations of Milvus:
- - [Index Vetor Fields](index-vector-fields.md)
- - [Single-Vector Search](single-vector-search.md)
- - [Hybrid Search](multi-vector-search.md)
-- Explore API references for Milvus SDKs:
- - [PyMilvus API reference](/api-reference/pymilvus/v2.4.x/tutorial.html)
- - [Node.js API reference](/api-reference/node/v2.4.x/tutorial.html)
-
diff --git a/v2.4.x/site/en/userGuide/load_balance.md b/v2.4.x/site/en/userGuide/load_balance.md
index b342acdfa..c1011e3b7 100644
--- a/v2.4.x/site/en/userGuide/load_balance.md
+++ b/v2.4.x/site/en/userGuide/load_balance.md
@@ -3,6 +3,7 @@ id: load_balance.md
related_key: Load balance
summary: Learn how to balance query load in Milvus.
title: Balance Query Load
+deprecate: true
---
# Balance Query Load
diff --git a/v2.4.x/site/en/userGuide/manage-indexes/index-vector-fields.md b/v2.4.x/site/en/userGuide/manage-indexes/index-vector-fields.md
index 6c9a0a45c..df4b4782b 100644
--- a/v2.4.x/site/en/userGuide/manage-indexes/index-vector-fields.md
+++ b/v2.4.x/site/en/userGuide/manage-indexes/index-vector-fields.md
@@ -183,7 +183,7 @@ const fields = [
name: "id",
data_type: DataType.Int64,
is_primary_key: true,
- auto_id: false
+ autoID: false
},
{
name: "vector",
diff --git a/v2.4.x/site/en/userGuide/search-query-get/get-and-scalar-query.md b/v2.4.x/site/en/userGuide/search-query-get/get-and-scalar-query.md
index 017fcfc0a..8a69ca489 100644
--- a/v2.4.x/site/en/userGuide/search-query-get/get-and-scalar-query.md
+++ b/v2.4.x/site/en/userGuide/search-query-get/get-and-scalar-query.md
@@ -1578,7 +1578,7 @@ A __boolean expression__ is always __a string comprising field names joined by o
| __Operator__ | __Description__ |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
-| __add (&&)__ | True if both operands are true |
+| __and (&&)__ | True if both operands are true |
| __or (||)__ | True if either operand is true |
| __+, -, *, /__ | Addition, subtraction, multiplication, and division |
| __**__ | Exponent |
@@ -1604,4 +1604,4 @@ A __boolean expression__ is always __a string comprising field names joined by o
-
\ No newline at end of file
+
diff --git a/v2.4.x/site/en/userGuide/search-query-get/single-vector-search.md b/v2.4.x/site/en/userGuide/search-query-get/single-vector-search.md
index 51a363667..7dc2625a6 100644
--- a/v2.4.x/site/en/userGuide/search-query-get/single-vector-search.md
+++ b/v2.4.x/site/en/userGuide/search-query-get/single-vector-search.md
@@ -1841,6 +1841,11 @@ The parameter settings for `radius` and `range_filter` vary with the metric type
| ---------------- | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `L2` | Smaller L2 distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
+
+As for the argument of `MilvusClient`:
+
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
Check if the collection already exists and drop it if it does.
diff --git a/v2.4.x/site/en/tutorials/dna_sequence_classification.md b/v2.4.x/site/en/tutorials/dna_sequence_classification.md
index 39a792c48..e84371264 100644
--- a/v2.4.x/site/en/tutorials/dna_sequence_classification.md
+++ b/v2.4.x/site/en/tutorials/dna_sequence_classification.md
@@ -7,8 +7,6 @@ title: DNA Sequence Classification
# DNA Sequence Classification
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a DNA sequence classification model.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/medical/dna_sequence_classification/dna_sequence_classification.ipynb)
-- [Quick deploy](https://github.com/milvus-io/bootcamp/blob/master/solutions/medical/dna_sequence_classification/quick_deploy)
The ML model and third-party software used include:
- CountVectorizer
diff --git a/v2.4.x/site/en/tutorials/image_similarity_search.md b/v2.4.x/site/en/tutorials/image_similarity_search.md
index 83e08e280..f4263b001 100644
--- a/v2.4.x/site/en/tutorials/image_similarity_search.md
+++ b/v2.4.x/site/en/tutorials/image_similarity_search.md
@@ -90,7 +90,7 @@ class FeatureExtractor:
```
## Create a Milvus Collection
-Then we need to create Milvus collection to store the image embeddings.
+Then we need to create Milvus collection to store the image embeddings
```python
@@ -111,6 +111,16 @@ client.create_collection(
)
```
+
+
+As for the argument of `MilvusClient`:
+
+- Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.
+- If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.
+- If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud.
+
+
+
## Insert the Embeddings to Milvus
We will extract embeddings of each image using the ResNet34 model and insert images from the training set into Milvus.
@@ -174,7 +184,7 @@ display(concatenated_image)
-
+
diff --git a/v2.4.x/site/en/tutorials/recommendation_system.md b/v2.4.x/site/en/tutorials/recommendation_system.md
index 3bdebe99f..cee4bf2db 100644
--- a/v2.4.x/site/en/tutorials/recommendation_system.md
+++ b/v2.4.x/site/en/tutorials/recommendation_system.md
@@ -7,7 +7,6 @@ title: Recommender System
# Recommender System
This tutorial demonstrates how to use Milvus, the open-source vector database, to build a recommendation system.
-- [Open Jupyter notebook](https://github.com/milvus-io/bootcamp/blob/master/solutions/nlp/recommender_system/recommender_system.ipynb)
The ML model and third-party software used include:
- PaddlePaddle
diff --git a/v2.4.x/site/en/userGuide/import_data.md b/v2.4.x/site/en/userGuide/import_data.md
deleted file mode 100644
index f2485a0ef..000000000
--- a/v2.4.x/site/en/userGuide/import_data.md
+++ /dev/null
@@ -1,198 +0,0 @@
----
-id: import_data.md
-related_key: bulk load
-summary: Learn how to bulk load data in Milvus.
-title: Import Data
----
-
-# Import Data
-
-This topic describes how to import data in Milvus via bulk load.
-
-Regular method to insert a large batch of entities to Milvus usually leads to a massive network transmission across client, proxy, Pulsar and data nodes. To avoid such situation, Milvus 2.1 supports loading data from files via bulk load. You can import large amounts of data into a collection by just a few lines of code, and endow atomicity to a whole batch of entities.
-
-You can also migrate data to Milvus with [MilvusDM](migrate_overview.md), an open-source tool designed specifically for importing and exporting data with Milvus.
-
-
-
-The code snippets on this page use new MilvusClient (Python) to interact with Milvus. New MilvusClient SDKs for other languages will be released in future updates.
-
-
-
-## Prepare data file
-
-You can prepare the data file on row base or column base.
-
-- Row-based data file
-
-A row-based data file is a JSON file containing multiple rows. The root key must be "rows". The file name can be specified arbitrarily.
-
-```json
-{
- "rows":[
- {"book_id": 101, "word_count": 13, "book_intro": [1.1, 1.2]},
- {"book_id": 102, "word_count": 25, "book_intro": [2.1, 2.2]},
- {"book_id": 103, "word_count": 7, "book_intro": [3.1, 3.2]},
- {"book_id": 104, "word_count": 12, "book_intro": [4.1, 4.2]},
- {"book_id": 105, "word_count": 34, "book_intro": [5.1, 5.2]},
- ]
-}
-```
-
-- Column-based data file
-
-A column-based data file can be a JSON file containing multiple columns, several Numpy files, each contains a single column, or a JSON file contains multiple columns and some Numpy files.
-
- - JSON file containing multiple columns
- ```json
- {
- "book_id": [101, 102, 103, 104, 105],
- "word_count": [13, 25, 7, 12, 34],
- "book_intro": [
- [1.1, 1.2],
- [2.1, 2.2],
- [3.1, 3.2],
- [4.1, 4.2],
- [5.1, 5.2]
- ]
- }
- ```
-
- - Numpy files
-
- ```python
- import numpy
- numpy.save('book_id.npy', numpy.array([101, 102, 103, 104, 105]))
- numpy.save('word_count.npy', numpy.array([13, 25, 7, 12, 34]))
- arr = numpy.array([[1.1, 1.2],
- [2.1, 2.2],
- [3.1, 3.2],
- [4.1, 4.2],
- [5.1, 5.2]])
- numpy.save('book_intro.npy', arr)
- ```
-
- - A JSON file contains multiple columns and some Numpy files.
-
- ```json
- {
- "book_id": [101, 102, 103, 104, 105],
- "word_count": [13, 25, 7, 12, 34]
- }
- ```
-
- ```python
- {
- "book_id": [101, 102, 103, 104, 105],
- "word_count": [13, 25, 7, 12, 34]
- }
- ```
-
-## Upload data file
-
-Upload data files to object storage.
-
-You can upload data file to MinIO or local storage (available only in Milvus Standalone).
-
-- Upload to MinIO
-
-upload the data files to the bucket which is defined by [`minio.bucketName`](configure_minio.md#miniobucketName) in the configuration file `milvus.yml`.
-
-- Upload to local storage
-
-copy the data files into the directory which is defined by [`localStorage.path`](configure_localstorage.md#localStoragepath) in the configuration file `milvus.yml`.
-
-
-## Insert data to Milvus
-
-Import the data to the collection.
-
-- For row-based files
-
-```python
-from pymilvus import utility
-tasks = utility.bulk_load(
- collection_name="book",
- is_row_based=True,
- files=["row_based_1.json", "row_based_2.json"]
-)
-```
-
-- For column-based files
-
-```python
-from pymilvus import utility
-tasks = utility.bulk_load(
- collection_name="book",
- is_row_based=False,
- files=["columns.json", "book_intro.npy"]
-)
-```
-
-| Parameter | -Description | -
|---|---|
collection_name |
- Name of the collection to load data into. | -
is_row_based |
- Boolean value to indicate if the file is row-based. | -
files |
- List of file names to load into Milvus. | -
partition_name (optional) |
- Name of the partition to insert data into. | -
`range_filter` <= distance < `radius` | | `IP` | Larger IP distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
`radius` < distance <= `range_filter` | +| `COSINE` | Larger cosine value indicates higher similarity. | To exclude the closest vectors from results, ensure that:
`radius` < distance <= `range_filter` | +| `JACCARD` | Smaller Jaccard distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
`range_filter` <= distance < `radius` | +| `HAMMING` | Smaller Hamming distances indicate higher similarity. | To exclude the closest vectors from results, ensure that:
`range_filter` <= distance < `radius` | + +To learn more about distance metric types, refer to [Similarity Metrics](metric.md). ## Grouping search diff --git a/v2.4.x/site/en/userGuide/search-query-get/with-iterators.md b/v2.4.x/site/en/userGuide/search-query-get/with-iterators.md index 10e17eab9..e9b9d8254 100644 --- a/v2.4.x/site/en/userGuide/search-query-get/with-iterators.md +++ b/v2.4.x/site/en/userGuide/search-query-get/with-iterators.md @@ -11,7 +11,7 @@ Milvus provides search and query iterators for iterating results with a large vo ## Overview -Iterators are powerful tools that help you navigate through large datasets by using primary key values and Boolean expressions. This can significantly improve the way you retrieve data. Unlike the traditional use of __offset__ and __limit__ parameters, which may become less efficient over time, iterators offer a more scalable solution. +Iterators are powerful tools that help you iterate through a large volume of or all data within a collection using primary key values and Boolean expressions. This can significantly improve the way you retrieve data. Unlike the traditional use of __offset__ and __limit__ parameters, which may become less efficient over time, iterators offer a more scalable solution. ### Benefits of using iterators diff --git a/v2.4.x/site/en/userGuide/tune_consistency.md b/v2.4.x/site/en/userGuide/tune_consistency.md index a8d8f0a2a..3a5d6af1f 100644 --- a/v2.4.x/site/en/userGuide/tune_consistency.md +++ b/v2.4.x/site/en/userGuide/tune_consistency.md @@ -3,6 +3,7 @@ id: tune_consistency.md related_key: tune consistency title: Tune Consistency summary: Learn how to tune consistency level in Milvus. +deprecate: true --- # Tune consistency diff --git a/v2.4.x/site/en/userGuide/use-json-fields.md b/v2.4.x/site/en/userGuide/use-json-fields.md index a74986f18..5ac039651 100644 --- a/v2.4.x/site/en/userGuide/use-json-fields.md +++ b/v2.4.x/site/en/userGuide/use-json-fields.md @@ -1755,7 +1755,7 @@ The following table assumes that the value of a JSON field named `json_key` has | __not__ | `'not json_field["A"] == 1'` | This expression evaluates to true if
- `json_field` does not have a key named `A`.
- `json_field["A"]` is not equal to `1`.
| | __in__ | `'json_field["A"] in [1, 2, 3]'` | This expression evaluates to true if the value of `json_field["A"]` is `1`, `2`, or `3`. | | __and (&&)__ | `'json_field["A"] > 1 && json_field["A"] < 3'` | This expression evaluates to true if the value of `json_field["A"]` is greater than 1 and less than `3`. | -| __or (\|\|)__ | `'json_field["A"] > 1 \|\| json_field["A"] < 3'` | This expression evaluates to true if the value of `json_field["A"]` is greater than `1` or less than `3`. | +| __or (||)__ |
'json_field["A"] > 1 || json_field["A"] < 3' | This expression evaluates to true if the value of `json_field["A"]` is greater than `1` or less than `3`. |
| __exists__ | `'exists json_field["A"]'` | This expression evaluates to true if `json_field` has a key named `A`. |
### Advanced Operators
diff --git a/v2.4.x/site/en/userGuide/using_milvusclient.md b/v2.4.x/site/en/userGuide/using_milvusclient.md
index a7c184bca..15a41da00 100644
--- a/v2.4.x/site/en/userGuide/using_milvusclient.md
+++ b/v2.4.x/site/en/userGuide/using_milvusclient.md
@@ -3,6 +3,7 @@ id: using_milvusclient.md
related_key: Milvus Client Python
summary: Learn how to use the MilvusClient Python API.
title: Use the MilvusClient
+deprecate: true
---
# Use the MilvusClient
diff --git a/version.json b/version.json
index 79f759454..8085d97f0 100644
--- a/version.json
+++ b/version.json
@@ -1,4 +1,4 @@
{
- "version": "v2.3.x",
+ "version": "v2.4.x",
"released": "yes"
}
Use
-- [Create a Collection](create_collection.md) -- [Manage Data](insert_data.md) -- [Build an Index](build_index.md) -- [Search](search.md) -- [Query](query.md) +- [Manage Collections](manage-collections.md) +- [Insert, Upsert, and Delete](insert-update-delete.md) +- [Index Vector Fields](index-vector-fields.md) +- [Single-Vector Search](single-vector-search.md) +- [Get & Scalar Query](get-and-scalar-query.md)