|
| 1 | +--- |
| 2 | +sidebar_label: 'Amazon DocumentDB' |
| 3 | +description: 'Step-by-step guide on how to set up Amazon DocumentDB as a source for ClickPipes' |
| 4 | +slug: /integrations/clickpipes/mongodb/source/docdb |
| 5 | +title: 'Amazon DocumentDB source setup guide' |
| 6 | +doc_type: 'guide' |
| 7 | +keywords: ['clickpipes', 'mongodb', 'documentdb', 'cdc', 'data ingestion', 'real-time sync'] |
| 8 | +--- |
| 9 | + |
| 10 | +import docdb_select_parameter_group from '@site/static/images/integrations/data-ingestion/clickpipes/mongodb/docdb-select-parameter-group.png' |
| 11 | +import docdb_modify_parameter_group from '@site/static/images/integrations/data-ingestion/clickpipes/mongodb/docdb-modify-parameter-group.png' |
| 12 | +import docdb_apply_parameter_group from '@site/static/images/integrations/data-ingestion/clickpipes/mongodb/docdb-apply-parameter-group.png' |
| 13 | +import docdb_parameter_group_status from '@site/static/images/integrations/data-ingestion/clickpipes/mongodb/docdb-parameter-group-status.png' |
| 14 | +import Image from '@theme/IdealImage'; |
| 15 | + |
| 16 | +# Amazon DocumentDB source setup guide |
| 17 | + |
| 18 | +## Supported DocumentDB versions {#supported-documentdb-versions} |
| 19 | + |
| 20 | +ClickPipes supports DocumentDB version 5.0. |
| 21 | + |
| 22 | +## Configure change stream log retention {#configure-change-stream-log-retention} |
| 23 | + |
| 24 | +By default, Amazon DocumentDB has a 3-hour change stream log retention period, initial load may take much longer depending on existing data volume in your DocumentDB. We recommend setting the change stream log retention to 72 hours or longer to ensure that it is not truncated before the initial snapshot is completed. |
| 25 | + |
| 26 | +### Update change stream log retention via AWS Console {#update-change-stream-log-retention-via-aws-console} |
| 27 | + |
| 28 | +1. Click `Parameter groups` in the left panel, find the parameter group used by your DocumentDB cluster (if you are using the default parameter group, you will need to create a new parameter group first in order to modify it). |
| 29 | +<Image img={docdb_select_parameter_group} alt="Select parameter group" size="lg" border/> |
| 30 | + |
| 31 | +2. Search for `change_stream_log_retention_duration`, select and edit it to `259200` (72 hours) |
| 32 | +<Image img={docdb_modify_parameter_group} alt="Modify parameter group" size="lg" border/> |
| 33 | + |
| 34 | +3. Click `Apply Changes` to apply the modified parameter group to your DocumentDB cluster immediately. You should see the status of the parameter group transition to `applying`, and then to `in-sync` when the change is applied. |
| 35 | +<Image img={docdb_apply_parameter_group} alt="Apply parameter group" size="lg" border/> |
| 36 | + |
| 37 | +<Image img={docdb_parameter_group_status} alt="Parameter group status" size="lg" border/> |
| 38 | + |
| 39 | +### Update change stream log retention via AWS CLI {#update-change-stream-log-retention-via-aws-cli} |
| 40 | + |
| 41 | +To check the current change stream log retention period via AWS CLI: |
| 42 | +```shell |
| 43 | +aws docdb describe-db-cluster-parameters --db-cluster-parameter-group-name <PARAMETER_GROUP_NAME> --query "Parameters[?ParameterName=='change_stream_log_retention_duration'].{Name:ParameterName,Value:ParameterValue}" |
| 44 | +``` |
| 45 | + |
| 46 | +To set the change stream log retention period to 72 hours via AWS CLI: |
| 47 | +```shell |
| 48 | +aws docdb modify-db-cluster-parameter-group --db-cluster-parameter-group-name <PARAMETER_GROUP_NAME> --parameters "ParameterName=change_stream_log_retention_duration,ParameterValue=259200,ApplyMethod=immediate" |
| 49 | +``` |
| 50 | + |
| 51 | +## Configure a database user {#configure-database-user} |
| 52 | + |
| 53 | +Connect to your DocumentDB cluster as an admin user and execute the following command to create a database user for MongoDB CDC ClickPipes: |
| 54 | + |
| 55 | +```javascript |
| 56 | +db.getSiblingDB("admin").createUser({ |
| 57 | + user: "clickpipes_user", |
| 58 | + pwd: "some_secure_password", |
| 59 | + roles: ["readAnyDatabase", "clusterMonitor"], |
| 60 | +}) |
| 61 | +``` |
| 62 | + |
| 63 | +:::note |
| 64 | +Make sure to replace `clickpipes_user` and `some_secure_password` with your desired username and password. |
| 65 | +::: |
| 66 | + |
| 67 | +## What's next? {#whats-next} |
| 68 | + |
| 69 | +You can now [create your ClickPipe](../index.md) and start ingesting data from your DocumentDB instance into ClickHouse Cloud. |
| 70 | +Make sure to note down the connection details you used while setting up your DocumentDB cluster as you will need them during the ClickPipe creation process. |
0 commit comments