Skip to content

Commit d415deb

Browse files
committed
add a page for docdb
1 parent baea0db commit d415deb

File tree

9 files changed

+75
-1
lines changed

9 files changed

+75
-1
lines changed

docs/integrations/data-ingestion/clickpipes/mongodb/faq.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ For more examples, see our [Working with JSON guide](./quickstart).
5252

5353
### How do I handle `resume of change stream was not possible, as the resume point may no longer be in the oplog.` error? {#resume-point-may-no-longer-be-in-the-oplog-error}
5454

55-
This error typically occurs when the oplog is truncated and ClickPipe is unable to resume the change stream at the expected point. To resolve this issue, [resync the ClickPipe](./resync.md). To avoid this issue from recurring, we recommend [increasing the oplog retention period](./source/atlas#enable-oplog-retention) (or [here](./source/generic#enable-oplog-retention) if you are on a self-managed MongoDB).
55+
This error typically occurs when the oplog is truncated and ClickPipe is unable to resume the change stream at the expected point. To resolve this issue, [resync the ClickPipe](./resync.md). To avoid this issue from recurring, we recommend increasing the oplog retention period. See instructions for [MongoDB Atlas](./source/atlas#enable-oplog-retention), [self-managed MongoDB](./source/generic#enable-oplog-retention), or [DocumentDB](./source/docdb##configure-change-stream-log-retention).
5656

5757
### How is replication managed? {#how-is-replication-managed}
5858

docs/integrations/data-ingestion/clickpipes/mongodb/index.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ To get started, you first need to ensure that your MongoDB database is correctly
3838

3939
2. [Generic MongoDB](./mongodb/source/generic)
4040

41+
3. [Amazon DocumentDB](./mongodb/source/docdb)
42+
4143
Once your source MongoDB database is set up, you can continue creating your ClickPipe.
4244

4345
## Create your ClickPipe {#create-your-clickpipe}
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
sidebar_label: 'Amazon DocumentDB'
3+
description: 'Step-by-step guide on how to set up Amazon DocumentDB as a source for ClickPipes'
4+
slug: /integrations/clickpipes/mongodb/source/docdb
5+
title: 'Amazon DocumentDB source setup guide'
6+
doc_type: 'guide'
7+
keywords: ['clickpipes', 'mongodb', 'documentdb', 'cdc', 'data ingestion', 'real-time sync']
8+
---
9+
10+
import docdb_select_parameter_group from '@site/static/images/integrations/data-ingestion/clickpipes/mongodb/docdb-select-parameter-group.png'
11+
import docdb_modify_parameter_group from '@site/static/images/integrations/data-ingestion/clickpipes/mongodb/docdb-modify-parameter-group.png'
12+
import docdb_apply_parameter_group from '@site/static/images/integrations/data-ingestion/clickpipes/mongodb/docdb-apply-parameter-group.png'
13+
import docdb_parameter_group_status from '@site/static/images/integrations/data-ingestion/clickpipes/mongodb/docdb-parameter-group-status.png'
14+
import Image from '@theme/IdealImage';
15+
16+
# Amazon DocumentDB source setup guide
17+
18+
## Supported DocumentDB versions {#supported-documentdb-versions}
19+
20+
ClickPipes supports DocumentDB version 5.0.
21+
22+
## Configure change stream log retention {#configure-change-stream-log-retention}
23+
24+
By default, Amazon DocumentDB has a 3-hour change stream log retention period, initial load may take much longer depending on existing data volume in your DocumentDB. We recommend setting the change stream log retention to 72 hours or longer to ensure that it is not truncated before the initial snapshot is completed.
25+
26+
### Update change stream log retention via AWS Console {#update-change-stream-log-retention-via-aws-console}
27+
28+
1. Click `Parameter groups` in the left panel, find the parameter group used by your DocumentDB cluster (if you are using the default parameter group, you will need to create a new parameter group first in order to modify it).
29+
<Image img={docdb_select_parameter_group} alt="Select parameter group" size="lg" border/>
30+
31+
2. Search for `change_stream_log_retention_duration`, select and edit it to `259200` (72 hours)
32+
<Image img={docdb_modify_parameter_group} alt="Modify parameter group" size="lg" border/>
33+
34+
3. Click `Apply Changes` to apply the modified parameter group to your DocumentDB cluster immediately. You should see the status of the parameter group transition to `applying`, and then to `in-sync` when the change is applied.
35+
<Image img={docdb_apply_parameter_group} alt="Apply parameter group" size="lg" border/>
36+
37+
<Image img={docdb_parameter_group_status} alt="Parameter group status" size="lg" border/>
38+
39+
### Update change stream log retention via AWS CLI {#update-change-stream-log-retention-via-aws-cli}
40+
41+
To check the current change stream log retention period via AWS CLI:
42+
```shell
43+
aws docdb describe-db-cluster-parameters --db-cluster-parameter-group-name <PARAMETER_GROUP_NAME> --query "Parameters[?ParameterName=='change_stream_log_retention_duration'].{Name:ParameterName,Value:ParameterValue}"
44+
```
45+
46+
To set the change stream log retention period to 72 hours via AWS CLI:
47+
```shell
48+
aws docdb modify-db-cluster-parameter-group --db-cluster-parameter-group-name <PARAMETER_GROUP_NAME> --parameters "ParameterName=change_stream_log_retention_duration,ParameterValue=259200,ApplyMethod=immediate"
49+
```
50+
51+
## Configure a database user {#configure-database-user}
52+
53+
Connect to your DocumentDB cluster as an admin user and execute the following command to create a database user for MongoDB CDC ClickPipes:
54+
55+
```javascript
56+
db.getSiblingDB("admin").createUser({
57+
user: "clickpipes_user",
58+
pwd: "some_secure_password",
59+
roles: ["readAnyDatabase", "clusterMonitor"],
60+
})
61+
```
62+
63+
:::note
64+
Make sure to replace `clickpipes_user` and `some_secure_password` with your desired username and password.
65+
:::
66+
67+
## What's next? {#whats-next}
68+
69+
You can now [create your ClickPipe](../index.md) and start ingesting data from your DocumentDB instance into ClickHouse Cloud.
70+
Make sure to note down the connection details you used while setting up your DocumentDB cluster as you will need them during the ClickPipe creation process.

scripts/aspell-ignore/en/aspell-dict.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -366,6 +366,7 @@ DistributedFilesToInsert
366366
DistributedProductMode
367367
DistributedSend
368368
DockerHub
369+
DocumentDB
369370
Doron
370371
DoubleDelta
371372
Doxygen

sidebars.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -698,6 +698,7 @@ const sidebars = {
698698
items: [
699699
"integrations/data-ingestion/clickpipes/mongodb/source/atlas",
700700
"integrations/data-ingestion/clickpipes/mongodb/source/generic",
701+
"integrations/data-ingestion/clickpipes/mongodb/source/docdb",
701702
],
702703
},
703704
],
377 KB
Loading
308 KB
Loading
166 KB
Loading
214 KB
Loading

0 commit comments

Comments
 (0)