Skip to content

Commit

Permalink
doc/md: add schema monitoring drift detection doc (#3082)
Browse files Browse the repository at this point in the history
  • Loading branch information
ronenlu authored Sep 3, 2024
1 parent 119f9a5 commit 32769b8
Show file tree
Hide file tree
Showing 4 changed files with 119 additions and 14 deletions.
31 changes: 17 additions & 14 deletions doc/md/monitoring/02-quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,16 @@ This is what this guide will walk you through:
Create a docker network and a database container attached to this network. This is required for the Atlas agent docker
to see the database.

After the database is up, create the database schema.

```shell
docker network create atlas-agent-network
docker run --network atlas-agent-network -p 3306:3306 -d --rm --name sakila sakiladb/mysql:8
docker run --network atlas-agent-network -e MYSQL_ROOT_PASSWORD=pass -e MYSQL_DATABASE=dev -p 3306:3306 -d --name dev mysql:8
docker exec -i dev mysql -uroot -ppass dev -e "
CREATE TABLE users(id int NOT NULL, name varchar(100) NULL, PRIMARY KEY(id));
CREATE TABLE blog_posts(id int NOT NULL, title varchar(100) NULL, body text NULL, author_id int NULL, PRIMARY KEY(id), CONSTRAINT author_fk FOREIGN KEY(author_id) REFERENCES users(id));"
```

[Sakila](https://dev.mysql.com/doc/sakila/en/) is a demo database by the MySQL team. It serves as an example database
with an already existing schema for us to use in this guide.

### 2. Install Atlas agent

Head over to your Atlas Cloud account and click on the top level **Monitoring** navigation entry. Click on **Add Agent**,
Expand All @@ -41,47 +43,47 @@ chose a name for your agent and click on Proceed. Copy the token.
</details>

Run the following command on your local machine to start the agent (replace `<your-token>` with the token shown to you
in the modal). Notice the PASS environment variable containing the password for the Sakila db (`p_ssW0rd`).
in the modal). Notice the PASS environment variable containing the password for the dev db we set up earlier (`pass`).

```shell
docker run -e PASS=p_ssW0rd -e ATLAS_CLOUD_TOKEN="<your-token>" -e ATLAS_AUTO_UPDATE=false --network atlas-agent-network -d arigaio/atlas-agent
docker run -e PASS=pass -e ATLAS_CLOUD_TOKEN="<your-token>" -e ATLAS_AUTO_UPDATE=false --network atlas-agent-network -d arigaio/atlas-agent
```

Once you see the success message, save the Agent configuration.

### 3. Connect Atlas to your database

Click on **New** and assign your database instance a name, Sakila in this case. Select driver (**MYSQL**) and agent
(there should only be the one you just created), and as address use `sakila:3306`, username is `sakila`. Atlas does not
Click on **New** and assign your database instance a name, dev in this case. Select driver (**MYSQL**) and agent
(there should only be the one you just created), and as address use `dev:3306`, username is `root`. Atlas does not
store any credentials, the agent just needs to know how to obtain them. In our case the password is given to the agent
as the environment variable `PASS`. Once you have filled out all fields, click on **Check Connection**. It should go
green, and you can save the changes.

<details>
<summary>Screenshot Example</summary>

![](https://atlasgo.io/uploads/cloud/monitoring/3.png)
![](https://atlasgo.io/uploads/cloud/monitoring/add-instance.png)
</details>

### 4. View the schema

Click on the newly created instance. You will see an empty state screen, click on **Create Scope**, give it a name
(e.g. Sakila), click **Add Scope** and wait for the agent to inspect the database and report back its database schema.
(e.g. all), click **Add Scope** and wait for the agent to inspect the database and report back its database schema.
Once that happened, you can view the schema ERD, docs and in the Changelog tab you can see the changes done to your
schema over time.

<details>
<summary>Screenshot Example</summary>

![](https://atlasgo.io/uploads/cloud/monitoring/4.png)
![](https://atlasgo.io/uploads/cloud/monitoring/show-instance.png)
</details>

### 5. Make a change

To simulate a change to the database schema, let's create a new table.

```shell
docker exec sakila mysql -h127.0.0.1 -P3306 -usakila -pp_ssW0rd sakila -e "create table some_table (some_column int)"
docker exec dev mysql -h127.0.0.1 -P3306 -uroot -ppass dev -e "create table some_table (some_column int)"
```

Hit the Refresh button and observe the ERD and schema docs reflect the changes. The Changelog will contain a new entry
Expand All @@ -90,11 +92,12 @@ with detailed information about the change.
<details>
<summary>Screenshot Example</summary>

![](https://atlasgo.io/uploads/cloud/monitoring/5.png)
![](https://atlasgo.io/uploads/cloud/monitoring/changelog.png)
</details>

### Next steps

- [Drift detection](06-drift-detection.mdx)
- [Arch overview](03-overview.mdx)
- [Real agent installation with Kubernetes](05-helm.mdx)
- [Learn more about security considerations](04-security.mdx)
- [Learn more about security considerations](04-security.mdx)
1 change: 1 addition & 0 deletions doc/md/monitoring/03-overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,5 +60,6 @@ detects such a drift, you will see it in the Change Log.

### Additional reading

- [Drift detection](06-drift-detection.mdx)
- [Real agent installation with Kubernetes](05-helm.mdx)
- [Learn more about security considerations](04-security.mdx)
97 changes: 97 additions & 0 deletions doc/md/monitoring/06-drift-detection.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
title: Drift Detection
---

## Schema Drift

Schema migrations on production databases should ideally be automated within CI/CD pipelines, keeping developers from
needing root access. However, this setup isn't always the case, leading to **Schema Drift**, where database schemas
drift from their intended state, causing inconsistencies and issues.

Schema drift is dangerous because it can lead to unexpected behavior in applications, data corruption, and even system outages.
When the actual database schema deviates from the expected schema, queries may fail, data integrity constraints may be violated,
and application logic may break. Additionally, schema migrations are planned based on the assumption the target
database is in a specific state and may fail and cause unknown side effects. This can result in hard-to-debug issues and
degraded system performance, ultimately affecting the reliability and stability of your application.

You can use Atlas [Schema Monitoring](/monitoring) to detect schema drift and ensure your database schema are always in sync with their
intended state. Atlas schema monitoring automatically detects schema drift by comparing the actual database schema with the
intended schema. If a drift is detected, Atlas will notify you and provide detailed information about the drift, including
an ERD, HCL and SQL of the diff.

#### Prerequisites:
- [Migration directory](/cloud/directories) in atlas cloud

#### Creating desired database schema

First, we will need to deploy our migration directory to a target database. This will be our desired state.

To [deploy](/cloud/deployment), create the next `atlas.hcl` file:
```hcl
env {
name = atlas.env
url = "docker://mysql/8/test"
migration {
dir = "atlas://<your-migration-directory>"
}
}
```

And run the following command:

```shell
atlas migrate apply --env local
```


#### Enabling Drift Detection

Atlas enables you to automatically monitor schema drifts by periodically checking if your deployed database schema align with the desired schema.

To enable periodic inspection, go the **settings** tab of the database you want to monitor and enable the **Periodic Inspection** toggle.
This will instruct Atlas to periodically take snapshots of the database schema.
<details>
<summary>Screenshot Example</summary>

![](https://atlasgo.io/uploads/cloud/monitoring/periodic-inspection.png)
</details>

Now, you can enable drift detection by toggling the **Drift Detection** switch,
you will need to choose which migration directory and database target to compare against as the desired state.

<details>
<summary>Screenshot Example</summary>

![](https://atlasgo.io/uploads/cloud/monitoring/enable-drift.png)
</details>

Once enabled, if there is a drift, Atlas will provide you with detailed information about the drift, including an ERD, HCL and SQL diff.

You can see the drift in the **Changelog** tab.

<details>
<summary>Screenshot Example</summary>

![](https://atlasgo.io/uploads/cloud/monitoring/diff-sql.png)
</details>

And you can also see the drift in the deployed database itself.

<details>
<summary>Screenshot Example</summary>

![](https://atlasgo.io/uploads/cloud/monitoring/database-drift.png)
</details>


### Notifications

You can instruct Atlas to notify you if there is a drift on the target database. Atlas supports various channels, such as Slack,
Workplace or by a plain webhook.

<details>
<summary>Screenshot Example</summary>

![](https://atlasgo.io/uploads/cloud/monitoring/webhooks.png)
![](https://atlasgo.io/uploads/cloud/images/drift-webhook.png)
</details>
4 changes: 4 additions & 0 deletions doc/website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,10 @@ module.exports = {
type: 'doc',
id: 'monitoring/quickstart'
},
{
type: 'doc',
id: 'monitoring/drift-detection'
},
{
type: 'doc',
id: 'monitoring/overview'
Expand Down

0 comments on commit 32769b8

Please sign in to comment.