From 32769b81b14d809e4a2b11d9adafb14fa4719223 Mon Sep 17 00:00:00 2001 From: Ronen Lubin <63970571+ronenlu@users.noreply.github.com> Date: Tue, 3 Sep 2024 10:18:53 +0300 Subject: [PATCH] doc/md: add schema monitoring drift detection doc (#3082) --- doc/md/monitoring/02-quickstart.mdx | 31 ++++---- doc/md/monitoring/03-overview.mdx | 1 + doc/md/monitoring/06-drift-detection.mdx | 97 ++++++++++++++++++++++++ doc/website/sidebars.js | 4 + 4 files changed, 119 insertions(+), 14 deletions(-) create mode 100644 doc/md/monitoring/06-drift-detection.mdx diff --git a/doc/md/monitoring/02-quickstart.mdx b/doc/md/monitoring/02-quickstart.mdx index 6174b2a155b..d87ad8a4770 100644 --- a/doc/md/monitoring/02-quickstart.mdx +++ b/doc/md/monitoring/02-quickstart.mdx @@ -20,14 +20,16 @@ This is what this guide will walk you through: Create a docker network and a database container attached to this network. This is required for the Atlas agent docker to see the database. +After the database is up, create the database schema. + ```shell docker network create atlas-agent-network -docker run --network atlas-agent-network -p 3306:3306 -d --rm --name sakila sakiladb/mysql:8 +docker run --network atlas-agent-network -e MYSQL_ROOT_PASSWORD=pass -e MYSQL_DATABASE=dev -p 3306:3306 -d --name dev mysql:8 +docker exec -i dev mysql -uroot -ppass dev -e " + CREATE TABLE users(id int NOT NULL, name varchar(100) NULL, PRIMARY KEY(id)); + CREATE TABLE blog_posts(id int NOT NULL, title varchar(100) NULL, body text NULL, author_id int NULL, PRIMARY KEY(id), CONSTRAINT author_fk FOREIGN KEY(author_id) REFERENCES users(id));" ``` -[Sakila](https://dev.mysql.com/doc/sakila/en/) is a demo database by the MySQL team. It serves as an example database -with an already existing schema for us to use in this guide. - ### 2. Install Atlas agent Head over to your Atlas Cloud account and click on the top level **Monitoring** navigation entry. Click on **Add Agent**, @@ -41,18 +43,18 @@ chose a name for your agent and click on Proceed. Copy the token. Run the following command on your local machine to start the agent (replace `` with the token shown to you -in the modal). Notice the PASS environment variable containing the password for the Sakila db (`p_ssW0rd`). +in the modal). Notice the PASS environment variable containing the password for the dev db we set up earlier (`pass`). ```shell -docker run -e PASS=p_ssW0rd -e ATLAS_CLOUD_TOKEN="" -e ATLAS_AUTO_UPDATE=false --network atlas-agent-network -d arigaio/atlas-agent +docker run -e PASS=pass -e ATLAS_CLOUD_TOKEN="" -e ATLAS_AUTO_UPDATE=false --network atlas-agent-network -d arigaio/atlas-agent ``` Once you see the success message, save the Agent configuration. ### 3. Connect Atlas to your database -Click on **New** and assign your database instance a name, Sakila in this case. Select driver (**MYSQL**) and agent -(there should only be the one you just created), and as address use `sakila:3306`, username is `sakila`. Atlas does not +Click on **New** and assign your database instance a name, dev in this case. Select driver (**MYSQL**) and agent +(there should only be the one you just created), and as address use `dev:3306`, username is `root`. Atlas does not store any credentials, the agent just needs to know how to obtain them. In our case the password is given to the agent as the environment variable `PASS`. Once you have filled out all fields, click on **Check Connection**. It should go green, and you can save the changes. @@ -60,20 +62,20 @@ green, and you can save the changes.
Screenshot Example -![](https://atlasgo.io/uploads/cloud/monitoring/3.png) +![](https://atlasgo.io/uploads/cloud/monitoring/add-instance.png)
### 4. View the schema Click on the newly created instance. You will see an empty state screen, click on **Create Scope**, give it a name -(e.g. Sakila), click **Add Scope** and wait for the agent to inspect the database and report back its database schema. +(e.g. all), click **Add Scope** and wait for the agent to inspect the database and report back its database schema. Once that happened, you can view the schema ERD, docs and in the Changelog tab you can see the changes done to your schema over time.
Screenshot Example -![](https://atlasgo.io/uploads/cloud/monitoring/4.png) +![](https://atlasgo.io/uploads/cloud/monitoring/show-instance.png)
### 5. Make a change @@ -81,7 +83,7 @@ schema over time. To simulate a change to the database schema, let's create a new table. ```shell -docker exec sakila mysql -h127.0.0.1 -P3306 -usakila -pp_ssW0rd sakila -e "create table some_table (some_column int)" +docker exec dev mysql -h127.0.0.1 -P3306 -uroot -ppass dev -e "create table some_table (some_column int)" ``` Hit the Refresh button and observe the ERD and schema docs reflect the changes. The Changelog will contain a new entry @@ -90,11 +92,12 @@ with detailed information about the change.
Screenshot Example -![](https://atlasgo.io/uploads/cloud/monitoring/5.png) +![](https://atlasgo.io/uploads/cloud/monitoring/changelog.png)
### Next steps +- [Drift detection](06-drift-detection.mdx) - [Arch overview](03-overview.mdx) - [Real agent installation with Kubernetes](05-helm.mdx) -- [Learn more about security considerations](04-security.mdx) \ No newline at end of file +- [Learn more about security considerations](04-security.mdx) diff --git a/doc/md/monitoring/03-overview.mdx b/doc/md/monitoring/03-overview.mdx index 7ff546ebe3b..3fbd6ac7b02 100644 --- a/doc/md/monitoring/03-overview.mdx +++ b/doc/md/monitoring/03-overview.mdx @@ -60,5 +60,6 @@ detects such a drift, you will see it in the Change Log. ### Additional reading +- [Drift detection](06-drift-detection.mdx) - [Real agent installation with Kubernetes](05-helm.mdx) - [Learn more about security considerations](04-security.mdx) \ No newline at end of file diff --git a/doc/md/monitoring/06-drift-detection.mdx b/doc/md/monitoring/06-drift-detection.mdx new file mode 100644 index 00000000000..6ffdd5f8dd6 --- /dev/null +++ b/doc/md/monitoring/06-drift-detection.mdx @@ -0,0 +1,97 @@ +--- +title: Drift Detection +--- + +## Schema Drift + +Schema migrations on production databases should ideally be automated within CI/CD pipelines, keeping developers from +needing root access. However, this setup isn't always the case, leading to **Schema Drift**, where database schemas +drift from their intended state, causing inconsistencies and issues. + +Schema drift is dangerous because it can lead to unexpected behavior in applications, data corruption, and even system outages. +When the actual database schema deviates from the expected schema, queries may fail, data integrity constraints may be violated, +and application logic may break. Additionally, schema migrations are planned based on the assumption the target +database is in a specific state and may fail and cause unknown side effects. This can result in hard-to-debug issues and +degraded system performance, ultimately affecting the reliability and stability of your application. + +You can use Atlas [Schema Monitoring](/monitoring) to detect schema drift and ensure your database schema are always in sync with their +intended state. Atlas schema monitoring automatically detects schema drift by comparing the actual database schema with the +intended schema. If a drift is detected, Atlas will notify you and provide detailed information about the drift, including +an ERD, HCL and SQL of the diff. + +#### Prerequisites: +- [Migration directory](/cloud/directories) in atlas cloud + +#### Creating desired database schema + +First, we will need to deploy our migration directory to a target database. This will be our desired state. + +To [deploy](/cloud/deployment), create the next `atlas.hcl` file: +```hcl +env { + name = atlas.env + url = "docker://mysql/8/test" + migration { + dir = "atlas://" + } +} +``` + +And run the following command: + +```shell +atlas migrate apply --env local +``` + + +#### Enabling Drift Detection + +Atlas enables you to automatically monitor schema drifts by periodically checking if your deployed database schema align with the desired schema. + +To enable periodic inspection, go the **settings** tab of the database you want to monitor and enable the **Periodic Inspection** toggle. +This will instruct Atlas to periodically take snapshots of the database schema. +
+Screenshot Example + +![](https://atlasgo.io/uploads/cloud/monitoring/periodic-inspection.png) +
+ +Now, you can enable drift detection by toggling the **Drift Detection** switch, +you will need to choose which migration directory and database target to compare against as the desired state. + +
+Screenshot Example + +![](https://atlasgo.io/uploads/cloud/monitoring/enable-drift.png) +
+ +Once enabled, if there is a drift, Atlas will provide you with detailed information about the drift, including an ERD, HCL and SQL diff. + +You can see the drift in the **Changelog** tab. + +
+Screenshot Example + +![](https://atlasgo.io/uploads/cloud/monitoring/diff-sql.png) +
+ +And you can also see the drift in the deployed database itself. + +
+Screenshot Example + +![](https://atlasgo.io/uploads/cloud/monitoring/database-drift.png) +
+ + +### Notifications + +You can instruct Atlas to notify you if there is a drift on the target database. Atlas supports various channels, such as Slack, +Workplace or by a plain webhook. + +
+Screenshot Example + +![](https://atlasgo.io/uploads/cloud/monitoring/webhooks.png) +![](https://atlasgo.io/uploads/cloud/images/drift-webhook.png) +
diff --git a/doc/website/sidebars.js b/doc/website/sidebars.js index 5e072706ee1..a760eaeadcb 100644 --- a/doc/website/sidebars.js +++ b/doc/website/sidebars.js @@ -130,6 +130,10 @@ module.exports = { type: 'doc', id: 'monitoring/quickstart' }, + { + type: 'doc', + id: 'monitoring/drift-detection' + }, { type: 'doc', id: 'monitoring/overview'