Skip to content

Commit

Permalink
doc/website/blog: v0.18 announcement (#2438)
Browse files Browse the repository at this point in the history
  • Loading branch information
rotemtam authored Jan 11, 2024
1 parent 5df7fae commit ddd1261
Show file tree
Hide file tree
Showing 2 changed files with 143 additions and 1 deletion.
2 changes: 1 addition & 1 deletion doc/md/guides/orms/sqlalchemy.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ A downside of this approach is that in order for it to work, a pre-existing data
In many production environments, databases should generally not be reachable from developer workstations,
which means this comparison is normally done against a local copy of the database which may have
undergone some changes that aren't reflected in the existing migrations.
In addition, Alembic auto-generation is [fails to detect many kinds of changes](https://alembic.sqlalchemy.org/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect)
In addition, Alembic auto-generation [fails to detect many kinds of changes](https://alembic.sqlalchemy.org/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect)
and cannot be relied upon to generate production-ready migration scripts without routine manual intervention.

Atlas, on the other hand, can automatically plan database schema migrations for SQLAlchemy
Expand Down
142 changes: 142 additions & 0 deletions doc/website/blog/2024-01-09-atlas-v0-18.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
---
title: "Announcing v0.18: Drift Detection, SQLAlchemy Support, Composite Schemas and More"
authors: rotemtam
tags: [release, drift-detection, docs, release]
---

import InstallationInstructions from '../../md/components/_installation_instructions.mdx'

Hi everyone,

Thanks for joining us today for another release announcement! We have a bunch of really exciting features to share with
you today, so let's get started! Here's what we'll cover:

* [Drift Detection](#drift-detection) - A common source of database trouble is that the schema in your database
doesn't match the schema in your code. This can happen for a variety of reasons, including manual changes to the
database, or changes made by other tools. Today, we are happy to announce the availability of a new feature that
lets you automatically detect these changes, and alerts you when they happen.
* [SQLAlchemy Support](#sqlalchemy-support) - SQLAlchemy is a popular Python ORM. Developers using SQLAlchemy can
use Atlas to automatically plan schema migrations for them, based on the desired state of their schema instead
of crafting them by hand.
* [VSCode ERDs](#vscode-erds) - We've added a new feature to our VSCode extension that lets you visualize your
database schema as an ERD diagram.
* [Composite Schemas](#composite-schemas) - The newly added `composite_schema` data source lets you combine
multiple schemas into one, which is useful for managing schemas that are loaded from multiple sources or
to describe applications that span multiple database schemas.

## Drift Detection

We believe, that in an ideal world, schema migrations on production databases should be done in an automated way,
preferably in your CI/CD pipelines, with developers not having root access. However, we know that this is oftentimes
is not the case. For this reason, it is also common to find databases which schemas differ from the ones they are
supposed to have. This phenomenon, called a **Schema Drift** can cause a lot of trouble for a team.

Atlas now can periodically check if your deployed databases schemas match their desired state. To function correctly, this
feature relies on [Atlas Cloud](https://atlasgo.cloud/) being able to communicate to your database. As it is uncommon for databases
to be directly accessible from the internet, we have added the option to run Atlas Agents in your database's network to facilitate this
communication. Agents register themselves via credentials against your Atlas Cloud account and continuously poll it for
work.

:::info PAID FEATURE
Drift Detection is currently only available in a [paid subscription](https://atlasgo.cloud/pricing).
:::

![](https://atlasgo.io/uploads/blog/v0.18/drift-detection.png)

To learn more about how to use this feature, check out our [Drift Detection Guide](/cloud/agents).

In addition, Atlas Agents enable you do use a lot more cool features, like
* Cloud mediated deployments (coming soon)
* Schema monitoring and auditing (coming soon)

## SQLAlchemy Support

#### Goodbye, Alembic. Hello, Atlas.

[SQLAlchemy](https://sqlalchemy.org) is a popular ORM toolkit widely used in the Python community. SQLAlchemy allows users to
describe their data model using its [declarative-mapping](https://docs.sqlalchemy.org/en/20/orm/declarative_tables.html)
feature. To actually create the underlying tables, users can use the `Base.metadata.create_all` method
which may be sufficient during development where tables can be routinely dropped and recreated.

However, at some point, teams need more control and decide to employ
the [versioned migrations](/concepts/declarative-vs-versioned#versioned-migrations) methodology,
which is a more robust way to manage a database schema.

The native way to manage migrations with SQLAlchemy is to use the [Alembic](https://alembic.sqlalchemy.org/en/latest/) migration tool.
Alembic can [automatically generate](https://alembic.sqlalchemy.org/en/latest/autogenerate.html#auto-generating-migrations)
migration scripts from the difference between the current state of the database and the desired state of the application.

A downside of this approach is that in order for it to work, a pre-existing database with the current version of the schema must be connected to.
In many production environments, databases should generally not be reachable from developer workstations,
which means this comparison is normally done against a local copy of the database which may have
undergone some changes that aren't reflected in the existing migrations.

In addition, Alembic auto-generation [fails to detect many kinds of changes](https://alembic.sqlalchemy.org/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect)
and cannot be relied upon to generate production-ready migration scripts without routine manual intervention.

Atlas, on the other hand, can automatically plan database schema migrations for SQLAlchemy
without requiring a connection to such a database and can detect almost any kind of schema change.
Atlas plans migrations by calculating the diff between the _current_ state of the database,
and its _desired_ state.

To learn how to use Atlas with SQLAlchemy, check out our [SQLAlchemy Guide](/guides/orms/sqlalchemy).

Special thanks to [No'am (Miko) Tamir](https://github.com/noamtamir) (who also doubles as my young brother)
for his fantastic work building the [prototype](https://github.com/noamtamir/atlas-provider-sqlalchemy) for this
feature and to [Ronen Lubin](https://github.com/ronenlu) for making it production-ready.

## VSCode ERDs

![](https://atlasgo.io/uploads/blog/v0.18/vsc-erd.png)

Starting with v0.4.2, our [VSCode Extension](https://marketplace.visualstudio.com/items?itemName=Ariga.atlas-hcl&ssr=false#overview)
can now visualize your database schema as an ERD diagram. To use this feature, simply open the command palette
(`Ctrl+Shift+P` on Windows/Linux, `Cmd+Shift+P` on Mac) and select `Atlas: View in ERD`.

## Composite Schemas

The `composite_schema` data source allows the composition of multiple Atlas schemas into a unified schema graph. This
functionality is useful when projects schemas are split across various sources such as HCL, SQL, or application ORMs.
For example, each service might have its own schema.

Referring to the `url` returned by this data source allows reading the entire project schemas as a single unit by any of
the Atlas commands, such as `migrate diff`, `schema apply`, or `schema inspect`.

#### Usage example

By running `atlas migrate diff` with the given configuration, Atlas loads the `inventory` schema from the [SQLAlchemy schema](/guides/orms/sqlalchemy),
the `graph` schema from [ent/schema](https://entgo.io), and the `auth` and `internal` schemas from HCL and SQL schemas defined in
Atlas format. Then, the composite schema, which represents these four schemas combined, will be compared against the
current state of the migration directory. In case of a difference between the two states, a new migration file will be
created with the necessary SQL statements.

```hcl title="atlas.hcl" {1-14}
data "composite_schema" "project" {
schema "inventory" {
url = data.external_schema.sqlalchemy.url
}
schema "graph" {
url = "ent://ent/schema"
}
schema "auth" {
url = "file://path/to/schema.hcl"
}
schema "internal" {
url = "file://path/to/schema.sql"
}
}
env "dev" {
src = data.composite_schema.project.url
dev = "docker://postgres/15/dev"
migration {
dir = "file://migrations"
}
}
```


### Wrapping up

That's it! I hope you try out (and enjoy) all of these new features and find them useful.
As always, we would love to hear your feedback and suggestions on our [Discord server](https://discord.gg/zZ6sWVg6NT).

0 comments on commit ddd1261

Please sign in to comment.