AutoMigrate for Postgres dialect

[bevzzz]

This PR is my second attempt at implementing #456 :)

This time around I decided not to meddle with ALTER TABLE query and focus on the functionality we want to bring -- AutoMigrator. I took into account our discussions in #726 and left it up to the dialects to implement "alter table" operations, keeping it internal. My thinking was that, if auto migration is done well, the users wouldn't need to work with ALTER TABLE all that much.

AutoMigrator exposes a simple and minimal API: CreateSQLMigrations(ctx) and Migrate(ctx, options). These are similar to Migrator except that they do not require writing any SQL/Go migrations manually.

What AutoMigrator can do:

• Create, drop, and rename tables
• Create, drop, and rename column
• Modify DEFAULT value
• Modify NOT NULL and UNIQUE constraints
• Modify primary key and foreign key constraints
• Change column data type
• Add or remove identity
• Create SQL migration files with support for transactional migrations (.tx.up.sql/.tx.down.sql)

What it doesn't do:
AutoMigrator performs schema migrations and not data migrations. For example, when changing a primary key on the table, related foreign keys will not be updated -- they will continue referencing the old PK; to "re-link" them would require updating the referencing values themselves, which is beyond AutoMigrator's scope.

Presently, only pgdialect supports database inspection and migration, for which it implements two additional interfaces. AutoMigrator will return an informative error if used with a dialect that doesn't support this feature.

Other changes:

• test.sh forwards additional arguments to go test
  (for example, now possible to run 1 test with ./test.sh -run=TestAutoMigrator_Run)
• added Schema string field to differentiate tables in different schemas.
• each Dialect now reports the name of the default schema
• had to update commit message in 95c4a8e because it was upsetting the linter

---

• In-code documentation
• Documentation on the website
• Example CLI to generate and run auto-migrations

[bevzzz]

I had to update a lot of to the existing unit tests (~5-7 files) because I noticed that many of them weren't properly cleaning up the database, causing side-effects in the auto-migrate tests.
All of those changes come in a single commit and I could open a different PR for them to be merged separately.

UPD: opened #927 for this specific change. Should make it easier to review the current one once the other changes are merged.

[bevzzz]

Short update: we've decided to drop multi-schema support, so the user would instead configure AutoMigrator to work with a particular schema by passing migrate.WithSchemaName("my_schema") option.
By default, AutoMigrator will continue to use the dialect's default schema, e.g. public for Postgres.

[vmihailenco]

Looks like we can now remove (Dialect) DefaultSchema(). Overall this looks pretty good.

[vmihailenco]

This looks pretty good. @bevzzz you should have necessary permissions to merge this so feel free to do it.

[bevzzz]

@vmihailenco sorry, I've completely overlooked the last suggested changes. Will fix them in a follow-up PR, if that's alright with you.

[bgrant0607]

Hi. Thanks for this change. It looks like it will be useful. My use case is that I just wanted to generate and write out the migrations automatically, but not execute them at the same time.

One piece of feedback regarding ordered.Map as a dependency: ordered map is very small. One small, simple file. The choice to use an internal type in exported fields of exported structs made this code unusable as is, and made extending and building upon it really really messy. I revisited this PR to see why that decision had been made when so many other types are fully exported.

One other tiny issue that had cascading consequences: Anyone who uses WithForeignKeys in their tables instead of or in addition to relation tags will have those foreign keys dropped. Adding a WithForeignKeys AutoMigratorOption is a small change (a few lines of code) that would address that. I may submit a PR for that at some point.

Also, it would be handy for migrate.comment to be an exported type. As it is, I had to use reflection to detect them.

[bevzzz]

hi @bgrant0607
thank you for taking the time to share your thoughts! you're right, that's something we've overlooked in the original implementation. I'll continue the discussion in the issue you've opened 👍

migrate.comment to be an exported type

Fair point, no harm in exporting it, given all other operation types are exposed as well.
Might need to think of a better name for the type; I'll see to it.

My use case is that I just wanted to generate and write out the migrations

Can your use case be served by CreateSQLMigrations()? It will generate migration files, but not execute them.
If not, would it be useful to have AutoMigrator write SQL to a custom Writer? *

The choice to use an internal type in exported fields of exported structs made this code unusable as is

Good point, I am going to look into that. Perhaps we could somehow push orderedmap to the inspectors' internals and export a slice instead.

---

* - Could you describe your current application in a bit more detail? You've mentioned that you're catching migration.comment with reflection, which has me a bit puzzled, because changeset is an unexported type, produced and iterated-over by AutoMigrator internally.

[bgrant0607]

@bevzzz I ran into multiple problems with the automigrate code and I was trying to build on it to work out solutions without forking all of bun.

I reported two issues: #1159 and #1160.

Another issue I had was that we have some tables with generated names that are managed separately, so I wanted to exclude tables based on a prefix or pattern.

And I needed to be able to set the directory and the output filenames. The options weren't plumbed through CreateSQLMigrations and/or exported through options in a way I could use.

I also had some issues with serializing multiple migration queries, like lack of semicolons and newlines IIRC, so I had to fork that part also to add the 2 characters. Maybe I was missing something from the pgdialect implementation.

My diff of bun/migrate ended up being 1777 lines even though my changes were very minor.

[bgrant0607]

My patched version of WriteTo:

func (c *changeset) WriteTo(w io.Writer, m bunsqlschema.Migrator) error {
        var err error

        b := MakeQueryBytes()
        for _, op := range c.operations {
                if isComment := reflect.TypeOf(op).String() == "*migrate.comment"; isComment {
                        // Just skip it so that we don't need to try to extract the value from a private type
                        // b = append(b, "/*\n"...)
                        // b = append(b, ...)
                        // b = append(b, "\n*/"...)
                        continue
                }

                b, err = m.AppendSQL(b, op)
                if err != nil {
                        return fmt.Errorf("write changeset: %w", err)
                }
                b = append(b, ";\n"...)
        }
        if _, err := w.Write(b); err != nil {
                return fmt.Errorf("write changeset: %w", err)
        }
        return nil
}

[bgrant0607]

Hmm. Looking at the current WriteTo code, I don't recall what problem I had with it now.

[bevzzz]

if isComment := reflect.TypeOf(op).String() == "*migrate.comment"; isComment {
  // Just skip it so that we don't need to try to extract the value from a private type
  // b = append(b, "/*\n"...)
  // b = append(b, ...)
  // b = append(b, "\n*/"...)
  continue
}

I recall you also mentioned this, that working with an unexpected comment type was unnecessarily cumbersome. We've since changed it to an exported migrate.Unimplemented.

I wanted to exclude tables based on a prefix or pattern

That's a good idea. We can add WithExcludeTablesLike() option that would accept the same patten syntax as SQL's LIKE clause. E.g., "generated_tbl_%" to exclude all "generated_tbl_" tables.

I also had some issues with serializing multiple migration queries, like lack of semicolons and newlines IIRC, so I had to fork that part also to add the 2 characters. Maybe I was missing something from the pgdialect implementation.

If you have the time to do it and the issue persists, would you care to post an MRE for me to look into? We do not any tests for this specifically, but it may be worth adding.

And I needed to be able to set the directory and the output filenames.

There's WithMigrationsDirectoryAuto() option:

bun/migrate/auto.go

Line 79 in 04da1b6

func WithMigrationsDirectoryAuto(directory string) AutoMigratorOption {

which is the AutoMigrator's double for WithMigrationsDirectory() ⬇️

bun/migrate/migrations.go

Line 16 in 04da1b6

func WithMigrationsDirectory(directory string) MigrationsOption {

I've added a TODO about providing another option to control filenames too 👍 .

My diff of bun/migrate ended up being 1777 lines even though my changes were very minor.

Started #1166 to track the progress on the improvements to AutoMigrator. If you feel like any of the other changes you've made will make a useful addition, please share your ideas there! Once again, @bgrant0607, appreciate the feedback you've provided so far, it's been very useful.

[bgrant0607]

Thanks! I will take a look.

I think my directory problem was a consequence of the way I partially forked just the migrate code and getDirectory() not being exported.

WithExcludeTablesLike would work for me, I think.