MultiTenancy - Large Migrations

[jmortlock]

When running large migrations with multi-tenancy setup (MySql) we are hitting this error

Error: driver: bad connection

This happens after a long migration and before the start of the next one, I assume the "bad connection' has infact timed out.

[a8m]

Hey, @jmortlock,

Which command do you use in the multi-tenant setup? Also, what do you consider a "long" migration? How long has it been running?

[jmortlock]

Hi

I can see an example here after 40mins

Error: driver: bad connection
[mysql] 2024/10/28 20:35:11 connection.go:49: bad connection
  -- 12 sql statements
  -- 5 migrations
  -- 39m23.268774378s
  -------------------------

I am running this command to apply the migrations

atlas migrate apply --exec-order non-linear --env aws

Using the setup outlined in your docs, i.e. this sorta thing

env "aws" {
  src = "db/schema.hcl"
  for_each = toset(data.sql.tenants.values)
  url      = urlsetpath(data.runtimevar.url, each.value)
  migration { dir = "file://db/migrations" }
}

I suspect whats happening is because we are using RDS proxy service which has a much tighter default Idle timeout of 30mins (mysql default is 8hours).

Current workaround is to restart the migrations when the error is encountered.