Database / Liquibase

[fxprunayre]

Database module using liquibase

Draft experiment/early proposal to identify edges cases

Main goals are:

• better manage migration which currently does not work with all possible migration paths
• be able to setup/migrate databases in pipelines
• be able to manage database from GeoNetwork 4 and 5 with one system

Content of the draft README.md

---

Database initialization

Liquibase is used to manage database schema creation and changes and the configuration is made in changelog.xml.

When adding new entities or making changes to existing ones, please make sure
to update the database changelog accordingly by adding a new file in the changesets directory.

Numbering is important to keep track of the order of changes, so please follow the existing pattern when naming new changeset files.
When a feature is backported in various branches, the numbering ensure that the changesets are applied in the correct order.

eg. main can contain changesets 001 to 010, while 4.2.x contains changesets 001 to 006 and 009 which was backported.

Liquibase track changes applied to the current database in the DATABASECHANGELOG and DATABASECHANGELOGLOCK tables.

Old database initialization and migration system proposal

GeoNetwork using its own mechanism based on Java and SQL to migrate from version to version.
This mechanism does not work well when backporting changes to previous versions and does not support all possible migration paths.

It is proposed the following strategy to move to Liquibase: Old GeoNetwork system is used until version 4.4.x, then starting from version 4.6.x Liquibase only is used.

This means:

• Users upgrading to version 4.6.x will first have to migrate to the latest 4.4.x version using the old system, then upgrade to 4.6.x which will use Liquibase.
• New installations of version 4.6.x and later will use Liquibase only.

This approach avoid to migrate all existing migration steps to Liquibase, which is hard to do because of:

• Java migration steps
• Hibernate automatic schema generation not described in SQL
• Complexity to test all migration paths (which are not even working well with the old system)

Liquibase implementation status

Draft experiment to implement initial database creation and initial data loading using Liquibase.

• Databases tested (test: first run works, restart works)
  • Postgres
  • H2 (needed for test in GN4)
  • Other - discuss which database do we support and test?
• Liquibase bean configuration with changelog.xml as main changelog file
• Schema creation / Precondition: no table metadata exists
• Initial data / Precondition: table settings is empty
• Initial languages / Precondition: table language does not have eng
• Other languages
• Remove past database migrations
• Remove old configuration initial_data.xml and database_migration.xml
• Remove db.migration_onstartup property / Hibernate hbm2ddl property set to none
• Remove TestDatabasePopulator
• ...

GeoNetwork 5 and Liquibase

GeoNetwork 5 can depend on the GeoNetwork 4 database module to initialize the database using Liquibase.

To enable liquibase in GeoNetwork 5 by:

• Adding liquibase dependency in pom.xml:

<dependency>
  <groupId>org.liquibase</groupId>
  <artifactId>liquibase-core</artifactId>
  <version>4.29.2</version>
  <exclusions>
    <exclusion>
      <artifactId>liquibase-commercial</artifactId>
      <groupId>org.liquibase</groupId>
    </exclusion>
  </exclusions>
</dependency>

• Create a new GeoNetwork 5 database module which will contains changesets specific to GeoNetwork 5:

Example of a conditional changeset:

--liquibase formatted sql
--changeset francois:00001
--preconditions onFail:HALT onError:HALT
--precondition-sql-check expectedResult:1 SELECT COUNT(*) FROM settings WHERE name='system/platform/version' AND value='4.4.10';

UPDATE Settings SET value='5.0.0' WHERE name='system/platform/version';

• Adding dependency on GeoNetwork 4 database module and GeoNetwork 5 database module in shared/gn-domain/pom.xml:

    <dependency>
      <groupId>org.geonetwork-opensource</groupId>
      <artifactId>gn-database</artifactId>
      <version>4.4.10-SNAPSHOT</version>
    </dependency>
    <dependency>
      <groupId>org.geonetwork</groupId>
      <artifactId>gn-database</artifactId>
      <version>${project.version}</version>
    </dependency>

• Adding the following properties to application.yml:

 spring:
   application:
     name: GeoNetwork
   liquibase:
     enabled: true
     change-log: classpath:/db/changelog.xml

Then both GeoNetwork 4 and GeoNetwork 5 can use Liquibase to initialize the database schema and data.
Liquibase has a lock mechanism to avoid multiple instances to apply changes at the same time.

Questions

To be investigated:

• Check context and labels to flag GN4/GN5 changesets? (See https://docs.liquibase.com/reference-guide/changelog-attributes/what-are-contexts)
• Define process to manage changes from GN4 and GN5?
• How to generate changelogs? Intellij has support for liquibase https://www.jetbrains.com/help/idea/liquibase.html
• ...

Checklist

• I have read the contribution guidelines
• Pull request provided for main branch, backports managed with label
• Good housekeeping of code, cleaning up comments, tests, and documentation
• Clean commit history broken into understandable chucks, avoiding big commits with hundreds of files, cautious of reformatting and whitespace changes
• Clean commit messages, longer verbose messages are encouraged
• API Changes are identified in commit messages
• Testing provided for features or enhancements using automatic tests
• User documentation provided for new features or enhancements in manual
• Build documentation provided for development instructions in README.md files
• Library management using pom.xml dependency management. Update build documentation with intended library use and library tutorials or documentation

Inspired by work done by @joachimnielandt in https://metadata.vlaanderen.be/
Looking for funding

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[josegar74]

@fxprunayre, I think for this pull request it should be relevant to create a 4.4.x branch and update main to 4.6.x, what do you think?