Skip to content

Commit 8fe59c0

Browse files
fxprunayrefxprunayre
committed
Database / Liquibase.
1 parent eb0e8d8 commit 8fe59c0

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+1944
-437
lines changed

‎core/src/test/resources/core-repository-test-context.xml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@
8787
<bean class="org.fao.geonet.TestDatabasePopulator">
8888
<property name="interpolatedScripts">
8989
<list>
90-
<value>classpath:cleanoutdatabase.sql</value>
90+
<!--<value>classpath:cleanoutdatabase.sql</value>
9191
<value>file:/${webapp}/WEB-INF/classes/setup/sql/data/data-db-default.sql</value>
9292
<value>file:/${webapp}/WEB-INF/classes/setup/sql/data/loc-ara-default.sql</value>
9393
<value>file:/${webapp}/WEB-INF/classes/setup/sql/data/loc-cat-default.sql</value>
@@ -104,7 +104,7 @@
104104
<value>file:/${webapp}/WEB-INF/classes/setup/sql/data/loc-rus-default.sql</value>
105105
<value>file:/${webapp}/WEB-INF/classes/setup/sql/data/loc-spa-default.sql</value>
106106
<value>file:/${webapp}/WEB-INF/classes/setup/sql/data/loc-tur-default.sql</value>
107-
<value>file:/${webapp}/WEB-INF/classes/setup/sql/data/loc-vie-default.sql</value>
107+
<value>file:/${webapp}/WEB-INF/classes/setup/sql/data/loc-vie-default.sql</value>-->
108108
</list>
109109
</property>
110110
</bean>
@@ -141,7 +141,7 @@
141141

142142
<bean id="servletContext" class="org.fao.geonet.GeonetMockServletContext"/>
143143
<bean id="threadPool" class="org.fao.geonet.TestThreadPool"/>
144-
144+
145145
<bean class="org.fao.geonet.analytics.WebAnalyticsConfiguration" name="webAnalyticsConfiguration">
146146
<property name="service" value="" />
147147
<property name="javascriptCode" value="" />

‎database/README.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Database module
2+
3+
## Database initialization
4+
5+
Liquibase is used to manage database schema creation and changes and the configuration is made in [`changelog.xml`](src/main/resources/db/changelog.xml).
6+
7+
When adding new entities or making changes to existing ones, please make sure
8+
to update the database changelog accordingly by adding a new file in the [`changesets directory`](src/main/resources/db/changesets).
9+
10+
Numbering is important to keep track of the order of changes, so please follow the existing pattern when naming new changeset files.
11+
When a feature is backported in various branches, the numbering ensure that the changesets are applied in the correct order.
12+
13+
eg. `main` can contain changesets 001 to 010, while `4.2.x` contains changesets 001 to 006 and 009 which was backported.
14+
15+
Liquibase track changes applied to the current database in the `DATABASECHANGELOG` and `DATABASECHANGELOGLOCK` tables.
16+
17+
18+
## Old database initialization and migration system proposal
19+
20+
GeoNetwork using its own mechanism based on Java and SQL to migrate from version to version.
21+
This mechanism does not work well when backporting changes to previous versions and does not support all possible migration paths.
22+
23+
It is proposed the following strategy to move to Liquibase: **Old GeoNetwork system is used until version 4.4.x, then starting from version 4.6.x Liquibase only is used.**
24+
25+
This means:
26+
* Users upgrading to version 4.6.x will first have to migrate to the latest 4.4.x version using the old system, then upgrade to 4.6.x which will use Liquibase.
27+
* New installations of version 4.6.x and later will use Liquibase only.
28+
29+
This approach avoid to migrate all existing migration steps to Liquibase, which is hard to do because of:
30+
* Java migration steps
31+
* Hibernate automatic schema generation not described in SQL
32+
* Complexity to test all migration paths (which are not even working well with the old system)
33+
34+
35+
## Liquibase implementation status
36+
37+
Draft experiment to implement initial database creation and initial data loading using Liquibase.
38+
39+
- [ ] Databases tested (test: first run works, restart works)
40+
- [x] Postgres
41+
- [x] H2 (needed for test in GN4)
42+
- [ ] Other - discuss which database do we support and test?
43+
- [x] [Liquibase bean configuration](../domain/src/main/resources/config-spring-geonetwork.xml) with [`changelog.xml`](src/main/resources/db/changelog.xml) as main changelog file
44+
- [x] [Schema creation](src/main/resources/db/changesets/00000-initial-schema.xml) / Precondition: no table metadata exists
45+
- [x] [Initial data](src/main/resources/db/changesets/00001-initial-data.sql) / Precondition: table settings is empty
46+
- [x] [Initial languages](src/main/resources/db/changesets/00002-initial-data-languages-eng.sql) / Precondition: table language does not have eng
47+
- [ ] Other languages
48+
- [ ] Remove past database migrations
49+
- [ ] Remove old configuration `initial_data.xml` and `database_migration.xml`
50+
- [ ] Remove `db.migration_onstartup` property / Hibernate `hbm2ddl` property set to `none`
51+
- [ ] Remove `TestDatabasePopulator`
52+
- [ ] ...
53+
54+
## GeoNetwork 5 and Liquibase
55+
56+
GeoNetwork 5 can depend on the GeoNetwork 4 database module to initialize the database using Liquibase.
57+
58+
To enable liquibase in GeoNetwork 5 by:
59+
60+
* Adding liquibase dependency in `pom.xml`:
61+
62+
```xml
63+
<dependency>
64+
<groupId>org.liquibase</groupId>
65+
<artifactId>liquibase-core</artifactId>
66+
<version>5.0.1</version>
67+
<exclusions>
68+
<exclusion>
69+
<artifactId>liquibase-commercial</artifactId>
70+
<groupId>org.liquibase</groupId>
71+
</exclusion>
72+
</exclusions>
73+
</dependency>
74+
```
75+
76+
* Adding dependency on GeoNetwork 4 database module in `shared/gn-domain/pom.xml`: (to rework once the dependency is available in a maven repository)
77+
78+
```xml
79+
<dependency>
80+
<groupId>org.geonetwork</groupId>
81+
<artifactId>gn-database</artifactId>
82+
<version>${project.version}</version>
83+
</dependency>
84+
```
85+
86+
* Adding the following properties to `application.yml`:
87+
88+
```yaml
89+
spring:
90+
application:
91+
name: GeoNetwork
92+
liquibase:
93+
enabled: true
94+
change-log: classpath:/db/changelog.xml
95+
```
96+
97+
Then both GeoNetwork 4 and GeoNetwork 5 can use Liquibase to initialize the database schema and data.
98+
Liquibase has a lock mechanism to avoid multiple instances to apply changes at the same time.
99+
100+
101+
## Questions
102+
103+
To be investigated:
104+
* Check context and labels to flag GN4/GN5 changesets? (See https://docs.liquibase.com/reference-guide/changelog-attributes/what-are-contexts)
105+
* Define process to manage changes from GN4 and GN5?
106+
* How to generate changelogs? Intellij has support for liquibase https://www.jetbrains.com/help/idea/liquibase.html
107+

‎database/pom.xml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
~ Copyright (C) 2001-2016 Food and Agriculture Organization of the
4+
~ United Nations (FAO-UN), United Nations World Food Programme (WFP)
5+
~ and United Nations Environment Programme (UNEP)
6+
~
7+
~ This program is free software; you can redistribute it and/or modify
8+
~ it under the terms of the GNU General Public License as published by
9+
~ the Free Software Foundation; either version 2 of the License, or (at
10+
~ your option) any later version.
11+
~
12+
~ This program is distributed in the hope that it will be useful, but
13+
~ WITHOUT ANY WARRANTY; without even the implied warranty of
14+
~ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
15+
~ General Public License for more details.
16+
~
17+
~ You should have received a copy of the GNU General Public License
18+
~ along with this program; if not, write to the Free Software
19+
~ Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA
20+
~
21+
~ Contact: Jeroen Ticheler - FAO - Viale delle Terme di Caracalla 2,
22+
~ Rome - Italy. email: [email protected]
23+
-->
24+
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
25+
xmlns="http://maven.apache.org/POM/4.0.0"
26+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
27+
<parent>
28+
<artifactId>geonetwork</artifactId>
29+
<groupId>org.geonetwork-opensource</groupId>
30+
<version>4.4.10-SNAPSHOT</version>
31+
</parent>
32+
<modelVersion>4.0.0</modelVersion>
33+
<artifactId>gn-database</artifactId>
34+
<name>GeoNetwork database</name>
35+
<dependencies>
36+
<dependency>
37+
<groupId>org.liquibase</groupId>
38+
<artifactId>liquibase-core</artifactId>
39+
</dependency>
40+
</dependencies>
41+
</project>

‎database/src/main/resources/db/changelog.xml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
<?xml version="1.1" encoding="UTF-8" standalone="no"?>
2+
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
3+
xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
4+
xmlns:pro="http://www.liquibase.org/xml/ns/pro"
5+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
6+
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/pro http://www.liquibase.org/xml/ns/pro/liquibase-pro-latest.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">
7+
<includeAll path="classpath:db/changesets/"/>
8+
</databaseChangeLog>

0 commit comments

Comments
 (0)