Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactoring/fairspc 82 #1566

Open
wants to merge 82 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
f194327
FAIRSPC-82: started switching to Spring Boot
tgreenwood Jun 11, 2024
5147398
FAIRSPC-82: started switching to Spring Boot
tgreenwood Jun 11, 2024
f96bc0a
Merge remote-tracking branch 'origin/refactoring/FAIRSPC-82' into ref…
tgreenwood Jul 18, 2024
8e1ee28
FAIRSPC-82: config loader
tgreenwood Aug 13, 2024
22e00fb
FAIRSPC-82: updated security config
tgreenwood Aug 23, 2024
0d9d964
FAIRSPC-82: fixed webdav endpoints
tgreenwood Aug 23, 2024
923bfc1
FAIRSPC-82: fixed user service and security config
tgreenwood Aug 23, 2024
34620c5
FAIRSPC-82: fixed security issue in async processes
tgreenwood Sep 4, 2024
322325a
FAIRSPC-82: fixed webdav endpoints
tgreenwood Sep 4, 2024
f3564b3
FAIRSPC-82: added SPARQL query read-only endpoint
tgreenwood Sep 6, 2024
9fa7093
FAIRSPC-82: slightly cleaned up the code, removed obsoleted todo's
tgreenwood Sep 6, 2024
84d0914
FAIRSPC-82: added permission check for the sparql query
tgreenwood Sep 6, 2024
3b3a2c8
FAIRSPC-82: added features config and condition to switch it on/off
tgreenwood Sep 9, 2024
b8e8bb8
FAIRSPC-82: moved part of configs to properties files
tgreenwood Sep 10, 2024
fc0d25e
FAIRSPC-82: switched cache config to spring properties
tgreenwood Sep 10, 2024
88052d8
FAIRSPC-82:
tgreenwood Sep 11, 2024
2dd43ad
FAIRSPC-82: moved public url to application.yml
tgreenwood Sep 11, 2024
e50b5b0
FAIRSPC-82: moved webdav to application.yml
tgreenwood Sep 12, 2024
66d5476
FAIRSPC-82: moved auth config to keycloak properties
tgreenwood Sep 16, 2024
3363eb0
FAIRSPC-82: removed old-fashion readiness and health endpoints; added…
tgreenwood Sep 16, 2024
75c87b3
FAIRSPC-82: fixed pluto health check of saturn
tgreenwood Sep 17, 2024
96c501e
FAIRSPC-82: got rid of config file amd moved to application.yaml
tgreenwood Sep 17, 2024
8d113f4
Merge branch 'dev' into refactoring/FAIRSPC-82
tgreenwood Sep 18, 2024
77b247a
FAIRSPC-82: applied spotless rules + warnings resolve
tgreenwood Sep 18, 2024
2d0f542
FAIRSPC-82: fixed some tests + refactoring
tgreenwood Sep 23, 2024
9dbe166
FAIRSPC-82: fixed some tests
tgreenwood Sep 26, 2024
f2889a7
FAIRSPC-82: fixed UserService tests
tgreenwood Sep 27, 2024
6590c66
FAIRSPC-82: fixed a bunch of tests
tgreenwood Oct 1, 2024
164c986
FAIRSPC-82: upgraded version of jena
tgreenwood Oct 2, 2024
0710bb3
FAIRSPC-82: fixed some tests
tgreenwood Oct 2, 2024
5f76dd4
FAIRSPC-82: fixed the remaining failing tests
tgreenwood Oct 2, 2024
718c152
FAIRSPC-82: cleaned up warnings
tgreenwood Oct 2, 2024
009acc2
FAIRSPC-82: cleaned up warnings
tgreenwood Oct 2, 2024
c951859
FAIRSPC-82: removed redundant ttl files
tgreenwood Oct 2, 2024
1e72d47
FAIRSPC-82 removed accidentally added yarn command
tgreenwood Oct 8, 2024
0c9043a
Update README.adoc
tgreenwood Oct 8, 2024
29adbdb
FAIRSPC-81: migrated views endpoint to spring mvc
tgreenwood Oct 3, 2024
d3f11cd
FAIRSPC-81: migrated workspace endpoints to spring mvc
tgreenwood Oct 4, 2024
51b3ce9
FAIRSPC-81: refactoring of controller tests
tgreenwood Oct 4, 2024
ea2e4d6
FAIRSPC-81: migrated metadata API to spring mvc
tgreenwood Oct 7, 2024
e070d9d
FAIRSPC-81: migrated vocabulary API to spring mvc
tgreenwood Oct 7, 2024
0e2a77b
FAIRSPC-81: migrated features endpoint to spring mvc
tgreenwood Oct 8, 2024
1351e93
FAIRSPC-81: migrated maintain endpoint to spring mvc
tgreenwood Oct 8, 2024
e344891
FAIRSPC-81: migrated user endpoints to spring mvc
tgreenwood Oct 8, 2024
1500c8a
FAIRSPC-82: fixed typo
tgreenwood Oct 9, 2024
766f84b
FAIRSPC-82: rollback of accidentally commited change in security config
tgreenwood Oct 9, 2024
86203c3
FAIRSPC-82: fixed health endpoints
tgreenwood Oct 9, 2024
d96b8c9
FAIRSPC-82: added timeout to sparql endpoint
tgreenwood Oct 9, 2024
7c8bbd0
Merge branch 'refactoring/FAIRSPC-82' into refactoring/FAIRSPC-81
tgreenwood Oct 9, 2024
b81b005
FAIRSPC-82: migrated search endpoints to spring mvc
tgreenwood Oct 9, 2024
b480ae7
FAIRSPC-82: removed redundant logout from Saturn as logging out happe…
tgreenwood Oct 10, 2024
e53bc8e
Merge branch 'refactoring/FAIRSPC-82' into refactoring/FAIRSPC-81
tgreenwood Oct 10, 2024
17e7514
FAIRSPC-81: got rid of java-spark framework and Jetty
tgreenwood Oct 10, 2024
921ade7
FAIRSPC-81: tuned global exception handling
tgreenwood Oct 11, 2024
c8217f6
FAIRSPC-82: fixed liveness and readiness probes of Saturn in helm charts
tgreenwood Oct 11, 2024
41d5807
FAIRSPC-82: got rid of PermissionService
tgreenwood Oct 11, 2024
8f85b5f
FAIRSPC-82: removed unused method and rolled back accidentally commit…
tgreenwood Oct 14, 2024
d4c338b
Merge branch 'refactoring/FAIRSPC-82' into refactoring/FAIRSPC-81
tgreenwood Oct 14, 2024
62eb244
FAIRSPC-79: partially refactored code with some services marked as bean
tgreenwood Oct 14, 2024
a1fbee9
FAIRSPC-79: migrated all web DAV instantiation under Spring's control
tgreenwood Oct 14, 2024
ac7b6e4
FAIRSPC-81: aligned all controllers with /api/ path
tgreenwood Oct 14, 2024
b18d47a
FAIRSPC-81: named reasonable get view data
tgreenwood Oct 14, 2024
c35bae3
FAIRSPC-81: removed redundant try-catch
tgreenwood Oct 14, 2024
a0c37d1
FAIRSPC-81: aligned the way we map resources across all controllers
tgreenwood Oct 14, 2024
001022e
FAIRSPC-81: removed mentioning of spark on log config
tgreenwood Oct 14, 2024
91ef6af
FAIRSPC-81: replaced two value classes with records
tgreenwood Oct 14, 2024
8397e6f
Merge branch 'refactoring/FAIRSPC-81' into refactoring/FAIRSPC-79
tgreenwood Oct 14, 2024
fc40d62
FAIRSPC-81: set servlet context path to /api
tgreenwood Oct 15, 2024
3ce8b8b
Merge branch 'refactoring/FAIRSPC-81' into refactoring/FAIRSPC-79
tgreenwood Oct 15, 2024
f8d27ea
FAIRSPC-81: got rid of Service.java (first iteration)
tgreenwood Oct 15, 2024
59e870e
Merge pull request #1567 from thehyve/refactoring/FAIRSPC-81
tgreenwood Oct 17, 2024
0d93e23
FAIRSPC-79: made configs more granular and other changes towards IoC …
tgreenwood Oct 17, 2024
dd99457
FAIRSPC-79: got rid of redundant qualifier annotations
tgreenwood Oct 17, 2024
9fd8994
FAIRSPC-79: migrated vocabularies from static context to IoC managed
tgreenwood Oct 17, 2024
00ccac3
FAIRSPC-79: migrated validators to spring initialisation
tgreenwood Oct 17, 2024
5725884
FAIRSPC-79: light refactoring + important todo
tgreenwood Oct 17, 2024
9a2890f
FAIRSPC-79: fixed all tests after migration
tgreenwood Oct 17, 2024
bd77df5
FAIRSPC-79: removed redundant init block in tests
tgreenwood Oct 23, 2024
c36d3a1
FAIRSPC-79: changed default transaction simple to bulk
tgreenwood Oct 23, 2024
9d07d42
FAIRSPC-79: removed dependency of URI validator from DavFactory
tgreenwood Oct 23, 2024
56e4d30
FAIRSPC-79: updated todos
tgreenwood Oct 23, 2024
19e7e18
Merge pull request #1568 from thehyve/refactoring/FAIRSPC-79
tgreenwood Oct 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2238,10 +2238,10 @@ The data models for these metadata are defined using the {SHACL}[Shapes Constrai
* The customisable data model includes the custom (shared)
metadata entities, custom controlled vocabulary types,
and custom properties of the system entities.
The default custom data model is defined in link:projects/saturn/vocabulary.ttl[vocabulary.ttl].
The default custom data model is defined in link:projects/saturn/src/main/resources/vocabulary.ttl[vocabulary.ttl].
This data model can be overriden by a data more suitable for your organisation.

A schematic overview of the default data model in link:projects/saturn/vocabulary.ttl[vocabulary.ttl]:
A schematic overview of the default data model in link:projects/saturn/src/main/resources/vocabulary.ttl[vocabulary.ttl]:

image:docs/images/diagrams/CDR data model.png[CDR data model]

Expand Down Expand Up @@ -2462,8 +2462,8 @@ use `--set-file` option:
+
This should also restart the Saturn pod. If not, trigger the restart manually.
+
For local development - replace vocabulary file in link:projects/saturn/vocabulary.ttl[projects/saturn/vocabulary.ttl]
and views configuration in link:projects/saturn/views.ttl[projects/saturn/views.ttl].
For local development - replace vocabulary file in link:projects/saturn/src/main/resources/vocabulary.ttl[vocabulary.ttl]
and views configuration in link:projects/saturn/src/main/resources/views.yaml[views.yaml].
Restart Saturn run.
+
. Load data for new entities or properties.
Expand All @@ -2474,7 +2474,7 @@ Restart Saturn run.

For controlled vocabulary types, e.g., _Gender_ and _Species_ in the example, you should insert the allowed values in the database by uploading
a taxonomies file using the <<RDF metadata>> API.
An example taxonomy is in link:projects/saturn/taxonomies.ttl[taxonomies.ttl].
An example taxonomy is in link:projects/saturn/src/main/resources/taxonomies.ttl[taxonomies.ttl].

It is preferred to use existing standard taxonomies and labels.
If that is not possible, please define your own namespaces for
Expand Down Expand Up @@ -2526,7 +2526,7 @@ ncbitaxon:10090 a example:Species ;

For the metadata pages in the user interface, a view configuration needs to be created
that specifies the tables and columns.
An example can be found in link:projects/saturn/views.yaml[views.yaml]
An example can be found in link:projects/saturn/src/main/resources/views.yaml[views.yaml]



Expand Down
8 changes: 4 additions & 4 deletions charts/fairspace/templates/project/stateful-set.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -100,16 +100,16 @@ spec:
{{- end }}
livenessProbe:
httpGet:
path: /liveness
path: /actuator/health/liveness
port: 8091
initialDelaySeconds: {{ .Values.saturn.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.saturn.livenessProbe.periodSeconds }}
successThreshold: {{ .Values.saturn.livenessProbe.successThreshold }}
timeoutSeconds: {{ .Values.saturn.livenessProbe.timeoutSeconds }}
readinessProbe:
httpGet:
path: /api/health/
port: 8090
path: /actuator/health/readiness
port: 8091
periodSeconds: {{ .Values.saturn.readinessProbe.periodSeconds }}
successThreshold: {{ .Values.saturn.readinessProbe.successThreshold }}
timeoutSeconds: {{ .Values.saturn.readinessProbe.timeoutSeconds }}
Expand Down Expand Up @@ -222,4 +222,4 @@ spec:
storageClassName: {{ index .Values "saturn" "persistence" "extra-file-storage" "storageClass" | quote }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
4 changes: 2 additions & 2 deletions projects/mercury/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
"dev-be": "concurrently \"yarn localdevelopment\" \"yarn saturn\" \"yarn pluto\"",
"dev-fe": "yarn build && yarn start",
"localdevelopment": "cd ../../local-development && docker compose up",
"saturn": "wait-on http://localhost:5100/ && wait-on tcp:9432 && cd ../saturn/ && KEYCLOAK_CLIENT_SECRET=********** ./gradlew run",
"pluto": "wait-on http://localhost:8090/api/health/ && cd ../pluto/ && KEYCLOAK_CLIENT_SECRET=********** ./gradlew bootRun --args='--spring.profiles.active=local'",
"saturn": "wait-on http://localhost:5100/ && wait-on tcp:9432 && cd ../saturn/ && KEYCLOAK_CLIENT_SECRET=********** ./gradlew bootRun",
"pluto": "wait-on http://localhost:8091/actuator/health/ && cd ../pluto/ && KEYCLOAK_CLIENT_SECRET=********** ./gradlew bootRun --args='--spring.profiles.active=local'",
"start": "craco start",
"build": "yarn check-format && craco build",
"test": "craco test --env=jsdom",
Expand Down
2 changes: 1 addition & 1 deletion projects/pluto/src/main/resources/application-local.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ pluto:
domains:
- http://localhost:8080
force-https: false
downstreamServiceHealthUrl: http://localhost:8090/api/health/
downstreamServiceHealthUrl: http://localhost:8091/actuator/health/
oauth2:
base-url: http://localhost:5100
realm: fairspace
Expand Down
68 changes: 0 additions & 68 deletions projects/saturn/application.yaml

This file was deleted.

37 changes: 17 additions & 20 deletions projects/saturn/build.gradle
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
buildscript {
ext {
jena_version = '4.10.0' // todo: upgrade to 5+ (FAIRSPC-69)
milton_version = '3.1.1.488' // Milton >= 4 is migrated to Jakarta EE 9, which is not compatible with Jena < 5 (no stable release of Jena 5 yet). To be updated when Jena 5 is released.
jena_version = '5.1.0'
milton_version = '4.0.2.2101'
mockitoVersion = '5.11.0'
jacksonVersion = '2.15.3' // check what version is used by Jena
postgresqlVersion = '42.7.2'
Expand All @@ -10,6 +10,8 @@ buildscript {

plugins {
id 'java'
id 'org.springframework.boot' version '3.3.0'
id 'io.spring.dependency-management' version '1.1.5'
id "io.freefair.lombok" version "8.6"
id 'application'
id 'jacoco'
Expand All @@ -26,7 +28,7 @@ compileJava {
}

application {
mainClassName = "io.fairspace.saturn.App"
mainClassName = "io.fairspace.saturn.SaturnApplication"
applicationDefaultJvmArgs = ['-XX:+ShowCodeDetailsInExceptionMessages']
}

Expand All @@ -38,53 +40,47 @@ repositories {
}
}

lombok.version = "1.18.30"
lombok.version = "1.18.32"

jacoco.toolVersion = "0.8.11"

dependencies {
// implementation fileTree(dir: 'libs', include: '*.jar') // jena-fuseki-server-*.jar jena-text-es-*.jar
implementation "org.apache.jena:jena-fuseki-main:${jena_version}"
implementation 'org.springframework.boot:spring-boot-starter-oauth2-resource-server'
implementation 'org.springframework.boot:spring-boot-starter-security'
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation 'org.springframework.boot:spring-boot-starter-validation'
implementation 'org.springframework.boot:spring-boot-starter-actuator'

implementation "org.apache.jena:jena-text:${jena_version}"
implementation "io.milton:milton-server-ce:${milton_version}"

implementation "com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:${jacksonVersion}"
implementation "com.fasterxml.jackson.datatype:jackson-datatype-jsr310:${jacksonVersion}"
implementation "com.sparkjava:spark-core:2.9.4"

implementation 'com.pivovarit:throwing-function:1.5.1'
implementation 'com.google.guava:guava:33.0.0-jre'
// implementation 'jakarta.servlet:jakarta.servlet-api:6.0.0' // To be updated when Jena 5 is released.
implementation('com.io-informatics.oss:jackson-jsonld:0.1.1') {
exclude group: 'com.github.jsonld-java'
}
implementation('org.bouncycastle:bcprov-jdk15on:1.70')

implementation 'org.apache.logging.log4j:log4j-layout-template-json:2.23.0'

implementation 'org.eclipse.jetty:jetty-proxy:9.4.54.v20240208'

implementation 'org.keycloak:keycloak-jetty94-adapter:23.0.7'
implementation 'org.keycloak:keycloak-admin-client:23.0.7'
implementation 'org.keycloak:keycloak-policy-enforcer:23.0.7'

implementation "org.postgresql:postgresql:${postgresqlVersion}"
implementation "com.zaxxer:HikariCP:5.1.0"

runtimeOnly 'org.apache.logging.log4j:log4j-api:2.23.0'
runtimeOnly 'org.apache.logging.log4j:log4j-core:2.23.0'
runtimeOnly 'org.apache.logging.log4j:log4j-slf4j2-impl:2.23.0'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is it not needed anymore?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log4j-slf4j2-impl is not used, all the rest come with other dependencies

by the way, I noticed you use several different log providers in Saturn (slf4j, lo4j, log4j2) - I will create ticket to align it across Saturn


testImplementation "junit:junit:4.13.2"
testImplementation "org.mockito:mockito-core:${mockitoVersion}"
testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'org.junit.vintage:junit-vintage-engine:5.10.2'
testImplementation "org.testcontainers:postgresql:1.19.6"
testImplementation('com.github.stefanbirkner:system-rules:1.19.0') {
exclude group: 'junit', module:'junit-dep'
}

constraints {
// implementation('com.fasterxml.jackson.core:jackson-databind:2.9.10.1') {
// because 'previous versions have security vulnerabilities'
// }
}
}

jacocoTestReport {
Expand Down Expand Up @@ -117,4 +113,5 @@ test {
// They also blocks usage of testing library mocking environment variables,
// That is why this additional arg for tests is needed
jvmArgs = ['--add-opens', 'java.base/java.util=ALL-UNNAMED']
useJUnitPlatform()
}
69 changes: 0 additions & 69 deletions projects/saturn/src/main/java/io/fairspace/saturn/App.java

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
package io.fairspace.saturn;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class SaturnApplication {

public static void main(String[] args) {
SpringApplication.run(SaturnApplication.class, args);
}
}
19 changes: 11 additions & 8 deletions projects/saturn/src/main/java/io/fairspace/saturn/audit/Audit.java
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@

import java.util.Objects;

import org.apache.logging.log4j.*;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;

import static io.fairspace.saturn.auth.RequestContext.getAccessToken;
import static io.fairspace.saturn.auth.RequestContext.getClaims;

public class Audit {
private static final Logger log = LogManager.getLogger("audit");
Expand All @@ -19,16 +20,18 @@ public static void audit(String event, Object... params) {
}
}

var token = getAccessToken();
var claims = getClaims();
String preferredUsername = claims.getPreferredUsername();

if (token.getPreferredUsername() != null) {
ThreadContext.put("user_name", token.getPreferredUsername());
if (preferredUsername != null) {
ThreadContext.put("user_name", preferredUsername);
}

if (token.getEmail() != null) {
ThreadContext.put("user_email", token.getEmail());
String email = claims.getEmail();
if (email != null) {
ThreadContext.put("user_email", email);
}
ThreadContext.put("user_id", token.getSubject());
ThreadContext.put("user_id", claims.getSubject());

log.trace(event);

Expand Down
Loading
Loading