Skip to content

feat(core): DSPX-2126 simplify docker compose UX #2958

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
b-long wants to merge 12 commits into
base: main
Choose a base branch
from
Draft

feat(core): DSPX-2126 simplify docker compose UX #2958

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
45bb685
feat(core): tweak docker-compose to allow a self-contained UX
b-long Dec 17, 2025
48927ec
feat(core): tweak Consuming.md, to use self-contained docker-compose UX
b-long Dec 17, 2025
fe8a1df
feat(core): tweak Consuming.md, to use self-contained docker-compose UX
b-long Dec 17, 2025
dc25a60
remove 'go' reference, no longer needed
b-long Dec 17, 2025
557851f
distinguish consumer and default compose files
b-long Dec 17, 2025
3c0effb
update docs/Consuming.md
b-long Dec 17, 2025
2c3fe64
DRY compose files & consolidate using "extends:"
b-long Dec 17, 2025
bf8222e
feat(core): tweak docker-compose to allow a self-contained UX
b-long Dec 18, 2025
18375ee
feat(core): tweak Consuming.md, to use self-contained docker-compose UX
b-long Dec 18, 2025
e2d734d
feat(core): tweak Consuming.md, to use self-contained docker-compose UX
b-long Dec 18, 2025
698197e
feat(core): tweak Consuming.md, to use self-contained docker-compose UX
b-long Dec 18, 2025
7e2705c
feat(core): tweak Consuming.md, to use self-contained docker-compose UX
b-long Dec 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 222 additions & 13 deletions docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,22 @@ networks:
services:
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This docker-compose.yaml file uses extends to inherit service configurations from docker-compose.base.yaml, which is great for reducing duplication. However, top-level keys like networks and volumes are not inherited via extends. This means that running docker compose up with just this file will fail or not work as intended because the opentdf_platform network and the ers_* volumes are not defined.

To fix this, you can use the include directive (available in Compose v2.20+) at the top of this file to properly include the base configuration:

include:
  - docker-compose.base.yaml

services:
  ...

Alternatively, you would need to instruct developers to always use docker compose -f docker-compose.yaml -f docker-compose.base.yaml up, which is less convenient.

keycloak:
volumes:
- ${KEYS_DIR:-./keys}/localhost.crt:/etc/x509/tls/localhost.crt
- ${KEYS_DIR:-./keys}/localhost.key:/etc/x509/tls/localhost.key
- ${KEYS_DIR:-./keys}/ca.jks:/truststore/truststore.jks
- keys:/keys:ro
image: keycloak/keycloak:25.0
restart: always
depends_on:
fix-keys-permissions:
condition: service_completed_successfully
command:
- "start-dev"
- "--verbose"
- "-Djavax.net.ssl.trustStorePassword=password"
- "-Djavax.net.ssl.HostnameVerifier=AllowAll"
- "-Djavax.net.ssl.trustStore=/truststore/truststore.jks"
- "-Djavax.net.ssl.trustStore=/keys/ca.jks"
- "--spi-truststore-file-hostname-verification-policy=ANY"
environment:
KC_PROXY: edge
KC_HTTP_RELATIVE_PATH: /auth
KC_DB_VENDOR: postgres
KC_DB_URL_HOST: keycloakdb
KC_DB_URL_PORT: 5432
KC_DB_URL_DATABASE: keycloak
KC_DB_USERNAME: keycloak
KC_DB_PASSWORD: changeme
KC_HOSTNAME_STRICT: "false"
KC_HOSTNAME_STRICT_BACKCHANNEL: "false"
KC_HOSTNAME_STRICT_HTTPS: "false"
Expand All @@ -38,9 +33,9 @@ services:
KC_FEATURES: "preview,token-exchange"
KC_HEALTH_ENABLED: "true"
KC_HTTPS_KEY_STORE_PASSWORD: "password"
KC_HTTPS_KEY_STORE_FILE: "/truststore/truststore.jks"
KC_HTTPS_CERTIFICATE_FILE: "/etc/x509/tls/localhost.crt"
KC_HTTPS_CERTIFICATE_KEY_FILE: "/etc/x509/tls/localhost.key"
KC_HTTPS_KEY_STORE_FILE: "/keys/ca.jks"
KC_HTTPS_CERTIFICATE_FILE: "/keys/localhost.crt"
KC_HTTPS_CERTIFICATE_KEY_FILE: "/keys/localhost.key"
KC_HTTPS_CLIENT_AUTH: "request"
###
# The following environment variable resolves SIGILL with Code 134 when running Java processes on Apple M4 chips
Expand Down Expand Up @@ -204,7 +199,221 @@ services:
ers-ldap:
condition: service_healthy

# Provision Keycloak with initial configuration
platform-provision-keycloak:
image: registry.opentdf.io/platform:nightly
command: ["provision", "keycloak", "-e", "http://keycloak:8888/auth", "-f", "/configs/keycloak_data.yaml"]
depends_on:
keycloak:
condition: service_healthy
opentdfdb:
condition: service_healthy
download-platform-config:
condition: service_completed_successfully
download-keycloak-data:
condition: service_completed_successfully
generate-keys:
condition: service_completed_successfully
volumes:
- configs:/configs:ro
- keys:/keys:ro
environment:
- OPENTDF_CONFIG_FILE=/configs/opentdf.yaml
restart: "no"

# Prepare fixtures directory structure - create symlink to expected location
prepare-fixtures:
image: alpine:latest
volumes:
- configs:/configs
depends_on:
download-fixtures:
condition: service_completed_successfully
command:
- sh
- -c
- |
mkdir -p /configs/service/internal/fixtures
cd /configs
ln -sf /configs/service/internal/fixtures ./service
restart: "no"

# Add sample attributes and metadata
platform-provision-fixtures:
image: registry.opentdf.io/platform:nightly
command: ["provision", "fixtures", "--config-file", "/configs/opentdf.yaml"]
working_dir: /configs
depends_on:
platform-provision-keycloak:
condition: service_completed_successfully
opentdfdb:
condition: service_healthy
prepare-fixtures:
condition: service_completed_successfully
generate-keys:
condition: service_completed_successfully
volumes:
- configs:/configs:ro
- keys:/keys:ro
restart: "no"

# Main OpenTDF Platform server
platform:
image: registry.opentdf.io/platform:nightly
command: ["start", "--config-file", "/configs/opentdf.yaml"]
depends_on:
platform-provision-fixtures:
condition: service_completed_successfully
keycloak:
condition: service_healthy
opentdfdb:
condition: service_healthy
generate-keys:
condition: service_completed_successfully
ports:
- "8080:8080"
volumes:
- configs:/configs:ro
- keys:/keys:ro
restart: unless-stopped

# Initialize volume permissions
init-volumes:
image: alpine:latest
volumes:
- configs:/configs
- keys:/keys
command:
- sh
- -c
- |
chmod 777 /configs /keys
mkdir -p /configs/service/internal/fixtures
chmod -R 777 /configs
restart: "no"

# Fix keys permissions after generation
fix-keys-permissions:
image: alpine:latest
volumes:
- keys:/keys
depends_on:
generate-keys:
condition: service_completed_successfully
command:
- sh
- -c
- |
chmod -R 755 /keys
chmod 644 /keys/*
restart: "no"

# Download platform configuration file
download-platform-config:
image: curlimages/curl:latest
volumes:
- configs:/configs
depends_on:
init-volumes:
condition: service_completed_successfully
command: ['-o', '/configs/opentdf.yaml', 'https://raw.githubusercontent.com/opentdf/platform/main/opentdf-example.yaml']
restart: "no"

# Download Keycloak provisioning data
download-keycloak-data:
image: curlimages/curl:latest
volumes:
- configs:/configs
depends_on:
init-volumes:
condition: service_completed_successfully
command: ['-o', '/configs/keycloak_data.yaml', 'https://raw.githubusercontent.com/opentdf/platform/main/service/cmd/keycloak_data.yaml']
restart: "no"

# Download fixtures data
download-fixtures:
image: curlimages/curl:latest
volumes:
- configs:/configs
depends_on:
init-volumes:
condition: service_completed_successfully
command: ['-o', '/configs/service/internal/fixtures/policy_fixtures.yaml', 'https://raw.githubusercontent.com/opentdf/platform/main/service/internal/fixtures/policy_fixtures.yaml']
restart: "no"

# Download init-temp-keys script
download-init-script:
image: curlimages/curl:latest
volumes:
- configs:/configs
depends_on:
init-volumes:
condition: service_completed_successfully
command: ['-o', '/configs/init-temp-keys.sh', 'https://raw.githubusercontent.com/opentdf/platform/main/.github/scripts/init-temp-keys.sh']
restart: "no"

# Generate keys without Docker dependency
generate-keys:
image: alpine:latest
volumes:
- configs:/configs
- keys:/keys
depends_on:
download-init-script:
condition: service_completed_successfully
init-volumes:
condition: service_completed_successfully
entrypoint: /bin/sh
command:
- -c
- |
apk add --no-cache openssl openjdk11-jre bash
cd /keys

# Generate KAS RSA private key
openssl genpkey -algorithm RSA -out /keys/kas-private.pem -pkeyopt rsa_keygen_bits:2048
openssl rsa -in /keys/kas-private.pem -pubout -out /keys/kas-cert.pem

# Generate ECC Key
openssl ecparam -name prime256v1 > /tmp/ecparams.tmp
openssl req -x509 -nodes -newkey ec:/tmp/ecparams.tmp -subj "/CN=kas" -keyout /keys/kas-ec-private.pem -out /keys/kas-ec-cert.pem -days 365

# Generate CA
openssl req -x509 -nodes -newkey RSA:2048 -subj "/CN=ca" -keyout /keys/keycloak-ca-private.pem -out /keys/keycloak-ca.pem -days 365

# Generate localhost certificate
printf "subjectAltName=DNS:localhost,IP:127.0.0.1" > /tmp/sanX509.conf
printf "[req]\ndistinguished_name=req_distinguished_name\n[req_distinguished_name]\n[alt_names]\nDNS.1=localhost\nIP.1=127.0.0.1" > /tmp/req.conf
openssl req -new -nodes -newkey rsa:2048 -keyout /keys/localhost.key -out /tmp/localhost.req -batch -subj "/CN=localhost" -config /tmp/req.conf
openssl x509 -req -in /tmp/localhost.req -CA /keys/keycloak-ca.pem -CAkey /keys/keycloak-ca-private.pem -CAcreateserial -out /keys/localhost.crt -days 3650 -sha256 -extfile /tmp/sanX509.conf

# Generate sample user certificate
openssl req -new -nodes -newkey rsa:2048 -keyout /keys/sampleuser.key -out /tmp/sampleuser.req -batch -subj "/CN=sampleuser"
openssl x509 -req -in /tmp/sampleuser.req -CA /keys/keycloak-ca.pem -CAkey /keys/keycloak-ca-private.pem -CAcreateserial -out /keys/sampleuser.crt -days 3650

# Convert to PKCS12
openssl pkcs12 -export -in /keys/keycloak-ca.pem -inkey /keys/keycloak-ca-private.pem -out /keys/ca.p12 -nodes -passout pass:password

# Convert PKCS12 to JKS using keytool (no Docker needed)
keytool -importkeystore \
-srckeystore /keys/ca.p12 \
-srcstoretype PKCS12 \
-destkeystore /keys/ca.jks \
-deststoretype JKS \
-srcstorepass "password" \
-deststorepass "password" \
-noprompt

echo "Keys generated successfully"
environment:
JAVA_OPTS_APPEND: "${JAVA_OPTS_APPEND:-}"
restart: "no"

volumes:
keys:
name: opentdf_keys
configs:
name: opentdf_configs
ers_postgres_data:
name: ers_test_postgres_data
ers_ldap_data:
Expand Down
76 changes: 35 additions & 41 deletions docs/Consuming.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,55 +15,49 @@ To get started with the OpenTDF platform make sure you are running the same Go v
<!-- markdownlint-disable MD034 github embedded sourcecode -->
https://github.com/opentdf/platform/blob/main/service/go.mod#L3

> **Note for Apple M4 chip users:**
> If you are running on an Apple M4 chip, set the Java environment variable before running any commands:
> ```sh
> export JAVA_OPTS_APPEND="-XX:UseSVE=0"
> ```
> This resolves SIGILL with Code 134 errors when running Java processes.

**Start Platform Services**

Start all services including automated provisioning with [compose-spec](https://compose-spec.io).

1. **Initialize Platform Configuration**
```shell
cp opentdf-dev.yaml opentdf.yaml
```shell
# If running on Apple M4 chip
JAVA_OPTS_APPEND="-XX:UseSVE=0" docker compose up

# Generate development keys/certs for the platform infrastructure.
./.github/scripts/init-temp-keys.sh
# Or on other architectures
docker compose up
```

# The following command is for macOS to trust the local certificate.
# For Linux, you may need to use a different command, e.g.:
# sudo cp ./keys/localhost.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ./keys/localhost.crt
```
- Optional: Update the [configuration](./Configuring.md) as needed.
- Optional: To remove the certificate, run:
```shell
sudo security delete-certificate -c "localhost"
```
2. **Start Background Services**

Start the required infrastructure with [compose-spec](https://compose-spec.io).

```shell
docker compose up
```
3. **Provision Keycloak**
```shell
go run ./service provision keycloak
```
4. **Add Sample Attributes and Metadata**
```shell
go run ./service provision fixtures
```
5. **Start Server**
```shell
go run ./service start
```
This will automatically:
- Download configuration files from GitHub
- Generate development keys and certificates
- Start the background services (Keycloak, PostgreSQL)
- Provision Keycloak with initial configuration
- Add sample attributes and metadata
- Start the OpenTDF Platform server

## 🎉 Your platform is ready to use!

You can now access platform services at http://localhost:8080/ , and Keycloak at http://localhost:8888/auth/ .

### Optional: Trust the Local Certificate

If you want to trust the auto-generated certificate on your host machine:

```shell
# For macOS
docker cp platform-keycloak-1:/keys/localhost.crt ./localhost.crt
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ./localhost.crt

# For Linux
docker cp platform-keycloak-1:/keys/localhost.crt ./localhost.crt
sudo cp ./localhost.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates
```

To remove the certificate later:
```shell
sudo security delete-certificate -c "localhost" # macOS
```

## Next steps
* Try out our CLI (`otdfctl`): https://github.com/opentdf/otdfctl
```sh
Expand Down
Loading