feat: officially release clean feature

Author: bethesque

Dockerfile

@@ -37,14 +37,15 @@ RUN set -ex && \
 
 # Install source
 COPY pact_broker $HOME/
+RUN mv $HOME/clean.sh /usr/local/bin/clean
 
 RUN ln -s /pact_broker/script/db-migrate.sh /usr/local/bin/db-migrate
 RUN ln -s /pact_broker/script/db-version.sh /usr/local/bin/db-version
 
 # Start Puma
 ENV RACK_ENV=production
 ENV PACT_BROKER_PORT_ENVIRONMENT_VARIABLE_NAME=PACT_BROKER_PORT
-ENV PACT_BROKER_DATABASE_BETA_CLEAN_ENABLED=false
+ENV PACT_BROKER_DATABASE_CLEAN_ENABLED=false
 ENV PACT_BROKER_DATABASE_CLEAN_CRON_SCHEDULE="0 2 15 * *"
 ENV PACT_BROKER_DATABASE_CLEAN_KEEP_VERSION_SELECTORS='[{ "latest": true, "tag": true }, { "max_age": 90 }]'
 ENV PACT_BROKER_DATABASE_CLEAN_DELETION_LIMIT=500

README.md

@@ -129,6 +129,38 @@ Set the environment variable `PACT_BROKER_LOG_LEVEL` to one of `DEBUG`, `INFO`,
 
 Documentation for the Pact Broker application itself can be found in the Pact Broker [docs][pact-broker-docs].
 
+## Automatic data clean up
+
+Performance can degrade when too much data accumulates in the Pact Broker. To read about the automatic data clean up feature, please see the [Maintenance](https://docs.pact.io/pact_broker/administration/maintenance) page of the Pact Broker documentation.
+
+### Running the clean task on a cron schedule within the application container
+
+If you have exactly one Pact Broker container running at a time, you can configure cron on the container to run the clean up.
+
+* `PACT_BROKER_DATABASE_CLEAN_ENABLED`: set to `true` to enable the clean. Default is `false`.
+* `PACT_BROKER_DATABASE_CLEAN_CRON_SCHEDULE`: set to a cron schedule that will run when your Broker is under the least operational load. Default is 2:15am - `15 2 * * *`
+* `PACT_BROKER_DATABASE_CLEAN_DELETION_LIMIT`: The maximum number of records to delete at a time for each of the categories listed in the [Categories of removable data](https://docs.pact.io/pact_broker/administration/maintenance#categories-of-removable-data). Defaults to `500`.
+* `PACT_BROKER_DATABASE_CLEAN_OVERWRITTEN_DATA_MAX_AGE`: The maximum number of days to keep "overwritten" data as described in the [Categories of removable data](https://docs.pact.io/pact_broker/administration/maintenance#categories-of-removable-data)
+* `PACT_BROKER_DATABASE_CLEAN_KEEP_VERSION_SELECTORS`:  a JSON string containing a list of the "keep" selectors described in [Configuring the keep selectors](https://docs.pact.io/pact_broker/administration/maintenance#configuring-the-keep-selectors) e.g `[{"latest": true, "tag": true}, {"max_age": 90}]` (remember to escape the quotes if necessary in your configuration files/console).
+* `PACT_BROKER_DATABASE_CLEAN_DRY_RUN`: defaults to `false`. Set to `true` to see the output of what *would* have been deleted if the task had run. This is helpful when experimenting with or fine tuning the clean feature. As nothing is deleted when in dry-run mode, the same output will be printed in the logs each time the task runs.
+
+### Running the clean task from an external source
+
+If you are running more than one Pact Broker Docker container at a time for the same database, then you will end up with two clean up tasks fighting with each other to delete the data. In this situation, it is best to run the clean task from an external location at a regular interval. To do this, run an instance of the pact-broker docker image with the entrypoint `clean`, the same database connection credentials as the application, and the same environment variables described in the section above *except the PACT_BROKER_DATABASE_CLEAN_ENABLED and PACT_BROKER_DATABASE_CLEAN_CRON_SCHEDULE* vars.
+
+You can see a working example in the [docker-compose-clean.yml](./docker-compose-clean.yml) file. To run the example locally, run:
+
+```
+docker-compose -f docker-compose-clean.yml up pact-broker
+
+# in another console
+docker-compose -f docker-compose-clean.yml up clean
+```
+
+### Known issues with the data clean up task
+
+* When the pact-broker docker container gets restarted because of an internal error, another supercronic (the application that runs the cron task in the background) process seems to get started each time, leading to multiple clean tasks running at once. This issue has been noticed in local testing, but we do not know if it is likely to be an issue under normal production use. Please raise an issue if you are observing it. The mitigation for this is to run the clean from an external source as documented above.
+
 ## Running with Docker Compose
 
 For a quick start with the Pact Broker and Postgres, we have an example

docker-compose-clean-cron.yml

@@ -0,0 +1,32 @@
+version: "3"
+
+services:
+  postgres:
+    image: postgres:12
+    healthcheck:
+      test: psql postgres --command "select 1" -U postgres
+    environment:
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: password
+      POSTGRES_DB: postgres
+    volumes:
+      - ${PWD}/docker/postgres-entrypoint.sh:/docker-entrypoint-initdb.d/init-db.sh
+      - ${PWD}/docker/pg-dump.sql:/tmp/pg-dump.sql
+
+  pact-broker:
+    build: .
+    ports:
+      - "9393:9393"
+    depends_on:
+      - postgres
+    environment:
+      PACT_BROKER_DATABASE_URL: "postgres://postgres:password@postgres/postgres"
+      PACT_BROKER_LOG_LEVEL: "INFO"
+      PACT_BROKER_SQL_LOG_LEVEL: "DEBUG"
+      PACT_BROKER_DATABASE_CLEAN_ENABLED: "true"
+      PACT_BROKER_DATABASE_CONNECT_MAX_RETRIES: "5"
+      PACT_BROKER_DATABASE_CLEAN_CRON_SCHEDULE: "* * * * *"
+      PACT_BROKER_DATABASE_CLEAN_DELETION_LIMIT: "500"
+      PACT_BROKER_SQL_LOG_WARN_DURATION: "60"
+      # Keep all prod versions, AND the latest version for every pacticipant/tag, and all versions less than 30 days old
+      PACT_BROKER_DATABASE_CLEAN_KEEP_VERSION_SELECTORS: '[{"latest": true, "tag": true}, {"max_age": 30}]'

docker-compose-clean.yml

@@ -0,0 +1,39 @@
+version: "3"
+
+services:
+  postgres:
+    image: postgres:12
+    healthcheck:
+      test: psql postgres --command "select 1" -U postgres
+    environment:
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: password
+      POSTGRES_DB: postgres
+    volumes:
+      - ${PWD}/docker/postgres-entrypoint.sh:/docker-entrypoint-initdb.d/init-db.sh
+      - ${PWD}/docker/pg-dump.sql:/tmp/pg-dump.sql
+
+  pact-broker:
+    build: .
+    ports:
+      - "9393:9393"
+    depends_on:
+      - postgres
+    environment:
+      PACT_BROKER_DATABASE_URL: "postgres://postgres:password@postgres/postgres"
+      PACT_BROKER_DATABASE_CONNECT_MAX_RETRIES: "5"
+
+  clean:
+    build: .
+    depends_on:
+      - postgres
+    environment:
+      PACT_BROKER_DATABASE_URL: "postgres://postgres:password@postgres/postgres"
+      PACT_BROKER_DATABASE_CONNECT_MAX_RETRIES: "5"
+      PACT_BROKER_DATABASE_CLEAN_DELETION_LIMIT: "500"
+      PACT_BROKER_SQL_LOG_WARN_DURATION: "60"
+      # Keep all prod versions, AND the latest version for every pacticipant/tag, and all versions less than 30 days old
+      PACT_BROKER_DATABASE_CLEAN_KEEP_VERSION_SELECTORS: '[{"latest": true, "tag": true}, {"max_age": 30}]'
+      PACT_BROKER_LOG_LEVEL: INFO
+      PACT_BROKER_SQL_LOG_LEVEL: DEBUG
+    entrypoint: clean

docker-compose-dev-migrate.yml

@@ -0,0 +1,64 @@
+version: "3"
+
+services:
+  postgres:
+    image: postgres:12
+    healthcheck:
+      test: psql postgres --command "select 1" -U postgres
+    # ports:
+    #   - "5432:5432"
+    environment:
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: password
+      POSTGRES_DB: postgres
+    volumes:
+      - ${PWD}/docker/postgres-entrypoint.sh:/docker-entrypoint-initdb.d/init-db.sh
+      - ${PWD}/docker/pg-dump.sql:/tmp/pg-dump.sql
+
+  pact-broker:
+    build: .
+    ports:
+      - "9393:9393"
+    depends_on:
+      - postgres
+    environment:
+      PACT_BROKER_PORT_ENVIRONMENT_VARIABLE_NAME: PORT
+      PACT_BROKER_DATABASE_URL_ENVIRONMENT_VARIABLE_NAME: DATABASE_URL
+      DATABASE_URL: "postgres://postgres:password@postgres/postgres"
+      PACT_BROKER_DATABASE_BETA_CLEAN_ENABLED: "true"
+      PACT_BROKER_DATABASE_CLEAN_CRON_SCHEDULE: "* * * * *"
+      PACT_BROKER_DATABASE_CLEAN_DRY_RUN: "false"
+      PACT_BROKER_DATABASE_CLEAN_DELETION_LIMIT: "500"
+      # Keep all prod versions, AND the latest version for every pacticipant/tag, and all versions less than 30 days old
+      PACT_BROKER_DATABASE_CLEAN_KEEP_VERSION_SELECTORS: '[{"latest": true, "tag": true}, {"max_age": 30}]'
+      # PACT_BROKER_DATABASE_USERNAME: postgres
+      # PACT_BROKER_DATABASE_PASSWORD: password
+      # PACT_BROKER_DATABASE_HOST: postgres
+      # PACT_BROKER_DATABASE_NAME: postgres
+      # PACT_BROKER_PORT: "9292"
+      PORT: '9393'
+      PACT_BROKER_LOG_LEVEL: INFO
+      PACT_BROKER_SQL_LOG_LEVEL: DEBUG
+    volumes:
+      - ./script/docker/docker-compose-entrypoint.sh:/custom-entrypoint.sh
+
+  db-migrate:
+    build: .
+    depends_on:
+      - postgres
+    environment:
+      PACT_BROKER_DATABASE_URL: "postgres://postgres:password@postgres/postgres"
+      PACT_BROKER_LOG_LEVEL: INFO
+      PACT_BROKER_SQL_LOG_LEVEL: DEBUG
+      PACT_BROKER_MIGRATION_TARGET: "20200922"
+    entrypoint: db-migrate
+
+  db-version:
+    build: .
+    depends_on:
+      - postgres
+    environment:
+      PACT_BROKER_DATABASE_URL: "postgres://postgres:password@postgres/postgres"
+      PACT_BROKER_LOG_LEVEL: INFO
+      PACT_BROKER_SQL_LOG_LEVEL: DEBUG
+    entrypoint: db-version

docker-compose-dev.yml

@@ -14,6 +14,8 @@ services:
     volumes:
       - ${PWD}/docker/postgres-entrypoint.sh:/docker-entrypoint-initdb.d/init-db.sh
       - ${PWD}/docker/pg-dump.sql:/tmp/pg-dump.sql
+    logging:
+        driver: none
 
   pact-broker:
     build: .
@@ -25,10 +27,11 @@ services:
       PACT_BROKER_PORT_ENVIRONMENT_VARIABLE_NAME: PORT
       PACT_BROKER_DATABASE_URL_ENVIRONMENT_VARIABLE_NAME: DATABASE_URL
       DATABASE_URL: "postgres://postgres:password@postgres/postgres"
-      PACT_BROKER_DATABASE_BETA_CLEAN_ENABLED: "true"
+      PACT_BROKER_DATABASE_CLEAN_ENABLED: "false"
       PACT_BROKER_DATABASE_CLEAN_CRON_SCHEDULE: "* * * * *"
       PACT_BROKER_DATABASE_CLEAN_DRY_RUN: "false"
       PACT_BROKER_DATABASE_CLEAN_DELETION_LIMIT: "500"
+      PACT_BROKER_SQL_LOG_WARN_DURATION: "60"
       # Keep all prod versions, AND the latest version for every pacticipant/tag, and all versions less than 30 days old
       PACT_BROKER_DATABASE_CLEAN_KEEP_VERSION_SELECTORS: '[{"latest": true, "tag": true}, {"max_age": 30}]'
       PACT_BROKER_DATABASE_CONNECT_MAX_RETRIES: "10"
@@ -40,6 +43,4 @@ services:
       PORT: '9393'
       PACT_BROKER_LOG_LEVEL: INFO
       PACT_BROKER_SQL_LOG_LEVEL: DEBUG
-    entrypoint: /custom-entrypoint.sh
-    volumes:
-      - ./script/docker/docker-compose-entrypoint.sh:/custom-entrypoint.sh
+      PACT_BROKER_DATABASE_CONNECT_MAX_RETRIES: "5"

pact_broker/Rakefile

@@ -5,7 +5,7 @@ require 'pact_broker/tasks'
 
 def create_database_configuration
   dc = PactBroker::DockerConfiguration.new(ENV, OpenStruct.new)
-  create_database_connection_from_config($logger, dc.database_configuration)
+  create_database_connection_from_config($logger, dc.database_configuration, dc.database_connect_max_retries)
 end
 
 def env(name)

pact_broker/clean.sh

@@ -1,4 +1,4 @@
-#/bin/sh
+#!/bin/sh
 
 bundle exec rake pact_broker:db:clean
 bundle exec rake pact_broker:db:delete_overwritten_data

pact_broker/entrypoint.sh

@@ -1,9 +1,8 @@
 #!/bin/sh
 
-if [ "${PACT_BROKER_DATABASE_BETA_CLEAN_ENABLED}" = "true" ] && [ ! -f .supercronic ]; then
+if [ "${PACT_BROKER_DATABASE_CLEAN_ENABLED}" = "true" ]; then
   echo "Creating crontab with schedule ${PACT_BROKER_DATABASE_CLEAN_CRON_SCHEDULE} to clean database"
   echo "${PACT_BROKER_DATABASE_CLEAN_CRON_SCHEDULE} /pact_broker/clean.sh" >> /pact_broker/crontab
-  touch .supercronic
   /usr/local/bin/supercronic -quiet -passthrough-logs /pact_broker/crontab &
 fi