CCAP-295: Updated how database credentials are read. (#8)

* Updated onedrive destination documentation to note potential issues with omitting the filename parameter.
* Updated database documentation.

Author: jamesiarmes

.pryrc

@@ -6,15 +6,15 @@ require_relative 'lib/config/application'
 require_relative 'lib/model'
 
 config = DocumentTransfer::Config::Application.from_environment
-Sequel.connect(config.database_url)
+Sequel.connect(config.database_credentials)
 
 DocumentTransfer::Model.load
 
 color = "\e[1;32m"
 color = "\e[1;31m" if config.prod?
 color = "\e[1;33m" if config.prod_like?
 
-Pry.config.prompt_name = "document_transfer(#{config.environment})"
+Pry.config.prompt_name = "document-transfer(#{config.environment})"
 Pry.config.prompt = Pry::Prompt.new(
   :document_transfer,
   'Document transfer console prompt',

CODEOWNER renamed to CODEOWNERS

@@ -73,11 +73,9 @@ recommend using a virtual environment manager such as [RVM] to manage your Ruby
 installations. The required version of Ruby is defined in the
 [`.ruby-version`][ruby-version] file.
 
-You will also need a database for the service to store data. The service is
-designed to work with a PostgreSQL database. You can configure the database
-using the `DATABASE_URL` environment variable. The `sample.env` file assumes
-you can connect to a database at `localhost:5432`. You can update this in your
-`.env` file.
+You will also need a database for the service to store data. Please review the
+[database documentation][database] for more information on how to configure the
+service to connect to your database before proceeding.
 
 With ruby installed and the database configured, install gem dependencies and
 set up the database with the following:
@@ -129,6 +127,7 @@ service.
 [.env]: ./sample.env
 [api]: ./doc/api.md
 [create-key]: ./doc/runbooks/create_auth_key.md
+[database]: ./doc/database.md
 [destination]: ./doc/destinations.md
 [Dockerfile]: ./Dockerfile
 [docker compose]: ./docker-compose.yaml

README.md

@@ -16,7 +16,7 @@ require_relative 'lib/model'
 
 # Connect to the database.
 config = DocumentTransfer::Config::Application.from_environment
-Sequel.connect(config.database_url)
+Sequel.connect(config.database_credentials)
 
 # Load all models.
 DocumentTransfer::Model.load

config.ru

@@ -0,0 +1,90 @@
+# Database
+
+The Document Transfer service is designed to work with a PostgreSQL database.
+
+## Configuration
+
+You can configure the database using the environment variables below. The
+`sample.env` file assumes that you can connect to a database at
+`localhost:5432`. You can update this in your `.env` file.
+
+| Name                | Description                                                                   | Default             | Required |
+|---------------------|-------------------------------------------------------------------------------|---------------------|----------|
+| `BASE_DATABASE`     | The base database to use when the expected database has not yet been created. | `postgres`          | no       |
+| `DATABASE_ADAPTER`  | The adapter to use for the database connection.                               | `postgresql`        | no       |
+| `DATABASE_HOST`     | The host of the database server.                                              | `localhost`         | no       |
+| `DATABASE_NAME`     | The name of the database to connect to.                                       | `document_transfer` | no       |
+| `DATABASE_PASSWORD` | The password for the database user.                                           | `""`                | no       |
+| `DATABASE_PORT`     | The port of the database server.                                              | `5432`              | no       |
+| `DATABASE_USER`     | The user to connect to the database as.                                       | `postgres`          | no       |
+
+Note that while the adapted can be changed, the service is designed to work with
+PostgreSQL, and therefore only bundles the `pg` gem by default in production.
+`sqlite3` is included in test environments for testing purposes, and is not
+supported for production use.
+
+## Creating, updating, and dropping the database
+
+Once you have the database configured, you can use the included [rake] commands
+to manage the database.
+
+To create the database, run the following command. If the database exists, this
+command will still exit 0 without making any changes, making it safe to run
+repeatedly.
+
+```bash
+bundle exec rake db:create
+```
+
+To update the schema by running [migrations]:
+
+```bash
+bundle exec rake db:migrate
+```
+
+You can also run these steps together:
+
+```bash
+bundle exec rake db:setup
+```
+
+You can restore the database to its initial state with the command below. This
+will drop all existing tables and recreate the schema.
+
+```bash
+bundle exec rake db:reset
+````
+
+Finally, you can drop the entire database with:
+
+```bash
+bundle exec rake db:drop
+```
+
+Note that both `db:reset` and `db:drop` will refuse to run in production.
+
+## Schema and migrations
+
+The service uses the [sequel] gem to manage the database schema via
+[migrations][sequel-migrations]. These migrations are stored in the
+[`db/migrate`][migrate] directory and are prefixed with a timestamp to ensure
+they run in proper order and avoid collisions.
+
+Migrations can be run using the following command:
+
+```bash
+bundle exec rake db:migrate
+```
+
+You can migrate to a specific version (up or down) by passing the version
+number:
+
+```bash
+bundle exec rake db:migrate\[202407082156]
+```
+
+[migrate]: ./db/migrate
+[migrations]: #schema-and-migrations
+[rake]: https://ruby.github.io/rake/
+[sequel]: https://sequel.jeremyevans.net/
+[sequel-migrations]: https://sequel.jeremyevans.net/rdoc/files/doc/migration_rdoc.html

doc/database.md

@@ -40,6 +40,10 @@ The following environment variables must be set on the service:
 | filename | The path in the drive to upload the file to. | `string` | `source.filename` | NO       |
 | path     | The path in the drive to upload the file to. | `string` | `""`              | NO       |
 
+If the `filename` parameter is not provided, the service will use the filename
+from the source. If the source URL does not contain a filename, such as when
+using an S3 presigned url, this may result in your transfer failing.
+
 ### Example request
 
 ```json

doc/destinations.md

@@ -4,12 +4,16 @@ services:
     depends_on:
       db:
         condition: service_healthy
+    restart: always
     environment:
       ONEDRIVE_CLIENT_ID: ${ONEDRIVE_CLIENT_ID}
       ONEDRIVE_CLIENT_SECRET: ${ONEDRIVE_CLIENT_SECRET}
       ONEDRIVE_DRIVE_ID: ${ONEDRIVE_DRIVE_ID}
       ONEDRIVE_TENANT_ID: ${ONEDRIVE_TENANT_ID}
-      DATABASE_URL: postgresql://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-postgres}@db/${POSTGRES_DB:-document_transfer}
+      DATABASE_USER: ${DATABASE_USER:-postgres}
+      DATABASE_PASSWORD: ${DATABASE_PASSWORD:-postgres}
+      DATABASE_HOST: db
+      DATABASE_NAME: ${POSTGRES_DB:-document_transfer}
     volumes:
       - .:/opt/app
     ports:

docker-compose.yaml

@@ -7,10 +7,26 @@ module Config
     # Configuration for the document transfer application.
     class Application < Base
       option :base_database, type: String, default: 'postgres'
-      option :database_url, type: String, required: true
+      option :database_adapter, type: String, default: 'postgresql'
+      option :database_host, type: String, default: 'localhost'
+      option :database_name, type: String, default: 'document_transfer'
+      option :database_password, type: String, default: ''
+      option :database_port, type: Integer, default: 5432
+      option :database_user, type: String, default: 'postgres'
       option :environment, type: String, default: 'development',
                            env_variable: 'RACK_ENV'
 
+      def database_credentials
+        {
+          adapter: database_adapter,
+          database: database_name,
+          host: database_host,
+          password: database_password,
+          port: database_port,
+          user: database_user
+        }
+      end
+
       def prod?
         %w[production prod].include?(environment)
       end

lib/config/application.rb

@@ -26,9 +26,10 @@ def respond_to_missing?(name, include_private = false)
       def format_value(option, value)
         return value if value.is_a?(options[option][:type])
 
-        case options[option][:type].name
-        when 'Symbol' then value.to_sym
-        when 'String' then value.to_s
+        case options[option][:type].name.to_sym
+        when :Integer then value.to_i
+        when :Symbol then value.to_sym
+        when :String then value.to_s
         else value
         end
       end

lib/config/dsl.rb

@@ -8,9 +8,9 @@ module Destination
     # Microsoft OneDrive destination.
     class OneDrive < Base
       def transfer(source)
-        result = service.upload(source, path: @config.path, filename: @config.filename)
+        service.upload(source, path: @config.path, filename: @config.filename)
 
-        { path: File.join(@config.path, result.name) }
+        { path: File.join(@config.path, @config.filename) }
       rescue Microsoft::Graph::Error => e
         raise DestinationError, "Failed to upload to OneDrive: #{e.message}"
       end

lib/destination/one_drive.rb

@@ -15,15 +15,14 @@ class Base < Rake::Base
         #
         # @return [String]
         def db_name
-          url = URI.parse(config.database_url)
-          url.path[1..] || ''
+          config.database_name
         end
 
         # Connect to the database for the duration of the provided block.
         #
         # @yieldparam [Sequel::Database]
         def db_connection(&)
-          Sequel.connect(config.database_url, &)
+          Sequel.connect(config.database_credentials, &)
         end
 
         # Connect to the base database for the duration of the provided block.
@@ -32,9 +31,10 @@ def db_connection(&)
         #
         # @yieldparam [Sequel::Database]
         def base_db_connection(&)
-          url = URI.parse(config.database_url)
-          url.path = "/#{config.base_database}"
-          Sequel.connect(url.to_s, &)
+          Sequel.connect(
+            config.database_credentials.merge(database: config.base_database),
+            &
+          )
         end
       end
     end

lib/rake/database/base.rb

@@ -20,7 +20,7 @@ def define(args, &task_block)
             version = task_args.to_a.first if task_args
             db_connection do |db|
               yield(*[self, :pre, task_args].slice(0, task_block.arity)) if task_block
-              Sequel::Migrator.run(db, 'db/migrations', target: version)
+              Sequel::Migrator.run(db, 'db/migrations', target: version&.to_i)
               yield(*[self, :post, task_args].slice(0, task_block.arity)) if task_block
             end
           end

lib/rake/database/migrate.rb

@@ -16,6 +16,8 @@ def initialize(name = :reset, *args, &)
         def define(args, &task_block)
           desc 'Reset the database'
           task(name, *args) do |_, _task_args|
+            raise EnvironmentError, 'Cannot drop the production database' if config.prod?
+
             ::Rake::Task['db:drop'].invoke(task_block)
             ::Rake::Task['db:setup'].invoke(task_block)
           end

lib/rake/database/reset.rb

@@ -2,7 +2,11 @@ export COVERAGE=1
 export RACK_ENV=development
 
 # Update to match your local database configuration, if necessary.
-export DATABASE_URL="postgresql://localhost:5432/document_transfer"
+export DATABASE_USER=""
+export DATABASE_PASSWORD=""
+export DATABASE_HOST="localhost"
+export DATABASE_PORT="5432"
+export DATABASE_NAME="document_transfer"
 
 # Uncomment and set to the name of your AWS profile to use the aws cli.
 #export AWS_PROFILE=

sample.env

@@ -19,10 +19,11 @@
 end
 
 # Connect to a test database and run migrations.
-ENV['DATABASE_URL'] = 'sqlite://memory'
+ENV['DATABASE_ADAPTER'] = 'sqlite'
+ENV['DATABASE_NAME'] = ':memory:'
 ENV['BASE_DATABASE'] = ''
 Sequel.extension :migration
-db = Sequel.connect(ENV.fetch('DATABASE_URL'))
+db = Sequel.connect('sqlite::memory:')
 Sequel::Migrator.run(db, 'db')
 
 # We need to build a Rack app for testing. This ensures that we're including the