Gatherer

Docker Image

This is a Docker container that uses the cisagov fork of 18F's domain-scan to gather domains as a precursor to scanning by pshtt, trustymail, and sslyze.

This Docker container is intended to be run via orchestrator.

N.B.: The secrets in the src/secrets directory are only used when testing via the compose.yml composition. Normally this Docker container is run via the Docker composition in cisagov/orchestrator, which expects the secrets in a different location.

Running

Running with Docker

To run the cisagov/gatherer image via Docker:

docker run cisagov/gatherer:1.5.18

Running with Docker Compose

1. Create a compose.yml file similar to the one below to use Docker Compose.

   ---
   name: gatherer
   
   services:
     gatherer:
       image: cisagov/gatherer:1.5.18
       volumes:
         - source: <your_output_dir>
           target: /home/cisa/shared
           type: bind

2. Start the container and detach:

   docker compose up --detach

Using secrets with your container

This container also supports passing sensitive values via Docker secrets. Passing sensitive values like your credentials can be more secure using secrets than using environment variables. See the secrets section below for a table of all supported secret files.

1. To use secrets, create a database_creds.yml file in this format:

   ---
   version: '1'
   
   database:
     name: cyhy
     uri: mongodb://readonly:[email protected]:27017/cyhy
   

2. Then add the secret to your compose.yml file:

   ---
   name: gatherer
   
   secrets:
     database_creds:
       file: database_creds.yml
   
   services:
     gatherer:
       image: cisagov/gatherer:1.5.18
       secrets:
         - source: database_creds
           target: database_creds.yml
       volumes:
         - source: <your_output_dir>
           target: /home/cisa/shared
           type: bind

Updating your container

Docker Compose

1. Pull the new image from Docker Hub:

   docker compose pull

2. Recreate the running container by following the previous instructions:

   docker compose up --detach

Docker

1. Stop the running container:

   docker stop <container_id>

2. Pull the new image:

   docker pull cisagov/gatherer:1.5.18

3. Recreate and run the container by following the previous instructions.

Updating Python dependencies

This image uses Pipenv to manage Python dependencies using a Pipfile. Both updating dependencies and changing the Pipenv configuration in src/Pipfile will result in a modified src/Pipfile.lock file that should be committed to the repository.

Warning

The src/Pipfile.lock as generated will fail pre-commit checks due to JSON formatting.

Updating dependencies

If you want to update existing dependencies you would run the following command in the src/ subdirectory:

pipenv lock

Modifying dependencies

If you want to add or remove dependencies you would update the src/Pipfile file and then update dependencies as you would above.

Note

You should only specify packages that are direct requirements of your Docker configuration. Allow Pipenv to manage the dependencies of the specified packages.

Image tags

The images of this container are tagged with semantic versions of the underlying gatherer project that they containerize. It is recommended that most users use a version tag (e.g. :1.5.18).

Image:tag	Description
cisagov/gatherer:1.5.18	An exact release version.
cisagov/gatherer:1.5	The most recent release matching the major and minor version numbers.
cisagov/gatherer:1	The most recent release matching the major version number.
cisagov/gatherer:edge	The most recent image built from a merge into the develop branch of this repository.
cisagov/gatherer:nightly	A nightly build of the develop branch of this repository.
cisagov/gatherer:latest	The most recent release image pushed to a container registry. Pulling an image using the :latest tag should be avoided.

See the tags tab on Docker Hub for a list of all the supported tags.

Volumes

Mount point	Purpose
/home/cisa/shared	Output

Ports

There are no ports exposed by this container.

Environment variables

Required

There are no required environment variables.

Optional

There are no optional environment variables.

Secrets

Filename	Purpose
database_creds.yml	Cyber Hygiene database credentials in this format

Building from source

Build the image locally using this git repository as the build context:

docker build \
  --tag cisagov/gatherer:1.5.18 \
  https://github.com/cisagov/gatherer.git#develop

Cross-platform builds

To create images that are compatible with other platforms, you can use the buildx feature of Docker:

1. Copy the project to your machine using the Code button above or the command line:

   git clone https://github.com/cisagov/gatherer.git
   cd gatherer

2. Create the Dockerfile-x file with buildx platform support:

   ./buildx-dockerfile.sh

3. Build the image using buildx:

   docker buildx build \
     --file Dockerfile-x \
     --platform linux/amd64 \
     --output type=docker \
     --tag cisagov/gatherer:1.5.18 .

Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

License

This project is in the worldwide public domain.

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.