Building and deploying images to production

This document covers how to build and deploy Cyber Hygiene (CyHy) images to production.

Step 1: Rebuild image(s)

Prepare

If this is a clean install:

• Clone cyhy-amis

git switch develop
git pull

• Set up pyenv virtual env:

pyenv virtualenv <Python version> <environment name>
pyenv local <environment name>

• Install dependencies:
  • ansible, boto3
    • Should convert to a requirements.txt - Nick will make a PR
  • TODO: switch to pip install -r requirements.txt

pip install --upgrade pip
pip install ansible boto3

• brew install aws-cli, packer, and terraform
  • Should be installed via brew to make the process easier
• Credentials and SSM
  • AWS_PROFILE will need to be set up
  • TODO: Find and document usage of our script that maintains aws profiles

Terraform prep

TODO: Add content

Packer prep

• Check which AWS_PROFILE is set so it can connect and build on AWS
• Verify we have packer installed
• Verify we have ansible-galaxy installed
• Install ansible requirements for packer
  • ansible-galaxy install --force -r packer/ansible/requirements.yml
• Verify we're in the correct role in the AWS_PROFILE - check terminal output for cyhy-production
• Build via packer from the json files - there are 7 images that we use and run the command for each:
  • e.g. packer build packer/bastion.json
  • Won't always need to rebuild all of them - sometimes they'll have been rebuilt in the interim to push an update
  • TODO: set up a shell script to build all images instead of having to manually run individually - some concerns will need to be addressed re: volume of logs if one of the builds fails
  • TODO: Documented in the repo README.md - crib notes!
• Verify image gets deployed and is copied across regions via EC2 web interface in AWS

Step 2: Deploying Image(s)

• Use AWS CLI to fetch latest TF vars
  • ./scripts/fetch_production_tfvars.sh
• Configure the dynamic modules based on environment
  • terraform/configure.py sets up the dynamic modules
• terraform init -upgrade=true in base terraform dir
• terraform workspace list
  • Should be on prod-a - the * indicates active workspace
• terraform workspace select prod-a
  • terraform workspace show to verify you're on prod-a
• Run ./configure.py and verify it sets up the workspace
• terraform init -upgrade=true in base terraform dir
• To deploy bastion, run the helper script ./deploy_new_bastion_ami.sh
• Second round of ansible applies roles from ansible/roles
• NOTE: Redeploying bastion will bork the CyHy team's access, so warn them via chat with a when and estimated duration.
  • Q: About how long does it take to re-deploy?
  • Per-image or overall estimate?
• Deploy the new AMI
  • ./deploy_new_bastion_ami.sh us-east-1 prod-a

TODO: Is this complete?