Merge pull request #39 from bird-house/autodeploy-the-autodeploy-phase-2

Autodeploy the autodeploy phase 2: everything operational but a few compatibility issues remain

Part of #27

Activating the `./components/scheduler` will do everything.  All configurations are centralized in the `env.local` file.

One missing feature is piece-wise choice of platform or notebook autodeploy only, like with the old manual `install-*` stcripts under https://github.com/bird-house/birdhouse-deploy/tree/master/birdhouse/deployment.  Right now it's all or nothing.  I can work on this if you guys think it's needed.

Remaining compatibility issues with Medus (Vagrant box works fine):

* Notebook autodeploy do not work. It looks like using the `bash` docker image, I am unable to wget any httpS address.  This same `docker run` command works fine on my Vagrant box as well.  So there's something on Medus.

```
$ docker run --rm --name debug_wget_httpS -u root bash bash -c "wget https://google.com -O -"
Connecting to google.com (172.217.13.206:443)
wget: error getting response: Connection reset by peer
```

* All the containers are being recreated when `./pavics-compose.sh` runs inside the container (first migration to the new autodeploy mechanism).  To investigate but I suspect this might be due to older version of `docker` and `docker-compose` on Medus.

* This one looks like due to older kernel on Medus:
```
sysctl: error: 'net.ipv4.tcp_tw_reuse' is an unknown key
sh: 0: unknown operand
```

* All the files updated by `git pull` are now owned by `root` (the user inside the container).  I'll have to undo this ownership change, somehow.  This one is super weird, I should have got it on my Vagrant box.  Probably Vagrant did some magic to always ensure files under `/vagrant` is always owned by the user even if changed by user `root`.

* Documentation: update README and list relevant configuration variables in `env.local` for this new `./component/scheduler`.


Migrating to this new mechanism requires manual deletion of all the artifacts created by the old install scripts: `sudo rm /etc/cron.d/PAVICS-deploy /etc/cron.hourly/PAVICS-deploy-notebooks /etc/logrotate.d/PAVICS-deploy /usr/local/sbin/triggerdeploy.sh`.  Both can not co-exist at the same time.

Maximum backward-compatibility has been kept with the old existing install scripts style:
* Still log to the same existing log files under `/var/log/PAVICS`.
* Old single ssh deploy key is still compatible, but the new mechanism allows for different ssh deploy keys for each extra repos (again, public repos should use https clone path to avoid dealing with ssh deploy keys in the first place)
* Old install scripts are kept

Features missing in old existing install scripts or how this improves on the old install scripts:
* Autodeploy of the autodeploy itself !  This is the biggest win.  Previously, if `triggerdeploy.sh` or `PAVICS-deploy-notebooks` script changes, they have to be deployed manually.  It's very annoying.  Now they are volume-mount in so are fresh on each run.
* `env.local` now drive absolutely everything, source control that file and we've got a true DevOPS pipeline.
* Configurable platform and notebook autodeploy frequency.  Previously, this means manually editing the generated cron file, less ideal.
* Do not need any support on the local host other than `docker` and `docker-compose`.  cron/logrotate/git/ssh versions are all locked-down in the docker images used by the autodeploy.  Recall previously we had to deal with git version too old on some hosts.
* Each cron job run in its own docker image meaning the runtime environment is traceable and reproducible.
* The newly introduced scheduler component is made extensible so other jobs can added into it as well (ex: backup), via `env.local`, which should source control, meaning all surrounding maintenance related tasks can also be traceable and reproducible.

This is a rather large PR.  For a less technical overview, start with the diff of README.md, env.local.example, common.env.  If a change looks funny to you, read the commit description that introduce that change, the reasoning should be there.

Author: tlvu

Vagrantfile

@@ -40,7 +40,6 @@ Vagrant.configure("2") do |config|
 
   if settings.has_key?('ssh_deploy_key')
       config.vm.provision :file, source: settings['ssh_deploy_key'], destination: ".ssh/id_rsa_git_ssh_read_only"
-      config.vm.provision :shell, path: "birdhouse/deployment/install-automated-deployment.sh", args: ["/vagrant", "vagrant", "5-mins"]
   end
 
   if settings.has_key?('datasets_dirs')

birdhouse/common.env

@@ -19,3 +19,16 @@ export JUPYTER_LOGIN_BANNER_BOTTOM_SECTION=""
 
 # Folder inside "proxy" container to drop extra monitoring config
 export CANARIE_MONITORING_EXTRA_CONF_DIR="/conf.d"
+
+# Folder containing ssh deploy keys for all extra git repos
+#
+# Note when overriding this variable in env.local, do not use HOME environment
+# var, use its fully resolved value.  This default value is suitable only for
+# backward-compatibility when autodeploy do not run in its own container.
+export AUTODEPLOY_DEPLOY_KEY_ROOT_DIR="$HOME/.ssh"
+
+# Daily at 5:07 AM
+export AUTODEPLOY_PLATFORM_FREQUENCY="7 5 * * *"
+
+# Hourly
+export AUTODEPLOY_NOTEBOOK_FREQUENCY="@hourly"

birdhouse/components/scheduler/.gitignore

@@ -0,0 +1 @@
+config.yml

birdhouse/components/scheduler/config.json

@@ -0,0 +1,46 @@
+---
+- name: logrotate
+  comment: Rotate log files under /var/log/PAVICS
+  schedule: '@daily'
+  command: bash -c 'cp /etc/logrotate.conf.orig /etc/logrotate.conf && chown root:root /etc/logrotate.conf && chmod 644 /etc/logrotate.conf && /usr/sbin/logrotate -v /etc/logrotate.conf'
+  dockerargs: >-
+    --rm --name logrotate
+    --volume /var/log/PAVICS:/var/log/PAVICS:rw
+    --volume /data/logrotate:/var/lib:rw
+    --volume ${COMPOSE_DIR}/deployment/PAVICS-deploy.logrotate:/etc/logrotate.conf.orig:ro
+  image: 'stakater/logrotate:3.13.0'
+
+- name: notebookdeploy
+  comment: Auto-deploy tutorial notebooks
+  schedule: '${AUTODEPLOY_NOTEBOOK_FREQUENCY}'
+  command: '${COMPOSE_DIR}/deployment/trigger-deploy-notebook'
+  dockerargs: >-
+    --rm --name notebookdeploy
+    --volume /var/run/docker.sock:/var/run/docker.sock:ro
+    --volume /var/log/PAVICS:/var/log/PAVICS:rw
+    --volume ${COMPOSE_DIR}:${COMPOSE_DIR}:ro
+    --volume ${JUPYTERHUB_USER_DATA_DIR}:${JUPYTERHUB_USER_DATA_DIR}:rw
+    --volume /tmp/notebookdeploy:/tmp/notebookdeploy:rw
+    --env COMPOSE_DIR=${COMPOSE_DIR}
+    --env TMP_BASE_DIR=/tmp/notebookdeploy
+    --env JUPYTERHUB_USER_DATA_DIR=${JUPYTERHUB_USER_DATA_DIR}
+  image: 'docker:19.03.6-git'
+
+- name: autodeploy
+  comment: Auto-deploy entire PAVICS platform
+  schedule: '${AUTODEPLOY_PLATFORM_FREQUENCY}'
+  command: '${COMPOSE_DIR}/deployment/triggerdeploy.sh ${COMPOSE_DIR}'
+  dockerargs: >-
+    --rm --name autodeploy${AUTODEPLOY_EXTRA_REPOS_AS_DOCKER_VOLUMES}
+    --volume /var/run/docker.sock:/var/run/docker.sock:ro
+    --volume /var/log/PAVICS:/var/log/PAVICS:rw
+    --volume ${COMPOSE_DIR}/..:${COMPOSE_DIR}/..:rw
+    --volume ${AUTODEPLOY_DEPLOY_KEY_ROOT_DIR}:${AUTODEPLOY_DEPLOY_KEY_ROOT_DIR}:ro
+    --volume ${JUPYTERHUB_USER_DATA_DIR}:${JUPYTERHUB_USER_DATA_DIR}:rw
+    --env COMPOSE_DIR=${COMPOSE_DIR}
+    --env AUTODEPLOY_DEPLOY_KEY_ROOT_DIR=${AUTODEPLOY_DEPLOY_KEY_ROOT_DIR}
+    --env JUPYTERHUB_USER_DATA_DIR=${JUPYTERHUB_USER_DATA_DIR}
+    --env AUTODEPLOY_SILENT=true
+  image: 'pavics/docker-compose-git:docker-18.09.7-compose-1.25.1'
+
+${AUTODEPLOY_EXTRA_SCHEDULER_JOBS}

birdhouse/components/scheduler/config.yml.template

@@ -2,14 +2,15 @@ version: '2.1'
 
 services:
   scheduler:
-    image: willfarrell/crontab:0.5.0
+    image: pavics/crontab:0.6-rc2
     container_name: scheduler
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock:ro
-      - ./components/scheduler/config.json:/opt/crontab/config.json:ro
+      - ./components/scheduler/config.yml:/opt/crontab/config.yml:ro
     environment:
       COMPOSE_DIR: ${PWD}
       JUPYTERHUB_USER_DATA_DIR: ${JUPYTERHUB_USER_DATA_DIR}
+      AUTODEPLOY_DEPLOY_KEY_ROOT_DIR: ${AUTODEPLOY_DEPLOY_KEY_ROOT_DIR}
     restart: always
 
 # vi: tabstop=8 expandtab shiftwidth=2 softtabstop=2

birdhouse/components/scheduler/docker-compose-extra.yml

@@ -7,4 +7,5 @@
     minsize 500k
     copytruncate
     compress
+    dateext
 }

birdhouse/deployment/PAVICS-deploy.logrotate

@@ -1,22 +1,24 @@
 #!/bin/sh
 # Script to automate local deployment process.
 #
+# Log to "/var/log/PAVICS/autodeploy.log" if AUTODEPLOY_SILENT is not empty.
+#
 # Still have to ssh to target machine but at least this single script
 # takes care of all the common steps for a standard deployment (see corner
 # cases not handled below).
 #
-# One time Setup for a *private* PAVICS repository:
+# One time Setup for each *private* repository (not applicable for *public* repos):
 #
 # * create a ssh deploy key (do not re-use your personal key)
-#   `ssh-keygen -b 4096 -t rsa -f ~/.ssh/id_rsa_git_ssh_read_only` (no passphrase)
+#   `ssh-keygen -b 4096 -t rsa -f ~/.ssh/{repo-name}_deploy_key` (no passphrase)
 #
-# * add the public key ~/.ssh/id_rsa_git_ssh_read_only.pub to the repo as read-only deploy key
+# * add the public key ~/.ssh/{repo-name}_deploy_key.pub to the repo as read-only deploy key
 #   see https://developer.github.com/v3/guides/managing-deploy-keys/#deploy-keys
 #
 # * if on host with git version less than 2.3 (no support for GIT_SSH_COMMAND),
 #   have to configure your ~/.ssh/config  with the following:
 #     Host github.com
-#     IdentityFile ~/.ssh/id_rsa_git_ssh_read_only
+#     IdentityFile ~/.ssh/{repo-name}_deploy_key
 #     UserKnownHostsFile /dev/null
 #     StrictHostKeyChecking no
 #
@@ -34,7 +36,8 @@
 #   Just run this script pointing to the checkout repo and optionally the env.local file.
 #
 #   It will find the rest that it needs from the checkout repo.  Only external
-#   depency is ~/.ssh/id_rsa_git_ssh_read_only.
+#   dependencies are the multiple ~/.ssh/{repo-name}_deploy_key (one deploy key
+#   for each repo).
 #
 # Corner cases not handled:
 #
@@ -50,12 +53,10 @@
 #   are re-read.  docker-compose is not aware of any changes outside of the
 #   docker-compose.yml file.
 
-
-GREEN=$(tput setaf 2)
-YELLOW=$(tput setaf 3)
-RED=$(tput setaf 1)
-NORMAL=$(tput sgr0)
-
+if [ ! -z "$AUTODEPLOY_SILENT" ]; then
+    LOG_FILE="/var/log/PAVICS/autodeploy.log"
+    exec >>$LOG_FILE 2>&1
+fi
 
 usage() {
     echo "USAGE: $0 <path to folder with docker-compose.yml file> [path to env.local]"
@@ -65,7 +66,7 @@ COMPOSE_DIR="$1"
 ENV_LOCAL_FILE="$2"
 
 if [ -z "$COMPOSE_DIR" ]; then
-    echo "${RED}ERROR:${NORMAL} please provide path to PAVICS docker-compose dir." 1>&2
+    echo "ERROR: please provide path to PAVICS docker-compose dir." 1>&2
     usage
     exit 2
 else
@@ -83,24 +84,17 @@ REPO_ROOT="`realpath "$COMPOSE_DIR/.."`"
 
 if [ ! -f "$COMPOSE_DIR/docker-compose.yml" -o \
      ! -f "$COMPOSE_DIR/pavics-compose.sh" ]; then
-    echo "${RED}ERROR:${NORMAL} missing docker-compose.yml or pavics-compose.sh file in '$COMPOSE_DIR'" 1>&2
+    echo "ERROR: missing docker-compose.yml or pavics-compose.sh file in '$COMPOSE_DIR'" 1>&2
     exit 2
 fi
 
 if [ ! -f "$ENV_LOCAL_FILE" ]; then
-    echo "${RED}ERROR:${NORMAL} env.local '$ENV_LOCAL_FILE' not found, please instantiate from '$COMPOSE_DIR/env.local.example'" 1>&2
-    exit 2
-fi
-
-SSH_DEPLOY_KEY="$HOME/.ssh/id_rsa_git_ssh_read_only"
-
-if [ ! -f "$SSH_DEPLOY_KEY" ]; then
-    echo "${RED}ERROR:${NORMAL} '$SSH_DEPLOY_KEY' not found, please create it and set it up for this repo" 1>&2
+    echo "ERROR: env.local '$ENV_LOCAL_FILE' not found, please instantiate from '$COMPOSE_DIR/env.local.example'" 1>&2
     exit 2
 fi
 
 if [ -f "$COMPOSE_DIR/docker-compose.override.yml" ]; then
-    echo "${YELLOW}WARNING:${NORMAL} docker-compose.override.yml found, should use EXTRA_CONF_DIRS in env.local instead"
+    echo "WARNING: docker-compose.override.yml found, should use EXTRA_CONF_DIRS in env.local instead"
 fi
 
 START_TIME="`date -Isecond`"
@@ -117,11 +111,11 @@ for adir in $COMPOSE_DIR $AUTODEPLOY_EXTRA_REPOS; do
 
         # fail fast if unclean checkout
         if [ ! -z "`git status -u --porcelain`" ]; then
-            echo "${RED}ERROR:${NORMAL} unclean repo '$adir'" 1>&2
+            echo "ERROR: unclean repo '$adir'" 1>&2
             exit 1
         fi
     else
-        echo "${YELLOW}WARNING:${NORMAL} extra repo '$adir' do not exist"
+        echo "WARNING: extra repo '$adir' do not exist"
     fi
 done
 
@@ -137,19 +131,31 @@ set -x
 # stop all to force reload any changed config that are volume-mount into the containers
 ./pavics-compose.sh stop
 
-# override git ssh command because this repo is private and need proper credentials
-#
-# https://git-scm.com/docs/git-config#Documentation/git-config.txt-sshvariant
-export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=$SSH_DEPLOY_KEY"
-
 for adir in $COMPOSE_DIR $AUTODEPLOY_EXTRA_REPOS; do
     if [ -d "$adir" ]; then
         cd $adir
 
+        EXTRA_REPO="`git rev-parse --show-toplevel`"
+        DEPLOY_KEY="$AUTODEPLOY_DEPLOY_KEY_ROOT_DIR/`basename "$EXTRA_REPO"`_deploy_key"
+        DEFAULT_DEPLOY_KEY="$AUTODEPLOY_DEPLOY_KEY_ROOT_DIR/id_rsa_git_ssh_read_only"
+        if [ ! -e "$DEPLOY_KEY" -a -e "$DEFAULT_DEPLOY_KEY" ]; then
+            DEPLOY_KEY="$DEFAULT_DEPLOY_KEY"
+        fi
+
+        export GIT_SSH_COMMAND=""  # git ver 2.3+
+        if [ -e "$DEPLOY_KEY" ]; then
+            # override git ssh command for private repos only
+            #
+            # https://git-scm.com/docs/git-config#Documentation/git-config.txt-sshvariant
+            export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=$DEPLOY_KEY"
+        else
+            unset GIT_SSH_COMMAND
+        fi
+
         # pull the current branch, so this deploy script supports any branches, not just master
         git pull
     else
-        echo "${YELLOW}WARNING:${NORMAL} extra repo '$adir' do not exist"
+        echo "WARNING: extra repo '$adir' do not exist"
     fi
 done
 

birdhouse/deployment/deploy.sh

@@ -10,6 +10,10 @@
 #
 #   Follow same instructions in deploy.sh.
 
+if [ ! -z "$AUTODEPLOY_SILENT" ]; then
+    LOG_FILE="/var/log/PAVICS/autodeploy.log"
+    exec >>$LOG_FILE 2>&1
+fi
 
 usage() {
     echo "USAGE: $0 <path to folder with docker-compose.yml file> [path to env.local]"
@@ -47,7 +51,23 @@ fi
 
 should_trigger() {
     EXTRA_REPO="`git rev-parse --show-toplevel`"
-    echo "triggerdeploy: checking repo '$EXTRA_REPO'"
+
+    DEPLOY_KEY="$AUTODEPLOY_DEPLOY_KEY_ROOT_DIR/`basename "$EXTRA_REPO"`_deploy_key"
+    DEFAULT_DEPLOY_KEY="$AUTODEPLOY_DEPLOY_KEY_ROOT_DIR/id_rsa_git_ssh_read_only"
+    if [ ! -e "$DEPLOY_KEY" -a -e "$DEFAULT_DEPLOY_KEY" ]; then
+        DEPLOY_KEY="$DEFAULT_DEPLOY_KEY"
+    fi
+    DEPLOY_KEY_DISPLAY=""
+
+    export GIT_SSH_COMMAND=""  # git ver 2.3+
+    if [ -e "$DEPLOY_KEY" ]; then
+        DEPLOY_KEY_DISPLAY=" deploy_key='$DEPLOY_KEY'"
+        export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=$DEPLOY_KEY"
+    else
+        unset GIT_SSH_COMMAND
+    fi
+
+    echo "triggerdeploy: checking repo '$EXTRA_REPO'$DEPLOY_KEY_DISPLAY"
 
     # fetch remote branches, not affecting current checkout
     git fetch --prune --all
@@ -90,8 +110,6 @@ should_trigger() {
 }
 
 
-SSH_DEPLOY_KEY="$HOME/.ssh/id_rsa_git_ssh_read_only"
-
 START_TIME="`date -Isecond`"
 echo "==========
 triggerdeploy START_TIME=$START_TIME"
@@ -101,8 +119,6 @@ triggerdeploy START_TIME=$START_TIME"
 
 set -x
 
-export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=$SSH_DEPLOY_KEY"
-
 SHOULD_TRIGGER=""
 for adir in $COMPOSE_DIR $AUTODEPLOY_EXTRA_REPOS; do
     if [ -d "$adir" ]; then

birdhouse/deployment/triggerdeploy.sh

@@ -1,6 +1,9 @@
 #############################################################################
 # Mandatory vars (will be enforced by pavics-compose.sh)
 # Can add new vars but do not remove, else automated deployment will break
+#
+# Do NOT use environment variables in here since when pavics-compose.sh runs
+# inside a container, the environment vars do not have the same value.
 #############################################################################
 
 export SSL_CERTIFICATE="/path/to/ssl/cert.pem"  # path to the nginx ssl certificate, path and key bundle
@@ -72,6 +75,53 @@ export POSTGRES_MAGPIE_PASSWORD=postgres-qwerty
 #export AUTODEPLOY_EXTRA_REPOS="/path/to/dir1 /path/to/dir2 /path/to/dir3"
 #export AUTODEPLOY_EXTRA_REPOS="/path/to/private-config-containing-env.local"
 
+# For each git repo in AUTODEPLOY_EXTRA_REPOS that use ssh to clone/fetch
+# instead of https, provide its corresponding ssh deploy key in this dir.
+#
+# See instructions in deployment/deploy.sh or
+# https://developer.github.com/v3/guides/managing-deploy-keys/#deploy-keys for
+# how to create deploy key for your git repos.
+#
+# The autodeploy mechanism runs inside its own container so environment
+# variables are not the same inside and outside the container.  Do not use
+# any environment vars, use their fully resolved values.
+#
+# Format of keys inside the dir: {repo-name-1}_deploy_key,
+#   {repo-name-2}_deploy_key, ...
+#
+# If '{repo-name}_deploy_key' file is not found, default to
+# 'id_rsa_git_ssh_read_only' so if multiple private repos share the same ssh
+# deploy key, you can just name that shared key id_rsa_git_ssh_read_only and
+# create {repo-name}_deploy_key only for repo specific key.
+#
+# Example of keys inside the dir: dir1_deploy_key, dir2_deploy_key,
+#   private-config-containing-env.local_deploy_key,
+#   id_rsa_git_ssh_read_only
+#
+#export AUTODEPLOY_DEPLOY_KEY_ROOT_DIR="/path/to/ssh-deploy-keys-for-all-repos"
+
+# Frequency to trigger the various autodeploy tasks.
+# See common.env for default.
+#
+# For all possible syntax, see implementation at
+# https://github.com/Ouranosinc/docker-crontab/blob/3ac8cfa363b3f2ffdd0ead6089d355ff84521dc9/docker-entrypoint#L137-L184
+#
+# Ex:
+# - daily at 5:07 AM: "7 5 * * *"
+# - daily at midnight: "0 0 * * *" or "@daily"/"@midnight"
+# - hourly: ""0 * * * *" or "@hourly"
+# - every 2 hours: "*/120 * * * *" or "@every 2h"
+# - every 5 minutes: "*/5 * * * *" or "@every 5m"
+#
+#export AUTODEPLOY_PLATFORM_FREQUENCY="@every 5m"
+#export AUTODEPLOY_NOTEBOOK_FREQUENCY="@every 5m"
+
+# Add more jobs to ./components/scheduler/config.yml
+#
+# Potential usages: other deployment, backup jobs on the same machine
+#
+#export AUTODEPLOY_EXTRA_SCHEDULER_JOBS=""
+
 # Public (on the internet) fully qualified domain name of this Pavics
 # installation.  This is optional so default to the same internal PAVICS_FQDN if
 # not set.