Orchestrate infra

.gitignore

@@ -1,18 +1,30 @@
+venv
+.idea
 *.pyc
-*~
-__pycache__
+*.*~
+vagrant-fabric-lxc
+django_project/media/images
+visual_changelog.db
+.vagrant
+
+node_modules
+id_dsa.pub
+id_rsa.pub
+docs/build
+fabgis_resources
+private.py
 .coverage
 sample_config.yaml
 fabfile.py
 .sass-cache
 npm-debug.log
-.idea
 django_blog_it
-node_modules/
 django_dev.log
 logs/
 celerybeat-schedule
 celerybeat.pid
+docs/build
+
 .env
 resume/
 docs/build

CHANGELOG

@@ -1,2 +1,2 @@
-0.1
-basic site functionality.
+1.0
+Stable functionality.

README-dev.md

@@ -0,0 +1,121 @@
+# Developer Documentation
+
+**Note:** This documentation is intentionally generic so that it can
+be copy-pasted between projects - do not put project specific details here.
+
+## Application architecture under docker
+
+The following diagram provides and overview of the core architecture
+components (Database, uwsgi server, web server):
+
+![dockerdjangoarchitecture - new page 1](https://cloud.githubusercontent.com/assets/178003/5024388/750b85c8-6b12-11e4-97b0-c73b2d07e539.png)
+
+
+The blue box is there to provide a means to develop on the same environment
+as you deploy and would not be relevant for server side deployments.
+Everything is managed using docker containers, with pycharm
+making ssh connections into the developer container and using the
+python interpreter found therein. Newer versions of PyCharm have 'native' docker support
+but after testing we opted to continue to use the ssh method described here. You may want to
+test when new versions of PyCharm come out in case they have improved the direct docker support.
+
+**Note:** You don't need to use this architecture, you can deploy as a standard
+django app using virtualenv and locally installed postgis/mysql, nginx etc.
+
+## Setup pycharm to work with a remove docker development environment
+
+### Build and run your dev docker image
+
+This image extends the production one, adding ssh to it. You must
+have built the production one first by following the "Quick Installation Guide"!
+When it's done, you can continue with this command:
+
+Linux and MacOS:
+
+```
+make build-devweb
+make devweb
+```
+
+Windows:
+
+```
+make-devbuild.bat
+make-devweb.bat
+```
+
+### Create a remote interpreter in pycharm
+
+Open the project in pycharm then do:
+
+* File -> Settings
+* Project Interpreter
+* Click on the gear icon next to project interpreter
+* Add remote...
+
+Now use these credentials:
+
+* SSH Credentials (tick)
+* Host: localhost
+* Port: (use the ssh port specified in the docker-compose.yml file)
+* User name: root
+* Auth type: password (and tick 'save password')
+* Password: docker
+* Python interpreter path: ``/usr/local/bin/python``
+
+When prompted about host authenticity, click Yes.
+After the python interpreter is set, set its path mapping:
+
+  * **Local path:** `<path to your git repo>/django_project`
+  * **Remote path:** `/home/web/django_project`
+
+After that you should see something like this:
+   `<Project root>/django_project→/home/web/django_project`
+
+In settings, django support:
+
+* tick to enable django support.
+* Set django project root to the path on your host that holds django code e.g.
+  ``<path to code base>/django_project``
+* Set the settings option to your setting profile e.g.
+  ``jobsp.settings_server``. If you need to use custom settings, copy
+  this file e.g. ``jobsp.settings_local.py`` and place your modifications in your
+  personalised copy. The first import of your custom file should import from
+  jobsp.settings_server.
+
+* manage script (leave default)
+
+
+### Create the django run configuration
+
+* Run -> Edit configurations
+* Click the `+` icon in the top left corner
+* Choose ``Django server`` from the popup list
+
+Now set these options:
+
+* **Name:** Django Server
+* **Host:** 0.0.0.0
+* **Port:** (use the http port specified in the docker-compose.yml file, e.g. 8080)`*` **Run browser** If checked, it will open the url after you click run. You should be able to access the running projecta on 0.0.0.0:61202 (the port that mapped to 8080)
+* **Additional options:** ``--settings=jobsp.settings_server``
+* **Run browser:** Optionally set this to your IP address (MacOS/Linux) or your specific IP address (Windows) followed by the port forward address for port 8080 specified in your ``docker-compose.yml`` file. For example: ``http://0.0.0.0:65202``.
+* **Environment vars:** Leave as default unless you need to add something to the env
+* **Python interpreter:** Ensure it is set you your remote interpreter (should be
+  set to that by default)
+* **Interpreter options:** Leave blank
+* **Path mappings:** Here you need to indicate path equivalency between your host
+  filesystem and the filesystem in the remote (docker) host. Click the ellipsis
+  and add a run that points to your git checkout on your local host and the
+  /home/web directory in the docker host. e.g.
+  * **Local path:** <path to your git repo>/django_project
+  * **Remote path:** /home/web/django_project
+* click OK to save your run configuration
+
+Now you can run the server using the green triangle next to the Django server
+label in the run configurations pull down. Debug will also work and you will be
+able to step through views etc as you work.
+
+
+## Running Tests
+
+sonlinux write stuff here....

README-docker.md

@@ -0,0 +1,123 @@
+# Managing your docker deployed site
+
+**Note:** This documentation is intentionally generic so that it can
+be copy-pasted between projects - do not put project specific details here.
+
+This document explains how to do various sysadmin related tasks when your
+site has been deployed under docker. Three deployment modes are supported:
+
+* **production**: no debug etc is enabled, has its own discrete database. Configure
+  your production environment in jobsp.settings_server - this
+  DJANGO_SETTINGS_MODULE is used when running in production mode.
+* **development**: Configure your development environment in jobsp.settings_server -
+  this DJANGO_SETTINGS_MODULE is used when running in production mode. Please see
+  README-dev.md for more information on setting up a developer environment.
+
+**Note:** We really recommend that you use docker 1.8 or greatr so that you
+  can take advantage of the exec command as well as other newer features.
+
+## Build your docker images and run them
+
+### Production
+
+You can simply run the provided Makefile commands and it will build and deploy the docker
+images for you in **production mode**.
+
+```
+cd deployment
+# make commands coming soon ...
+make build
+make permissions
+make web
+# Wait a few seconds for the DB to start before to do the next command
+make migrate
+make collectstatic
+```
+
+#### Using make
+
+Using the make commands is probably simpler - the following make commands are
+provided for production (you can obtain this list by typing ``make help`. All commands
+should be run from in the ``deployment`` directory.
+
+
+* **build** - builds all required containers.
+* **build-devweb** - build the development container. See [development notes](README-dev.md).
+* **collectstatic** - run the django collectstatic command.
+* **create-machine** .
+* **db** - build and run the db container.
+* **dbbackup** - make a snapshot of the database, saving it to deployments/backups/YYYY/MM/project-DDMMYYYY.dmp. It also creates a symlink to backups/latest.dmp for the latest backup.
+* **dbbash** - open a bash shell inside the database container.
+* **dblogs** - view the database logs.
+* **dbrestore** - restore deployment/backups/latest.dmp over the active database. Will delete any existing data in your database and replace with the restore, so **use with caution**.
+* **dbschema** - dump the current db schema (without data) to stdio. Useful if you want to compare changes between instances.
+* **dbshell** - get a psql prompt into the db container.
+* **dbsnapshot** - as above but makes the backup as deployment/snapshot.smp - replacing any pre-existing snapshot.
+* **dbsync** - use this from a development or offsite machine. It will rsync all database backups from deployment/backups to your offsite machine.
+* **default** .
+* **deploy** .
+* **devweb** - create an ssh container derived from uwsgi that can be used as a remote interpreter for PyCharm. See [development notes](README-dev.md).
+* **enable-machine** -
+* **kill** - kills all running containers. Does not remove them.
+* **logs** - view the logs of all running containers. Note that you can also view individual logs in the deployment/logs directory.
+* **mailerrorlogs** - View the error logs from the mail server.
+* **maillogs** - view the transaction logs from the mail server.
+* **mediasync** - use this from a development or offsite machine. It will rsync all media backups from deployment/media to your offsite machine.
+* **migrate** - run any pending migrations.
+* **nginx** - builds and runs the nginx container.
+* **nginxlogs** - view just the nginx activity logs.
+* **permissions** - Update the permissions of shared volumes. Note this will destroy any existing permissions you have in place.
+* **reload** - reload the uwsgi process. Useful when you need django to pick up any changes you may have deployed.
+* **rm** - remove all containers.
+* **rm-only** - remove any containers without trying to kill them first.
+* **run** - builds and runs the complete orchestrated set of containers.
+* **sentry** - **currently not working I think.** The idea is to spin up a sentry instance together with your app for fault reporting.
+* **shell** - open a bash shell in the uwsgi (where django runs) container.
+* **superuser** - create a django superuser account.
+* **update-migrations** - freshen all migration definitions to match the current code base.
+* **web** - same as **run** - runs the production site.
+
+
+e.g. ``make web``
+
+#### Arbitrary commands
+
+Running arbitrary management commands is easy (assuming you have docker >= 1.3)
+e.g.:
+
+```
+docker exec foo_web_1 /usr/local/bin/python /home/web/django_project/manage.py --help
+```
+
+**Note:** rm should not destroy any data since it only removes containers
+and not host volumes for db and django. All commands should be non-destructive
+to existing data - though **smart people make backups before changing things**.
+
+
+## Setup nginx reverse proxy
+
+You should create a new nginx virtual host - please see
+``*-nginx.conf`` in the deployment directory of the source for an example.
+
+Simply add the example file (symlinking is best) to your ``/etc/nginx/sites-enabled/`` directory
+and then modify the contents to match your domain. Then use
+
+```
+sudo nginx -t
+```
+
+To verify that your configuration is correct and then reload / restart nginx
+e.g.
+
+```
+sudo /etc/init.d/nginx restart
+```
+
+**Note that the default configuration runs the service directly on port 80 since
+we assume there is a dedicated server for deployment.**
+
+# Configuration options
+
+You can configure the base port used and various other options like the
+image organisation namespace and postgis/mysql user/pass by editing the ``docker-compose.yml``
+files.