Centralised configuration and building of eLife journal applications locally (Vagrant) and remotely (AWS, GCS).
builder is a Python 3.8+ project and supports Linux and MacOS.
Download:
git clone ssh://[email protected]/elifesciences/builder
Install (Linux, see docs/osx.md for macs).
./update.sh
Fix any missing pre-requisites and call ./update.sh again until you see the 'all done' message.
Exclude all pre-requisite checks with --exclude all. For example:
./update.sh --exclude all
Exclude specific pre-requisite checks with --exclude foo bar baz. For example:
./update.sh --exclude virtualbox vault
Checked pre-requisites:
- git
- virtualenv
- make
- virtualbox
- vagrant
- ssh-credentials
- ssh-agent
- aws-credentials
- vault
- brew (MacOS only)
- [email protected] (MacOS only)
- libffi (MacOS only)
- libssh2 (MacOS only)
- cmake (MacOS only)
git pull
./update.sh
The project file ./projects/elife.yaml describes the eLife projects that can be built and their environments.
See here for more project file documentation.
Multiple project files can be configured by copying settings.yaml.dist to settings.yaml and
then modifying project-files.
After successfully installing and configuring builder, try launching a Vagrant machine to test all is working correctly:
PROJECT=basebox vagrant up
builder is a Python project and it's dependencies are captured in a Pipfile.
It's virtualenv is found in ./venv and can be activated with ./venv/bin/activate.
To update a dependency, modify the Pipfile and run ./update-dependencies.sh to refresh the Pipfile.lock and
requirements.txt files. You will need pipenv installed.
./test.sh
The Vagrantfile can build any project:
$ PROJECT=journal vagrant up
or use the menu:
$ vagrant up
You must select a project:
1 - journal--vagrant
2 - api-gateway--vagrant
3 - ...
>
The Vagrantfile will call a Python script to discover which projects are available. To execute that script with Docker:
touch .use-docker.flag
Note: if you wish to use a private key not in ~/.ssh/id_rsa, you can customize the SSH key path.
Note: if you wish to use a hypervisor other than virtualbox, you can use the vagrant-mutate plugin
to rebuild the ubuntu/trusty64 box for your own hypervisor. See the vagrant and virtualbox documentation.
Project formulas are cloned to the local ./cloned-projects directory and become shared directories within Vagrant.
Changes to formulas including their branches are available immediately.
Run the following command inside Vagrant to change the remote commit or branch:
$ set_local_revision $commitOrBranch
Then apply the formula inside Vagrant with sudo salt-call state.highstate or from outside with vagrant provision.
The other half of the builder project is the ability to create and manage AWS (Amazon Web Services) and
GCP (Google Cloud Platform) resources. This is controlled with the bldr script:
$ ./bldr -l
Will list all available tasks.
builder relies on a ~/.aws/credentials file containing configuration for accessing your AWS account.
A master-server instance must exist before project instances can be brought up. See here.
To create a project instance:
$ ./bldr launch
Or specify a project and an instance ID directly with:
$ ./bldr launch:journal,myinstanceid
To ssh into this project instance:
$ ./bldr ssh:journal--myinstanceid
If the instance ID used matches the name of an alternate config (under 'aws-alt') in ./projects/elife.yaml then
that alternate configuration will be used.
Some alternate configurations are unique (like most prod configurations) and you won't be able to use that ID.
General:
AWS:
Troubleshooting:
Development:
The builder project is MIT licenced.