Devops is a set of tools for building a continuous deployment pipeline for GitLab CI/CD with serverless infrastructure on AWS.
Check out the full presentation that covers how to setup your GitLab CI/CD pipeline that uses autoscaling GitLab Runners on AWS.
Devops handles creating AWS resources and serverless deployments for your project using Go code (hopefully the primary language your project is coded in). This is known Configuration as Code where code is the formal migration of config between your applications and your deployment environment. What does this entail?
-
All configuration for your project is check into version control.
-
Configurations are migrated as apart of your build pipeline and is therefore treated the same as application code.
-
You can customize any of the configuration code without having to deal with JSON or YAML files.
This project was developed to support the SaaS Startup Kit to facilitate getting code to production with minimal additional configuration.
Multiple AWS services are already integrated into the SaaS Startup Kit, using the AWS SDK for Go. Leveraging this existing AWS integration reduces new project dependencies and limits the scope of additional technologies required to get your code successfully up and running on production. If you understand Golang, then you will be a master at devops with this tool.
This project has three main components:
-
pkg/devdeploy - A package which provides configuration for AWS resources and handles executing that configuration for you.
-
build/cicd - An example implementation of the devdeploy package that includes configuration for two example applications:
-
Go Web API - An API service written in GO that is deployed to AWS Fargate with built in support for HTTPS.
AWS Fargate is a compute engine for Amazon ECS that allows you to run containers without having to manage servers or clusters. With AWS Fargate, you no longer have to provision, configure, and scale clusters of virtual machines to run containers.
-
Python Datadog Log Collector - An python script that is deployed to AWS Lambda to ship logs from AWS to Datadog.
AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume, there is no charge when your code is not running.
-
Build with Base Image - A service that uses a separate step to build the base image. Sometimes we need to compile a bunch of libraries to support a Go library like gopkg.in/gographics/imagick.v3/imagick. The pipeline for this example service first builds build/docker/go-imagemagick7 which adds the stage
imageto the GitLab CI/CD pipeline. After this completes, the normal build stage is run.
-
-
cmd/devops - A tool developed to help make it easy to get starting with this project. This tool will copy the example build/cicd to a desired project directory updating Go imports as necessary.
Make sure you have a working Go environment. Go version 1.2+ is supported. See the install instructions for Go.
To install devops, simply run:
$ go get -v github.com/rogaha/devops/cmd/devops
Make sure your PATH includes the $GOPATH/bin directory so your commands can
be easily used:
export PATH=$PATH:$GOPATH/bin
Make a copy of build/cicd to your specified project path.
$ devops inject-build cicd -project $GOPATH/src/gitlab.com/geeks-accelerator/oss/saas-starter-kitYou should only run this command once as it will only create files that don't exist. It will not update or overwrite existing files. Once this command is executed, you are in charge of maintaining your copy cicd as it will contain configuration details only relevant to your project. Don't forget to add this to folder to git.
The build/cicd directory should have been added to your project path with the following structure:
.
├── ...
├── build/cicd
│ ├── internal
│ │ ├── config # Project configuration for build and deployment.
│ │ └── schema # Database schema migration helper.
│ ├── main.go # Command line entry point for executing config and schema.
│ └── README.md # Instructions focused on using cicd for a project.
└── ...
No changes should be necessary to main.go. You should review README.md instructions as it will cover the current
capabilities currently coded in config. Once you have completed updated the configuration for your services and
functions, ensure the README.md reflects the changes you have made.
The directory build/cicd/internal/config is where all the configuration for deployment exists. This code should be
updated to reflect your desired configuration.
config
├── config.go # Configuration for AWS infrastructure.
├── function.go # Defines functions that will be deployed to AWS Lambda.
├── service.go # Defines services that will be deployed to AWS Fargate.
└── schema.go # Handles executution of schema migrations.
-
config.go- Defines the configuration for AWS infrastructure required for serverless deployment. This includes details for AWS VPC, security group, RDS postgres database, Redis cache cluster, etc. -
function.go- Defines your functions that will be deployed to AWS Lambda. This includes settings for the runtime, amount of memory, and timeout. The code has one function defined, Python Datadog Log Collector. Additional functions can easily be defined here. -
service.go- Defines your services that will be deployed to AWS Fargate. This includes settings for your AWS ECS Cluster, the specific service and task definitions. The code as one service defined, Go Web API. Additional services can easily be defined here. -
schema.go- Handles execution of schema migrations for target the deployment environment. Database credentials are loaded from AWS Secrets Manager.
The directory build/cicd/internal/schema is a minimalistic database migration script that implements
github.com/geeks-accelerator/sqlxmigrate. Database schema
for the entire project should be defined globally. The SaaS Startup Kit
also uses this package to dynamically spin up database containers on-demand and automatically include all the
migrations. This allows the testing package to programmatically execute schema migrations before running any unit tests.
schema
├── schema.go # Entry point for executing schema migration.
├── init_schema.go # SQL queries executed on new databases.
└── migrations.go # Versioned SQL queries that be applied to the database.
-
init_schema.go- SQL queries that will run as-if no migration was run before (in a new clean database). -
migrations.go- List of direct SQL statements for each migration with defined version ID. A database table is created to persist executed migrations. Upon run of each schema migration run, the migration logic checks the migration database table to check if it’s already been executed. Thus, schema migrations are only ever executed once. Migrations are defined as a function to enable complex migrations so results from query manipulated before being piped to the next query.
Migrations should be backwards compatible with the existing deployed code. Refrain from drop table. Instead of
renaming columns, add a new column and copy the data from the old column using an update.
Ideally migrations should be idempotent to avoid possible data loss since data could have been generated between migration runs.
cicd command is primary executed by a GitLab runner. After you have updated the configuration for your project, you will need to configure GitLab CI/CD to execute the build and deployment. This project has an example .gitlab-ci.yml that should be placed in your project root.
The project includes a Postgres database which adds an additional resource dependency when deploying the project. It is important to know that the tasks running schema migration for the Postgres database can not run as shared GitLab Runners since they will be outside the deployment AWS VPC. There are two options here:
- Enable the AWS RDS database to be publicly available (not recommended).
- Run your own GitLab runners inside the same AWS VPC and grant access for them to communicate with the database.
This project has opted to implement option 2 and thus setting up the deployment pipeline requires a few more additional steps.
Note that using shared runners hosted by GitLab also requires AWS credentials to be input into GitLab for configuration.
Hosting your own GitLab runners will use an AWS Role instead of hardcoding the access key ID and secret access key in GitLab and in other configuration files. And since this project is open-source, we want to avoid sharing our AWS credentials, while also making a smaller surface area of privileged infrastructure information (only AWS).
Below outlines the basic steps to setup Autoscaling GitLab Runner on AWS.
You can also check out the full presentation that covers the same steps.
If you don't have an AWS account, signup for one now and then proceed with the deployment setup.
We assume that if you are deploying the SaaS Stater Kit, you are starting from scratch with no existing dependencies. This however, excludes any domain names that you would like to use for resolving your services publicly. To use any pre-purchased domain names, make sure they are added to Route 53 in the AWS account. Or you can let the deploy script create a new zone is Route 53 and update the DNS for the domain name when your ready to make the transition. It is required to hosted the DNS on Route 53 so DNS entries can be managed by this deploy tool. It is possible to use a subdomain that uses Route 53 as the DNS service without migrating the parent domain.
-
If you have already setup AWS Permissions for local usage, then the IAM policy
saas-starter-kit-deployshould already have been created. If not, create a new AWS IAM Policy calledsaas-starter-kit-deploywith defined JSON statement instead of using the visual editor. The statement is rather large as each permission is granted individually. A copy of the statement is stored in the repo at configs/aws-aim-deploy-policy.json -
Define an AWS IAM Role that will be attached to the GitLab Runner instances. The role will need permission to scale (EC2), update the cache (via S3) and perform the project specific deployment commands.
Trusted Entity: AWS Service Service that will use this role: EC2 Attach permissions policies: AmazonEC2FullAccess, AmazonS3FullAccess, saas-starter-kit-deploy Role Name: SaasStarterKitEc2RoleForGitLabRunner Role Description: Allows GitLab runners hosted on EC2 instances to call AWS services on your behalf. -
Launch a new AWS EC2 Instance.
GitLab Runnerwill be installed on this instance and will serve as the bastion that spawns new instances. This instance will be a dedicated host since we need it always up and running, thus it will be the standard costs apply.Note: Since this machine will not run any jobs itself, it does not need to be very powerful. A t2.micro instance will be sufficient.
Amazon Machine Image (AMI): Amazon Linux AMI 2018.03.0 (HVM), SSD Volume Type - ami-0f2176987ee50226e Instance Type: t2.micro -
Configure Instance Details.
Note: Do not forget to select the IAM Role SaasStarterKitEc2RoleForGitLabRunner
Number of instances: 1 Network: default VPC Subnet: no Preference Auto-assign Public IP: Use subnet setting (Enable) Placement Group: not checked/disabled Capacity Reservation: Open IAM Role: SaasStarterKitEc2RoleForGitLabRunner Shutdown behavior: Stop Enable termination project: checked/enabled Monitoring: not checked/disabled Tenancy: Shared Elastic Interence: not checked/disabled T2/T3 Unlimited: not checked/disabled Advanced Details: none -
Add Storage. Increase the volume size for the root device to 30 GiB.
Volume Type | Device | Size (GiB) | Volume Type Root | /dev/xvda | 30 | General Purpose SSD (gp2) -
Add Tags.
Name: gitlab-runner -
Configure Security Group. Create a new security group with the following details:
Name: gitlab-runner Description: Gitlab runners for running CICD. Rules: Type | Protocol | Port Range | Source | Description SSH | TCP | 22 | My IP | SSH access for setup. -
Review and Launch instance. Select an existing key pair or create a new one. This will be used to SSH into the instance for additional configuration.
-
Update the security group to reference itself. The instances need to be able to communicate between each other.
Navigate to edit the security group and add the following two rules where
SECURITY_GROUP_IDis replaced with the name of the security group created in step 6.Rules: Type | Protocol | Port Range | Source | Description Custom TCP | TCP | 2376 | SECURITY_GROUP_ID | Gitlab runner for Docker Machine to communicate with Docker daemon. SSH | TCP | 22 | SECURITY_GROUP_ID | SSH access for setup. -
SSH into the newly created instance.
ssh -i ~/saas-starter-kit-uswest2-gitlabrunner.pem [email protected]
- If you get the error
Permissions 0666 are too open, then you will need tochmod 400 FILENAME. - If you get the error
permission denied, check that they're usingec2-useras the username.
- If you get the error
-
Install GitLab Runner from the official GitLab repository
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh | sudo bash yes | sudo yum install -y gitlab-runner
-
Install Docker Community Edition.
yes | sudo yum install docker -
curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && chmod +x /tmp/docker-machine && sudo cp /tmp/docker-machine /usr/sbin/docker-machine
-
You will need to navigate to the
CI / CDunderSettingsfor your GitLab repo. This will provide the first two bits of information you will need to register a new runner.
Now you can execute the register command.
sudo gitlab-runner register
Notes:
- When asked for gitlab-ci tags, enter
prod- If you would like to setup a stage environment, then you could add the additional tags
stage
- If you would like to setup a stage environment, then you could add the additional tags
- When asked the executor type, enter
docker+machine - When asked for the default Docker image, enter
geeksaccelerator/docker-library:golang1.12-docker
- When asked for gitlab-ci tags, enter
-
sudo vim /etc/gitlab-runner/config.toml
Update the
[runners.docker]configuration section inconfig.tomlto match the example below replacing the obvious placeholderXXXXXwith the relevant value.Few notes:
privileged = trueallows for the build pipeline to take advantage of caching of multistage docker containers.- No AWS access/secret keys should be necessary in this file as the attached AWS IAM role should handle dynamically providing credentials.
environment = ["GOPROXY=https://goproxy.io"] [runners.docker] tls_verify = false privileged = true disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = true volumes = ["/cache"] shm_size = 0 [runners.machine] IdleCount = 0 IdleTime = 1800 MachineDriver = "amazonec2" MachineName = "gitlab-runner-machine-%s" MachineOptions = [ "amazonec2-iam-instance-profile=SaasStarterKitEc2RoleForGitLabRunner", "amazonec2-region=us-west-2", "amazonec2-vpc-id=XXXXX", "amazonec2-subnet-id=XXXXX", "amazonec2-zone=d", "amazonec2-use-private-address=true", "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true", "amazonec2-security-group=gitlab-runner", "amazonec2-instance-type=t2.large" ]
You will need use the same VPC subnet and availability zone as the instance launched in step 2. We are using AWS region
us-west-2. Under MachineOptions you can add anything that the AWS Docker Machine driver supports.
Below are some example values for the placeholders to ensure for format of your values are correct. The AWS web console lists the full availability zone as
us-west-2a, GitLab requires only the letter of the availability zone, in this caseato be used foramazonec2-zone.amazonec2-vpc-id=vpc-5f43f027 amazonec2-subnet-id=subnet-693d3110 amazonec2-zone=a
Once complete, restart the runner.
sudo gitlab-runner restart
It's optional to enable cache to speed up your jobs. For more details on this, refer to The runners.cache section. If you decide to enable cache, an AWS S3 Bucket is required. We normally manually create an S3 bucket with the same name as the primate S3 bucket configured in config.go. The deployment will finish applying any additional details required for the project to the manually created S3 bucket even though you already created the bucket.
[runners.cache] Type = "s3" Shared = true [runners.cache.s3] ServerAddress = "s3.us-west-2.amazonaws.com" BucketName = "XXXXX" BucketLocation = "us-west-2"
The ServerAddress for S3 will need to be updated if the region is changed. For
us-east-1the ServerAddress iss3.amazonaws.com. Read more here about accessing a bucket -
Build the gitlab base image that includes golang and the devops tool.
docker login registry.gitlab.com
cd ./build/docker
docker build -t golang1.13-docker -t registry.github.com/rogaha/devops:golang1.13-docker golang/1.13/docker
docker push registry.github.com/rogaha/devops:golang1.13-dockerUpdate the image in .gitlab-ci.yml to match your project registry.
image: registry.github.com/rogaha/devops:golang1.13-docker- Optionally enable a locally hosted proxy for Go modules to speed up build times using goproxy.io.
sudo chkconfig docker on
sudo service docker start
sudo usermod -a -G docker $USER
sudo docker run -d -p8081:8081 -v /tmp:/go --restart always goproxy/goproxy -proxy https://goproxy.ioGet the public DNS name for the instance. This will be used by runners to access goproxy running on the bastion.
echo "http://"$(curl -s http://169.254.169.254/latest/meta-data/public-hostname)":8081"Open up /etc/gitlab-runner/config.toml in vim to edit the configuration file. In the [[runners]] section add the
following line after executor:
environment = ["GOPROXY=xxxx"]If you change any of the environment variables in the config and an instance has already been spun up to execute a pending job, you will need to manually terminate the instance to force a new instance to be created before the change to the environment will be included.
Example after update:
[[runners]]
name = "oss-devops-dev"
url = "https://gitlab.com/"
executor = "docker+machine"
environment = ["GOPROXY=http://ec2-52-34-34-34.us-west-2.compute.amazonaws.com:8081"]Restart the gitlab runner:
sudo gitlab-runner restartAdd port 8081 to the AWS security group gitlab-runner using the same process outlined in step 9.
- Setup complete. You should now be able navigate back to the
CI / CDunderSettingsfor your GitLab repo and see the newly deployed instance listed as an active runner.
You can execute the cicd command locally as an alternative to having GitLab CI/CD execute the commands. Before you are able to run the build and deploy sub commands, you will need an AWS access/secret key.
$ cicd help
NAME:
cicd - Provides build and deploy for GitLab to Amazon AWS
USAGE:
cicd [global options] command [command options] [arguments...]
VERSION:
1.0
COMMANDS:
build, b build a service or function
deploy, d deploy a service or function or infrastructure
schema, s manage the database schema
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--env value target environment, one of [dev, stage, prod]
--aws-access-key value AWS Access Key [$AWS_ACCESS_KEY_ID]
--aws-secret-key value AWS Secret Key [$AWS_SECRET_ACCESS_KEY]
--aws-region value AWS Region [$AWS_DEFAULT_REGION]
--aws-use-role Use an IAM Role else AWS Access/Secret Keys are required [$AWS_USE_ROLE]
--help, -h show help
--version, -v print the versionRefer to the cicd readme for full command details
Create an AWS user for development purposes. This user is not needed for the build pipeline using GitLab CI/CD.
-
You will need an existing AWS account or create a new AWS account.
-
An AWS IAM Policy is needed for cdcd to setup the configured AWS infrastructure and deploy services/functions. If you haven't setup the Setup GitLab CI/CD then you will need to define a new IAM policy called
saas-starter-kit-deploywith a defined JSON statement instead of using the visual editor. The statement is rather large as each permission is granted individually. A copy of the statement is stored in the repo at configs/aws-aim-deploy-policy.json -
Create new AWS User called
saas-starter-kit-deploywith Programmatic Access and Attach existing policies directly with the policy created from step 2saas-starter-kit-deploy -
Set your AWS credentials as environment variables. These can also be passed into cicd as command line options.
export AWS_ACCESS_KEY_ID=XXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXX
export AWS_DEFAULT_REGION="us-west-2"
export AWS_USE_ROLE=falseWe ❤️ contributions.
Have you had a good experience with devops ? Why not share some love and contribute code?
Thank you to all those that have contributed to this project and are using it in their projects. You can find a CONTRIBUTORS file where we keep a list of contributors to the project. If you contribute a PR please consider adding your name there.
If you are having problems installing, getting the project running, or would like to contribute, join the
channel #saas-starter-kit on Gopher Slack
Please read the LICENSE file here.
Copyright 2019, Geeks Accelerator [email protected]