Skip to content

Experiment with running rails/rails builds on Buildkite Hosted Agents #141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Experiment with running rails/rails builds on Buildkite Hosted Agents #141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4e17648
Update rails-ci pipeline sto run on Buildkite hosted agents
Nov 10, 2024
da0ac85
inspect the docker image store at the start of each job
Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion lib/buildkite/config/build_context.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ def min_ruby
end

def remote_image_base
"973266071021.dkr.ecr.us-east-1.amazonaws.com/#{"#{build_queue}-" unless standard_queues.include?(build_queue)}builds"
ENV.fetch("REGISTRY") + "/#{"#{build_queue}-" unless standard_queues.include?(build_queue)}builds"
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We now fetch the registry hostname from the environment dynamically in the initial job

end
end
end
17 changes: 2 additions & 15 deletions lib/buildkite/config/docker_build.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,7 @@ def cache_from(build_context)
end

def build_push(build_context)
[
build_context.local_branch =~ /:/ ?
build_context.image_name_for("pr-#{build_context.pull_request}") :
build_context.image_name_for("br-#{build_context.local_branch}"),
]
build_context.image_name_for(build_context.build_id, prefix: nil)
end
end

Expand Down Expand Up @@ -66,20 +62,11 @@ def builder(ruby)
compressed: ".buildkite.tgz"
}

plugin :docker_compose, {
build: "base",
config: ".buildkite/docker-compose.yml",
env: %w[PRE_STEPS RACK],
"image-name" => build_context.ruby.image_name_for(build_context.build_id),
"cache-from" => cache_from(build_context),
push: build_push(build_context),
"image-repository" => build_context.image_base,
}
command "docker build --push --build-arg RUBY_IMAGE=#{build_context.ruby.ruby_image} --tag #{build_push(build_context)} --file .buildkite/Dockerfile ."

env({
BUNDLER: build_context.bundler,
RUBYGEMS: build_context.rubygems,
RUBY_IMAGE: build_context.ruby.ruby_image,
encrypted_0fb9444d0374_key: nil,
encrypted_0fb9444d0374_iv: nil
})
Expand Down
6 changes: 4 additions & 2 deletions lib/buildkite/config/rake_command.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,12 +48,14 @@ def install_plugins(service = "default", env = nil, dir = ".")
],
compressed: ".buildkite.tgz"
}
plugin :metahook, {
"pre-command": "echo \"+++ inspect docker image store\"\ndocker image ls"
}
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for debugging only, we can remove it before considering merging

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✍️

inspect docker image store
REPOSITORY                            TAG                                             IMAGE ID       CREATED         SIZE
nscr.io/0meljd4iqsga4/hosted-builds   ruby-3-4-0194ff6b-5960-4b8d-b22f-03d45ddc78b9   5da7504497bf   7 minutes ago   3.58GB
memcached                             alpine                                          7944216f8983   8 days ago      15.9MB
postgres                              alpine                                          7e5df973a748   9 days ago      397MB
mysql                                 latest                                          b842a59bfaf8   3 weeks ago     1.09GB
redis                                 alpine                                          1bf97f21f01b   5 weeks ago     60.6MB


plugin :docker_compose, {
"env" => env,
"run" => service,
"pull" => service,
"pull-retries" => 3,
"tty" => "true",
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tty changed to default:false in v5 of the plugin

"config" => ".buildkite/docker-compose.yml",
"shell" => ["runner", *dir],
}.compact
Expand Down
17 changes: 15 additions & 2 deletions pipelines/rails-ci/initial.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,15 @@
# This file is never read -- it's just a copy of the pipeline's
# configuration in the Buildkite UI.
env:
CONFIG_REPO: "https://github.com/yob/buildkite-config"
CONFIG_BRANCH: "hosted"

steps:
- name: ":pipeline: rails-initial-pipeline"
command: |
echo "Fetching registry details"
export REGISTRY="$$(nsc workspace describe -o json -k registry_url)"

PATH=/bin:/usr/bin
set -e

Expand All @@ -30,17 +36,22 @@ steps:
echo "Fetching pull-request metadata:"
(docker run --rm \
-v "$$PWD":/app:ro -w /app \
-v "$$PWD/cache/bundler":/usr/local/bundle \
-e GITHUB_PUBLIC_REPO_TOKEN \
-e BUILDKITE_REPO \
-e BUILDKITE_PULL_REQUEST \
ruby:latest \
.buildkite/bin/fetch-pr > .buildkite/tmp/.pr-meta.json) || true


echo "Generating pipeline:"
sh -c "$$PIPELINE_COMMAND"

([ -f .buildkite/.dockerignore ] && cp .buildkite/.dockerignore .dockerignore) || true

cache:
paths:
- "cache/bundler"
name: "rails-initial-bundler-cache"
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create a cache volume for the gems used in this initial step. In most cases the cache will be warm and it shaves 10s of seconds of the initial step

plugins:
- artifacts#v1.9.3:
upload: ".dockerignore"
Expand All @@ -58,6 +69,7 @@ steps:
PIPELINE_COMMAND: >-
docker run --rm
-v "$$PWD":/app:ro -w /app
-v "$$PWD/cache/bundler":/usr/local/bundle
-e CI
-e BUILDKITE
-e BUILDKITE_AGENT_META_DATA_QUEUE
Expand All @@ -72,9 +84,10 @@ steps:
-e DOCKER_IMAGE
-e RUN_QUEUE
-e QUEUE
-e REGISTRY
ruby:latest
.buildkite/bin/pipeline-generate rails-ci |
buildkite-agent pipeline upload
timeout_in_minutes: 5
agents:
queue: "${QUEUE-builder}"
queue: hosted
3 changes: 2 additions & 1 deletion pipelines/rails-ci/pipeline.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@
use Buildkite::Config::RakeCommand
use Buildkite::Config::RubyGroup

plugin :docker_compose, "docker-compose#v4.16.0"
plugin :docker_compose, "docker-compose#v5.4.1"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yob Was there a particular reason to update the compose plugin? Or just wanted to try latest?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC it was just to get to the latest major version while I was changing things. If sticking on v4.x helps with the transition, I doubt that'll be a problem on the hosted agents

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Thanks for the quick reply! Makes sense, I just wasn't sure if there was a requirement to using hosted for this. I will try it out with the updated plugin.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/buildkite-plugins/docker-compose-buildkite-plugin/releases/tag/v5.0.0 does mention that it makes docker-compose v2 the default, and I assume hosted agents have v2. just a guess, but if you have issues maybe you'll need to explicit about the cli-version: https://github.com/buildkite-plugins/docker-compose-buildkite-plugin/tree/v4.16.0?tab=readme-ov-file#cli-version-optional-string-or-integer

plugin :artifacts, "artifacts#v1.9.3"
plugin :metahook, "improbable-eng/metahook#v0.4.1"

if build_context.nightly?
build_context.rubies << Buildkite::Config::RubyConfig.master_ruby
Expand Down