Use Docker images locally and in CI by EngHabu · Pull Request #6671 · flyteorg/flyte

EngHabu · 2025-10-10T18:36:28Z

Signed-off-by: Haytham Abuelfutuh [email protected]

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Please add one or more of the following labels to categorize your PR:

added: For new features.
changed: For changes in existing functionality.
deprecated: For soon-to-be-removed features.
removed: For features being removed.
fixed: For any bug fixed.
security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

Signed-off-by: Haytham Abuelfutuh <[email protected]>

flyte-bot · 2025-10-10T18:36:53Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

Signed-off-by: Haytham Abuelfutuh <[email protected]>

github-actions · 2025-10-10T18:52:03Z

🐳 Docker CI Image Built

The CI Docker image has been built and pushed for this PR!

Image: ghcr.io/flyteorg/flyte/ci:pr-6671

This image will be automatically used by CI workflows in this PR.

To test locally:

docker pull ghcr.io/flyteorg/flyte/ci:pr-6671
docker run --rm -it -v $(pwd):/workspace -w /workspace ghcr.io/flyteorg/flyte/ci:pr-6671 bash

Signed-off-by: Haytham Abuelfutuh <[email protected]>

bito-code-review · 2025-10-10T19:45:13Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

Signed-off-by: Haytham Abuelfutuh <[email protected]>

bito-code-review · 2025-10-10T22:51:02Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

Signed-off-by: Haytham Abuelfutuh <[email protected]>

bito-code-review · 2025-10-11T00:01:06Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

Signed-off-by: Haytham Abuelfutuh <[email protected]>

bito-code-review · 2025-10-14T01:31:03Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

Signed-off-by: Haytham Abuelfutuh <[email protected]>

bito-code-review · 2025-10-14T02:33:22Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

bito-code-review · 2025-10-14T03:43:36Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Copilot

Pull Request Overview

This PR introduces Docker-based development and CI workflows to ensure consistent environments across local development and CI. The changes standardize tool versions and eliminate "works on my machine" issues by containerizing the development environment.

Key changes:

Creates a unified Docker image (ci.Dockerfile) with pinned tool versions for Go, Python, Node.js, Rust, and Buf
Implements intelligent CI workflows that automatically build PR-specific Docker images when the Dockerfile is modified
Updates the Makefile to support both Docker-based (default) and local tool-based development workflows

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`scripts/docker-dev.sh`	Helper script to simplify Docker container usage for development
`ci.Dockerfile`	Multi-stage Docker image with development tools and optimized caching
`Makefile`	Updated with Docker-based targets as defaults, local tool targets as alternatives
`.github/workflows/build-ci-image.yml`	Workflow to build and publish Docker images with PR-specific tagging
`.github/workflows/check-generate.yml`	Updated to use Docker containers and wait for PR-specific images
`docs/`	Comprehensive documentation for Docker workflows and development environment

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

.github/workflows/build-ci-image.yml

.github/workflows/check-generate.yml

.github/workflows/regenerate-on-comment.yml

Co-authored-by: Copilot <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]>

bito-code-review · 2025-10-15T01:51:06Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

Signed-off-by: Haytham Abuelfutuh <[email protected]>

machichima · 2025-10-15T08:58:58Z

Makefile

-gen: buf mocks go_tidy ## Generates everything in the 'gen' directory
-	@echo '⚡  Finished generating everything in the gen directory'
+gen: buf mocks go-tidy ## Generate everything (uses Docker - no local tools required)
+	@echo '⚡  Finished generating everything in the gen directory (Docker)'


Can we use:

$(DOCKER_RUN) bash -c "git config --global --add safe.directory /workspace && make gen-local"

Same as what we did in docker-gen-local? In this case we do not need buf-... and buf-...-local. We can just have buf-... as what we have originally

I was keeping it so you can continue to run individual commands (but now using docker).... I'll follow your guidance and we can bring them back if needed.

machichima · 2025-10-15T09:04:24Z

Makefile


+# Docker CI image configuration
+DOCKER_CI_IMAGE := ghcr.io/flyteorg/flyte/ci:v2
+DOCKER_LOCAL_IMAGE := flyte-ci:local


Can we also name the image built locally as ghcr.io/flyteorg/flyte/ci:v2?

In this case, no matter using remote image or build locally, we can run make gen. If no changes in ci.Dockerfile, we use make docker-pull before make gen, otherwise use make docker-build.

The process will become:

typical usage:

pull CI image make docker-pull

run make gen

If there's modification in ci.Dockerfile:

build the CI image make docker-build

also run make gen

I think it can make the process easier to follow and also make the Makefile cleaner

machichima · 2025-10-15T09:07:52Z

gen.Dockerfile

Maybe we can use name gen.Dockerfile instead? As I think this is mainly for generating

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Copilot

Pull Request Overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 8 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

.github/workflows/regenerate-on-comment.yml

.github/workflows/check-generate.yml

.github/workflows/regenerate-on-comment.yml

Copilot · 2025-10-15T20:31:43Z

docs/docker-image-workflow.md

+│ check-generate workflow triggers                            │
+│ • Checks if Dockerfile.ci modified: NO                      │


Documentation references Dockerfile.ci, but the repository adds gen.Dockerfile; update all occurrences to avoid confusion.

Copilot · 2025-10-15T20:31:44Z

docs/docker-image-workflow.md

+
+### Image Detection Logic
+
+Workflows check if `Dockerfile.ci` or `.github/workflows/build-ci-image.yml` were modified:


Example detection logic uses Dockerfile.ci instead of gen.Dockerfile; this is inconsistent with the actual file name and will mislead users.

DOCKER_QUICK_START.md

Copilot · 2025-10-15T20:31:44Z

scripts/docker-dev.sh

+    docker run --rm -it \
+        -v "$REPO_ROOT:$WORKSPACE" \
+        -w "$WORKSPACE" \
+        "$IMAGE" \
+        bash


[nitpick] Running as root in the container will create root-owned files on the host, complicating cleanup. Consider adding --user $(id -u):$(id -g) to preserve host ownership.

Copilot · 2025-10-15T20:31:44Z

docs/docker-image-workflow.md

+| Context            | Image Tag                              | When Created                    |
+|--------------------|----------------------------------------|---------------------------------|


Table rows start with double pipe characters, which can break Markdown table rendering. Remove the extra leading pipe on these lines.

Co-authored-by: Copilot <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]>

Signed-off-by: Haytham Abuelfutuh <[email protected]>

…bu/dockerize-ci

flyte-bot · 2025-10-15T21:05:13Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

machichima

Not familiar with github action YAML, but tried running make docker-pull gen DOCKER_CI_IMAGE=ghcr.io/flyteorg/flyte/ci:pr-6671 locally and it works.

Overall LGTM! Just some small nits on docs

README.md

docs/docker-image-workflow.md

CONTRIBUTING.md

Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]>

Signed-off-by: Haytham Abuelfutuh <[email protected]>

flyte-bot · 2025-10-16T03:01:08Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

machichima

Sorry I miss this. I think it should be master instead of main branch

machichima · 2025-10-16T07:51:25Z

.github/workflows/build-ci-image.yml

+on:
+  push:
+    branches:
+      - main


Suggested change

- main

- master

So I created a main branch.. it's where I deleted everything from the master branch... I think what we will do is when we are ready to switch over, we will switch the default branch for the repo from master to main... and let master be the v1 stuff

machichima · 2025-10-16T07:51:47Z

.github/workflows/build-ci-image.yml

+            # Use PR number for pull requests
+            type=ref,event=pr
+            # Use 'latest' tag for main branch
+            type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}


Suggested change

type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}

type=raw,value=latest,enable=${{ github.ref == 'refs/heads/master' }}

machichima · 2025-10-16T07:53:04Z

.github/workflows/build-ci-image.yml

+            echo "tag=pr-${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT
+          elif [ "${{ github.ref }}" == "refs/heads/v2" ]; then
+            echo "tag=v2" >> $GITHUB_OUTPUT
+          elif [ "${{ github.ref }}" == "refs/heads/main" ]; then


Suggested change

elif [ "${{ github.ref }}" == "refs/heads/main" ]; then

elif [ "${{ github.ref }}" == "refs/heads/master" ]; then

machichima · 2025-10-16T07:55:53Z

.github/workflows/build-ci-image.yml

+            type=ref,event=branch
+            # Use PR number for pull requests
+            type=ref,event=pr
+            # Use 'latest' tag for main branch


Suggested change

# Use 'latest' tag for main branch

# Use 'latest' tag for master branch

machichima · 2025-10-16T07:56:14Z

docs/docker-image-workflow.md

+
+1. **`.github/workflows/build-ci-image.yml`**
+   - Builds Docker image
+   - Runs on: PR with Dockerfile changes, push to main/v2


Suggested change

- Runs on: PR with Dockerfile changes, push to main/v2

- Runs on: PR with Dockerfile changes, push to master/v2

machichima · 2025-10-16T07:56:28Z

docs/docker-image-workflow.md

+| Regular PR         | `v2` or `latest`                      | Uses existing branch image      |
+| PR with Dockerfile | `pr-123`                              | Built when PR is created/updated|
+| v2 branch push     | `v2`                                  | Built on every push to v2       |
+| main branch push   | `latest`                              | Built on every push to master   |


Use Docker images locally and in CI

09e3a42

Signed-off-by: Haytham Abuelfutuh <[email protected]>

EngHabu added 2 commits October 10, 2025 11:42

fix branch prefix tag

db9e21c

Signed-off-by: Haytham Abuelfutuh <[email protected]>

use github.head_ref instead

60dc38a

Signed-off-by: Haytham Abuelfutuh <[email protected]>

fix tag in comment

c199ad0

Signed-off-by: Haytham Abuelfutuh <[email protected]>

EngHabu added 5 commits October 10, 2025 14:47

make local docker experience

f56d3bf

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Merge branch 'v2' into enghabu/dockerize-ci

8bfcf23

try

398fb37

Signed-off-by: Haytham Abuelfutuh <[email protected]>

wip

41d0300

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Faster python install?

ddfe2bd

Signed-off-by: Haytham Abuelfutuh <[email protected]>

EngHabu added 2 commits October 10, 2025 15:57

wait for image build wf

f8b05e9

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Optimize image

d04e266

Signed-off-by: Haytham Abuelfutuh <[email protected]>

cleanups

9b085e2

Signed-off-by: Haytham Abuelfutuh <[email protected]>

EngHabu added 2 commits October 13, 2025 18:32

wip

2e4e90b

Signed-off-by: Haytham Abuelfutuh <[email protected]>

regen

ba15967

Signed-off-by: Haytham Abuelfutuh <[email protected]>

EngHabu marked this pull request as ready for review October 14, 2025 02:41

EngHabu added 2 commits October 14, 2025 17:49

cleanup not needed commands

97ddb4a

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Merge branch 'v2' into enghabu/dockerize-ci

3096ecd

EngHabu requested review from Copilot, pingsutw and troychiu October 15, 2025 01:06

Copilot AI reviewed Oct 15, 2025

View reviewed changes

Update .github/workflows/regenerate-on-comment.yml

446134c

Co-authored-by: Copilot <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]>

EngHabu added 2 commits October 14, 2025 19:46

test changes

e55f209

Signed-off-by: Haytham Abuelfutuh <[email protected]>

PR Comments

51d90b7

Signed-off-by: Haytham Abuelfutuh <[email protected]>

flyteorg deleted a comment from bito-code-review bot Oct 15, 2025

machichima reviewed Oct 15, 2025

View reviewed changes

EngHabu added 5 commits October 15, 2025 12:54

PR Comments

309b7cb

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Trying with :v2 image

a53f28f

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Merge branch 'v2' into enghabu/dockerize-ci

ded4418

Fix WF

c5a051e

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Update guides

eba57fe

Signed-off-by: Haytham Abuelfutuh <[email protected]>

EngHabu requested review from Copilot and machichima October 15, 2025 20:30

Copilot AI reviewed Oct 15, 2025

View reviewed changes

EngHabu and others added 6 commits October 15, 2025 13:35

Update .github/workflows/regenerate-on-comment.yml

5c6e0be

Co-authored-by: Copilot <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]>

Update .github/workflows/check-generate.yml

6cd7027

Co-authored-by: Copilot <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]>

Update .github/workflows/regenerate-on-comment.yml

8a7c729

Co-authored-by: Copilot <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]>

Update DOCKER_QUICK_START.md

562c7fc

Co-authored-by: Copilot <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]>

PR Comments

5d24eaf

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Merge remote-tracking branch 'origin/enghabu/dockerize-ci' into engha…

1b4565d

…bu/dockerize-ci

machichima reviewed Oct 16, 2025

View reviewed changes

README.md Show resolved Hide resolved

docs/docker-image-workflow.md Outdated Show resolved Hide resolved

CONTRIBUTING.md Show resolved Hide resolved

EngHabu and others added 3 commits October 15, 2025 18:55

Update README.md

fe87a21

Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]>

Update CONTRIBUTING.md

350a4d7

Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: Haytham Abuelfutuh <[email protected]>

PR Comments

f26c0bd

Signed-off-by: Haytham Abuelfutuh <[email protected]>

EngHabu requested a review from machichima October 16, 2025 01:58

machichima reviewed Oct 16, 2025

View reviewed changes

EngHabu merged commit 2378cd5 into v2 Oct 16, 2025
5 checks passed

EngHabu deleted the enghabu/dockerize-ci branch October 16, 2025 19:02

mhotan mentioned this pull request Feb 9, 2026

Add ExecutedBy EnrichedIdentity to RunSpec #6906

Open

		│ check-generate workflow triggers │
		│ • Checks if Dockerfile.ci modified: NO │


		### Image Detection Logic

		Workflows check if `Dockerfile.ci` or `.github/workflows/build-ci-image.yml` were modified:

		\| Context \| Image Tag \| When Created \|
		\|--------------------\|----------------------------------------\|---------------------------------\|

	type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}
	type=raw,value=latest,enable=${{ github.ref == 'refs/heads/master' }}

	elif [ "${{ github.ref }}" == "refs/heads/main" ]; then
	elif [ "${{ github.ref }}" == "refs/heads/master" ]; then

	# Use 'latest' tag for main branch
	# Use 'latest' tag for master branch

	- Runs on: PR with Dockerfile changes, push to main/v2
	- Runs on: PR with Dockerfile changes, push to master/v2

	\| main branch push \| `latest` \| Built on every push to master \|
	\| master branch push \| `latest` \| Built on every push to master \|

Conversation

EngHabu commented Oct 10, 2025

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

Uh oh!

flyte-bot commented Oct 10, 2025

Uh oh!

github-actions bot commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🐳 Docker CI Image Built

Uh oh!

bito-code-review bot commented Oct 10, 2025

Uh oh!

bito-code-review bot commented Oct 10, 2025

Uh oh!

bito-code-review bot commented Oct 11, 2025

Uh oh!

bito-code-review bot commented Oct 14, 2025

Uh oh!

bito-code-review bot commented Oct 14, 2025

Uh oh!

bito-code-review bot commented Oct 14, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bito-code-review bot commented Oct 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

flyte-bot commented Oct 15, 2025

Uh oh!

machichima left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flyte-bot commented Oct 16, 2025

Uh oh!

github-actions bot commented Oct 10, 2025 •

edited

Loading