ci: Split tests run in CI #12064

JagjeevanAK · 2025-06-21T06:57:38Z

What

Screenshot

Related issue(s) and discussion

Fixes #[ISSUE NUMBER]

Signed-off-by: Jagjeevan Kashid <[email protected]>

codecov-commenter · 2025-06-21T07:44:21Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 50.20%. Comparing base (d766aa2) to head (8266725).

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12064      +/-   ##
==========================================
- Coverage   50.46%   50.20%   -0.26%     
==========================================
  Files          90       90              
  Lines       23900    23900              
  Branches     5743     5743              
==========================================
- Hits        12062    12000      -62     
  Misses      10357    10357              
- Partials     1481     1543      +62

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Jagjeevan Kashid <[email protected]>

.github/workflows/pull_request.yml

Signed-off-by: Jagjeevan Kashid <[email protected]>

.github/workflows/pull_request.yml

hangy · 2025-06-21T14:42:17Z

.github/workflows/pull_request.yml

-        download_tmp_dir: ${{ runner.temp }}
-    - name: tests
+        image: "openfoodfacts-server/backend:dev"       
+    # downloadbackendimage task loads the image into docker and keeps the original file.


This shouldn't be necessary for ishworkh/[email protected]

really ? I will see docs and will update it thanks for pointing it out.

.github/workflows/pull_request.yml

hangy · 2025-06-21T14:45:35Z

Makefile

+# Unit test groups for parallel execution in CI
+# Usage: make unit_test_group TEST_GROUP=1
+# Groups are balanced by number of test files
+define UNIT_TEST_GROUPS


Could the groups be determined algorithmically? All jobs of the actions matrix use the same source, so that the groups could be built automatically to avoid having to add each test file here in the Makefile individually

I have made group on there size as small are combined with large one so that run it in equal time

I don't like the idea of having hard-coded groups. It will be too easy to add a test and forget to include it which means the test will run locally but not in CI.

good point never though of this will look into it.

Hey @john-gom, @hangy what do you think instead of hard coding making directory inside of existing directory of tests.

@JagjeevanAK I think it's better to create a script that you call and gives you back the test group, it should also take the number of group as parameter.

This script would list all tests, using file system) and separate into groups in a predictable manner (eg. after sorting the list), and print the list on stdout.

Then in makefile you can just call the script to get the test group.

The script might be written in python, as it's found on most machines today.

Please put the script in a sub folder, eg. scripts/tests/

Oh, in bash this is as simple as eg.:

find tests/unit -name "*.t"|sort|split -n 1/10 find tests/unit -name "*.t"|sort|split -n 2/10

alex problem is that some test are taking time so I have mixed it with small and big.

there are some e2e test like some what 2-3 if I am not wrong.

Copilot

Pull Request Overview

This PR enhances the CI setup by adding grouped, parallel execution for both unit and integration tests in the Makefile and the GitHub Actions workflow.

Introduce unit_test_group and integration_test_group targets in the Makefile to run test subsets in parallel.
Update .github/workflows/pull_request.yml to set up Docker Buildx, cache BuildKit, build the backend image, and use a matrix strategy for parallel test jobs.

Reviewed Changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File	Description
Makefile	Add grouped parallel test targets (`unit_test_group`, `integration_test_group`) and supporting variables.
.github/workflows/pull_request.yml	Configure Docker Buildx, caching layers, and matrix-based parallel unit/integration test jobs.

Comments suppressed due to low confidence (3)

.github/workflows/pull_request.yml:80

The hashFiles call includes single quotes around patterns, which can be interpreted literally. Remove the internal quotes so patterns are passed directly (e.g., hashFiles(Dockerfile*, docker-compose.yml, docker/**, cpanfile*, conf/apache*)).

          key: ${{ runner.os }}-docker-${{ hashFiles('Dockerfile*', 'docker-compose.yml', 'docker/**', 'cpanfile*', 'conf/apache*') }}

.github/workflows/pull_request.yml:21

[nitpick] The filter job is marked with a TODO and has no steps. If it’s not currently used, consider removing it to simplify the workflow.

  filter:

Makefile:611

[nitpick] The grouping logic spans many test files. Adding a brief explanation of the grouping strategy or moving the lists to a separate include file could improve readability and maintenance.

# Unit test groups for parallel execution in CI

Makefile

my mistake while testing it remained Co-authored-by: Copilot <[email protected]>

Signed-off-by: Jagjeevan Kashid <[email protected]>

fix increased no. of groups form 4 to6 Co-authored-by: Copilot <[email protected]>

Signed-off-by: Jagjeevan Kashid <[email protected]>

sonarqubecloud · 2025-06-21T15:38:29Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

JagjeevanAK · 2025-06-21T16:12:30Z

hey @alexgarel,

This PR is ready to merge.

john-gom · 2025-06-21T16:45:25Z

It would be good to see some kind of comparison with not using groups but just running the unit and integration tests as separate parallel tasks. That way we can see how much time the group approach is actually saving

JagjeevanAK · 2025-06-21T17:21:23Z

group approach is good according to me because every time we need to spin the resource for each test and that might take time.

alexgarel · 2025-06-23T07:33:10Z

Makefile

+# Unit test groups for parallel execution in CI
+# Usage: make unit_test_group TEST_GROUP=1
+# Groups are balanced by number of test files
+define UNIT_TEST_GROUPS


@JagjeevanAK I think it's better to create a script that you call and gives you back the test group, it should also take the number of group as parameter.

This script would list all tests, using file system) and separate into groups in a predictable manner (eg. after sorting the list), and print the list on stdout.

Then in makefile you can just call the script to get the test group.

The script might be written in python, as it's found on most machines today.

Please put the script in a sub folder, eg. scripts/tests/

alexgarel · 2025-06-23T07:34:19Z

Makefile

+	mkdir -p tests/unit/outputs/
+	${DOCKER_COMPOSE_TEST} up -d memcached postgres mongodb
+	@echo "🥫 Running all tests in group $(TEST_GROUP) with parallel execution and JUnit XML generation..."
+	${DOCKER_COMPOSE_TEST} run ${COVER_OPTS} -e JUNIT_TEST_FILE="tests/unit/outputs/junit_group_$(TEST_GROUP).xml" -T --rm backend yath test --renderer=Formatter --renderer=JUnit --job-count=${CPU_COUNT} $(addprefix tests/unit/,$(call get_unit_group_tests,$(TEST_GROUP)))


Don't we loose coverage with this new run type ?

according to me it should not and this is ordered this way because each test was running double previously.

alexgarel · 2025-06-23T07:40:41Z

Makefile

+# Unit test groups for parallel execution in CI
+# Usage: make unit_test_group TEST_GROUP=1
+# Groups are balanced by number of test files
+define UNIT_TEST_GROUPS


Oh, in bash this is as simple as eg.:

find tests/unit -name "*.t"|sort|split -n 1/10 find tests/unit -name "*.t"|sort|split -n 2/10

alexgarel · 2025-06-23T07:48:03Z

Makefile

@@ -289,10 +289,12 @@ unit_test: create_folders
 	@echo "🥫 Running unit tests …"
 	mkdir -p tests/unit/outputs/
 	${DOCKER_COMPOSE_TEST} up -d memcached postgres mongodb
+	@echo "🥫 Running tests with yath (parallel jobs: ${CPU_COUNT})"
 	${DOCKER_COMPOSE_TEST} run ${COVER_OPTS} -e JUNIT_TEST_FILE="tests/unit/outputs/junit.xml" -T --rm backend yath test --renderer=Formatter --renderer=JUnit --job-count=${CPU_COUNT} tests/unit


Here unit tests might run in parallel directly thanks to yath, without having to run them in different github runner. Is it that CPU_COUNT is too low in each runner ?

need to check

Testing gnu parllel

66c6fa2

Signed-off-by: Jagjeevan Kashid <[email protected]>

JagjeevanAK requested a review from a team as a code owner June 21, 2025 06:57

github-actions bot assigned JagjeevanAK Jun 21, 2025

teolemon added this to 🍊 Open Food Facts Server issues Jun 21, 2025

github-project-automation bot moved this to To discuss and validate in 🍊 Open Food Facts Server issues Jun 21, 2025

JagjeevanAK changed the title ~~CI: Testing gnu parllel for unit tests~~ ci: Testing gnu parllel for unit tests Jun 21, 2025

Testing gnu parllel and llel statergy in integration tests

68c4d1c

Signed-off-by: Jagjeevan Kashid <[email protected]>

github-actions bot added the GitHub Actions Pull requests that update Github_actions code label Jun 21, 2025

Testing gnu parllel and llel statergy in integration tests

60e6171

Signed-off-by: Jagjeevan Kashid <[email protected]>

github-actions bot added the 🐋 Docker https://docker-curriculum.com/ label Jun 21, 2025

Testing gnu parllel and llel statergy in integration tests

0d9ae04

Signed-off-by: Jagjeevan Kashid <[email protected]>

github-actions bot added 🧪 tests dependencies Pull requests that update a dependency file labels Jun 21, 2025

JagjeevanAK added 12 commits June 21, 2025 14:57

Testing gnu parllel and llel statergy in integration tests

d07d38a

Signed-off-by: Jagjeevan Kashid <[email protected]>

Testing gnu parllel and llel statergy in integration tests

34970df

Signed-off-by: Jagjeevan Kashid <[email protected]>

Testing gnu parllel and llel statergy in integration tests

efa8246

Signed-off-by: Jagjeevan Kashid <[email protected]>

Testing gnu parllel and llel statergy in integration tests

0ac66bb

Signed-off-by: Jagjeevan Kashid <[email protected]>

Testing gnu parllel and llel statergy in integration tests

7c9161c

Signed-off-by: Jagjeevan Kashid <[email protected]>

Testing gnu parllel and llel statergy in integration tests

e7f4546

Signed-off-by: Jagjeevan Kashid <[email protected]>

Testing gnu parllel and llel statergy in integration tests

f2efce3

Signed-off-by: Jagjeevan Kashid <[email protected]>

Testing gnu parllel and llel statergy in integration tests

9513739

Signed-off-by: Jagjeevan Kashid <[email protected]>

Testing gnu parllel and llel statergy in integration tests

d676fd5

Signed-off-by: Jagjeevan Kashid <[email protected]>

Testing gnu parllel and llel statergy in integration tests

2a80b7d

Signed-off-by: Jagjeevan Kashid <[email protected]>

minor fix

4fbdc8a

Signed-off-by: Jagjeevan Kashid <[email protected]>

minor fix

b62444c

Signed-off-by: Jagjeevan Kashid <[email protected]>

hangy reviewed Jun 21, 2025

View reviewed changes

.github/workflows/pull_request.yml Outdated Show resolved Hide resolved

minor fix

e1dd786

Signed-off-by: Jagjeevan Kashid <[email protected]>

hangy reviewed Jun 21, 2025

View reviewed changes

.github/workflows/pull_request.yml Outdated Show resolved Hide resolved

hangy reviewed Jun 21, 2025

View reviewed changes

.github/workflows/pull_request.yml Show resolved Hide resolved

hangy reviewed Jun 21, 2025

View reviewed changes

hangy requested review from a team and Copilot June 21, 2025 14:47

Copilot AI reviewed Jun 21, 2025

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

Makefile Show resolved Hide resolved

JagjeevanAK and others added 6 commits June 21, 2025 20:21

Update Makefile

978f60b

my mistake while testing it remained Co-authored-by: Copilot <[email protected]>

minor fix

851916e

Signed-off-by: Jagjeevan Kashid <[email protected]>

minor fix

4b332c5

Signed-off-by: Jagjeevan Kashid <[email protected]>

Update Makefile

87253b1

fix increased no. of groups form 4 to6 Co-authored-by: Copilot <[email protected]>

minor fix

befa7df

Signed-off-by: Jagjeevan Kashid <[email protected]>

minor fix

8266725

Signed-off-by: Jagjeevan Kashid <[email protected]>

alexgarel changed the title ~~ci: Testing gnu parllel for unit tests~~ ci: Split tests run in CI Jun 23, 2025

alexgarel reviewed Jun 23, 2025

View reviewed changes

github-actions bot added the 💥 Merge Conflicts 💥 Merge Conflicts label Jul 2, 2025

teolemon moved this from To discuss and validate to PRs in 🍊 Open Food Facts Server issues Jul 16, 2025

Uh oh!

ci: Split tests run in CI #12064

Are you sure you want to change the base?

ci: Split tests run in CI #12064

Uh oh!

Conversation

JagjeevanAK commented Jun 21, 2025

What

Screenshot

Related issue(s) and discussion

Uh oh!

codecov-commenter commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JagjeevanAK Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Jun 21, 2025

Quality Gate passed

Uh oh!

JagjeevanAK commented Jun 21, 2025

Uh oh!

john-gom commented Jun 21, 2025

Uh oh!

JagjeevanAK commented Jun 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov-commenter commented Jun 21, 2025 •

edited

Loading

JagjeevanAK Jun 21, 2025 •

edited

Loading