CLIENT-3852 tests against multi node cluster #536

gagan405 · 2025-11-07T22:27:24Z

Adds an action to create Aerospike cluster using Aerolab. Uses the 7.9.0 release of Aerolab, latest as of today.
A minimal set of tests are run against the cluster
Workflow is configured for nightly runs and manual
Minor fixes in the existing build yaml to properly print the workflow variables

Tested here: https://github.com/aerospike/aerospike-client-go/actions/runs/19178189525

.github/workflows/build-multi-node.yml

.github/actions/deploy-aerolab/action.yml

mirzakaracic

Not part of this PR but we should think about how we can tie CI flows with reusable actions and workflows in the future. It is also worth checking in with julien since he is working devops folks on runnable aerospike server action standard.

.github/workflows/build-multi-node.yml

mirzakaracic · 2025-11-13T18:05:17Z

None of the actions runs show these changes as valid. Do you have a run that shows

Aerolab cluster starting a mulitnode cluster?
Client connecting and running tests against multi node cluster?

Checkin in this PR are all against single node instance.

gagan405 · 2025-11-13T18:09:28Z

Successful run: https://github.com/aerospike/aerospike-client-go/actions/runs/19178189525
From the logs, it connects to 3100 - which is started from aerolab: 2025/11/07 18:54:51 aerospike_suite_test.go:110: Connecting to seeds: [127.0.0.1:3100]

gagan405 · 2025-11-13T18:11:03Z

Some manually triggered runs: https://github.com/aerospike/aerospike-client-go/actions/workflows/build-multi-node.yml

mirzakaracic · 2025-11-13T19:59:09Z

When you run tests against a containerized cluster you need to set use-services-alternate to true. The test invocation command does not do that currently.

go run -mod=vendor github.com/onsi/ginkgo/v2/ginkgo -coverprofile=./cover_native.out -covermode=atomic -coverpkg=./... -race -keep-going -succinct -randomize-suites -skip="HyperLogLog" -- -p=3100

See

aerospike-client-go/aerospike_suite_test.go

Line 53 in b794a56

    
           UseServicesAlternate = flag.Bool("use-services-alternate", false, "Will set ClientPolicy.UseServicesAlternate to true.")

.github/workflows/build-multi-node.yml

+    runs-on: ${{ github.event_name == 'workflow_dispatch' && inputs.runner_version || 'ubuntu-24.04' }}
+    outputs:
+      input-matrix: ${{ steps.create-server-matrix.outputs.matrix }}
+      go-version: ${{ steps.get-go-version.outputs.go-version }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Get Go version
+        id: get-go-version
+        run: |
+          echo "go-version=$(grep '^go [0-9]' go.mod | awk '{print $2}')" >> $GITHUB_OUTPUT
+
+      - id: create-server-matrix
+        uses: ./.github/actions/make-matrix
+        with:
+          default_json: |
+            {
+              "stable": { "version": "${{ github.event_name == 'workflow_dispatch' && inputs.aerospike_version || env.AEROSPIKE_VERSION }}" }
+            }
+          go_version: ${{ steps.get-go-version.outputs.go-version }}
+
+      - name: debug - print input variables
+        run: |
+          echo "Aerospike version: ${{ github.event_name == 'workflow_dispatch' && inputs.aerospike_version || env.AEROSPIKE_VERSION }}"
+          echo "Nodes: ${{ github.event_name == 'workflow_dispatch' && inputs.nodes || env.NODES }}"
+          echo "Cluster name: ${{ github.event_name == 'workflow_dispatch' && inputs.cluster_name || env.CLUSTER_NAME }}"
+          echo "Go version: ${{ steps.get-go-version.outputs.go-version }}"
+          echo 'Matrix output:'
+          echo '${{ steps.create-server-matrix.outputs.matrix }}' | jq .
+
+  build:


To fix the problem, we should set a minimal explicit permissions block in the workflow’s root section, granting only read access to repository contents, unless other permissions are also needed. The best change is to add:

permissions: contents: read

just below the workflow name, i.e., between lines 1 and 2. If later jobs require additional permissions (e.g., to write status or manipulate issues/PRs), these can be added, but for the current workflow (checkout, build, test) contents: read is sufficient. No additional code, imports, or dependencies are needed—this is a standard configuration for GitHub workflows.

.github/workflows/build-multi-node.yml

+    timeout-minutes: 30
+    needs: make-matrix
+    runs-on: ${{ github.event_name == 'workflow_dispatch' && inputs.runner_version || 'ubuntu-24.04' }}
+    strategy:
+      matrix:
+        include: ${{ fromJson(needs.make-matrix.outputs.input-matrix) }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Deploy Aerolab Cluster
+        uses: ./.github/actions/deploy-aerolab
+        with:
+          aerospike_version: ${{ github.event_name == 'workflow_dispatch' && inputs.aerospike_version || env.AEROSPIKE_VERSION }}
+          nodes: ${{ github.event_name == 'workflow_dispatch' && inputs.nodes || env.NODES }}
+          cluster_name: ${{ github.event_name == 'workflow_dispatch' && inputs.cluster_name || env.CLUSTER_NAME }}
+
+      - name: "Setup Go ${{ needs.make-matrix.outputs.go-version }}"
+        uses: actions/setup-go@v4
+        with:
+          go-version: "${{ needs.make-matrix.outputs.go-version }}"
+          cache: true
+      - name: Fetch dependencies
+        env:
+          GODEBUG: fips140=on
+        run: |
+          # Install all dependencies
+          go mod download
+
+          # Install ginkgo CLI and gocovmerge for testing
+          go install github.com/onsi/ginkgo/v2/[email protected]
+          go mod download github.com/wadey/gocovmerge
+
+          # Vendor the dependencies
+          go mod vendor
+
+      - name: Display Go version
+        run: |
+          echo "Go version: ${{ needs.make-matrix.outputs.go-version }}"
+      - name: Run the tests
+        env: {GOPROXY: off, GOSUMDB: off, GOFLAGS: -mod=vendor}
+        run: go run -mod=vendor github.com/onsi/ginkgo/v2/ginkgo \
+          -coverprofile=./cover_native.out -covermode=atomic -coverpkg=./... \
+          -race -keep-going -succinct -randomize-suites -skip="HyperLogLog" \
+          -- -p=3100 -use-services-alternate=true
+      - name: Combine Cover Profiles
+        env: {GOPROXY: off, GOSUMDB: off, GOFLAGS: -mod=vendor}
+        run: go run -mod=vendor github.com/wadey/gocovmerge cover_*.out > cover_all.out
+      - name: Check Code Coverage
+        uses: vladopajic/go-test-coverage@v2
+        with:
+          # Configure action using config file (option 1)
+          config: ./.testcoverage.yml


To fix this problem, a permissions block should be added to the workflow YAML file, either at the global level (just after the workflow name: and on: keys, before the jobs: block) or at each job, specifying the minimal set of permissions needed. Generally, for CI builds and tests where no repository contents are being modified, the minimal permissions should be set—usually contents: read. If jobs require higher permissions, those can be set at the job level. In the provided workflow, it appears only code is checked out and tested, so contents: read suffices. The best fix is to add:

permissions: contents: read

at the root of the YAML file, after the on: block and before env: or jobs:. No changes to imports, definitions, or other code are needed.

gagan405 · 2025-11-14T07:47:09Z

Successful run against a multi node cluster: https://github.com/aerospike/aerospike-client-go/actions/runs/19357434983
logs:

  go run -mod=vendor github.com/onsi/ginkgo/v2/ginkgo -output-dir=./ -coverprofile=cover_native.out -covermode=atomic -coverpkg=./... -race -keep-going -succinct -randomize-suites -skip="HyperLogLog" -- -hosts=127.0.0.1:3100,127.0.0.1:3101,127.0.0.1:3102 -use-services-alternate=true
  shell: /usr/bin/bash -e {0}
  env:
    AEROSPIKE_VERSION: 8.1.0.1c
    NODES: 3
    CLUSTER_NAME: ce
    RUNNER_VERSION: ubuntu-24.04
    AEROLAB_HOME: /tmp/aerolab-home
    GOPROXY: off
    GOSUMDB: off
    GOFLAGS: -mod=vendor
2025/11/14 07:26:04 aerospike_suite_test.go:110: Connecting to seeds: [127.0.0.1:3100 127.0.0.1:3101 127.0.0.1:3102]
Node: BB9BA7252636986, Host: 127.0.0.1, Port: 3101
Node: BB90505C1457A9E, Host: 127.0.0.1, Port: 3100
Node: BB9A93627FB1126, Host: 127.0.0.1, Port: 3102
[1763105127] Aerospike Client Library Suite - 767/783 specs •••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••SSSSSSSSSSSSSSS••••••••••••••••••••••••••••••••••••••••••••••••••••••••SSSSSSSSSSSSSSSS••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••SSSSSSSS••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••SS•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••SSS••••••••••••••S•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••SSSSSSSSSSSSSSSSSSSSSSSSS•••••• SUCCESS! 9m28.39496085s PASS
coverage: 51.8% of statements in ./...
composite coverage: 51.8% of statements

Ginkgo ran 1 suite in 10m8.34048618s
Test Suite Passed

gagan405 · 2025-11-14T07:50:16Z

connection_test.go

-			gm.Expect(expTotalDeadline).To(gm.BeTemporally("~", matrix.expTotalDeadline, time.Millisecond))
-			gm.Expect(expSocketDeadline).To(gm.BeTemporally("~", matrix.expSocketDeadline, time.Millisecond))
-			gm.Expect(expSocketTimeout).To(gm.BeNumerically("~", matrix.expSocketTimeout, time.Millisecond))
+			gm.Expect(expTotalDeadline).To(gm.BeTemporally("~", matrix.expTotalDeadline, 2*time.Millisecond))


This was occasionally failing. Increased it to 2ms to give it some space.

mirzakaracic

👍

gagan405 self-assigned this Nov 7, 2025

gagan405 added the enhancement label Nov 7, 2025

gagan405 requested a review from mirzakaracic November 7, 2025 22:27

github-advanced-security bot found potential problems Nov 7, 2025

View reviewed changes

.github/workflows/build-multi-node.yml Fixed Show fixed Hide fixed

.github/workflows/build-multi-node.yml Fixed Show fixed Hide fixed

mirzakaracic reviewed Nov 10, 2025

View reviewed changes

.github/workflows/build-multi-node.yml Outdated Show resolved Hide resolved

mirzakaracic reviewed Nov 10, 2025

View reviewed changes

.github/actions/deploy-aerolab/action.yml Outdated Show resolved Hide resolved

mirzakaracic reviewed Nov 10, 2025

View reviewed changes

github-advanced-security bot found potential problems Nov 11, 2025

View reviewed changes

.github/workflows/build-multi-node.yml Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Nov 13, 2025

View reviewed changes

.github/workflows/build-multi-node.yml Fixed Show fixed Hide fixed

.github/workflows/build-multi-node.yml Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Nov 13, 2025

View reviewed changes

gagan405 added 15 commits November 13, 2025 23:37

wip: add build for multi node

46ccd39

fix debug print

1ff7514

force run workflow on this branch

208945d

add aerolab action

4553ca7

add nightly runs

c8bfd64

pin runner version and add aerolab version to inputs

eba5fc8

enable use-services-alternate

3c3fd81

enable use service alternate in build script

b489f89

updated build script to remove hardcode

0294da0

fixed build script

82a0f84

fix run command

721daa1

fix tests

8c31f58

use community edition

bcc59c7

update coverprofile

ff98d7b

updated build workflow to remove feature branch

954c8ba

gagan405 force-pushed the CLIENT-3852-tests-against-multi-node-cluster branch from 12f7b03 to 954c8ba Compare November 14, 2025 07:38

gagan405 commented Nov 14, 2025

View reviewed changes

mirzakaracic approved these changes Nov 14, 2025

View reviewed changes

mirzakaracic merged commit 50d8825 into stage Nov 14, 2025
12 checks passed

@@ -1,4 +1,6 @@
             name: Build and test against multi-node cluster
+            permissions:
+              contents: read
             on:
               workflow_dispatch:

@@ -22,6 +22,8 @@
               schedule:
                 - cron: "0 2 * * *" # Nightly at 02:00 UTC
+            permissions:
+              contents: read
             env:
               # defaults to use against workflow triggered through cron
               AEROSPIKE_VERSION: 8.1.0.1c

CLIENT-3852 tests against multi node cluster #536

CLIENT-3852 tests against multi node cluster #536

Uh oh!

Conversation

gagan405 commented Nov 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mirzakaracic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mirzakaracic commented Nov 13, 2025

Uh oh!

gagan405 commented Nov 13, 2025

Uh oh!

gagan405 commented Nov 13, 2025

Uh oh!

mirzakaracic commented Nov 13, 2025

Uh oh!

Uh oh!

Uh oh!

Check warning

Copilot Autofix

Check warning

Copilot Autofix

gagan405 commented Nov 14, 2025

Uh oh!

gagan405 Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

mirzakaracic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants