[CI] Test packages basic subscription - WIP #13377

mrodm · 2025-04-01T17:51:23Z

Proposed commit message

Update CI scripts and tools (ReportFailedTests) to allow run Elastic stack with other subscriptions.
Currently, it is just supported "basic" and "trial".

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Test packages with "basic" subscription.
Ensure that packages with "enterprise" or "platinum" subscriptions are filtered out if "basic" is set.
Test in Dry-Run mode creation of scripts.
Remove changes for debugging in go.mod, go.sum and .buildkite/scripts/common.sh.

How to test this PR locally

# test some package
cd packages/<package>
elastic-package stack up -v -d -U stack.elastic_subscription=basic
elastic-package test -v
elastic-package stack down -v

# test creation of scripts
DRY_RUN=true CI_MAX_TESTS_REPORTED=1 BUILDKITE_BUILD_URL=https://buildkite.com/elastic/integrations/builds/20643 SERVERLESS=false STACK_VERSION=9.0.0-SNAPSHOT ELASTIC_SUBSCRIPTION=basic TESTSREPORTER_VERBOSE_MODE_ENABLED=false  mage -v ReportFailedTests build/test-results

Related issues

Relates Allow to run Elastic stack with other licenses elastic-package#2501

Verbose mode allows us to show the full description of the issue that it is going to be created or updated.

elastic-vault-github-plugin-prod · 2025-04-01T18:16:12Z

🚀 Benchmarks report

Package `o365_metrics` 👍(9) 💚(5) 💔(7)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`outlook_activity`	12658.23	7633.59	-5024.64 (-39.69%)	💔
`service_health`	83333.33	55555.56	-27777.77 (-33.33%)	💔
`sharepoint_site_usage_detail`	6578.95	4149.38	-2429.57 (-36.93%)	💔
`teams_user_activity_user_counts`	7692.31	6134.97	-1557.34 (-20.25%)	💔
`viva_engage_device_usage_user_counts`	9900.99	3759.4	-6141.59 (-62.03%)	💔
`onedrive_usage_file_counts`	11111.11	8849.56	-2261.55 (-20.35%)	💔
`onedrive_usage_storage`	19607.84	16393.44	-3214.4 (-16.39%)	💔

To see the full report comment with /test benchmark fullreport

mrodm · 2025-04-01T18:24:45Z

dev/testsreporter/testsreporter.go

@@ -137,7 +145,7 @@ func errorsFromTests(resultsPath string, options CheckOptions) ([]*packageError,
 }

 // packagesFromTests returns the sorted packages failing given the results file
-func packagesFromTests(resultsPath string, options CheckOptions) ([]string, error) {
+func packagesFromTests(resultsPath string) ([]string, error) {


Removed unused parameter

mrodm · 2025-04-01T18:28:52Z

dev/testsreporter/builderror.go

-		"owners":            p.teams,
-	}
+func (b *buildError) SummaryData() map[string]any {
+	data := b.dataError.Data()


Moved the creation of the common data from dataError struct to its own method.

mrodm · 2025-04-01T18:29:18Z

dev/testsreporter/builderror.go

-		"previousBuilds": p.errorLinks.previousBuilds,
-	}
+func (b *buildError) DescriptionData() map[string]any {
+	return b.errorLinks.Data()


Same as in dataError, moved the creation of the common data from errorLinks struct to its own method.

mrodm · 2025-04-01T18:30:30Z

dev/testsreporter/packageerror.go

-		sb.WriteString(p.stackVersion)
-		sb.WriteString("] ")
-	}
+	sb.WriteString(p.dataError.String())


String built from the common data (dataError) is now built in just one place, as a method of the given struct.

mrodm · 2025-04-01T18:31:26Z

dev/testsreporter/reporter.go

-		maxPreviousLinks: maxPreviousLinks,
+		ghCli:            options.GhCli,
+		maxPreviousLinks: options.MaxPreviousLinks,
+		verbose:          options.Verbose,


Added a new verbose parameter that it can be enabled via the TESTSREPORTER_VERBOSE_MODE_ENABLED environment variable.

Currently, this will show the full description of the issue that it is going to be created or updated.

mrodm · 2025-04-01T18:32:00Z

.buildkite/scripts/common.sh

@@ -757,7 +809,7 @@ teardown_test_package() {
 }

 list_all_directories() {
-    find . -maxdepth 1 -mindepth 1 -type d | xargs -I {} basename {} | sort
+    find . -maxdepth 1 -mindepth 1 -type d | xargs -I {} basename {} | sort |grep -E '^elastic_package_registry|cloud_defend|beaconing$'


To be removed.

mrodm · 2025-04-02T08:47:52Z

dev/testsreporter/_static/summary.tmpl

@@ -20,7 +23,7 @@
    - {{ . }}
 {{- end }}
 {{ end -}}
-{{ if ne (len .owners) 0 -}}
+{{ if and ( ne .owners nil) (ne (len .owners) 0) -}}


Avoid errors if owners list is not defined.

mrodm · 2025-04-03T09:35:51Z

.buildkite/pipeline.schedule-daily.yml

+  - label: "Check integrations local stacks and basic subscription"
+    trigger: "integrations"
+    build:
+      env:
+        SERVERLESS: "false"
+        FORCE_CHECK_ALL: "true"
+        PUBLISH_COVERAGE_REPORTS: "false"
+        ELASTIC_SUBSCRIPTION: "basic"
+    depends_on:
+      - step: "check"
+        allow_failure: false
+
+  - label: "Check integrations local stacks and basic subscription - LogsDB"
+    trigger: "integrations"
+    build:
+      env:
+        SERVERLESS: "false"
+        FORCE_CHECK_ALL: "true"
+        STACK_LOGSDB_ENABLED: "true"
+        PUBLISH_COVERAGE_REPORTS: "false"
+        ELASTIC_SUBSCRIPTION: "basic"
+    depends_on:
+      - step: "check"
+        allow_failure: false


mrodm · 2025-04-03T09:37:20Z

Split this PR into two different PRs:

elasticmachine · 2025-04-04T15:53:55Z

💔 Build Failed

Buildkite Build
Commit: 067f900

Failed CI Steps

History

💚 Build #24344 succeeded 28e6578
💚 Build #24305 succeeded 5522c3d
💔 Build #24303 failed d1a7db4
💔 Build #24264 failed ce0b4a7

elastic-sonarqube · 2025-04-04T17:13:18Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

mrodm added 6 commits April 1, 2025 16:22

Add support to run Elastic stack with other licenses

9679c7c

Add support to report issues related to subscription

6e586b7

Refactor to build common summary and string from dataError

495da16

Refactor to buld common error links data from its object

e74ae7e

Add option to enable verbose mode

081ffd3

Verbose mode allows us to show the full description of the issue that it is going to be created or updated.

Test setting subscription basic - to be removed

969e38b

mrodm commented Apr 1, 2025

View reviewed changes

Rename function

cf0da7b

mrodm commented Apr 1, 2025

View reviewed changes

Check first if owners is not nil in summary template

099dea6

mrodm commented Apr 2, 2025

View reviewed changes

mrodm added 13 commits April 2, 2025 11:46

Test with trial subscription

2247914

Test with other packages with other scenarios

d7e479c

Test again with basic subscription

b37ac87

Add target mage for subscription validation

3c114bd

Add some more packages to test

0ee660e

Test with bash functions

51bbb78

Try with mage targets - added missing installation

5d03168

Test all packages with basic license

1dc0401

Not shown files from git-diff checks

69edc0a

Show annotation with skipped files

bee404b

Fix collapsed annotation

7252015

Test annotation

571eca9

Just show skipped packages

ce0b4a7

mrodm mentioned this pull request Apr 3, 2025

[CI] Refactor testsreporter tool #13411

Open

2 tasks

mrodm mentioned this pull request Apr 3, 2025

[CI] Update CI scripts #13410

Open

4 tasks

mrodm commented Apr 3, 2025

View reviewed changes

mrodm mentioned this pull request Apr 3, 2025

Add a new validation to use or not upload API to install packages elastic/elastic-package#2511

Open

mrodm added 4 commits April 3, 2025 19:15

Test elastic-package PR 2511 - 62704d87

d1a7db4

Update version in .go-version file

5522c3d

Revert changes moved to other PRs

28e6578

Test all packages with basic license

067f900

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Test packages basic subscription - WIP #13377

[CI] Test packages basic subscription - WIP #13377

mrodm commented Apr 1, 2025

elastic-vault-github-plugin-prod bot commented Apr 1, 2025 •

edited

Loading

mrodm Apr 1, 2025

mrodm Apr 1, 2025

mrodm Apr 1, 2025

mrodm Apr 1, 2025

mrodm Apr 1, 2025

mrodm Apr 1, 2025

mrodm Apr 2, 2025

mrodm Apr 3, 2025

mrodm commented Apr 3, 2025

elasticmachine commented Apr 4, 2025 •

edited

Loading

elastic-sonarqube bot commented Apr 4, 2025

[CI] Test packages basic subscription - WIP #13377

Are you sure you want to change the base?

[CI] Test packages basic subscription - WIP #13377

Conversation

mrodm commented Apr 1, 2025

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

elastic-vault-github-plugin-prod bot commented Apr 1, 2025 • edited Loading

🚀 Benchmarks report

Package o365_metrics 👍(9) 💚(5) 💔(7)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrodm commented Apr 3, 2025

elasticmachine commented Apr 4, 2025 • edited Loading

💔 Build Failed

Failed CI Steps

History

elastic-sonarqube bot commented Apr 4, 2025

Quality Gate passed

elastic-vault-github-plugin-prod bot commented Apr 1, 2025 •

edited

Loading

Package `o365_metrics` 👍(9) 💚(5) 💔(7)

elasticmachine commented Apr 4, 2025 •

edited

Loading