Skip to content

[scout] support flaky test runner #234918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Sep 18, 2025
Merged

[scout] support flaky test runner #234918

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8b804a0
[scout] extend script and flaky-test-runner pipeline to support scout
dmlemeshko Sep 12, 2025
3804588
test SCOUT_CONFIG var in scout script
dmlemeshko Sep 12, 2025
546b62b
update the script
dmlemeshko Sep 12, 2025
dcc4222
Revert "test SCOUT_CONFIG var in scout script"
dmlemeshko Sep 12, 2025
fd77631
Merge branch 'main' into scout/support-flaky-test-runner
dmlemeshko Sep 15, 2025
7acb6e7
Merge branch 'main' into scout/support-flaky-test-runner
dmlemeshko Sep 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions .buildkite/pipelines/flaky_tests/pipeline.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,19 @@ if (Number.isNaN(concurrency)) {
const BASE_JOBS = 1;
const MAX_JOBS = 500;

function getScoutConfigGroupType(configPath: string): string | null {
// Match platform paths: x-pack/platform/... or src/platform/...
if (/^(x-pack|src)\/platform\//.test(configPath)) {
return 'platform';
}
// Match solution paths: x-pack/solutions/<solution>/plugins/...
const match = configPath.match(/^x-pack\/solutions\/([^/]+)\/plugins\//);
if (match) {
return match[1];
}
return null;
}

function getTestSuitesFromJson(json: string) {
const fail = (errorMsg: string) => {
console.error('+++ Invalid test config provided');
Expand All @@ -55,6 +68,7 @@ function getTestSuitesFromJson(json: string) {
const testSuites: Array<
| { type: 'group'; key: string; count: number }
| { type: 'ftrConfig'; ftrConfig: string; count: number }
| { type: 'scoutConfig'; scoutConfig: string; count: number }
Copy link
Member Author

@dmlemeshko dmlemeshko Sep 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When Scout config is selected in Flaky test runner, server should return in json:

{
  type: 'scoutConfig',
  title: 'Scout: path/to/config.ts',
  id: 'scout:path/to/config.ts',
  scoutConfig: 'path/to/config.ts',
  count: 15
}

> = [];
for (const item of parsed) {
if (typeof item !== 'object' || item === null) {
Expand Down Expand Up @@ -160,6 +174,33 @@ for (const testSuite of testSuites) {
continue;
}

if (testSuite.type === 'scoutConfig') {
const usesParallelWorkers = testSuite.scoutConfig.endsWith('parallel.playwright.config.ts');
const scoutConfigGroupType = getScoutConfigGroupType(testSuite.scoutConfig);

steps.push({
command: `.buildkite/scripts/steps/test/scout_configs.sh`,
env: {
SCOUT_CONFIG: testSuite.scoutConfig,
Copy link
Member Author

@dmlemeshko dmlemeshko Sep 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if SCOUT_CONFIG is set, running script will run explicitly defined config (Flaky test runner support) instead of fetching the json artifact and executing all the configs for the plugin (normal flow)

SCOUT_CONFIG_GROUP_TYPE: scoutConfigGroupType!,
},
key: `scout-suite-${suiteIndex++}`,
label: `${testSuite.scoutConfig}`,
parallelism: testSuite.count,
concurrency,
concurrency_group: process.env.UUID,
concurrency_method: 'eager',
agents: expandAgentQueue(usesParallelWorkers ? 'n2-8-spot' : 'n2-4-spot'),
depends_on: 'build',
timeout_in_minutes: 30,
cancel_on_build_failing: true,
retry: {
automatic: [{ exit_status: '-1', limit: 3 }],
},
});
continue;
}

const [category, suiteName] = testSuite.key.split('/');
switch (category) {
case 'cypress':
Expand Down
39 changes: 23 additions & 16 deletions .buildkite/scripts/steps/test/scout_configs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,44 +7,51 @@ source .buildkite/scripts/steps/functional/common.sh
BUILDKITE_PARALLEL_JOB=${BUILDKITE_PARALLEL_JOB:-}
SCOUT_CONFIG_GROUP_KEY=${SCOUT_CONFIG_GROUP_KEY:-}
SCOUT_CONFIG_GROUP_TYPE=${SCOUT_CONFIG_GROUP_TYPE:-}
SCOUT_CONFIG=${SCOUT_CONFIG:-}

if [ "$SCOUT_CONFIG_GROUP_KEY" == "" ] && [ "$BUILDKITE_PARALLEL_JOB" == "" ]; then
echo "Missing SCOUT_CONFIG_GROUP_KEY env var"
exit 1
fi
EXTRA_ARGS=${FTR_EXTRA_ARGS:-}
test -z "$EXTRA_ARGS" || buildkite-agent meta-data set "ftr-extra-args" "$EXTRA_ARGS"

configs=""

if [ "$SCOUT_CONFIG_GROUP_TYPE" == "" ]; then
echo "Missing SCOUT_CONFIG_GROUP_TYPE env var"
exit 1
fi

EXTRA_ARGS=${FTR_EXTRA_ARGS:-}
test -z "$EXTRA_ARGS" || buildkite-agent meta-data set "ftr-extra-args" "$EXTRA_ARGS"

export JOB="$SCOUT_CONFIG_GROUP_KEY"

FAILED_CONFIGS_KEY="${BUILDKITE_STEP_ID}${SCOUT_CONFIG_GROUP_KEY}"

configs=""
group=$SCOUT_CONFIG_GROUP_TYPE

if [ "$SCOUT_CONFIG" != "" ]; then
configs="$SCOUT_CONFIG"
export JOB="$SCOUT_CONFIG"
FAILED_CONFIGS_KEY="${BUILDKITE_STEP_ID}${SCOUT_CONFIG}"
elif [ "$SCOUT_CONFIG_GROUP_KEY" != "" ]; then
export JOB="$SCOUT_CONFIG_GROUP_KEY"
FAILED_CONFIGS_KEY="${BUILDKITE_STEP_ID}${SCOUT_CONFIG_GROUP_KEY}"
else
if [ "$BUILDKITE_PARALLEL_JOB" == "" ]; then
echo "Missing SCOUT_CONFIG_GROUP_KEY or SCOUT_CONFIG env var"
exit 1
fi
fi

# The first retry should only run the configs that failed in the previous attempt
# Any subsequent retries, which would generally only happen by someone clicking the button in the UI, will run everything
if [[ ! "$configs" && "${BUILDKITE_RETRY_COUNT:-0}" == "1" ]]; then
if [[ -z "$configs" && "${BUILDKITE_RETRY_COUNT:-0}" == "1" ]]; then
configs=$(buildkite-agent meta-data get "$FAILED_CONFIGS_KEY" --default '')
if [[ "$configs" ]]; then
if [[ -n "$configs" ]]; then
echo "--- Retrying only failed configs"
echo "$configs"
fi
fi

if [ "$configs" == "" ] && [ "$SCOUT_CONFIG_GROUP_KEY" != "" ]; then
if [ -z "$configs" ] && [ "$SCOUT_CONFIG_GROUP_KEY" != "" ]; then
echo "--- downloading scout test configuration"
download_artifact scout_playwright_configs.json .
configs=$(jq -r '.[env.SCOUT_CONFIG_GROUP_KEY].configs[]' scout_playwright_configs.json)
fi

if [ "$configs" == "" ]; then
if [ -z "$configs" ]; then
echo "unable to determine configs to run"
exit 1
fi
Expand Down