feat(e2e): add tests for GitHub plugins (PR statistics & security insights) #2998

guyoron1 · 2025-06-17T08:15:03Z

Description

This PR adds end-to-end tests for two GitHub-related plugins in RHDH:

GitHub Pull Requests Statistics plugin
GitHub Security Insights plugin

Key changes:

Created new test files:
- e2e-tests/playwright/e2e/plugins/github-pull-requests.spec.ts: Tests PR statistics functionality including metrics like average PR time, merge ratio, etc.
- e2e-tests/playwright/e2e/plugins/github-security-insights.spec.ts: Tests security insights functionality including Dependabot alerts severity levels
Enhanced common.ts with improved GitHub authentication:
- Added githubLoginPopUpModal function to handle GitHub login in a popup context
- Implemented robust 2FA handling
- Prevents context switching issues by maintaining the popup state

The new authentication approach improves test reliability by handling the GitHub login flow in a popup modal without navigating away from the main application context.

Which issue(s) does this PR fix

Adds test coverage for GitHub plugins (no specific issue)

PR acceptance criteria

GitHub Actions are completed and successful
E2E Tests are updated and passing
- Added new E2E tests for GitHub PR statistics plugin
- Added new E2E tests for GitHub Security Insights plugin
- All tests pass locally with proper environment variables set
Documentation is updated if necessary (not required - test additions only)

How to test changes / Special notes to the reviewer

To run the tests locally:

Set required environment variables:

export BASE_URL="your-rhdh-url"
export GH_USER_ID="your-github-username"
export GH_USER_PASS="your-github-password"
export GH_2FA_SECRET="your-github-2fa-secret"

Run specific test files:

cd e2e-tests
npx playwright test playwright/e2e/plugins/github-pull-requests.spec.ts --project=showcase --headed
npx playwright test playwright/e2e/plugins/github-security-insights.spec.ts --project=showcase --headed

The tests verify:

GitHub PR Statistics:
- Average Time of PR Until Merge
- Merged To Closed Ratio
- Average Size of PR
- Average Changed Files of PR
- Average Coding Time of PR
Security Insights:
- Critical severity alerts
- High severity alerts
- Medium severity alerts
- Low severity alerts

openshift-ci · 2025-06-17T08:15:15Z

Hi @guyoron1. Thanks for your PR.

I'm waiting for a redhat-developer member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

github-actions · 2025-06-17T08:53:03Z

The image is available at:

gustavolira · 2025-06-17T12:36:18Z

/ok-to-test

gustavolira · 2025-06-17T12:44:12Z

my_changes.diff

@@ -0,0 +1,66 @@
+diff --git a/.ibm/pipelines/value_files/values_showcase.yaml b/.ibm/pipelines/value_files/values_showcase.yaml


Why did you commit this file?

gustavolira · 2025-06-17T13:11:39Z

e2e-tests/playwright/utils/common.ts

+    await githubPage.fill('input[name="login"]', username);
+    await githubPage.fill('input[name="password"]', password);


I think you can reuse uiHelper.fillTextInputByLabel for all those page.fill

gustavolira · 2025-06-17T13:15:02Z

e2e-tests/playwright/e2e/plugins/github-pull-requests.spec.ts

+  await uiHelper.openSidebar("Catalog");
+  // Step 3: Search and click 'Backstage Showcase'
+  await page.fill('input[placeholder="Search"]', repoName);
+  await page.waitForSelector('a:has-text("Backstage Showcase")', { timeout: 20000 });


await catalog.goToByName("Backstage Showcase");

export class CatalogHelper {
private page: Page;
private uiHelper: UIhelper;

constructor(page: Page) {
this.page = page;
this.uiHelper = new UIhelper(page);
}

async goToByName(name: string): Promise {
await this.uiHelper.openSidebar("Catalog");
await this.page.fill('input[placeholder="Search"]', name);
await this.page.waitForSelector(a:has-text("${name}"), { timeout: 20000 });
await this.uiHelper.clickLink(name);
}
}

and then it would be easier to maintain and much clearer

gustavolira · 2025-06-17T13:15:17Z

e2e-tests/playwright/e2e/plugins/github-pull-requests.spec.ts

+  await page.fill('input[placeholder="Search"]', repoName);
+  await page.waitForSelector('a:has-text("Backstage Showcase")', { timeout: 20000 });
+  await uiHelper.clickLink(repoName);
+  await page.waitForLoadState("networkidle");


don't use await page.waitForLoadState("networkidle");

gustavolira · 2025-06-17T13:16:55Z

e2e-tests/playwright/e2e/plugins/github-pull-requests.spec.ts

+  // Step 5: Click 'Sign in' inside the PR statistics card
+  await uiHelper.clickBtnInCard('GitHub Pull Requests Statistics', 'Sign in', true);
+  // Wait for the login modal to appear
+  const modalLoginButton = page.locator('button:has-text("Log in")');


uiHelper.isBtnVisible
uiHelper.getButtonSelector

gustavolira · 2025-06-17T13:21:37Z

e2e-tests/playwright/utils/common.ts

+    ]);
+    await githubPage.waitForLoadState();
+    await githubPage.waitForSelector('input[name="login"]', { timeout: 30000 });
+    await githubPage.fill('input[name="login"]', username);


uiHelper.fillTextInputByLabel

gustavolira · 2025-06-17T13:23:26Z

e2e-tests/playwright/utils/common.ts

+
+    // Handle 2FA
+    let otpSelector = null;
+    if (await githubPage.isVisible('input[name="otp"]', { timeout: 10000 }).catch(() => false)) {


maybe you can use uiHelper.isElementVisible

gustavolira · 2025-06-17T13:28:12Z

e2e-tests/playwright/utils/common.ts

+
+  async GetChildGroupsDisplayed(): Promise<string[]> {
+    await this.page.waitForSelector("p:has-text('Child Groups')");
+    const parent = await this.page


private async getGroupLinksByLabel(label: string): Promise<string[]> { const section = this.page.locator(`p:has-text("${label}")`).first(); await section.waitFor({ state: "visible" }); return await section.locator("..").locator("a").allInnerTexts(); } async getParentGroupsDisplayed(): Promise<string[]> { return await this.getGroupLinksByLabel("Parent Group"); } async getChildGroupsDisplayed(): Promise<string[]> { return await this.getGroupLinksByLabel("Child Groups"); }

gustavolira · 2025-06-17T13:29:38Z

e2e-tests/playwright/utils/common.ts

+      .filter((u) => u.spec.profile && u.spec.profile.displayName)
+      .map((u) => u.spec.profile.displayName);
+    LOGGER.info(
+      `Checking ${JSON.stringify(catalogUsersDisplayNames)} contains users ${JSON.stringify(users)}`,


type EntityWithDisplay = { spec?: { profile?: { displayName?: string } } }; export class CatalogVerifier { private api: APIHelper; constructor(token: string) { this.api = new APIHelper(); this.api.UseStaticToken(token); } async assertUsersInCatalog(expected: string[]): Promise<void> { await this.assertEntities( () => this.api.getAllCatalogUsersFromAPI(), expected, 'users', ); } async assertGroupsInCatalog(expected: string[]): Promise<void> { await this.assertEntities( () => this.api.getAllCatalogGroupsFromAPI(), expected, 'groups', ); } private async assertEntities( fetch: () => Promise<{ items?: EntityWithDisplay[] }>, expected: string[], label: string, ): Promise<void> { const { items = [] } = await fetch(); expect(items.length).toBeGreaterThan(0); const displayNames = items .flatMap(e => e.spec?.profile?.displayName ?? []); const catalogSet = new Set(displayNames); const missing = expected.filter(name => !catalogSet.has(name)); LOGGER.info( `Catalog ${label}: [${displayNames.join(', ')}] – expecting [${expected.join(', ')}]`, ); expect(missing, `Missing ${label}: ${missing.join(', ')}`).toHaveLength(0); } }

gustavolira · 2025-06-17T13:31:40Z

e2e-tests/playwright/utils/common.ts

-    const secret = secrets[userid];
-    if (!secret) {
-      throw new Error("Invalid User ID");
+  async githubLoginPopUpModal(context, username, password, otpSecret) {


async function githubLoginPopUpModal(
context: BrowserContext,
username: string,
password: string,
otpSecret: string
): Promise {
const [githubPage] = await Promise.all([
context.waitForEvent('page'),
]);

await githubPage.waitForLoadState('domcontentloaded');
await githubPage.fill('input[name="login"]', username);
await githubPage.fill('input[name="password"]', password);
await githubPage.click('button[type="submit"], input[type="submit"]');

const otpSelector = await findOtpSelector(githubPage);

if (otpSelector) {
if (githubPage.isClosed()) return;

await githubPage.waitForSelector(otpSelector, { timeout: 10000 }); if (githubPage.isClosed()) return; const otp = authenticator.generate(otpSecret); await githubPage.fill(otpSelector, otp); if (githubPage.isClosed()) return; await Promise.race([ githubPage.waitForEvent('close', { timeout: 20000 }), githubPage.click('button[type="submit"], input[type="submit"]'), ]);

} else {
await githubPage.waitForEvent('close', { timeout: 20000 });
}
}

async function findOtpSelector(page: Page): Promise<string | null> {
const selectors = ['input[name="otp"]', '#app_totp'];

for (const selector of selectors) {
try {
if (await page.isVisible(selector, { timeout: 5000 })) {
return selector;
}
} catch {
// Ignore visibility timeout errors
}
}
return null;
}

openshift-ci · 2025-06-17T17:42:39Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign gustavolira for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

.ibm/OWNERS
e2e-tests/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gustavolira · 2025-06-18T13:16:16Z

.ibm/pipelines/value_files/values_showcase.yaml

@@ -213,6 +213,9 @@ global:
        disabled: false
      - package: ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-global-floating-action-button
        disabled: false
+      # Enable Security Insights plugin


nit: Please remove this comment. It's clear that it's enabling security insights plugin based on the name of the imported plugin name

gustavolira · 2025-06-18T13:18:28Z

e2e-tests/package.json

    "prettier": "3.5.3",
    "typescript": "5.8.3"
  },
  "dependencies": {
    "@azure/identity": "4.9.1",
+    "@backstage/catalog-model": "^1.7.4",


I think we don't need to import this library. You are importing this lib only to type a field as far as I could see

gustavolira · 2025-06-18T13:45:06Z

e2e-tests/playwright/e2e/plugins/github-security-insights.spec.ts

+
+  await common.loginAsKeycloakUser(process.env.GH_USER_ID, process.env.GH_USER_PASS);
+  await uiHelper.openSidebar("Catalog");
+  await page.fill('input[placeholder="Search"]', repoName);


you can use uiHelper.searchInputPlaceholder. Please replace all occurrences with it.

gustavolira · 2025-06-18T13:47:23Z

e2e-tests/playwright/utils/common.ts

+    await githubPage.waitForLoadState('domcontentloaded');
+
+    await githubPage.getByLabel('Username or email address').fill(username);
+    await githubPage.getByLabel('Password').fill(password);


uiHelper.fillTextInputByLabel replace all occurrences with it

gustavolira · 2025-06-18T13:47:47Z

e2e-tests/playwright/utils/common.ts

+
+    await githubPage.getByLabel('Username or email address').fill(username);
+    await githubPage.getByLabel('Password').fill(password);
+    await githubPage.click('button[type="submit"], input[type="submit"]');


uiHelper.clickButton replace all

Please update this one

github-actions · 2025-06-19T08:49:39Z

The image is available at:

gustavolira · 2025-06-23T14:42:07Z

e2e-tests/playwright/utils/common.ts

+    const [githubPage] = await Promise.all([
+      context.waitForEvent('page'),
+    ]);
+    await githubPage.waitForLoadState('domcontentloaded');


it's better to use specific locators instead of load state.
for example: await page.waitForSelector('text=something');

gustavolira · 2025-06-23T14:42:28Z

e2e-tests/playwright/utils/common.ts

+
+    await githubPage.getByLabel('Username or email address').fill(username);
+    await githubPage.getByLabel('Password').fill(password);
+    await githubPage.click('button[type="submit"], input[type="submit"]');


Please update this one

gustavolira · 2025-06-23T14:42:50Z

e2e-tests/playwright/utils/common.ts

+      if (githubPage.isClosed()) return;
+      await Promise.race([
+        githubPage.waitForEvent('close', { timeout: 20000 }),
+        githubPage.click('button[type="submit"], input[type="submit"]'),


uiHelper.clickButton

gustavolira · 2025-06-23T14:52:35Z

e2e-tests/playwright/utils/common.ts

+    for (const selector of selectors) {
+      try {
+        if (await page.isVisible(selector, { timeout: 5000 })) {
+          return selector;


The silent catch could lead to false positives. If a timeout or other issue occurs, it gets ignored and the function may return null even when something went wrong. Might be worth adding logging or refining the error handling.

/** * Looks for an OTP input field and returns its selector. * Throws if no field becomes visible within the overall timeout. */ async function findOtpSelector(page: Page): Promise<string> { const selectors = ['input[name="otp"]', '#app_totp']; for (const sel of selectors) { try { // Wait until the element is visible (up to 10 s) await page.locator(sel).waitFor({ state: 'visible', timeout: 10_000 }); return sel; // Found a visible OTP field } catch (err) { // Timeout for this selector — log and try the next one console.debug(`Selector ${sel} not visible yet, continuing…`); } } // None of the selectors became visible in time throw new Error('OTP field not found on the page within 10 s'); }

github-actions · 2025-06-23T15:09:52Z

The image is available at:

github-actions · 2025-06-24T11:08:56Z

The image is available at:

gustavolira · 2025-06-26T11:40:52Z

/test e2e-tests

github-actions · 2025-06-30T12:51:41Z

The image is available at:

gustavolira · 2025-06-30T13:19:01Z

/ok-to-test

gustavolira · 2025-06-30T21:49:34Z

/test e2e-tests

gustavolira · 2025-07-09T14:53:38Z

/test e2e-tests

github-actions · 2025-07-10T11:47:11Z

The image is available at:

gustavolira · 2025-07-10T13:01:09Z

/test e2e-tests

github-actions · 2025-07-13T16:24:35Z

The image is available at:

gustavolira · 2025-07-14T12:53:45Z

/test e2e-tests

…lugins - Created Playwright spec files for GitHub PR statistics and security insights plugins - Enabled the plugins in values_showcase.yaml - Added new GitHub sign-in logic using popup modal

…ectly

…login, address review feedback

…ndencies

…devDependencies

…ling

…omments: use only UIhelper methods for Backstage UI, robust popup handling, CatalogVerifier, and code clarity

…t reliability

…review

github-actions · 2025-07-15T12:35:37Z

The image is available at:

….json

github-actions · 2025-07-16T11:59:02Z

The image is available at:

openshift-ci · 2025-07-16T12:00:09Z

@guyoron1: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-tests-sealight-helm-nightly	`e47da43`	link	false	`/test e2e-tests-sealight-helm-nightly`
ci/prow/e2e-tests	`2c59fdc`	link	true	`/test e2e-tests`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot requested review from dzemanov and josephca June 17, 2025 08:15

openshift-ci bot added the needs-ok-to-test label Jun 17, 2025

openshift-ci bot added ok-to-test and removed needs-ok-to-test labels Jun 17, 2025

gustavolira requested changes Jun 17, 2025

View reviewed changes

gustavolira requested changes Jun 18, 2025

View reviewed changes

openshift-merge-robot added the needs-rebase label Jun 23, 2025

guyoron1 force-pushed the pr-statistics-security-insights-branch branch from 1925ab6 to dcc2cf2 Compare June 23, 2025 13:44

openshift-merge-robot removed the needs-rebase label Jun 23, 2025

guyoron1 changed the title ~~Pr statistics security insights branch~~ feat(e2e): add tests for github plugins - pr statistics and security insights Jun 23, 2025

gustavolira requested changes Jun 23, 2025

View reviewed changes

openshift-merge-robot added the needs-rebase label Jun 30, 2025

guyoron1 force-pushed the pr-statistics-security-insights-branch branch from 125e042 to 82bc0bf Compare June 30, 2025 12:13

openshift-merge-robot removed the needs-rebase label Jun 30, 2025

guyoron1 changed the title ~~feat(e2e): add tests for github plugins - pr statistics and security insights~~ feat(e2e): add tests for GitHub plugins (PR statistics & security insights) Jun 30, 2025

openshift-merge-robot added the needs-rebase label Jun 30, 2025

guyoron1 force-pushed the pr-statistics-security-insights-branch branch from 82bc0bf to f7c8913 Compare July 1, 2025 06:09

openshift-merge-robot removed the needs-rebase label Jul 1, 2025

openshift-merge-robot added the needs-rebase label Jul 9, 2025

github-actions bot removed the Stale label Jul 10, 2025

guyoron1 force-pushed the pr-statistics-security-insights-branch branch from f504e33 to 940639a Compare July 10, 2025 11:07

openshift-merge-robot removed the needs-rebase label Jul 10, 2025

guyoron1 force-pushed the pr-statistics-security-insights-branch branch from 940639a to f6f4286 Compare July 13, 2025 15:43

guyoron1 force-pushed the pr-statistics-security-insights-branch branch from 42f4517 to f6f4286 Compare July 15, 2025 11:45

guyoron1 added 14 commits July 15, 2025 14:55

feat(e2e): add tests for GitHub PR statistics and security insights p…

794d2ea

…lugins - Created Playwright spec files for GitHub PR statistics and security insights plugins - Enabled the plugins in values_showcase.yaml - Added new GitHub sign-in logic using popup modal

fix: update GitHub login flow in plugin tests to use popup modal corr…

4cf2145

…ectly

chore: remove my_changes.diff from PR

7643298

refactor(e2e): use CatalogHelper and CatalogVerifier, improve GitHub …

0ee3a9e

…login, address review feedback

chore(e2e): update test specs, remove compliance.spec.ts, update depe…

870de73

…ndencies

Address review comments: use UIhelper methods, move catalog-model to …

06e6aa9

…devDependencies

fix: remove duplicate isElementVisible function

537087a

fix: address all outstanding review comments

4faa564

fix(e2e): address final review comments and improve OTP selector hand…

1a3f626

…ling

Align E2E GitHub plugin tests and helpers with all Gustavo's review c…

aef5007

…omments: use only UIhelper methods for Backstage UI, robust popup handling, CatalogVerifier, and code clarity

Increase timeout for plugin card text verification to improve E2E tes…

42a5e7d

…t reliability

Strictly enforce OTP field presence in findOtpSelector per Gustavo's …

0456bf2

…review

chore: save local changes before rebase

2a30e38

fix: restore package.json

1b0406c

guyoron1 force-pushed the pr-statistics-security-insights-branch branch from 6d3bcf0 to 1b0406c Compare July 15, 2025 11:55

create new yarn.lock file using the yarn install, delete package-lock…

2c59fdc

….json

		@@ -0,0 +1,66 @@
		diff --git a/.ibm/pipelines/value_files/values_showcase.yaml b/.ibm/pipelines/value_files/values_showcase.yaml

		await githubPage.fill('input[name="login"]', username);
		await githubPage.fill('input[name="password"]', password);

feat(e2e): add tests for GitHub plugins (PR statistics & security insights) #2998

Are you sure you want to change the base?

feat(e2e): add tests for GitHub plugins (PR statistics & security insights) #2998

Uh oh!

Conversation

guyoron1 commented Jun 17, 2025

Description

Which issue(s) does this PR fix

PR acceptance criteria

How to test changes / Special notes to the reviewer

Uh oh!

openshift-ci bot commented Jun 17, 2025

Uh oh!

github-actions bot commented Jun 17, 2025

Uh oh!

gustavolira commented Jun 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Jun 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 23, 2025

Uh oh!

github-actions bot commented Jun 24, 2025

Uh oh!

gustavolira commented Jun 26, 2025

Uh oh!

github-actions bot commented Jun 30, 2025

Uh oh!

gustavolira commented Jun 30, 2025

Uh oh!

gustavolira commented Jun 30, 2025

Uh oh!

gustavolira commented Jul 9, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

gustavolira commented Jul 10, 2025