✅ Playwright e2e tests (#404)

* 🔧 Moved build-only deps to dev-deps list

moved tailwindcss, postcss, autoprefixer, and daisyui to dev deps list given they are strictly build-time.

* 📝 Updated changeset with info on latest commit.

* ✅ Added Playwright test to main user story, updated scripts

Added accessibility-driven, automated Playwright test to verify main user story: enter GitHub repo, obtain preview URL, and see preview. Updated yarn testing scripts in package.json. Made repo.tsx more accessible.

* 📝 Updated changeset following last playwright commit

* ⬆️ Bump nanoid in the npm_and_yarn group across 1 directory

Bumps the npm_and_yarn group with 1 update in the / directory: [nanoid](https://github.com/ai/nanoid).


Updates `nanoid` from 3.3.7 to 3.3.8
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](ai/nanoid@3.3.7...3.3.8)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: indirect
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <[email protected]>

* ✅ Updated Playwright tests, config, and snapshots

selectively checked out #405 's updated working 100% passing Playwright tests and config to current branch. Resovled to not including stargazers issue mentioned in #404 comments. Updated all snapshots to reflect this new config. Also only using Chrome and Mobile Chrome to ensure small testing footprint while covering responsiveness of the app.

* 📝 Added contrib instructions and credits to devs in readme

added detailed project setup, forking, local dev server, and testing and contributing instructions in readme. also added "community" section crediting all contributors to socialify so far.

* 📝 Generated changeset to reflect latest commits.

* 👷 Configured Playwright for the build.yml CI script.

Given that GITHUB_TOKEN is a reserved env var, testing if renaming the secret to SOCIALIFY_GITHUB_TOKEN and mapping it to the GITHUB_TOKEN env is a viable workaround.

* 👷 Reverted back to using default GITHUB_TOKEN to test fix.

Previous commit of using a separate SOCIALIFY_GITHUB_TOKEN didn't work. Reverting back to using the default {{ secrets.GITHUB_TOKEN }} auto-assigned by GitHub Actions per run to test the fix.

* 💚 Used step-scoped env rewrite to ensure GITHUB_TOKEN is valid for ci.

GITHUB_TOKEN is reserved for GitHub Actions. Since Socialify uses the same named env, we must rewrite it only for the `yarn verify` step in the `build.yml` script. Requires repo-level actions secret ${{ secrets.SOCIALIFY_GITHUB_TOKEN }}.

* 💚 Added PROJECT_URL env to ci script.

Previous CI script might also be missing the PROJECT_URL env, added as a secret accessible at the same step level as the SOCIALIFY_GITHUB_TOKEN.

* 👷 Added upload playwright artifact to help debug behavior diff.

* 👷 Added continue-on-error: true to ci to ensure debugging artifact upload

* 👷 Reconfig ci Playwright artifact upload.

* 📝 Clarified changeset requirements in README.md.

* 👷 Added --frozen-lockfile to yarn install in build.yml

* 📝 Updated README.md contrib instr per owner's comments.

* ✅ Specified font usage for Socialify to maintain Playwright consistency.

* 💄 Updated to using the Inter variable font for best prod UI match.

* 👷 Testing if fixed weight Inter improves CI Playwright

* ✅ Added +1s to Playwright animation waiting to ensure consistency in CI.

* ✅ Gave UI transition timeout a total of 5s to try improving consistency.

Add multi-line, detailed description

* ✅ Added 1% allowed buffer to UI test, kept user story test 100%.

* ✅ Revised Playwright threshold: 99% for UI and 100% for output/user story

Due to known GitHub Actions CI Playwright inconsistency issue not resolvable by waiting longer nor changing fonts, I propose 99% threshold for UI tests while keeping output/user story tests threshold at 100%.

* 📝 Update README.md

* 💚 Fix CI Build

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Wei He <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Wei He <[email protected]>

Author: 3 people

.changeset/warm-singers-pump.md

@@ -0,0 +1,7 @@
+---
+"socialify": minor
+---
+
+Added playwright e2e testing to capture user stories and main UI.
+
+Updated contributor setup instructions and contributor list in README.md.

.dockerignore

@@ -1,5 +1,10 @@
 # See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
 
+# Playwright results
+/.playwright/test-report/
+/.playwright/test-results/
+/.playwright/.cache/
+
 # dependencies
 /node_modules
 /.pnp

.env.example

@@ -1,4 +1,5 @@
-# GitHub token
+# GitHub token with Public Repositories (read-only) access
+# Get one at https://github.com/settings/personal-access-tokens/new
 GITHUB_TOKEN=
 # API URL (Use public url in edge deployments, and localhost url in non-edge deployments)
 PROJECT_URL=http://localhost:3000

.github/workflows/build.yml

@@ -1,9 +1,15 @@
-name: Node.js CI
+name: Build
 
 on:
+  workflow_dispatch:
   push:
   pull_request:
-    types: [assigned, opened, synchronize, reopened]
+    types:
+      - opened
+      - synchronize
+      - reopened
+  schedule:
+    - cron: '0 6 * * 0'
 
 jobs:
   build:
@@ -16,21 +22,16 @@ jobs:
         uses: actions/setup-node@v4
         with:
           node-version: 22.x
+          cache: 'yarn'
 
-      - name: Get yarn cache directory path
-        id: yarn-cache-dir-path
-        run: echo "dir=$(yarn cache dir)" >> $GITHUB_OUTPUT
+      - name: Install dependencies
+        run: yarn install --frozen-lockfile
+  
+      - name: Lint
+        run: yarn lint
 
-      - name: Cache yarn and next
-        uses: actions/cache@v4
-        with:
-          path: |
-            ${{ steps.yarn-cache-dir-path.outputs.dir }}
-            ${{ github.workspace }}/.next/cache
-          # Generate a new cache whenever packages or source files change.
-          key: ${{ runner.os }}-nextjs-${{ hashFiles('**/yarn.lock') }}-${{ hashFiles('**.[jt]s', '**.[jt]sx') }}
-          # If source files changed but packages didn't, rebuild from a prior cache.
-          restore-keys: |
-            ${{ runner.os }}-nextjs-${{ hashFiles('**/yarn.lock') }}-
-      - run: yarn install
-      - run: yarn verify
+      - name: Unit Test
+        run: yarn test:unit
+
+      - name: Build
+        run: yarn build

.github/workflows/e2e.yml

@@ -0,0 +1,62 @@
+name: End-to-End Tests
+
+on:
+  workflow_dispatch:
+  push:
+  pull_request_target:
+    types:
+      - opened
+      - synchronize
+      - reopened
+  schedule:
+    - cron: '0 6 * * 0'
+
+jobs:
+  e2e:
+    runs-on: ubuntu-latest
+    env:
+      PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/../.ms-playwright
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v4
+        
+      - name: Setup node
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22.x
+          cache: 'yarn'
+
+      - name: Install dependencies
+        run: yarn install --frozen-lockfile
+
+      - name: Get Playwright's Version
+        shell: bash # necessary for it to work on Windows, which uses powershell by default
+        run: echo "PLAYWRIGHT_VERSION=$(node -p 'require("@playwright/test/package.json").version')" >> $GITHUB_OUTPUT
+        id: playwright-version
+
+      - name: Cache playwright binaries
+        uses: actions/cache@v4
+        id: playwright-cache
+        with:
+          path: |
+            ${{ env.PLAYWRIGHT_BROWSERS_PATH }}
+          key: ${{ runner.os }}-playwright-${{ steps.playwright-version.outputs.PLAYWRIGHT_VERSION }}
+  
+      - name: Install Playwrigtht dependencies
+        run: yarn playwright install --with-deps chromium
+        if: steps.playwright-cache.outputs.cache-hit != 'true'
+
+      - name: E2E Test
+        run: yarn test:e2e
+        env:
+          GITHUB_TOKEN: ${{ secrets.PAT }}
+          PROJECT_URL: http://127.0.0.1:3000
+
+      - name: Upload Playwright artifacts on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: playwright-artifacts
+          path: .playwright/**/*
+          include-hidden-files: true
+          if-no-files-found: error

.gitignore

@@ -1,5 +1,13 @@
 # See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
 
+# Local vscode
+.vscode
+
+# Playwright
+/.playwright/.cache
+/.playwright/test-report
+/.playwright/test-results
+
 # dependencies
 /node_modules
 /.pnp

.playwright/mainUIConsistency.spec.ts

@@ -0,0 +1,67 @@
+import { type Page, expect, test } from '@playwright/test'
+
+const customPageLoadTimeout = { timeout: 30000 }
+
+// As a known CI issue, allow max 1% deviation in pixel diff.
+const customDiffPixelRatio = { maxDiffPixelRatio: 0.01 }
+
+// Testing constants.
+const repoPreviewURL: string =
+  '/wei/socialify?language=1&owner=1&name=1&stargazers=1&theme=Light'
+
+test.describe('Socialify UI:', () => {
+  test('is consistent for landing page', async ({
+    page,
+  }: { page: Page }): Promise<void> => {
+    await page.goto('/', customPageLoadTimeout)
+
+    // Wait for the page to load/hydrate completely.
+    await page.waitForLoadState('networkidle', customPageLoadTimeout)
+    await page.waitForTimeout(5000)
+
+    const image = await page.screenshot()
+    expect(image).toMatchSnapshot(customDiffPixelRatio)
+  })
+
+  test('is consistent for error (404) page', async ({
+    page,
+  }: { page: Page }): Promise<void> => {
+    await page.goto('/404', customPageLoadTimeout)
+
+    // Wait for the page to load/hydrate completely.
+    await page.waitForLoadState('networkidle', customPageLoadTimeout)
+
+    const image = await page.screenshot()
+    expect(image).toMatchSnapshot(customDiffPixelRatio)
+  })
+
+  test('is consistent for preview config page', async ({
+    page,
+  }: { page: Page }): Promise<void> => {
+    await page.goto(repoPreviewURL, customPageLoadTimeout)
+
+    // Wait for the page to load/hydrate completely.
+    await page.waitForLoadState('networkidle', customPageLoadTimeout)
+
+    // To maintain consistency, de-select the 'Stars' checkbox,
+    // and selects the 'Description' checkbox.
+    await page.click('input[name="stargazers"]')
+    await page.click('input[name="description"]')
+
+    // Wait for the component transition/animation to finish completely.
+    await page.waitForTimeout(1000)
+
+    const image = await page.screenshot()
+    expect(image).toMatchSnapshot(customDiffPixelRatio)
+
+    // Also check the toaster UI consistency.
+    await page.click('button:has-text("URL")')
+    await page.waitForSelector('[role="alert"]', customPageLoadTimeout)
+
+    // Wait for the component transition/animation to finish completely.
+    await page.waitForTimeout(1000)
+
+    const toastImage = await page.screenshot()
+    expect(toastImage).toMatchSnapshot(customDiffPixelRatio)
+  })
+})

.playwright/mainUIConsistency.spec.ts-snapshots/Socialify-UI-is-consistent-for-error-404-page-1-Mobile-Chrome-linux.png

@@ -0,0 +1,62 @@
+import { type Page, expect, test } from '@playwright/test'
+
+const customTimeout = { timeout: 30000 }
+
+// Testing constants.
+const repo: string = 'wei/socialify'
+const expectedConfigURL: string =
+  '/wei/socialify?language=1&owner=1&name=1&stargazers=1&theme=Light'
+const expectedImageURLRegExp: RegExp =
+  /\/wei\/socialify\/image\?description=1&language=1&name=1&owner=1&theme=Light$/
+
+async function getClipboardText(page: Page): Promise<string> {
+  return await page.evaluate(async () => {
+    return await navigator.clipboard.readText()
+  })
+}
+
+test.beforeEach(async ({ page }: { page: Page }): Promise<void> => {
+  await page.goto('/', customTimeout)
+
+  // Wait for the page to load/hydrate completely.
+  await page.waitForLoadState('networkidle', customTimeout)
+})
+
+test.describe('A simple user story:', () => {
+  test(`user can enter a GitHub repo ("username/repo"), click submit button, click "URL", and get the social preview image`, async ({
+    page,
+  }: { page: Page }): Promise<void> => {
+    // Input and submit the repo following accessibility best practices.
+    await page.fill('input[name="repo-input"]', repo)
+    await page.click('button[type="submit"]')
+
+    // Wait for navigation to the preview config page.
+    await page.waitForSelector('button:has-text("URL")', customTimeout)
+
+    // Wait for the page to load/hydrate completely.
+    await page.waitForLoadState('networkidle', customTimeout)
+    await expect(page).toHaveURL(expectedConfigURL)
+
+    // To maintain consistency, de-select the 'Stars' checkbox,
+    // and selects the 'Description' checkbox.
+    await page.click('input[name="stargazers"]')
+    await page.click('input[name="description"]')
+
+    // Obtain the consistent preview image URL.
+    await page.click('button:has-text("URL")')
+
+    // Compare the clipboard content to the expected image URL.
+    // (Only check the end of the URL due to dynamic localhost port allocation.)
+    const url: string = await getClipboardText(page)
+    expect(url).toMatch(expectedImageURLRegExp)
+
+    // Visit the image URL and snapshot the image.
+    await page.goto(url, customTimeout)
+
+    // Wait for the page to load/hydrate completely.
+    await page.waitForLoadState('networkidle', customTimeout)
+
+    const image = await page.screenshot()
+    expect(image).toMatchSnapshot()
+  })
+})

.playwright/mainUIConsistency.spec.ts-snapshots/Socialify-UI-is-consistent-for-error-404-page-1-chromium-linux.png

@@ -54,23 +54,67 @@ You can use cli tool [mheap/github-social-image](https://github.com/mheap/github
   </a>
 </p>
 
-## Development
+## Contributing
 
-- Create a GitHub token from `Settings > Developer settings > Personal access tokens`, you'll need it in when setting up environemnt variables.
-- You'll need the `repo` scope
-- Run the following commands to set up the Development server:
+Please read and agree to our [Contributor Covenant Code of Conduct](https://github.com/wei/socialify/blob/master/CONTRIBUTING.md) to get started.
 
-  ```shell
-  # Clone
-  git clone https://github.com/wei/socialify.git && cd $_
+### Running Socialify Locally
 
-  # Set environment variables in .env
-  cp .env.example .env
+Run the following commands to get Socialify running locally:
 
-  yarn install
-  yarn build
-  yarn dev
-  ```
+```shell
+# Clone your fork of wei/socialify if contributing
+git clone https://github.com/wei/socialify.git && cd $_
+
+# Set environment variables in .env.
+cp .env.example .env
+
+# Install dependencies
+yarn install
+
+# Start local development server
+yarn dev
+```
+
+### Testing and Committing
+
+Socialify uses [`biomejs`](https://biomejs.dev/) as linter/formatter, [`Jest`](https://jestjs.io/) for unit testing, and [`Playwright`](https://playwright.dev/) for end-to-end testing. 
+
+Make sure to run and pass the linter, unit and end-to-end tests locally before committing your code. Please let us know in case you need to update the test snapshots. More in `"scripts"` section in your `package.json` file.
+
+```shell
+# Run linter/formatter
+yarn lint  # yarn lint:fix
+
+# Fix linter/formatter errors
+# yarn lint:fix
+
+# Run unit tests
+yarn test:unit
+
+# Install Playwright dependencies (first-time)
+# yarn playwright install --with-deps chrome
+
+# Run e2e tests
+yarn test:e2e
+```
+
+**One** changeset file is required per each PR.
+
+  1. Run `yarn changeset` and select the appropriate senmantic versioning type (major, minor, patch) based on the scope of the change, together with an concise message.
+  2. You will see a new markdown file with a silly name like `milk-honey-eggs.md` in the `.changeset` directory.
+  3. Test, commit, and push your code **WITH** this changeset file when submitting a new PR.
+  4. You can manually edit it to include more information. For example:
+
+      ```markdown
+      ---
+      "socialify": minor
+      ---
+
+      Added playwright e2e testing to capture user stories and main UI
+      ```
+
+That's it! We are tremendously grateful for your contribution to Socialify! :heart:
 
 ## Authors
 
@@ -79,6 +123,14 @@ You can use cli tool [mheap/github-social-image](https://github.com/mheap/github
 
 _Part of [@MLH-Fellowship](https://github.com/MLH-Fellowship) Pod 1.0.6_
 
+## Community
+
+These are the heros who continue to make Socialify the best social preview generator for the community.
+
+<a href="https://github.com/wei/socialify/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=wei/socialify" />
+</a>
+
 ## License
 
 - [MIT](https://wei.mit-license.org)

.playwright/mainUIConsistency.spec.ts-snapshots/Socialify-UI-is-consistent-for-landing-page-1-Mobile-Chrome-linux.png

@@ -20,6 +20,12 @@ const customJestConfig = {
     '<rootDir>/**/__tests__/**/*.{js,jsx,ts,tsx}',
     '<rootDir>/**/*.{spec,test}.{js,jsx,ts,tsx}',
   ],
+  testPathIgnorePatterns: [
+    '/node_modules/',
+    '/.next/',
+    '/.vercel/',
+    '/.playwright/',
+  ],
   moduleNameMapper: {
     '\\.(css|less)$': 'identity-obj-proxy',
   },