Add headless tests to starter templates #2868

FranjoMindek · 2025-07-01T09:36:32Z

Since starter templates are not a valid Wasp project by themselves, we need to use Wasp CLI to generate the projects to test them out.

For that I created the waspc/starters-headless-tests/run-starter-headless-tests.sh script.
It:

Creates a starter project from the template
Runs the headless test in DEV mode
If not using sqlite, runs the headless test in BUILD mode
Cleans up the generated project

Note: we are only testing minimal and basic starters here. open-saas has it's own tests, and we are not testing AI generated apps.

FranjoMindek · 2025-07-01T09:39:38Z

waspc/starters-headless-tests/run-starter-headless-tests.sh

@@ -0,0 +1,79 @@
+#!/bin/bash
+
+# Script to run E2E tests for a given Wasp template.


We need a script because we need to generate projects from the starter templates.
Starter templates are not valid wasp projects:

Missing files which are part of starters skeleton

main.wasp has interpolation placeholder values for e.g. version, which is invalid wasp syntax

FranjoMindek · 2025-07-01T09:39:54Z

waspc/starters-headless-tests/tests/health-check.spec.ts

@@ -0,0 +1,6 @@
+import { expect, test } from "@playwright/test";
+
+test("health check", async ({ page }) => {


"Text the app runs successfully and works."

FranjoMindek · 2025-07-01T09:40:12Z

waspc/starters-headless-tests/playwright.config.ts

+      use: { ...devices["Desktop Chrome"] },
+    },
+    // {
+    //   name: "firefox",


Since we are only testing that the app works, mobile is not really needed.
Though it might be useful to try it with firefox and webkit since we did have a firefox only bug?

After we decide I will of course remove all the commented code.

…-tests

cloudflare-workers-and-pages · 2025-07-01T12:17:20Z

Deploying wasp-docs-on-main with Cloudflare Pages

Latest commit:	`373f731`
Status:	✅ Deploy successful!
Preview URL:	https://b8b48221.wasp-docs-on-main.pages.dev
Branch Preview URL:	https://franjo-starters-headless-tes.wasp-docs-on-main.pages.dev

View logs

…ess-tests # Conflicts: # waspc/cli/src/Wasp/Cli/Command/CreateNewProject/StarterTemplates.hs # waspc/data/Cli/starters/README.md

…ess-tests

FranjoMindek · 2025-07-11T12:38:15Z

waspc/starters-headless-tests/package.json

+  "main": "index.js",
+  "keywords": [],
+  "scripts": {
+    "test": "for starter in basic minimal; do ./run-starter-headless-tests.sh $starter wasp || exit 1; done",


Is there a better way for this?

infomiho

Left some suggestions

infomiho · 2025-07-17T12:12:24Z

waspc/starters-headless-tests/run-starter-headless-tests.sh

@@ -0,0 +1,70 @@
+#!/bin/bash


Maybe the script can run npm install before starting to ensure Playwright is installed.

We could run npx -y playwright test to avoid asking the user if they want to install the Playwright CLI.

infomiho · 2025-07-17T12:43:26Z

waspc/starters-headless-tests/run-starter-headless-tests.sh

+  DEBUG=pw:webserver E2E_APP_PATH="./${TEMP_PROJECT_NAME}" WASP_CLI_CMD="${WASP_CLI_CMD}" HEADLESS_TEST_MODE=build npx playwright test
+}
+
+template_uses_sqlite() {


I feel like the sqlite detection is kinda fragile, can we do better?

Do we want to upgrade from bash to Node.js and use zx here - we can reuse same logic for detecting SQlite from the Wasp App Runner: https://github.com/wasp-lang/wasp/blob/main/wasp-app-runner/src/waspCli.ts#L94

Do we want to use zx regardless of reusing the SQLite detection logic, I feel like it's a quite common tool we use (Wasp App Runner, deploy package).

infomiho · 2025-07-17T12:43:44Z

waspc/starters-headless-tests/package.json

+    "test:dev": "for starter in basic minimal; do ./run-starter-headless-tests.sh $starter wasp-cli || exit 1; done"
+  },
+  "devDependencies": {
+    "@playwright/test": "^1.53.0",


I'd suggest we pin the Playwright version so browsers can be reused (Playwright versions and browser version are tied together).

We use the same version across kitchen sink app and example apps, I believe it's "@playwright/test": "1.51.1"

infomiho · 2025-07-17T12:44:16Z

waspc/starters-headless-tests/tests/health-check.spec.ts

@@ -0,0 +1,6 @@
+import { expect, test } from "@playwright/test";
+
+test("health check", async ({ page }) => {


I'd be much more comfortable with not testing starters manually we did something like added a new task with the basic template.

infomiho · 2025-07-17T12:44:25Z

waspc/starters-headless-tests/run-starter-headless-tests.sh

@@ -0,0 +1,70 @@
+#!/bin/bash


Can we test saas starter with this as well?

There is an issue that the saas template has the app in the /app folder.

Also the saas starter needs some .env vars set up to be able to run the app.

We can't test the saas starter in production mode with email sender set as Dummy.

infomiho · 2025-07-17T12:44:41Z

waspc/starters-headless-tests/run-starter-headless-tests.sh

@@ -0,0 +1,70 @@
+#!/bin/bash


We could be declarative around which tests we are doing, and define an object:

minimal / [sqlite] / use . as dir

basic / [sqlite] / use . as dir

saas / [postgres] / use app as dir

This way we don't bother with parsing the db type and we can declare stuff like app dir.

Also, do we see some value in testing all the starters with both database types?

FranjoMindek added 17 commits May 14, 2025 11:19

basic to minimal, prepare to move minimal to starter templates

889e06f

remove minimal from waspc

b466405

fix source

d843ee5

Merge remote-tracking branch 'origin/main' into franjo/starter-templates

31fb462

no css bcs basic

2fd70ad

Merge branch 'main' into franjo/starter-templates

b6c96a4

revert to todoTs instructions in basic

602f94a

rest of old files

bff4ed8

merge main + new direction for stearters

17c195a

Gitignore

a4b8c9d

desc

3f76b84

fix tests

f2de0a5

changes

b13a87c

gitignore and data-files

f5315bd

add tailwind plugin

0556a19

formatting

3489d11

headless tests

646c154

FranjoMindek self-assigned this Jul 1, 2025

FranjoMindek mentioned this pull request Jul 1, 2025

Starters headless tests wasp-lang/starters#29

Closed

FranjoMindek commented Jul 1, 2025

View reviewed changes

FranjoMindek added 2 commits July 1, 2025 14:13

Merge branch 'main' into franjo/starter-templates

3cacc70

Merge branch 'franjo/starter-templates' into franjo/starters-headless…

49a14bd

…-tests

Base automatically changed from franjo/starter-templates to main July 4, 2025 14:38

infomiho mentioned this pull request Jul 7, 2025

Automate templates testing in the CI wasp-lang/starters#8

Closed

FranjoMindek added 3 commits July 9, 2025 12:46

Merge remote-tracking branch 'origin/main' into franjo/starters-headl…

3e9459f

…ess-tests # Conflicts: # waspc/cli/src/Wasp/Cli/Command/CreateNewProject/StarterTemplates.hs # waspc/data/Cli/starters/README.md

rm gitignore

0b26d2f

fix

144940b

FranjoMindek added 5 commits July 9, 2025 12:50

reduce echo

03739cc

Merge remote-tracking branch 'origin/main' into franjo/starters-headl…

373f731

…ess-tests

Merge remote-tracking branch 'origin/main' into franjo/starters-headl…

0a23acf

…ess-tests

improve script

dd3ab8e

prod -> build

ba634ee

FranjoMindek commented Jul 11, 2025

View reviewed changes

infomiho self-requested a review July 11, 2025 13:37

FranjoMindek requested a review from sodic July 16, 2025 13:20

infomiho reviewed Jul 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add headless tests to starter templates #2868

Add headless tests to starter templates #2868

Uh oh!

FranjoMindek commented Jul 1, 2025 •

edited

Loading

Uh oh!

FranjoMindek Jul 1, 2025

Uh oh!

FranjoMindek Jul 1, 2025

Uh oh!

FranjoMindek Jul 1, 2025

Uh oh!

FranjoMindek Jul 16, 2025

Uh oh!

cloudflare-workers-and-pages bot commented Jul 1, 2025 •

edited

Loading

Uh oh!

FranjoMindek Jul 11, 2025

Uh oh!

infomiho left a comment

Uh oh!

infomiho Jul 17, 2025

Uh oh!

infomiho Jul 17, 2025

Uh oh!

infomiho Jul 17, 2025

Uh oh!

infomiho Jul 17, 2025

Uh oh!

infomiho Jul 17, 2025

Uh oh!

infomiho Jul 17, 2025

Uh oh!

infomiho Jul 17, 2025

Uh oh!

Uh oh!

		@@ -0,0 +1,79 @@
		#!/bin/bash

		# Script to run E2E tests for a given Wasp template.

		@@ -0,0 +1,6 @@
		import { expect, test } from "@playwright/test";

		test("health check", async ({ page }) => {

Uh oh!

Add headless tests to starter templates #2868

Are you sure you want to change the base?

Add headless tests to starter templates #2868

Uh oh!

Conversation

FranjoMindek commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloudflare-workers-and-pages bot commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying wasp-docs-on-main with Cloudflare Pages

Uh oh!

Choose a reason for hiding this comment

Uh oh!

infomiho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FranjoMindek commented Jul 1, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Jul 1, 2025 •

edited

Loading