WebKit 26 fails driver-integration-tests-webkit (cookies.cy.js)

[MikeMcC399]

Current behavior

If an attempt is made to update playwright-webkit to the latest version in Cypress source code, it fails the Cypress test spec

• packages/driver/cypress/e2e/commands/cookies.cy.js

which is part of driver-integration-tests-webkit.

This is reproducible on Ubuntu with versions of playwright-webkit starting with 1.37.0 and extending to the current version (with WebKit 17.4 18.0 26.x).

Cypress is currently configured to use playwright-webkit@1.24.2 released in July 2022:

cypress/package.json

Line 189 in d8884e0

"playwright-webkit": "1.24.2",

Desired behavior

WebKit should pass all tests from driver-integration-tests-webkit when using the latest version of playwright-webkit.

• Qualify if this is due to bug(s) in Playwright/WebKit or Cypress
• Depending on outcome, report bugs to Playwright and / or fix bugs in Cypress

Test code to reproduce

npm install yarn@latest -g
git clone https://github.com/cypress-io/cypress
cd cypress
yarn add playwright-webkit@latest -D -W
yarn playwright install-deps webkit # Linux only

then

yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js

or

yarn workspace @packages/driver cypress:open --e2e --browser webkit

and select cypress/e2e/commands/cookies.cy.js to run in UI.

Playwright Version

First reported on playwright-webkit@1.45.3
Continues to be reproducible on playwright-webkit@1.55.0

Cypress Version

First reported on 13.13.2
Continues to be reproducible on 15.0.0

Node version

v18.17.1 v20.18.1 and v22.18.0

Operating System

Ubuntu 22.04.4 LTS
Ubuntu 22.04.5 LTS and
Ubuntu 24.04.3 LTS

Debug Logs

See failures in CircleCI at
https://app.circleci.com/pipelines/github/cypress-io/cypress/63152/workflows/e49a5390-7dc4-432e-abce-391b205c080b/jobs/2615104

  1) src/cy/commands/cookies - no stub
       #getCookies
         returns cookies for the specified domain:

      AssertionError: expected 'key2' to equal 'key1'
      + expected - actual

      -'key2'
      +'key1'

      at  ([native code])
      at  (http://www.barbaz.com:3500/__cypress/tests:1717:41)
  getRet@
  tryCatcher@
  @
  thenFn@
  @
  tryCatcher@
  @
  @
  @
  _drainQueueStep@
  _drainQueue@
  @
  @

  2) src/cy/commands/cookies - no stub
       #getCookie
         returns the cookie from the specified domain:

      AssertionError: expected 'foobar.com value' to equal 'www.foobar.com value'
      + expected - actual

      -'foobar.com value'
      +'www.foobar.com value'

      at  ([native code])
      at  (http://www.barbaz.com:3500/__cypress/tests:1870:38)
  getRet@
  tryCatcher@
  @
  thenFn@
  @
  tryCatcher@
  @
  @
  @
  _drainQueueStep@
  _drainQueue@
  @
  @

  3) src/cy/commands/cookies
       #setCookie
         can set cookies with sameSite:

      AssertionError: expected { Object (name, value, ...) } to have property 'sameSite' of 'no_restriction', but got 'lax'
      + expected - actual

      -'lax'
      +'no_restriction'

      at  ([native code])
  applyChainer@
  arrayReduce@
  reduce@
  applyChainers@
  tryCatcher@
  @
  shouldFn@
  @
  tryCatcher@
  @
  @
  @
  _drainQueueStep@
  _drainQueue@
  @
  @

Other

• See WebKit (Experimental)
• https://playwright.dev/docs/browsers#webkit
• See chore(webkit): update playwright-webkit to 1.45.3 #29955 for failed attempt to update.

[MikeMcC399]

Further test results which may assist troubleshooting:

Results of yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js dependent on version of playwright-webkit

Playwright	pass	fail	WebKit	Notes
1.24.2	129	0	16.0	(1)
1.34.3	129	0	16.4	(2)
1.35.0	129	0	16.4	(3)
1.36.2	129	0	17.0	(3)
1.37.0	128	1	17.0	(4)
1.37.1	127	2	17.0	(5)
1.38.1	127	2	17.0	(5)
1.39.0	127	2	17.4	(5)
1.40.1	128	1	17.4	(4)
1.41.2	127	2	17.4	(5)
1.42.1	127	2	17.4	(5)
1.43.1	127	2	17.4	(5)
1.44.1	127	2	17.4	(5)
1.45.0	127	2	17.4	(6)
1.45.1	126	3	17.4	(7)
1.45.3	126	3	17.4	(7)
1.46.0	127	3	18.0	(7)

(1) Currently configured version
(2) Previously recommended version
(3) Test does not exit and instead wrongly opens Cypress runner at end in global mode
(4) Failure is
AssertionError: expected 'foobar.com value' to equal 'www.foobar.com value'
(5) Failures as (4) above, additionally
AssertionError: expected 'key2' to equal 'key1'
(6) Failure as (4) above, additionally
AssertionError: expected { Object (name, value, ...) } to have property 'sameSite' of 'no_restriction', but got 'lax'
(7) Failures as (4), (5) & (6) above

Change version with for instance:

yarn add playwright-webkit@1.45.3 -D -W
yarn playwright install-deps webkit

Test with:

yarn workspace @packages/driver cypress:run --browser webkit --spec cypress/e2e/commands/cookies.cy.js

View WebKit version with

yarn workspace @packages/driver cypress:open --e2e --browser webkit

Settings > Project settings > Resolved configuration (Fails to launch Cypress Runner with some old versions)

For debug logs use:

export DEBUG=cypress:server:browsers:webkit-automation

[MikeMcC399]

Since there hasn't been any progress made on this issue and a minimum version of playwright-webkit@1.45.0 is required to support Ubuntu 24.04, I suggest to modify packages/driver/cypress/e2e/commands/cookies.cy.js to skip tests that fail in current versions of WebKit.

There are already multiple exceptions made in this test code depending on whether WebKit is being tested.

[MikeMcC399]

@jennifer-shehane

3 out of 129 cookie tests fail for WebKit installed from playwright-webkit@1.37.0 and above

I would submit a PR that skips these tests for WebKit if you agree. This would unblock an upgrade to the latest version of playwright-webkit and the system tests would be able to run under Debian 12 and Ubuntu 24.04.

Let me know if you want a PR for this.

Edit: I may have been too optimistic as it appears that there are other tests that fail apart from the cookies test when an upgrade is attempted. I'll take a look at doing smaller steps and see if anything looks feasible.