Browser Engine Testing Status-Quo

[gsnedders]

We decided to have some summaries as to the status quo of testing browser engines within themselves.

As @jgraham said in the last meeting:

Obvious questions would be are you running on emulators, real hardware, would that be the same for WPT. Another thing that could be relevant is are you running [ToT] WPT.

Let's share summaries here (or link to them if lengthy), prior to the next meeting.

[gsnedders]

On WebKit, we do all testing on iOS and iPadOS Simulators via WebKitTestRunner (comparable to Chrome’s content_shell). This includes the same set of tests as we run on macOS and other platforms—including the same imported subset of WPT. It isn’t possible to run the system Safari against a local WebKit build, even on a Simulator. It also isn’t possible to do on device testing, as the only possible version of WebKit is the system-provided version.

For WPT CI, given the inability to do anything else with ToT WebKit, presumably the same path would need to be taken—though of course I can imagine there might be some interest in testing shipping stable Safari too.

As a reminder, I'm out and won't be there on Tuesday, but @JonWBedard will be showing up in lieu of me (and he can probably answer questions about WebKit CI better than I could anyway).

[WeizhongX]

https://docs.google.com/document/d/e/2PACX-1vQ6yAmdUkred6zUpKC4h1AMj7e6Qn4r21HptkhF0PIcJXXKP30jzYf9_S9uujcDXmWvdu9Vb_n8uEsy/pub

[jgraham]

@WeizhongX that document doesn't seem to be accessible

[jgraham]

I didn't have loads of content, so I'll just put it in a comment here:

Gecko Android Testing

• All web-platform-tests are run.
• Runs on try / autoland / central for all builds (nightly, beta, release, etc.)
• All tests run on Linux using the Android Emulator (real hardware is only used for a very limited set of tests, not including any wpt).
• Test jobs run inside a docker container
• Machine type is t-linux-kvm-gcp. KVM is required to run the Android emulator.
• Testing uses the geckoview test app rather than full Mobile Firefox.

wpt Status Quo

• Local support for android emulator testing based on Gecko implementation: docker image, support for downloading / installing emulator, installing browser, etc.
• Docker requires --privileged
• Not often used, so it might be broken.
• Can't deploy to CI because we don't have any KVM capable machines in community taskcluster.

[WeizhongX]

@WeizhongX that document doesn't seem to be accessible

I was not able to figure out what is wrong. The document is supposed to accessible to the public. I will just copy the content and put it here.

Components we had in our system.
Builders:

• One builder for Chrome Android and One builder for WebView
• Runs continuously on CI, each run takes ~3 hours with 36 shards
• Uses Android Emulator, version Android Pie
• Compiles the browser from ToT
• Download upstream Wpt repo on the fly, and run the tests using the latest 3h epoch tag.
• Wpt report saved to the result database in the end.
• Still working on Chrome iOS

Result Database: Holds Wpt reports json file from the builders.

Blink-kms: Authentication Service that stores username/password to authenticate with wpt.fyi.

Wpt Result Uploader:

• A cron task triggered every 3 hours
• Get authentication data from blink-kms
  Fetch Wpt report from Result Database for the latest successful run, and send that to wpt.fyi

[foolip]

@WeizhongX as a high-level summary, is it accurate to say that Chromium's CI is already running Chrome for Android and uploading the results to wpt.fyi? Is there anything missing from the picture, or is everything fine the way it is for Chrome?

[WeizhongX]

The answer is yes to the first question. We are using the 3h epoch tag from upstream WPT to run against ToT Chrome for Android, which is exactly same as how some other browser runs Wpt.

We might still need to set up the dashboard to monitor status, and there are work to do if we want results from stable release.

[past]

I believe the Chrome on Android and WebView runs are not properly tagged as coming from the master branch, as jgraham mentioned recently. @WeizhongX has anyone been working on this?

[WeizhongX]

We did not work on that, because we don't know what is the priority. This should be an easy fix, so I can create a PR for it if needed.