Agenda 2023-03-14

[jgraham]

• Introduction to investigation
• Roadmap

[gsnedders]

Minutes:

James G: Ensure everyone is on the same page.

James G: Let’s go over the original proposal, making sure everyone is on the same page here.

James G: Suggest making scoring [of mobile runs, for Interop 2024 purposes] out of scope for this investigation effort, and this investigation effort should focus on getting testing results on wpt.fyi, and leaving the scoring questions to Interop in future.
… End goal of the project would be uploading regular mobile testing runs to wpt.fyi.
… Current scope doesn’t include anything about specific mobile testing APIs. Q about whether we want to include things like viewport APIs and whether this should be added to it.

Panos: does it make sense to learn from current internal testing of different vendors, and how different vendors test things internally.

Sam: There was already some discussion about this in a previous GH issue. There's some summary of the state of mobile testing for browser engines there. Might be worth going through that, but there is prior discussion there.

Simon: I know you were referring to the scoring of the tests, but we also need to have a score for this investigation area.

James G: Yes. We do need to sort out that one. Strawman proposal: we have 5 items on the to-do list, weigh them equally, make them 20% each.

Simon: I also believe we have two mobile platforms we’re interested in; we could get to a state where one of them works and one of them doesn’t. Maybe that could be in the score somehow?

James G: I think that makes sense; we can also talk about this in terms of different browsers, if we got two out of three then it should presumably be 2/3rds credits. It’s also not clear we should make 20% of the score step 3 (“make a decision”). Maybe making decisions is hard. Could imagine combining 2&3. Then 4&5 are about the implementation. This would give a 50/50 split for “figuring things out” and “implementing”.

Simon: Maybe we should figure out the end goal first?

Panos: I expect step 4 will be the one that takes the largest effort. It might make sense to split that down into smaller steps like platforms.

James G: I think the problem with step 4 but we don’t know what it is gonna be until we’ve done step 2 or 3 to break it down. We could probably decide that’s 40% or 50% probably and getting the results into wpt.fyi is probably a smaller bit so maybe 50/10 and split the other 40% between the first three steps there.

Panos: Any objections to using that as a starting point? To make things more concrete: 20/20/10/40/10 [across all five]?

James G: Maybe combine 2&3 and 20/20/50/10?

Sam: Maybe we should, as Simon said, figure out the end goal because if we add extra WebDriver APIs that’s a bunch of extra work above and beyond desktop equality?

Patrick: I think my gut instinct is what’s already written in the roadmap and we should consider new things out of scope and just get the status-quo running on mobile, and maybe the viewport investigation should do that.

James G: There is no viewport investigation, so that doesn’t work. But limiting the scope is probably sensible. It probably makes sense to let people figure out if “implement WebDriver APIs and viewport tests” is a reasonable focus area for 2024 and we already have plenty of work with mobile testing in general.

Sam: One radical take would be to set out why we want mobile testing at all. i.e. is it for mobile-specific features or because mobile is popular. If we're doing this because there are mobile-only features we probably want an impetus for the WebDriver WG to specify those things.

Simon: In my view the main goal is testing things that are impossible to test today. But being able to run all tests would be nice. But if we’re constrained for how many tests we run we could run a subset of tests that are more interesting for mobile. And that would still be a net improvement over the status quo. And viewed as a success for this investigation area.

James G: On the question of whether we specifically need more mobile testing APIs, I’m sure there are things that require new mobile testing APIs which are probably mobile specific. I don’t know how much of that work should happen within BT&T and should happen with the domain experts for that specific feature. For the viewport example, BT&T doesn’t have experts on how the mobile viewports work and nor does this group. So while it might be good to add that, it’s not clear either of those groups is the right place, and it probably needs to be in collaboration with the WG specifying the feature. Internally we do see differences [between desktop and mobile], because of new/different features, so we do think there’s a value in general. And other groups can then have the impetus to go and do the work to get everything in WPT versus the much larger work currently where there aren't mobile results on wpt.fyi.

James S: I had a Q, once we have this all set up, do we want to add additional guidance for the RFC process—and maybe it would be too early to decide versus the roadmap items—but would we need to?

James G: Do you have an example of the kind of thing that might be needed?

James S: No, as you were saying, we might not be the exact group of people for a given feature, but we’ve e.g. worked with HTML repo to get new WebDriver APIs: is there other guidance for developers for when they want to add a test?

Panos: One idea that’s probably along the same lines as James, maybe we would like to relax some of the constraints about what changes require RFC (or tighten them); we did that with testdriver.js changes—for mobile we might want to make other changes? But that doesn’t seem like a very pressing matter.

Sam: I'd view discussions on wpt RFC as out of scope for this group. This group can have an opinion and pass it on, but that's primarily a wpt core team decision.

James G: Underlying assumption I’ve been making which is that we will still have access to WebDriver and testdriver APIs, if we have limits there then we’d have differences between desktop and mobile there. Or we’d have to figure out a way to make testdriver work on top of something that isn’t WebDriver that matches the semantics. Which has always been the promise of testdriver but you have to match the specification which is (usually) WebDriver, but we also have the Test Utils spec, and in principle for mobile-specific stuff you add more stuff there. But it seems more likely to be testing mobile-specific variants of existing behaviour like user interaction, but maybe there will be something like dumping a bunch of metadata about the viewport we don’t want to expose to pages.

James G: So it seems like we have a fuzzy JPEG rendering of what the roadmap will be and we’ll better resolve it as we get more data, so I guess we then have next steps. Panos mentioned earlier maybe next meeting presenting something about each engine.

Sam: When is the next meeting?

James G: Currently every two weeks. Currently an hour long. Maybe that’s too long, but it’s easier to cut back. I don’t know if people want to start with this schedule.

Panos: You said I think we should interleave this the WPT RFC meeting.

James G: Yes. I just couldn’t figure out how to do that without scheduling multiple events, and we have until June before the first conflict.

Sam: I wouldn’t be able to present in two weeks time, at least; could pull someone else in, but lack some of the context about both our infra and WPT.

James G: We could just do it in four weeks, but based on last year it would be good to have momentum from early on. It would be good to build momentum early on. We could get someone else to present. We could have a meeting in two weeks but not do a presentation. We could write it up in a document and not do a presentation.

Simon: I wonder how much prep work we need to present, or if people here could do it off hand.

Sam: We only have fifteen mins left in this meeting.

James G: I am happy if people want to. I can give the summary as to upstream WPT, at least.

Sam: I can also give the WebKit summary off-hand.

James G: It would seem good to have a written summary. Maybe that’s true of the background as well. It would probably be good to have it as a presentation or a document somewhere? I think if Sam has time to write it down or as a presentation then at least someone could that’s probably okay, especially if someone else can answer questions. From Chromium/Gecko then we could do something similar but with people in the room. Does that sound acceptable Sam?

Sam: Yes.

James G: Would you or someone else be able to do it from Chromium, Panos?

Panos: I probably can’t, but there’s others like Weizhong are probably better versed. But also have async documents in two weeks and resync in four weeks. But it’s probably also fine to meet again in two weeks, however many can attend. Is there any length you had in mind? A one pager? A five minute presentation?

James G: I was thinking quite brief; don’t know if I could do twenty slides. The goal would be so the others in the group understand what you already have and how that feeds into the requirements for running tests. Obvious questions would be are you running on emulators, real hardware, would that be the same for WPT. Another thing that could be relevant is are you running WPT. One of the potential options we have is just using vendor results, is that a thing your setup would support? For Gecko we typically don’t have something exactly like WPT, so it would be quite a lot of work to get it something identical to upstream WPT. But if you were already running ToT WPT that makes that option more appealing. But I imagine a relatively short, a couple of pages, up to ten minute presentation. Does that seem reasonable?

James G: Nobody is saying that’s unreasonable. Let’s minute that it was reasonable.

RESOLVED: IT’S REASONABLE.

James G: Okay so we have an action item for the next meeting (prepare documents and/or presentations), James will also put together a summary of WPT status-quo.

Sam: do we have an opinion on documents versus slides?

Simon: documents would be advantageous, slides are an extra.

Panos: a document then allows us to collate the documents into something longer.