(Selenium) `legacyRun` and `runPartial`/`finishRun` do not produce the same results 

`runLegacy()` and `runPartial`/`finishRun` do not produce the same results causing our test to fail as we assert that the axe-results generated are equal. Test in question: https://github.com/dequelabs/axe-core-maven-html/blob/3d13cc7662e7428c7217ba687e696a0622c9ac62/selenium/src/test/java/com/deque/html/axecore/selenium/Axe43xIntegrationTest.java#L257-L276

There are few things happening here: 

When legacy mode is enabled, selenium handles collecting all of the iframes for axe to inject: https://github.com/dequelabs/axe-core-maven-html/blob/3d13cc7662e7428c7217ba687e696a0622c9ac62/selenium/src/main/java/com/deque/html/axecore/extensions/WebDriverInjectorExtensions.java#L135-L171

Selenium now adds an attribute to each iframe called `cd_frame_id` which is some unique UUID. This is the first difference, as for `runPartial` we do not rely on Selenium to collect all of the frames thus no `cd_frame_id` attribute being added onto each iframe. This causes the equality check between results to fail. An HTML snippet captured by axe: 

```
"html": "<html lang=\"en\"><head>\n    <title>Foo</title>\n  </head>\n  <body>\n    <h1>Foo</h1>\n    <iframe src=\"bar.html\" id=\"foo-bar\" cd_frame_id_=\"5830db61d325a1a554e5e72eb2ba2959\"></iframe>\n    <iframe src=\"baz.html\" id=\"foo-baz\" cd_frame_id_=\"af6e22095931226bb0861676921f74ae\"></iframe>\n  \n</body></html>"
```

Moreover, `aria-hidden-body` rule appears to only capture the `body` element during a legacy run but everything including the `body` during a normal run: 


<details>
<summary>Legacy</summary>

```json
 {
      "id": "aria-hidden-body",
      "description": "Ensure aria-hidden=\"true\" is not present on the document body.",
      "help": "aria-hidden=\"true\" must not be present on the document body",
      "helpUrl": "https://dequeuniversity.com/rules/axe/4.10/aria-hidden-body?application=axeAPI",
      "impact": "",
      "tags": [
        "cat.aria",
        "wcag2a",
        "wcag131",
        "wcag412",
        "EN-301-549",
        "EN-9.1.3.1",
        "EN-9.4.1.2"
      ],
      "nodes": [
        {
          "html": "<body>",
          "target": [
            "body"
          ],
          "impact": null,
          "any": [
            {
              "id": "aria-hidden-body",
              "impact": "critical",
              "message": "No aria-hidden attribute is present on document body",
              "data": null,
              "relatedNodes": []
            }
          ],
          "all": [],
          "none": [],
          "failureSummary": null
        }
      ],
      "url": null,
      "createdDate": null
    }
```


</details>

<details>
<summary>Normal</summary>

```json
{
      "id": "aria-hidden-body",
      "description": "Ensure aria-hidden=\"true\" is not present on the document body.",
      "help": "aria-hidden=\"true\" must not be present on the document body",
      "helpUrl": "https://dequeuniversity.com/rules/axe/4.10/aria-hidden-body?application=axeAPI",
      "impact": "",
      "tags": [
        "cat.aria",
        "wcag2a",
        "wcag131",
        "wcag412",
        "EN-301-549",
        "EN-9.1.3.1",
        "EN-9.4.1.2"
      ],
      "nodes": [
        {
          "html": "<body>\n    <h1>This page has nested frames!</h1>\n    <iframe src=\"iframes/foo.html\" id=\"ifr-foo\"></iframe>\n    <iframe src=\"iframes/bar.html\" id=\"ifr-bar\"></iframe>\n    <iframe src=\"iframes/baz.html\" id=\"ifr-baz\"></iframe>\n  \n</body>",
          "target": [
            "body"
          ],
          "impact": null,
          "any": [
            {
              "id": "aria-hidden-body",
              "impact": "critical",
              "message": "No aria-hidden attribute is present on document body",
              "data": null,
              "relatedNodes": []
            }
          ],
          "all": [],
          "none": [],
          "failureSummary": null
        }
      ],
      "url": null,
      "createdDate": null
    }
```

</details>


This does not happen with our @axe-core/webdriverjs [test](https://github.com/dequelabs/axe-core-npm/blob/f8a761efd22870be27db4ad8a126e04d31b08ec8/packages/webdriverjs/test/axe-webdriverjs.spec.ts#L367-L383). The difference between the Java and TypeScript test is the Java test navigates to the page again before running a normal run: https://github.com/dequelabs/axe-core-maven-html/blob/3d13cc7662e7428c7217ba687e696a0622c9ac62/selenium/src/test/java/com/deque/html/axecore/selenium/Axe43xIntegrationTest.java#L267

Doing only one navigation in the WebDriverJS test causes some of the remanence of the legacy run notably the `cd_frame_id` attribute and causes the `aria-hidden-rule` to only capture the body. Removing the additional navigation in the Java test causes the test to pass. On the other hand, adding the additional navigation in the Typescript test causes the test to fail. 

Follow up that needs to happen: 

1. Identify if having the separate navigations between runs is the correct way for this test to be constructed. If we conclude that it is, we need to follow up on why axe-core is only capturing the `body` element for legacy and everything inside `body` for normal run. Second, how we can do true equality checks when selenium adds `cd_frame_id` onto each iframe this fundamentally breaks our test
2. Amend the test accordingly from the outcomes of 1 

(see below comments for full axe-results generated from each run)







	@Test
	public void returnsSameResultsRunPartialAndRun() throws Exception {
	webDriver.get(fixture("/nested-iframes"));
	Results legacyResults =
	new AxeBuilder()
	.setAxeScriptProvider(new StringAxeScriptProvider(axePost43x + axeForceLegacyJS))
	.analyze(webDriver);
	assertFalse(legacyResults.isErrored());
	assertEquals("axe-legacy", legacyResults.getTestEngine().getName());

	webDriver.get(fixture("/nested-iframes"));
	Results normalResults = new AxeBuilder().analyze(webDriver);

	normalResults.setTimestamp(legacyResults.getTimestamp());
	normalResults.getTestEngine().setName(legacyResults.getTestEngine().getName());
	ObjectMapper mapper = new ObjectMapper();
	Map normal = mapper.convertValue(normalResults, Map.class);
	Map legacy = mapper.convertValue(legacyResults, Map.class);
	assertEquals(normal, legacy);
	}

	/**
	* Recursively find frames and inject a script into them. If a frame errors when injecting due to
	* not being displayed, the error is ignored.
	*
	* @param driver An initialized WebDriver
	* @param script Script to inject
	*/
	private static void injectIntoFrames(
	final WebDriver driver,
	final String script,
	Consumer<WebDriver> injectCB,
	boolean doNotInjectAxe) {
	JavascriptExecutor js = (JavascriptExecutor) driver;
	List<WebElement> frames =
	driver.findElements(By.xpath(".//*[local-name()='frame' or local-name()='iframe']"));

	for (WebElement frame : frames) {
	try {
	driver.switchTo().frame(frame);
	if (!doNotInjectAxe) {
	js.executeScript(script);
	}
	if (injectCB != null) {
	injectCB.accept(driver);
	}

	injectIntoFrames(driver, script, injectCB, doNotInjectAxe);

	driver.switchTo().parentFrame();
	} catch (Exception e) {
	// Ignore all errors except those caused by the injected javascript itself
	if (e instanceof JavascriptException) {
	throw e;
	}
	}
	}
	}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

(Selenium) `legacyRun` and `runPartial`/`finishRun` do not produce the same results #481

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

(Selenium) legacyRun and runPartial/finishRun do not produce the same results #481

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

(Selenium) `legacyRun` and `runPartial`/`finishRun` do not produce the same results #481