Skip to content

Commit

Permalink
Move to w3c WebDriver spec (#616)
Browse files Browse the repository at this point in the history
* Move to w3c WebDriver spec

We've been using a `capabilities` syntax for create session that has long been
deprecated. Firefox finally forced our hand with geckodriver 0.35.0,
which has removed support for the obsolete syntax.

All supported WebDrivers understand the newer syntax.

We used to pass capabilities in as `desiredCapabilities`,
we now pass them in as a single item in the `firstMatch` vector.
This should match existing behaviour.

Moving to the new capabilities syntax tells `chromedriver`
(and `msedgedriver`) that we are running in "w3c mode". This has benefits on
Etaoin code. Our many customizations for these WebDrivers are no longer
necessary, they now support the W3C WebDriver endpoint spec.

In "w3c mode", chromedriver disallows any endpoints that could be served
by W3C WebDriver endpoints.

Notable examples are mouse and touch operations. These are now
handled by the W3C WebDriver action endpoints. WebDriver actions are not
stand-alone, they are submitted as transaction of steps. As such the
following Chrome-specific fns are no longer needed and would be confusing
if not deleted:

- `mouse-btn-down`
- `mouse-btn-up`
- `with-mouse-btn`
- `mouse-move-to` (was also available in Firefox)
- `mouse-click`
- `right-click`
- `left-click`
- `touch-down`
- `touch-move`
- `touch-up`

Users will instead express these manipulations via the existing
`perform-actions`.

Fns that represent a transaction of actions in themselves remain:
- `drag-and-drop` (from element to element)
- `double-click*` (on an element)
- `*-click-on` (on an element)
- `touch-tap` (on an element)

Verifying that everything worked required a full review of the API.

Some sweeping changes:
- Our recent deletion of PhantomJS support allowed me to streamline many
`defmulti`s into `defn`s.
- Added links to W3C WebDriver Spec endpoints to docstrings

Added some fns to expose at the W3C WebDriver Spec granularity:
- `get-timeouts`
- `set-timeouts`
- `get-element-rect`
- `get-element-rect-el`
- `get-element-rect`
- `set-window-rect`
- `get-window-rect`

We have finer grained versions of the above (ex. `get-element-position`,
`get-element-size`, etc). We've kept these for backwards compatibility,
but if we were starting today, we would have just matched today's W3C
WebDriver spec.

Minor fixes/changes:
- Uncomment and fix the maximize test
- Screenshots on elements work on safari, enable them
- Added notes on displayedness to docstrings
- Remove internal support for undocumented `:desired-capabilities`, the implementation
was either ultra legacy or misunderstood legacy APIs.

Add `bb test-coverage` task to check what we are not covering with tests

New tests include coverage for:
- `back`
- `forward`
- `set-cookie`
- `release-actions`
- `*-timeout*`
- `double-click`

Closes #467

Incidentally closes #522 via better docstrings on getting properties.

Closes #484 with W3C WebDriver spec endpoint links.

* address lint error

* test: capabilities: \ path separators on win
  • Loading branch information
lread authored Aug 12, 2024
1 parent 46a5642 commit e58ff88
Show file tree
Hide file tree
Showing 13 changed files with 841 additions and 948 deletions.
1 change: 1 addition & 0 deletions .clj-kondo/config.edn
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{:config-paths ^:replace ;; don't adopt any user preferences
["hooks" ;; keep our internal hooks separate from imported ones
"../resources/clj-kondo.exports/etaoin/etaoin"] ;; include our exported public config
:output {:linter-name true}
:cljc {:features [:clj]} ;; our bb reader conditionals might make some tools also assume cljs, state otherwise
:hooks
;; for internal stuff, I'm fine with using macroexpand, our external config uses analyze-call for
Expand Down
32 changes: 32 additions & 0 deletions CHANGELOG.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,38 @@ A release with an intentional breaking changes is marked with:
* Technically breaking
** {issue}613[#612]: Remove all support for long obsolete and long untested PhantomJS
({lread})
** {issue}467[#467]: Move to W3C WebDriver spec.
({lread})
*** The impetus was Firefox no longer supporting legacy `:capabilities` syntax.
The breaking impact was on Chrome/Edge, when in "w3c mode" it will fail on WebDriver endpoints where there is a viable w3c alternative.
This means some custom Chrome specific fns should now be expressed as w3c WebDriver actions.
The following Chrome-specific fns have been deleted:
**** `mouse-btn-down`
**** `mouse-btn-up`
**** `with-mouse-btn`
**** `mouse-move-to` (was also available in Firefox)
**** `mouse-click`
**** `right-click`
**** `left-click`
**** `touch-down`
**** `touch-move`
**** `touch-up`
*** Remove internal support for undocumented `:desired-capabilities`.
The implementation was either ultra legacy or misunderstood legacy APIs.

* Other changes
** Add new fns that more lightly abstract W3C WebDriver Spec (as part of {issue}467[#467] API review sweep)
({lread})
*** `get-timeouts` - as alternative to `get-*-timeout`
*** `set-timeouts` - as alternative to `set-*-timeout`
*** `get-element-rect` - as alternative to `get-element-size`, `get-element-location`
*** `get-element-rect-el` - as alternative to `get-element-size-el`, `get-element-location-el`
*** `get-window-rect` - as alternative to `get-window-size`, `get-window-position`
*** `set-window-rect` - as alternative to `set-window-size`, `set-window-position`
** Review tests and add some missing coverage (as part of {issue}467[#467] API review sweep)
({lread})
** {issue}467[#467]: Required a full sweep of the API, so also includes:
({lread})
** {pr}552[#552]: Add support for wide characters to input fill functions
({person}tupini07[@tupini07])
** {issue}566[#566]: Recognize `:driver-log-level` for Edge
Expand All @@ -48,6 +78,8 @@ A release with an intentional breaking changes is marked with:
({lread})
*** {issue}602[#602]: Document all `:fn/*` query pseudo-functions in a definitive list
({person}dgr[@dgr])
*** {issue}484[#484]: Add W3C WebDriver Spec links to docstrings
*** {issue}522[#522]: Via better docstrings on getting properties

== v1.0.40 - 2023-03-08 [[v1.0.40]]

Expand Down
4 changes: 3 additions & 1 deletion bb.edn
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,10 @@
:task test-server/-main}
test:bb {:doc "Runs tests under Babashka [--help]"
:task test/test-bb}
test-doc {:doc "test code blocks in user guide"
test-doc {:doc "Test code blocks in user guide"
:task test-doc/test-doc}
test-coverage {:doc "Run doc and unit tests on JVM and generate ./target/clofidence code coverage report"
:task test-coverage/-main}
test-matrix {:doc "Returns a test matrix for CI [--help]"
:task test-matrix/-main}
drivers {:doc "[list|kill] any running WebDrivers"
Expand Down
12 changes: 11 additions & 1 deletion deps.edn
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
:aliases
{;; we use babashka/neil for project attributes
;; publish workflow references these values (and automatically bumps patch component of version)
:neil {:project {:version "1.0.40" ;; describes last release and is template for next release
:neil {:project {:version "1.1.40" ;; describes last release and is template for next release
:name etaoin/etaoin
;; not neilisms - could potentially conflict with new neilisms
:github-coords clj-commons/etaoin}}
Expand Down Expand Up @@ -51,6 +51,16 @@
:vars [:symbol]}}
:main-opts ["-m" "babashka.cli.exec"]}

:clofidence {:classpath-overrides {org.clojure/clojure nil}
:extra-deps {com.github.flow-storm/clojure {:mvn/version "1.11.4"}
com.github.flow-storm/clofidence {:mvn/version "0.3.1"}}
:exec-fn clofidence.main/run
:exec-args {:report-name "Etaoin Test Coverage"
:output-folder "target/clofidence"
:test-fn cognitect.test-runner.api/test
:test-fn-args [{:dirs ["test" "target/test-doc-blocks/test"]}]}
:jvm-opts ["-Dclojure.storm.instrumentOnlyPrefixes=etaoin"]}

;; for consistent linting we use a specific version of clj-kondo through the jvm
:clj-kondo {:extra-deps {clj-kondo/clj-kondo {:mvn/version "2024.05.24"}}
:main-opts ["-m" "clj-kondo.main"]}
Expand Down
38 changes: 15 additions & 23 deletions doc/01-user-guide.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ Edge and `msedgedriver` must match so you might need to specify the version:
`scoop install [email protected]`
** Download: link:{url-edge-dl}[Official Microsoft download site]

Check your WebDriver installations launching by launching these commands.
Check your WebDriver installations by launching these commands.
Each should start a process that includes its own local HTTP server.
Use Ctrl-C to terminate.

Expand All @@ -190,7 +190,7 @@ You can optionally run the Etaoin test suite to verify your installation.
TIP: Some Etaoin API tests rely on ImageMagick.
Install it prior to running test.

From a clone of the https://github.com/clj-commons/etaoin[Etaoin GitHub repo]
From a clone of the https://github.com/clj-commons/etaoin[Etaoin GitHub repo]:

* To check tools of interest to Etaoin:
+
Expand Down Expand Up @@ -333,7 +333,7 @@ A portion of the above rewritten with `doto`:
----

== Playing Along in your REPL
We encourage you to try the examples in from this user guide in your REPL.
We encourage you to try the examples from this user guide in your REPL.

The Interwebs is constantly changing.
This makes testing against live sites impractical.
Expand Down Expand Up @@ -385,7 +385,7 @@ into:
;; => ["username2" "pass2" "some text2"]
----

If any exception occurs during a browser session, the WebDriver process might hang until you kill it manually.
If any exception occurs during a browser session, the WebDriver process might hang around until you kill it manually.
To prevent that, we recommend the `with-<browser>` macros:

[source,clojure]
Expand Down Expand Up @@ -939,7 +939,7 @@ The `query-shadow-root-el` and `query-all-shadow-root-el` allow you to specify t
(def shadow-root (e/get-element-shadow-root-el driver host))
(e/get-element-text-el driver (e/query-shadow-root-el driver shadow-root {:css "#in-shadow"}))
;; => "I'm in the shadow DOM"
(->> (e/query-all-shadow-root-el driver shadow-root {:css "span"})
(map #(e/get-element-text-el driver %)))
;; > ("I'm in the shadow DOM" "I'm also in the shadow DOM")
Expand Down Expand Up @@ -1069,17 +1069,7 @@ It can be used, for example, to check your handling of disallowing multiple form
(e/double-click driver {:tag :button :name "submit"})
----

There are also "blind" clicking functions.
They trigger mouse clicks on the current mouse position:

[source,clojure]
----
(e/left-click driver)
(e/middle-click driver)
(e/right-click driver)
----

Another set of functions do the same but move the mouse pointer to a specified element before clicking on them:
Functions that move the mouse pointer to a specified element before clicking on it:

[source,clojure]
----
Expand Down Expand Up @@ -1575,7 +1565,7 @@ The `:eager` option only works with Firefox at the moment.

Etaoin supports link:{actions}[Webdriver Actions].
They are described as "virtual input devices".
They act as little device input scripts that run simultaneously.
They act as little device input scripts that can even be run simultaneously.

Here, in raw form, we have an example of two actions.
One controls the keyboard, the other the pointer (mouse).
Expand Down Expand Up @@ -2315,6 +2305,8 @@ From least to most verbose:
* `:debug`
* `:all` for all messages.

Applies to Chrome and Edge only.

See <<console-logs>>

[id=opt-driver-log-level,reftext=`:driver-log-level`]
Expand Down Expand Up @@ -2473,9 +2465,9 @@ _Default:_ <not set>

_Example:_ See <<http-proxy>> for an example usage.

A *WebDriver*'s capabilities can be vendor specific and define preferred options.
A *WebDriver*'s capabilities can be vendor-specific and define preferred options.
Read WebDriver vendor docs before setting anything here.
While reading docs, note that Etaoin passes along `:capabilities` as `desiredCapabilties`.
While reading docs, note that Etaoin passes along `:capabilities` under `firstMatch`.

=== Using Headless Drivers [[headless]]

Expand All @@ -2489,7 +2481,7 @@ Running without a UI is helpful when:
Ensure your browser supports headless mode by checking if it accepts `--headless` command-line argument when running it from the terminal.

When starting a driver, pass the `:headless` boolean flag to switch into headless mode.
This flag is ignored for Safari which, as of June 2022, still does not support headless mode.
This flag is ignored for Safari which, as of August 2024, still does not support headless mode.

//{:test-doc-blocks/test-ns user-guide-headless-test}
[source,clojure]
Expand Down Expand Up @@ -2566,7 +2558,7 @@ To specify a directory where the browser should download files, use the `:downlo
----

Now, when you click on a download link, the file will be saved to that folder.
Currently, only Chrome and Firefox are supported.
Currently, only Chrome, Edge and Firefox are supported.

Firefox requires specifying MIME-types of the files that should be downloaded without showing a system dialog.
By default, when the `:download-dir` parameter is passed, the library adds the most common MIME-types: archives, media files, office documents, etc.
Expand Down Expand Up @@ -2598,7 +2590,7 @@ Set a custom `User-Agent` header with the `:user-agent` option when creating a d
----

Setting this header is important when using <<headless,headless browsers>> as many websites implement some sort of blocking when the User-Agent includes the "headless" string.
This can lead to 403 response or some weird behavior of the site.
This can lead to 403 responses or some weird behavior of the site.

=== HTTP Proxy [[http-proxy]]

Expand All @@ -2622,7 +2614,7 @@ To set proxy settings use environment variables `HTTP_PROXY`/`HTTPS_PROXY` or pa
NOTE: A `:pac-url` is for a https://en.wikipedia.org/wiki/Proxy_auto-config#The_PAC_File[proxy autoconfiguration file].
Used with Safari as other proxy options do not work in Safari.

To fine tune the proxy you use the original https://www.w3.org/TR/webdriver/#proxy[object] and pass it to capabilities:
To fine tune the proxy you use the original https://www.w3.org/TR/webdriver/#proxy[object] and pass it as capabilities:

//:test-doc-blocks/skip
[source,clojure]
Expand Down
14 changes: 14 additions & 0 deletions doc/02-developer-guide.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,20 @@ Run `bb cljdoc-preview --help` for help.
* `bb cljdoc-preview view` opens a view to your imported docs in your default web browser
* `bb cljdoc-preview stop` stops the docker image

=== Test Coverage
Sometimes it's nice to get an idea of what parts of Etaoin its unit and doc tests (or more importantly, don't) cover.

[source,shell]
----
bb test-coverage
----

The intent is not to strive for some percentage of coverage, just information on what is not covered.

When possible, run from macOS, the only OS where we hit all supported browsers (you'll need all browsers and WebDrivers installed and up to date).

It will take a while, but after tests are complete, crack open `./target/clofidence/index.html` for results.

== Other Notes

=== Logging
Expand Down
47 changes: 44 additions & 3 deletions env/test/resources/static/test.html
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,16 @@
</head>
<body>


<h3>Click section</h3>
<script>
function fill() {
document.getElementById("baz").innerHTML = "clicked";
}
</script>

</script>
<button id="foo" onclick="fill()">Click me</button>
<span id="baz"></span>
<hr>
<hr>

<h3>CSS section</h3>
<style>
Expand Down Expand Up @@ -257,7 +257,48 @@ <h3>Shadow DOM</h3>
</template>
</div>

<hr>

<h3>Pointer Click Target</h3>
<div id="pointerClickTarget" style="border: 1px solid black;">
[not clicked]
</div>

<h3>Mouse Button State</h3>
<pre id="mouseButtonState">[no clicks yet]</pre>

<h3 id="document-end">Document end</h3>
<script>
// Mouse button state, taken from:
// https://developer.mozilla.org/en-US/docs/Web/API/MouseEvent/buttons
const buttonNames = ["left", "right", "wheel", "back", "forward"];
function mouseButtonPressed(event, buttonName) {
// Use binary `&` with the relevant power of 2 to check if a given button is pressed
return Boolean(event.buttons & (1 << buttonNames.indexOf(buttonName)));
}

function format(event) {
const { type, buttons } = event;
const obj = { type };
for (const buttonName of buttonNames) {
obj[buttonName] = mouseButtonPressed(event, buttonName);
}
return JSON.stringify(obj, null, 2);
}

const log = document.getElementById("log");
function logButtons(event) {
// Avoid interfering with form elements
mouseButtonState.textContent = format(event);
}
document.addEventListener("mouseup", logButtons);
document.addEventListener("mousedown", logButtons);

// pointer click target
const pointerClickTarget = document.getElementById("pointerClickTarget");
pointerClickTarget.addEventListener("dblclick", (e) => {
pointerClickTarget.textContent = "[double clicked]";
});
</script>
</body>
</html>
17 changes: 17 additions & 0 deletions script/test_coverage.clj
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
(ns test-coverage
(:require [babashka.fs :as fs]
[helper.shell :as shell]
[lread.status-line :as status]))

(defn generate-doc-tests []
(status/line :head "Generating tests for code blocks in documents")
(shell/command "clojure -X:test-doc-blocks gen-tests"))

(defn run-all-tests []
(status/line :head "Running unit and code block tests under Clojure for coverage report")
(shell/command "clojure" "-X:test:test-docs:clofidence"))

(defn -main [& _args]
(fs/delete-tree "./target/clofidence")
(generate-doc-tests)
(run-all-tests))
Loading

0 comments on commit e58ff88

Please sign in to comment.