User Guide

Table of Contents

Introduction
Installation
- Add the Etaoin Library Dependency
  - For Clojure Users
  - For Bababashka Users
- Installing the Browser WebDrivers
Getting Started
Playing Along in your REPL
More Getting Started
Unit Tests as Docs
Creating and Quitting the Driver
Selecting Elements
Interactions
Capturing Screenshots
- Screenshots for Specific Elements
- Screenshots after each form
Print Page to PDF
Peeking deeper
Driver Options
Writing and Running Integration Tests For Your Application
Running Selenium IDE files
- CLI Arguments
Webdriver in Docker
Troubleshooting

Introduction

Etaoin offers the Clojure community a simple way to script web browser interactions from Clojure and Babashka. It is a thin abstraction atop the W3C WebDriver protocol that also endeavors to resolve real-world nuances and implementation differences.

History

Ivan Grishaev (@igrishaev) created Etaoin and published its first release to Clojars in February 2017. He and his band of faithful contributors grew Etaoin into a well-respected goto-library for browser automation.

In May 2022, finding his time had gravitated more to back-end development, Ivan offered Etaoin for adoption to clj-commons. It is now currently under the loving care of @lread and @borkdude.

Interesting Alternatives

If Etaoin is not your cup of tea, you might also consider:

Clojure based:

Wally - A Clojure Playwright wrapper
clj-chrome-devtools - Clojure API for controlling a Chrome DevTools remote

Other:

Selenium - A browser automation framework and ecosystem
Playwright - Reliable end-to-end testing for modern web apps
Puppeteer - A high-level API to control Chrome/Chromium over the DevTools Protocol

Supported OSes & Browsers

Etaoin’s test suite covers the following OSes and browsers for both Clojure and Babashka:

OS	Chrome	Firefox	Safari	Edge
Linux (ubuntu)	yes	yes	-	-
macOS ¹	yes	yes	yes	yes
Windows	yes	yes	-	yes

Our GitHub Actions macOS tests run on silicon (aka arm64, aarch64, or M*) hardware

Installation

There are two steps to installation:

Add the etaoin library as a dependency to your project
Install the WebDriver for each web browser that you want to control with Etaoin

Add the Etaoin Library Dependency

For Clojure Users

Etaoin supports Clojure v1.10 and above on JDK11 and above.

Add the following into the :dependencies vector in your project.clj file:

   [etaoin "1.1.42"]

Or the following under :deps in your deps.edn file:

   etaoin/etaoin {:mvn/version "1.1.42"}

For Bababashka Users

We recommend the current release of babashka.

Add the following under :deps to your bb.edn file:

   etaoin/etaoin {:mvn/version "1.1.42"}

Tip	Babashka uses timbre for logging. Timbre’s default logging level is debug. For a quieter Etaoin experience when using babashka, set the timbre default log level to info: (require '[taoensso.timbre :as timbre]) (timbre/set-level! :info)

Installing the Browser WebDrivers

Etaoin controls web browsers via their WebDrivers. Each browser has its own WebDriver implementation that must be installed.

Tip	If it is not already installed, you will need to install the web browser (Chrome, Firefox, Edge). This is usually done via a download from the browser’s official site. Safari comes bundled with macOS.

Tip	WebDrivers and browsers are updated regularly to fix bugs. Use current versions.

Some ways to install WebDrivers:

Google Chrome Driver
- macOS: brew install chromedriver
- Windows: scoop install chromedriver
- Download: Official Chromedriver download
Geckodriver for Firefox
- macOS: brew install geckodriver
- Windows: scoop install geckodriver
- Download: Official geckodriver release page
Safari Driver
- macOS only: Set up Safari options as the Webkit page instructs (scroll down to "Running the Example in Safari" section).
Microsoft Edge Driver
- macOS: (download manually)
- Windows: scoop install edgedriver
  Edge and msedgedriver must match, so you might need to specify the version: scoop install edgedriver@101.0.1210.0
- Download: Official Microsoft download site

Check your WebDriver installations by launching these commands. Each should start a process that includes its own local HTTP server. Use Ctrl-C to terminate.

chromedriver
geckodriver
safaridriver -p 0
msedgedriver

You can optionally run the Etaoin test suite to verify your installation.

Tip	Some Etaoin API tests rely on ImageMagick. Install it before running tests.

From a clone of the Etaoin GitHub repo:

To check tools of interest to Etaoin:
```
bb tools-versions
```
Run all tests:
```
bb test:bb
```
For a smaller sanity test, you might want to run api tests against browsers you are particularly interested in. Example:
```
bb test:bb --suites api --browsers chrome
```

During the test run, browser windows will open and close in series. The tests use a local handcrafted HTML file to validate most interactions.

See Troubleshooting if you have problems - or reach out on Clojurians Slack #etaoin or GitHub issues.

Getting Started

The great news is that you can automate your browser directly from your Babashka or Clojure REPL. Let’s interact with Wikipedia:

(require '[etaoin.api :as e]
         '[etaoin.keys :as k]
         '[clojure.string :as str])

;; Start WebDriver for Firefox
(def driver (e/firefox)) ;; a Firefox window should appear
(e/driver-type driver)
;; => :firefox

;; let's perform a quick Wiki session

;; navigate to Wikipedia
(e/go driver "https://en.wikipedia.org/")

;; make sure we aren't using a large screen layout
(e/set-window-size driver {:width 1280 :height 800})

;; wait for the search input to load
(e/wait-visible driver [{:tag :input :name :search}])

;; search for something interesting
(e/fill driver {:tag :input :name :search} "Clojure programming language")
(e/wait driver 1)
(e/fill driver {:tag :input :name :search} k/enter)
(e/wait-visible driver {:class :mw-search-results})

;; click on first match
(e/click driver [{:class :mw-search-results} {:class :mw-search-result-heading} {:tag :a}])
(e/wait-visible driver {:id :firstHeading})

;; check our new url location
;; (wikipedia can tack on a querystring, for result consistency we'll ignore it)
(-> (e/get-url driver) (str/split #"\?") first)
;; => "https://en.wikipedia.org/wiki/Clojure"

;; and our new title
(e/get-title driver)
;; => "Clojure - Wikipedia"

;; does the page have Clojure in it?
(e/has-text? driver "Clojure")
;; => true

;; navigate through history
(e/back driver)
(e/forward driver)
(e/refresh driver)
(e/get-title driver)
;; => "Clojure - Wikipedia"

;; let's explore the info box
;; What's its caption? Let's select it with a css query:
(e/get-element-text driver {:css "table.infobox caption"})
;; => "Clojure"

;; Ok, now let's try something trickier
;; Maybe we are interested in what value the infobox holds for the Family row:
(let [wikitable (e/query driver {:css "table.infobox.vevent tbody"})
      row-els (e/query-all-from driver wikitable {:tag :tr})]
  (for [row row-els
        :let [header-col-text (e/with-http-error
                                (e/get-element-text-el driver
                                                       (e/query-from driver row {:tag :th})))]
        :when (= "Family" header-col-text)]
    (e/get-element-text-el driver (e/query-from driver row {:tag :td}))))
;; => ("Lisp")

;; Etaoin gives you many options; we can do the same-ish in one swoop in XPath:
(e/get-element-text driver "//table[@class='infobox vevent']/tbody/tr/th[text()='Family']/../td")
;; => "Lisp"

;; When we are done, we quit, which stops the Firefox WebDriver
(e/quit driver) ;; the Firefox Window should close

Most api functions require the driver as the first argument. The doto macro can give your code a DSL feel. A portion of the above rewritten with doto:

(require '[etaoin.api :as e]
         '[etaoin.keys :as k])

(def driver (e/firefox))

(doto driver
  (e/go "https://en.wikipedia.org/")
  (e/set-window-size {:width 1280 :height 800})
  (e/wait-visible [{:tag :input :name :search}])
  (e/fill {:tag :input :name :search} "Clojure programming language")
  (e/wait 1)
  (e/fill {:tag :input :name :search} k/enter)
  (e/wait-visible {:class :mw-search-results})
  (e/click [{:class :mw-search-results} {:class :mw-search-result-heading} {:tag :a}])
  (e/wait-visible {:id :firstHeading})
  (e/quit))

Playing Along in your REPL

We encourage you to try the examples from this user guide in your REPL.

The Internet is constantly changing, making testing against live sites impractical. The code in this user guide has been tested to work against our little sample page.

Until we figure out something more clever, it might be easiest to clone the Etaoin GitHub repository and run a REPL from its project root.

Unless otherwise directed, our examples throughout the rest of this guide will assume you’ve already executed the equivalent of:

(require '[etaoin.api :as e]
         '[etaoin.keys :as k]
         '[clojure.java.io :as io])

(def sample-page (-> "doc/user-guide-sample.html" io/file .toURI str))

(def driver (e/chrome)) ;; or replace chrome with your preference
(e/go driver sample-page)

More Getting Started

You can use fill-multi to shorten the code like so:

(e/fill driver :uname "username")
(e/fill driver :pw "pass")
(e/fill driver :text "some text")

;; let's get what we just set:
(mapv #(e/get-element-value driver %) [:uname :pw :text])
;; => ["username" "pass" "some text"]

into:

;; issue a browser refresh
(e/refresh driver)
(e/fill-multi driver {:uname "username2"
                      :pw "pass2"
                      :text "some text2"})

;; to get what we just set:
(mapv #(e/get-element-value driver %) [:uname :pw :text])
;; => ["username2" "pass2" "some text2"]

If any exception occurs during a browser session, the WebDriver process might hang around until you kill it manually. To prevent that, we recommend the with-<browser> macros:

(e/with-firefox driver
  (doto driver
    (e/go "https://google.com")
    ;; ... your code here
    ))

This will ensure that the WebDriver process is closed regardless of what happens.

Unit Tests as Docs

The sections that follow describe how to use Etaoin in more depth.

In addition to these docs, the Etaoin api tests are also a good reference.

Creating and Quitting the Driver

Etaoin provides many ways to create a WebDriver instance.

Tip	As mentioned, we recommend the `with-<browser>` convention when you need proper cleanup.

Let’s say we want to create a chrome headless driver:

(require '[etaoin.api :as e])

;; at the base we have:
(def driver (e/boot-driver :chrome {:headless true}))
;; do stuff
(e/quit driver)

;; This can also be expressed as:
(def driver (e/chrome {:headless true}))
;; do stuff
(e/quit driver)

;; Or...
(def driver (e/chrome-headless))
;; do stuff
(e/quit driver)

The with-<browser> functions handle cleanup nicely:

(e/with-chrome {:headless true} driver
  (e/go driver "https://clojure.org"))

(e/with-chrome-headless driver
  (e/go driver "https://clojure.org"))

Replace chrome with firefox, edge or safari for other variants. See API docs for details.

See Driver Options for all options available when creating a driver.

Selecting Elements

Queries (aka selectors) are used to select the elements on the page that Etaoin will interact with.

;; let's start anew by refreshing the page:
(e/refresh driver)
;; select the element with an html attribute id of 'uname' and fill it with text:
(e/fill driver {:id "uname"} "Etaoin")
;; select the first element with an html button tag and click on it:
(e/click driver {:tag :button})

Tip

A query returns a unique element identifier typically meaningful only as a selector to other functions it is passed to.

Many functions accept a query directly. For example:

;; specifying query directly
(e/get-element-text driver {:tag :button})
;; => "Submit Form"
;; specifying the result of a query (notice the `-el` fn variant here)
(e/get-element-text-el driver (e/query driver {:tag :button}))
;; => "Submit Form"

Tip	An exception is thrown if a query does not find an element. Use `exists?` to check for element existence: (e/exists? driver {:tag :button}) ;; => true (e/exists? driver {:id "wont-find-me"}) ;; => false

Simple Queries, XPath, CSS

:active finds the currently active element. Note that querying for :active is deprecated. Users are encouraged to call get-active-element to retrieve the active element. The Google homepage, for example, automatically places the focus on the search input, so there is no need to click on it first.

;; Deprecated
(e/go driver "https://google.com")
(e/fill driver :active "Let's search for something" k/enter)
;; Better
(e/go driver "https://google.com")
(e/fill-el driver (e/get-active-element driver) "Let's search for something" k/enter)
;; Or best... This case is so common that the API includes `fill-active`
(e/go driver "https://google.com")
(e/fill-active driver "Let's search for something" k/enter)

any other keyword is translated to an HTML ID attribute. Note that you cannot query for an HTML element with an ID of "active" using this keyword syntax because the special keyword :active conflicts. If you want to query for an element with an ID of "active", use the map syntax, described below (e.g., {:id "active"} or {:id :active}).
```
(e/go driver sample-page)
(e/fill driver :uname "Etaoin" k/enter)
;; alternatively you can:
(e/fill driver {:id "uname"} "Etaoin Again" k/enter)
```

a string containing either an XPath or a CSS expression. The way query interprets this string depends on the driver’s :locator setting. By default, the driver’s :locator is set to "xpath". The :locator can be changed using the use-css, with-css, use-xpath, and with-xpath functions and macros. (Be careful when writing XPath manually; see Troubleshooting.) Here, we find an input tag with an attribute id of uname and an attribute name of username:

(e/refresh driver)
(e/fill driver ".//input[@id='uname'][@name='username']" "XPath can be tricky")

;; let's check if that worked as expected:
(e/get-element-value driver :uname)
;; => "XPath can be tricky"

;; let's modify the driver to use CSS rather than XPath
;; note that this returns a new, modified copy of the driver
;; the old driver is still valid, however
(def driver-css (e/use-css driver))
(e/refresh driver-css)
(e/fill driver-css "input#uname[name='username']" "CSS can be tricky, too")
(e/get-element-value driver-css :uname)
;; => "CSS can be tricky, too"

a map with either :xpath or :css key with a string in corresponding syntax:

(e/refresh driver)
(e/fill driver {:xpath ".//input[@id='uname']"} "XPath selector")
(e/fill driver {:css "input#uname[name='username']"} " CSS selector")

;; And here's what we should see in username input field now:
(e/get-element-value driver :uname)
;; => "XPath selector CSS selector"

This CSS selector reference may be of help.

Map Syntax Queries

A query can also be a map that represents an XPath expression as data. The rules are:

A :tag key represents a tag’s name. Defaults to *.
Any non-special key represents an attribute and its value.
:fn/ is a prefix followed by a supported query function.

There are several query functions of the form :fn/*. Each query function takes a parameter which is the value associated with the query function keyword in the map.

:fn/index: Takes a positive integer parameter. This expands into a trailing XPath [x] clause and is useful when you need to select a specific row in a table, for example.
:fn/text: Takes a string parameter. Matches if the element has the exact text specified.
:fn/has-text: Takes a string parameter. Matches if the element includes the specified text.
:fn/has-string: Takes a string parameter. Matches if the element string contains the specified string. The difference between :fn/has-text and :fn/has-string is the difference between the XPath text() and string() functions (text() is the text within a given element, and string() is the text of all descendant elements concatenated together in document order). Generally, if you are targeting an element at the top of the hierarchy, you probably want :fn/has-string. If you are targeting a single element at the bottom of the hierarchy, you probably want to use :fn/has-text.
:fn/has-class: Takes a string parameter. Matches if the element’s class attribute includes the string. Unlike using a :class key in the map, :fn/has-class can match single classes, whereas :class is an exact match of the whole class string.
:fn/has-classes: Takes a vector of strings parameter. Matches if the element’s class attribute includes all of the specified class strings.
:fn/link: Takes a string parameter. Matches if the element’s href attribute contains the specified string.
:fn/enabled: Takes a boolean (true or false) parameter. If the parameter is true, matches if the element is enabled. If the parameter is false, matches if the element is disabled.
:fn/disabled: Takes a boolean (true or false) parameter. If the parameter is true, matches if the element is disabled. If the parameter is true, matches if the element is enabled.

Here are some examples of the map syntax:

find the first div tag

(= (e/query driver {:tag :div})
   ;; equivalent via xpath:
   (e/query driver ".//div"))
;; => true

find the n-th (1-based) div tag

(= (e/query driver {:tag :div :fn/index 1})
   ;; equivalent via xpath:
   (e/query driver ".//div[1]"))
;; => true

find the tag a where the class attribute equals to active

(= (e/query driver {:tag :a :class "active"})
   ;; equivalent xpath:
   (e/query driver ".//a[@class='active']"))

find a form by its attributes:

(= (e/query driver {:tag :form :method :GET :class :formy})
   ;; equivalent in xpath:
   (e/query driver ".//form[@method=\"GET\"][@class='formy']"))

find a button by its text (exact match):

(= (e/query driver {:tag :button :fn/text "Submit Form"})
   ;; equivalent in xpath:
   (e/query driver ".//button[text() = 'Submit Form']"))

find an nth element (p, div, whatever, it does not matter) with "blarg" text:

(e/get-element-text driver {:fn/has-text "blarg" :fn/index 3})
;; => "blarg in a p"

;; equivalent in xpath:
(e/get-element-text driver ".//*[contains(text(), 'blarg')][3]")
;; => "blarg in a p"

find an element that includes a class:

(e/get-element-text driver {:tag :span :fn/has-class "class1"})
;; => "blarg in a span"

;; equivalent xpath:
(e/get-element-text driver ".//span[contains(@class, 'class1')]")
;; => "blarg in a span"

find an element that has the following domain in a href:

(e/get-element-text driver {:tag :a :fn/link "clojure.org"})
;; => "link 3 (clojure.org)"

;; equivalent xpath:
(e/get-element-text driver ".//a[contains(@href, \"clojure.org\")]")
;; => "link 3 (clojure.org)"

find an element that includes all of the specified classes:

(e/get-element-text driver {:fn/has-classes [:class2 :class3 :class5]})
;; => "blarg in a div"

;; equivalent in xpath:
(e/get-element-text driver ".//*[contains(@class, 'class2')][contains(@class, 'class3')][contains(@class, 'class5')]")
;; => "blarg in a div"

find explicitly enabled/disabled input widgets:

;; first enabled input
(= (e/query driver {:tag :input :fn/enabled true})
   ;; equivalent xpath:
   (e/query driver ".//input[@enabled=true()]"))
;; => true

;; first disabled input
(= (e/query driver {:tag :input :fn/disabled true})
   ;; equivalent xpath:
   (e/query driver ".//input[@disabled=true()]"))
;; => true

;; return a vector of all disabled inputs
(= (e/query-all driver {:tag :input :fn/disabled true})
   ;; equivalent xpath:
   (e/query-all driver ".//input[@disabled=true()]"))
;; => true

Vector Syntax Queries

A query can be a vector of any valid query expressions. For vector queries, every expression matches the output from the previous expression.

A simple, somewhat contrived, example:

(e/click driver [{:tag :html} {:tag :body} {:tag :button}])
;; our sample page shows form submits; did it work?
(e/get-element-text driver :submit-count)
;; => "1"

You may combine both XPath and CSS expressions

Tip	Reminder: the leading dot in an XPath expression means starting at the current node

;; under the html tag (using map query syntax),
;;  under a div tag with a class that includes some-links (using css query),
;;   click on a tag that has
;;    a class attribute equal to active (using xpath syntax):
(e/click driver [{:tag :html} {:css "div.some-links"} ".//a[@class='active']"])
;; our sample page shows link clicks, did it work?
(e/get-element-text driver :clicked)
;; => "link 2 (active)"

Advanced Queries

Querying the nth Element Matched

Sometimes, you may want to interact with the nth element of a query. Maybe you want to click on the second link within:

<ul>
    <li class="search-result">
        <a href="a">a</a>
    </li>
    <li class="search-result">
        <a href="b">b</a>
    </li>
    <li class="search-result">
        <a href="c">c</a>
    </li>
</ul>

You can use the :fn/index like so:

(e/click driver [{:tag :li :class :search-result :fn/index 2} {:tag :a}])
;; check click tracker from our sample page:
(e/get-element-text driver :clicked)
;; => "b"

or you can use the nth-child trick with the CSS expression like this:

;; start page anew
(e/refresh driver)
(e/click driver {:css "li.search-result:nth-child(2) a"})
(e/get-element-text driver :clicked)
;; => "b"

Finally it is also possible to obtain the nth element directly by using query-all:

;; start page anew
(e/refresh driver)
(e/click-el driver (nth (e/query-all driver {:css "li.search-result a"}) 1))
(e/get-element-text driver :clicked)
;; => "b"

Note	Notice: The use of `click-el` here. The `query-all` function returns an element, not a selector that can be passed to `click` directly The nth offset of 1 instead of 2. Clojure’s nth is 0-based, and our search indexes are 1-based.

Querying a Tree

query-tree pipes selectors. Every selector queries elements from the previous one. The first selector finds elements from the root, and subsequent selectors find elements downward from each of the previously found elements.

Given the following HTML:

<div id="query-tree-example">
  <div id="one">
    <a href="#">a1</a>
    <a href="#">a2</a>
    <a href="#">a3</a>
  </div>
  <div id="two">
    <a href="#">a4</a>
    <a href="#">a5</a>
    <a href="#">a6</a>
  </div>
  <div id="three">
    <a href="#">a7</a>
    <a href="#">a8</a>
    <a href="#">a9</a>
  </div>
</div>

The following query will find a vector of div tags then return a set of all a tags under those div tags:

(->> (e/query-tree driver :query-tree-example {:tag :div} {:tag :a})
     (map #(e/get-element-text-el driver %))
     sort)
;; => ("a1" "a2" "a3" "a4" "a5" "a6" "a7" "a8" "a9")

Querying the Shadow DOM

The shadow DOM provides a way to attach another DOM tree to a specified element in the normal DOM and have the internals of that tree hidden from JavaScript and CSS on the same page. When the browser renders the DOM, the elements from the shadow DOM appear at the location where the tree is rooted in the normal DOM. This provides a level of encapsulation, allowing "components" in the shadow DOM to be styled differently than the rest of the page and preventing conflicts between the normal page CSS and the component CSS. The shadow DOM is also hidden from normal Web Driver queries (query) and thus requires a separate set of API calls to query it. For more details about the shadow DOM, see this article at Mozilla Developer Network (MDN).

There are a few terms that are important to understand when dealing with the Shadow DOM. The "shadow root host" is the element in the standard DOM to which a shadow root is attached as a property. The "shadow root" is the top of the shadow DOM tree rooted at the shadow root host.

The following examples use this HTML fragment in the User Guide sample HTML that has a bit of shadow DOM in it.

<span id="not-in-shadow">I'm not in the shadow DOM</span>
<div id="shadow-root-host">
    <template shadowrootmode="open">
        <span id="in-shadow">I'm in the shadow DOM</span>
        <span id="also-in-shadow">I'm also in the shadow DOM</span>
    </template>
</div>

Everything in the template element is part of the shadow DOM. The div with the id of shadow-root-host is, as the ID suggests, the shadow root host element.

Given this HTML, you can run a standard query to find the shadow root host and then use get-element-property-el to return to the "shadowRoot" property. Note that the element IDs returned in the following examples will be unique to the specific Etaoin driver and driver session, and you will not see the same IDs.

(e/query driver {:id "shadow-root-host"})
;; an element ID similar to (but not the same as)
;; "78344155-7a53-46fb-a46e-e864210e501d"

(e/get-element-property-el driver (e/query driver {:id "shadow-root-host"}) "shadowRoot")
;; something similar to
;; {:shadow-6066-11e4-a52e-4f735466cecf "ac5ab914-7f93-427f-a0bf-f7e91098fd37"}

(e/get-element-property driver {:id "shadow-root-host"} "shadowRoot")
;; something similar to
;; {:shadow-6066-11e4-a52e-4f735466cecf "ac5ab914-7f93-427f-a0bf-f7e91098fd37"}

If you go this route, you’re going to have to pick apart the return values. The element-id of the shadow root is the string value of the first map key.

You can get the shadow root element ID more directly using Etaoin’s get-element-shadow-root API. The query parameter looks for a matching element in the standard DOM and returns its shadow root property.

(e/get-element-shadow-root driver {:id "shadow-root-host"})
;; something similar to
;; "ac5ab914-7f93-427f-a0bf-f7e91098fd37"

If you already have the shadow root host element, you can return its corresponding shadow root element ID using get-element-shadow-root-el.

(def host (e/query driver {:id "shadow-root-host"}))
(e/get-element-shadow-root-el driver host)
;; something similar to
;; "ac5ab914-7f93-427f-a0bf-f7e91098fd37"

You can test whether an element is a shadow root host using has-shadow-root? and has-shadow-root-el?.

(e/has-shadow-root? driver {:id "shadow-root-host"})
;; => true
(e/has-shadow-root-el? driver host)
;; => true
(e/has-shadow-root? driver {:id "not-in-shadow"})
;; => false

Now that you know how to retrieve the shadow root, you can query elements in the shadow DOM using query-from-shadow-root, query-all-from-shadow-root, query-from-shadow-root-el, and query-all-from-shadow-root-el.

For query-from-shadow-root and query-all-from-shadow-root, the q parameter specifies a query of the normal DOM to find the shadow root host. If the host is identified, the shadow-q parameter is a query that is executed within the shadow DOM rooted at the shadow root host.

The query-from-shadow-root-el and query-all-from-shadow-root-el allow you to specify the shadow root host element directly, rather than querying for it.

(def in-shadow (e/query-from-shadow-root driver {:id "shadow-root-host"} {:css "#in-shadow"}))
(e/get-element-text-el driver in-shadow)
;; => "I'm in the shadow DOM"

(->> (e/query-all-from-shadow-root driver {:id "shadow-root-host"} {:css "span"})
     (map #(e/get-element-text-el driver %)))
;; => ("I'm in the shadow DOM" "I'm also in the shadow DOM")

(def shadow-root (e/get-element-shadow-root-el driver host))
(e/get-element-text-el driver (e/query-from-shadow-root-el driver shadow-root {:css "#in-shadow"}))
;; => "I'm in the shadow DOM"

(->> (e/query-all-from-shadow-root-el driver shadow-root {:css "span"})
     (map #(e/get-element-text-el driver %)))
;; > ("I'm in the shadow DOM" "I'm also in the shadow DOM")

Note

In the previous shadow root queries, you should note that we used CSS selectors for the shadow-q argument in each case. This was done because current browsers do not support XPath, which is what the Etaoin map syntax is typically translated into under the hood. While it is expected that browsers will support XPath queries of the shadow DOM in the future, it is unclear when this support might appear. For now, use CSS.

For more information, see the Web Platforms Test Dashobard.

Interacting with Queried Elements

To interact with elements found via a query or query-all function call you have to pass the query result to either click-el or fill-el (note the -el suffix):

(e/click-el driver (first (e/query-all driver {:tag :a})))

You can collect elements into a vector and arbitrarily interact with them at any time:

(e/refresh driver)
(def elements (e/query-all driver {:tag :input :type :text :fn/disabled false}))

(e/fill-el driver (first elements) "This is a test")
(e/fill-el driver (rand-nth elements) "I like tests!")

Interactions

Some basic interactions are covered under Selecting Elements, here we go into other types of interactions and more detail.

UNICODE and Emojis

As of this writing, Chrome, and Edge only support filling inputs with UNICODE in the Basic Multilingual Plane. This includes many characters but not many emojis 😢.

Firefox and Safari seem to support UNICODE more generally 🙂.

(e/with-chrome driver
  (e/go driver sample-page)
  (e/fill driver :uname "ⱾⱺⱮⱸ ᢹⓂ Ᵽ")
  (e/get-element-value driver :uname))
;; => "ⱾⱺⱮⱸ ᢹⓂ Ᵽ"

(e/with-firefox driver
  (e/go driver sample-page)
  (e/fill driver :uname "ⱾⱺⱮⱸ ᢹⓂ Ᵽ plus 👍🔥🙂")
  (e/get-element-value driver :uname))
;; => "ⱾⱺⱮⱸ ᢹⓂ Ᵽ plus 👍🔥🙂"

Emulate how a Real Person Might Type

Real people type slowly and make mistakes. To emulate these characteristics, you can use the fill-human function. The following options are enabled by default:

{:mistake-prob 0.1 ;; a real number from 0.1 to 0.9, the higher the number, the more typos will be made
 :pause-max    0.2} ;; max typing delay in seconds

which you can choose to override if you wish:

(e/refresh driver)
(e/fill-human driver :uname "soslowsobad"
              {:mistake-prob 0.5
               :pause-max 1})

;; or just use default options by omitting them
(e/fill-human driver :uname " typing human defaults")

(e/get-element-value driver :uname)
;; => "soslowsobad typing human defaults"

For multiple inputs, use fill-human-multi

(e/refresh driver)
(e/fill-human-multi driver {:uname "login"
                            :pw "password"
                            :text "some text"}
                           {:mistake-prob 0.1
                            :pause-max 0.1})

Mouse Clicks

The click function triggers the left mouse click on an element found by a query term:

(e/click driver {:tag :button})

The click function uses only the first element found by the query, which sometimes leads to clicking on the wrong items. To ensure there is one and only one element found, use the click-single function. It acts the same but raises an exception when querying the page returns multiple elements:

(e/click-single driver {:tag :button :name "submit"})

Although double-clicking is rarely purposefully employed on web sites, some naive users might think it is the correct way to click on a button or link.

A double-click can be simulated with the double-click function. It can be used, for example, to check your handling of disallowing multiple form submissions.

(e/double-click driver {:tag :button :name "submit"})

Functions that move the mouse pointer to a specified element before clicking on it:

(e/left-click-on driver {:tag :a})
(e/middle-click-on driver {:tag :a})
(e/right-click-on driver {:tag :a})

A middle mouse click can open a link in a new background tab. The right-click sometimes is used to imitate a context menu in web applications.

Selecting an option from a dropdown

An <option> from a <select> can be selected via the click function.

Given the following HTML:

<select id="dropdown" name="options">
  <option value="o1">foo one</option>
  <option value="o2">bar two</option>
  <option value="o3">bar three</option>
  <option value="o4">bar four</option>
</select>

Click on the option with value o4:

(e/click driver [{:id :dropdown} {:value "o4"}])
(e/get-element-value driver :dropdown)
;; => "o4"

Click on the option with text bar three:

(e/click driver [{:id :dropdown} {:fn/text "bar three"}])
(e/get-element-value driver :dropdown)
;; => "o3"

Tip	Safari Quirk: You might need to first click on the `select` element, then the option.

Note

Etaoin also includes the select convenience function. It will select the first option from a dropdown that includes the specified text. It also automatically handles the Safari quirk.

Click the first matching option with text bar:

(e/select driver :dropdown "bar")
(e/get-element-value driver :dropdown)
;; => "o2"

The same operation, expressed with click:

(e/click driver :dropdown) ;; needed for Safari quirk only
(e/click driver [{:id :dropdown} {:fn/has-text "bar"}])
(e/get-element-value driver :dropdown)
;; => "o2"

Keyboard Chords

There is an option to input a series of keys simultaneously. This is useful to imitate holding a system key like Control, Shift, or whatever when typing.

The namespace etaoin.keys includes key constants as well as a set of functions related to keyboard input.

(require '[etaoin.keys :as k])

A quick example of entering ordinary characters while holding Shift:

(e/refresh driver)
(e/wait 1) ;; maybe we need a sec for active element to focus
(e/fill-active driver (k/with-shift "caps is great"))
(e/get-element-value-el driver (e/get-active-element driver))
;; => "CAPS IS GREAT"

The main input gets populated with "CAPS IS GREAT". Let’s duplicate the text via select-all, copy, and paste keyboard shortcuts:

(if (= "Mac OS X" (System/getProperty "os.name"))
  (e/fill-active driver (k/with-command "a") (k/with-command "c") k/arrow-right " " (k/with-command "v"))
  (e/fill-active driver (k/with-ctrl "a") (k/with-ctrl "c") k/arrow-right " " (k/with-ctrl "v")))
(e/get-element-value-el driver (e/get-active-element driver))
;; => "CAPS IS GREAT CAPS IS GREAT"

And now let’s clear the input by: 1. moving the cursor to the beginning of the input field with the home key 2. moving the cursor to the end field while holding shift to select all text 3. deleting the selected text with the delete key

(e/fill-active driver k/home (k/with-shift k/end) k/delete)
(e/get-element-value-el driver (e/get-active-element driver))
;; => ""

Note	These functions do not apply to the global browser’s shortcuts. For example, neither "Command + R" nor "Command + T" reload the page or open a new tab.

The etaoin.keys/with-* functions are just wrappers for the etaoin.keys/chord function that might be used for complex cases.

File Uploading

Clicking on a file input button opens an OS-specific dialog. You technically cannot interact with this dialog using the WebDriver protocol. Use the upload-file function to attach a local file to a file input widget. An exception will be thrown if the local file is not found.

;; open a web page that serves uploaded files
(e/go driver "http://nervgh.github.io/pages/angular-file-upload/examples/simple/")

;; bind element selector to variable; you may also specify an id, class, etc
(def file-input {:tag :input :type :file})

;; upload a file from your system to the first file input
(def my-file "env/test/resources/static/drag-n-drop/images/document.png")
(e/upload-file driver file-input my-file)

;; or pass a native Java File object:
(require '[clojure.java.io :as io])
(def my-file (io/file "env/test/resources/static/drag-n-drop/images/document.png"))
(e/upload-file driver file-input my-file)

When interacting with a remote WebDriver process, you’ll need to avoid the local file existence check by using remote-file like so:

(e/upload-file driver file-input (e/remote-file "/yes/i/really/do/exist.png"))

The remote file is assumed to exist where the WebDriver is running. The WebDriver will throw an error if it does not exist.

Scrolling

Etaoin includes functions to scroll the web page.

The most important one, scroll-query jumps the first element found with the query term:

(e/go driver sample-page)
;; scroll to the 5th h2 heading
(e/scroll-query driver {:tag :h2} {:fn/index 5})

;; and back up to the first h1
(e/scroll-query driver {:tag :h1})

To jump to the absolute pixel positions, use scroll:

(e/scroll driver 100 600)
;; or pass a map with x and y keys
(e/scroll driver {:x 100 :y 600})

To scroll relatively by pixels, use scroll-by with offset values:

;; scroll right by 100 and down by 300
(e/scroll-by driver 100 300)
;; use map syntax to scroll left by 50 and up by 200
(e/scroll-by driver {:x -50 :y -200})

There are two convenience functions to scroll vertically to the top or bottom of the page:

(e/scroll-bottom driver) ;; you'll see the footer...
(e/scroll-top driver)    ;; ...and the header again

The following functions scroll the page in all directions:

(e/scroll driver [0 0])     ;; let's start at the top left

(e/scroll-down driver 200)  ;; scrolls down by 200 pixels
(e/scroll-down driver)      ;; scrolls down by the default (100) number of pixels

(e/scroll-up driver 200)    ;; the same, but scrolls up...
(e/scroll-up driver)

(e/scroll-right driver 200) ;; ... and right
(e/scroll-right driver)

(e/scroll-left driver 200)  ;; ...left
(e/scroll-left driver)

Note	All scroll actions are carried out via Javascript. Ensure your browser has it enabled.

Working with frames and iframes

You can only interact with items within an individual frame or iframe by first switching to them.

Say you have an HTML layout like this:

<iframe id="frame1" src="...">
  <p id="in-frame1">In frame2 paragraph</p>
  <iframe id="frame2" src="...">
    <p id="in-frame2">In frame2 paragraph</p>
  </iframe>
</iframe>

Let’s explore switching to :frame1.

(e/go driver sample-page)
;; we start in the main page, we can't see inside frame1:
(e/exists? driver :in-frame1)
;; => false

;; switch context to frame with id of frame1:
(e/switch-frame driver :frame1)

;; now we can interact with elements in frame1:
(e/exists? driver :in-frame1)
;; => true
(e/get-element-text driver :in-frame1)
;; => "In frame1 paragraph"

;; switch back to top frame (the main page)
(e/switch-frame-top driver)

To reach nested frames, you can dig down like so:

;; switch to the first top-level iframe with the main page: frame1
(e/switch-frame-first driver)
;; downward to the first iframe with frame1: frame2
(e/switch-frame-first driver)
(e/get-element-text driver :in-frame2)
;; => "In frame2 paragraph"
;; back up to frame1
(e/switch-frame-parent driver)
;; back up to main page
(e/switch-frame-parent driver)

Use the with-frame macro to temporarily switch to a target frame, then do some work, returning its last expression while preserving your original frame context.

(e/with-frame driver {:id :frame1}
  (e/with-frame driver {:id :frame2}
    (e/get-element-text driver :in-frame2)))
;; => "In frame2 paragraph"

Executing Javascript

Use js-execute to evaluate a Javascript code in the browser:

(e/js-execute driver "alert('Hello from Etaoin!')")
(e/dismiss-alert driver)

Pass any additional parameters to the script with the arguments array-like object.

(e/js-execute driver "alert(arguments[2].foo)" 1 false {:foo "hello again!"})
(e/dismiss-alert driver)

We have passed 3 arguments:

1
false
{:foo "hello again!} which is automatically converted to JSON {"foo": "hello again!"}

The alert then presents the foo field of the 3rd (index 2) argument, which is "hello again!".

To return any data to Clojure, add return into your script:

(e/js-execute driver "return {foo: arguments[2].foo, bar: [1, 2, 3]}"
                     ;; same args as previous example:
                     1 false {:foo "hello again!"})
;; => {:bar [1 2 3], :foo "hello again!"}

Notice that the JSON has been automatically converted to edn.

Asynchronous Scripts

Use js-async to deal with scripts that rely on async strategies such as setTimeout. The WebDriver creates and passes a callback as the last argument to your script. To indicate that work is complete, you must call this callback.

Example:

(e/js-async
  driver
  "var args = arguments; // preserve the global args
  // WebDriver added the callback as the last arg; we grab it here
  var callback = args[args.length-1];
  setTimeout(function() {
    // We call the WebDriver callback, passing with what we want it to return
    // In this case we pass we chose to return 42 from the arg we passed in
    callback(args[0].foo.bar.baz);
  },
  1000);"
  {:foo {:bar {:baz 42}}})
;; => 42

If you’d like to override the default script timeout, you can do so for the WebDriver session:

;; optionally save the current value for later restoration
(def orig-script-timeout (e/get-script-timeout driver))
(e/set-script-timeout driver 5) ;; in seconds
;; do some stuff
(e/set-script-timeout driver orig-script-timeout)

or for a block of code via with-script-timeout:

(e/with-script-timeout driver 30
  (e/js-async driver "var callback = arguments[arguments.length-1];
                      //some long operation here
                      callback('phew,done!');"))
;; => "phew,done!"

Wait Functions

The main difference between a program and a human being is that the first operates very fast. A computer operates so fast that sometimes a browser cannot render new HTML in time. After each action, you might consider including a wait-<something> function that polls a browser until the predicate evaluates to true. Or just (wait <seconds>) if you don’t care about optimization.

The with-wait macro might be helpful when you need to prepend each action with (wait n). For example, the following form:

(e/with-wait 1
  (e/refresh driver)
  (e/fill driver :uname "my username")
  (e/fill driver :text "some text"))

is executed something along the lines of:

(e/wait 1)
(e/refresh driver)
(e/wait 1)
(e/fill driver :uname "my username")
(e/wait 1)
(e/fill driver :text "some text")

and thus returns the result of the last form of the original body.

The (doto-wait n driver & body) acts like the standard doto but prepends each form with (wait n). The above example, re-expressed with doto-wait:

(e/doto-wait 1 driver
  (e/refresh)
  (e/fill :uname "my username")
  (e/fill :text "some text"))

This is effectively the same as:

(doto driver
  (e/wait 1)
  (e/refresh)
  (e/wait 1)
  (e/fill :uname "my username")
  (e/wait 1)
  (e/fill :text "some text"))

In addition to with-wait and do-wait there are a number of waiting functions: wait-visible, wait-has-alert, wait-predicate, etc (see the full list in the API docs. They accept default timeout/interval values that can be redefined using the with-wait-timeout and with-wait-interval macros. They all throw if the wait timeout is exceeded.

(e/with-wait-timeout 15 ;; time in seconds
  (doto driver
    (e/refresh)
    (e/wait-visible {:id :last-section})
    (e/click {:tag :a})
    (e/wait-has-text :clicked "link 1")))

Wait text:

wait-has-text waits until an element has text anywhere inside it (including inner HTML).
```
(e/click driver {:tag :a})
(e/wait-has-text driver :clicked "link 1")
```
wait-has-text-everywhere like wait-has-text but searches for text across the entire page
```
(e/wait-has-text-everywhere driver "ipsum")
```

Load Strategy

When you navigate to a page, the driver waits until the whole page has been completely loaded. That’s fine in most cases but doesn’t reflect the way human beings interact with the Internet.

Change this default behavior with the :load-strategy option:

:normal (the default) wait for full page load (everything, include images, etc)
:none don’t wait at all
:eager wait for only DOM content to load

For example, the default :normal strategy:

(e/with-chrome driver
  (e/go driver sample-page)
  ;; by default you'll hang on this line until the page loads
  ;; (do-something)
)

Load strategy option of :none:

(e/with-chrome {:load-strategy :none} driver
  (e/go driver sample-page)
  ;; no pause, no waiting, acts immediately
  ;; (do-something)
)

The :eager option only works with Firefox at the moment.

Actions

Etaoin supports Webdriver Actions. They are described as "virtual input devices". They act as little device input scripts that can even be run simultaneously.

Here, in raw form, we have an example of two actions. One controls the keyboard, the other controls the pointer (mouse).

;; a keyboard input
{:type    "key"
 :id      "some name"
 :actions [{:type "keyDown" :value "a"}
           {:type "keyUp" :value "a"}
           {:type "pause" :duration 100}]}
;; some pointer input
{:type       "pointer"
 :id         "UUID or some name"
 :parameters {:pointerType "mouse"}
 :actions    [{:type "pointerMove" :origin "pointer" :x 396 :y 323}
              ;; double click
              {:type "pointerDown" :duration 0 :button 0}
              {:type "pointerUp" :duration 0 :button 0}
              {:type "pointerDown" :duration 0 :button 0}
              {:type "pointerUp" :duration 0 :button 0}]}

You can create a map manually and send it to the perform-actions method:

(def keyboard-input {:type    "key"
                     :id      "some name"
                     :actions [{:type "keyDown" :value "e"}
                               {:type "keyUp" :value "e"}
                               {:type "keyDown" :value "t"}
                               {:type "keyUp" :value "t"}
                               ;; duration is in ms
                               {:type "pause" :duration 100}]})
;; refresh so that we'll be at the active input field
(e/refresh driver)
;; perform our keyboard input action
(e/perform-actions driver keyboard-input)

Or, you might choose to use Etaoin’s action helpers. First, you create the virtual input device:

(def keyboard (e/make-key-input))

and then fill it with the actions:

(-> keyboard
    (e/add-key-down k/shift-left)
    (e/add-key-down "a")
    (e/add-key-up "a")
    (e/add-key-up k/shift-left))

Here’s a slightly larger working annotated example:

;; multiple virtual inputs run simultaneously, so we'll create a little helper to generate n pauses
(defn add-pauses [input n]
  (->> (iterate e/add-pause input)
       (take (inc n))
       last))

(let [username (e/query driver :uname)
      submit-button (e/query driver {:tag :button})
      mouse (-> (e/make-mouse-input)
                ;; click on username
                (e/add-pointer-click-el
                  username k/mouse-left)
                ;; pause 10 clicks to allow keyboard action to enter username
                ;; (key up and down for each of keypress for etaoin)
                (add-pauses 10)
                ;; click on submit button
                (e/add-pointer-click-el
                  submit-button k/mouse-left))
      keyboard (-> (e/make-key-input)
                   ;; pause 2 ticks to allow mouse action to first click on username
                   ;; (move to username element + click on it)
                   (add-pauses 2)
                   (e/with-key-down k/shift-left
                     (e/add-key-press "e"))
                   (e/add-key-press "t")
                   (e/add-key-press "a")
                   (e/add-key-press "o")
                   (e/add-key-press "i")
                   (e/add-key-press "n")) ]
  (e/perform-actions driver keyboard mouse))

To clear the state of virtual input devices, release all currently pressed keys etc, use the release-actions method:

(e/release-actions driver)

Capturing Screenshots

The screenshot function dumps the currently visible page to a PNG image file on your disk. Specify any absolute or relative path. Specify a string:

(e/screenshot driver "target/etaoin-play/screens1/page.png")

or a File object:

(require '[clojure.java.io :as io])
(e/screenshot driver (io/file "target/etaoin-play/screens2/test.png"))

Screenshots for Specific Elements

With Firefox and Chrome, you can also capture a single element within a page, say a div, an input widget, or whatever. It doesn’t work with other browsers at this time.

(e/screenshot-element driver {:tag :form :class :formy} "target/etaoin-play/screens3/form-element.png")

Screenshots after each form

Use with-screenshots to take a screenshot to the specified directory after each form is executed in the code block. The file naming convention is <webdriver-name>-<milliseconds-since-1970>.png

(e/refresh driver)
(e/with-screenshots driver "target/etaoin-play/saved-screenshots"
  (e/fill driver :uname "et")
  (e/fill driver :uname "ao")
  (e/fill driver :uname "in"))

this is equivalent to something along the lines of:

(e/refresh driver)
(e/fill driver :uname "et")
(e/screenshot driver "target/etaoin-play/saved-screenshots/chrome-1.png")
(e/fill driver :uname "ao")
(e/screenshot driver "target/etaoin-play/saved-screenshots/chrome-2.png")
(e/fill driver :uname "in")
(e/screenshot driver "target/etaoin-play/saved-screenshots/chrome-3.png")

Print Page to PDF

Use print-page to print the current page to a PDF file:

(e/with-firefox-headless driver
  (e/go driver sample-page)
  (e/print-page driver "target/etaoin-play/printed.pdf"))

See API docs for details.

Peeking deeper

Sometimes it is useful to go a little deeper.

Invoking WebDriver Implementation Specific Features

The Etaoin API exposes an abstraction of the W3C WebDriver protocol. This is normally all you need, but sometimes you’ll want to invoke a WebDriver implementation feature that is not part of the WebDriver protocol.

Etaoin talks to the WebDriver process via its execute function. You can use this lower level function to send whatever you like to the WebDriver process.

As a real-world example, Chrome supports taking screenshots with transparent backgrounds.

Here we use Etaoin’s execute function to ask Chrome to do this:

(e/with-chrome driver
  ;; navigate to our sample page
  (e/go driver sample-page)
  ;; send the Chrome-specific request for a transparent background
  (e/execute {:driver driver
              :method :post
              :path [:session (:session driver) "chromium" "send_command_and_get_result"]
              :data {:cmd "Emulation.setDefaultBackgroundColorOverride"
                     :params {:color {:r 0 :g 0 :b 0 :a 0}}}})
  ;; and here we take an element screenshot as per normal
  (e/screenshot-element driver
                        {:tag :form}
                        (str "target/etaoin-play/saved-screenshots/form.png")))

Reading a Browser’s Console Logs

Function get-logs returns the browser’s console logs as a vector of maps. Each map has the following structure:

(e/js-execute driver "console.log('foo')")
(e/get-logs driver)
;; [{:level :info,
;;   :message "console-api 2:32 \"foo\"",
;;   :source :console-api,
;;   :timestamp 1654358994253,
;;   :datetime #inst "2022-06-04T16:09:54.253-00:00"}]

;; on the 2nd call, for chrome, we'll find the logs empty
(e/get-logs driver)
;; => []

Currently, logs are available in Chrome only. The message text and the source type will vary by browser vendor. Chrome wipes the logs once they have been read.

DevTools: Tracking HTTP Requests, XHR (Ajax)

You can trace events that come from the DevTools panel. This means that everything you see in the developer console now is available through the Etaoin API. This currently only works for Google Chrome.

To start a driver with devtools support enabled, specify a :dev map.

(require '[etaoin.api :as e])

(e/with-chrome driver {:dev {}}
  ;; do some stuff
)

The value must not be a map (not nil). When :dev an empty map, the following defaults are used.

{:perf
 {:level :all
  :network? true
  :page? false
  :categories [:devtools.network]
  :interval 1000}}

We’ll work with a driver that enables everything:

(require '[etaoin.api :as e])

(def driver (e/chrome {:dev
                       {:perf
                        {:level :all
                         :network? true
                         :page? true
                         :interval 1000
                         :categories [:devtools
                                      :devtools.network
                                      :devtools.timeline]}}}))

Under the hood, Etaoin sets up a special perfLoggingPrefs dictionary inside the chromeOptions object.

Now that your browser is accumulating these events, you can read them using a special dev namespace.

The results will be different when you try this, but here’s what I experienced:

(require '[etaoin.dev :as dev])

(e/go driver "https://google.com")

(def reqs (dev/get-requests driver))

;; reqs is a vector of maps
(count reqs)
;; 23

;; what were the request types?
(frequencies (map :type reqs))
;; {:script 6,
;;  :other 2,
;;  :xhr 4,
;;  :image 5,
;;  :stylesheet 1,
;;  :ping 3,
;;  :document 1,
;;  :manifest 1}

;; Interesting, we've got Js requests, images, AJAX and other stuff

;; let's take a peek at the last image:
(last (filter #(= :image (:type %)) reqs))
;;    {:state 4,
;;     :id "14535.6",
;;     :type :image,
;;     :xhr? false,
;;     :url
;;     "https://www.google.com/images/searchbox/desktop_searchbox_sprites318_hr.webp",
;;     :with-data? nil,
;;     :request
;;     {:method :get,
;;      :headers
;;      {:Referer "https://www.google.com/?gws_rd=ssl",
;;       :sec-ch-ua-full-version-list
;;       "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"102.0.5005.61\", \"Google Chrome\";v=\"102.0.5005.61\"",
;;       :sec-ch-viewport-width "1200",
;;       :sec-ch-ua-platform-version "\"10.15.7\"",
;;       :sec-ch-ua
;;       "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
;;       :sec-ch-ua-platform "\"macOS\"",
;;       :sec-ch-ua-full-version "\"102.0.5005.61\"",
;;       :sec-ch-ua-wow64 "?0",
;;       :sec-ch-ua-model "",
;;       :sec-ch-ua-bitness "\"64\"",
;;       :sec-ch-ua-mobile "?0",
;;       :sec-ch-dpr "1",
;;       :sec-ch-ua-arch "\"x86\"",
;;       :User-Agent
;;       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"}},
;;     :response
;;     {:status nil,
;;      :headers
;;      {:date "Sat, 04 Jun 2022 00:11:36 GMT",
;;       :x-xss-protection "0",
;;       :x-content-type-options "nosniff",
;;       :server "sffe",
;;       :cross-origin-opener-policy-report-only
;;       "same-origin; report-to=\"static-on-bigtable\"",
;;       :last-modified "Wed, 22 Apr 2020 22:00:00 GMT",
;;       :expires "Sat, 04 Jun 2022 00:11:36 GMT",
;;       :cache-control "private, max-age=31536000",
;;       :content-length "660",
;;       :report-to
;;       "{\"group\":\"static-on-bigtable\",\"max_age\":2592000,\"endpoints\":[{\"url\":\"https://csp.withgoogle.com/csp/report-to/static-on-bigtable\"}]}",
;;       :alt-svc
;;       "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
;;       :cross-origin-resource-policy "cross-origin",
;;       :content-type "image/webp",
;;       :accept-ranges "bytes"},
;;      :mime "image/webp",
;;      :remote-ip "142.251.41.68"},
;;     :done? true}

Tip	The details of these responses come from Chrome and are subject to changes to Chrome.

Since we’re mostly interested in AJAX requests, there is a function get-ajax that does the same but filters XHR requests:

;; refresh to fill the logs again
(e/go driver "https://google.com")
(e/wait 2) ;; give ajax requests a chance to finish

(last (dev/get-ajax driver))
;; {:state 4,
;;  :id "14535.59",
;;  :type :xhr,
;;  :xhr? true,
;;  :url
;;    "https://www.google.com/complete/search?q&cp=0&client=gws-wiz&xssi=t&hl=en-CA&authuser=0&psi=OtuaYq-xHNeMtQbkjo6gBg.1654315834852&nolsbt=1&dpr=1",
;;  :with-data? nil,
;;  :request
;;  {:method :get,
;;   :headers
;;   {:Referer "https://www.google.com/",
;;    :sec-ch-ua-full-version-list
;;    "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"102.0.5005.61\", \"Google Chrome\";v=\"102.0.5005.61\"",
;;    :sec-ch-viewport-width "1200",
;;    :sec-ch-ua-platform-version "\"10.15.7\"",
;;    :sec-ch-ua
;;    "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
;;    :sec-ch-ua-platform "\"macOS\"",
;;    :sec-ch-ua-full-version "\"102.0.5005.61\"",
;;    :sec-ch-ua-wow64 "?0",
;;    :sec-ch-ua-model "",
;;    :sec-ch-ua-bitness "\"64\"",
;;    :sec-ch-ua-mobile "?0",
;;    :sec-ch-dpr "1",
;;    :sec-ch-ua-arch "\"x86\"",
;;    :User-Agent
;;    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"}},
;;  :response
;;  {:status nil,
;;   :headers
;;   {:bfcache-opt-in "unload",
;;    :date "Sat, 04 Jun 2022 04:10:35 GMT",
;;    :content-disposition "attachment; filename=\"f.txt\"",
;;    :x-xss-protection "0",
;;    :server "gws",
;;    :expires "Sat, 04 Jun 2022 04:10:35 GMT",
;;    :accept-ch
;;    "Sec-CH-Viewport-Width, Sec-CH-Viewport-Height, Sec-CH-DPR, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version, Sec-CH-UA-Arch, Sec-CH-UA-Model, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version-List, Sec-CH-UA-WoW64",
;;    :cache-control "private, max-age=3600",
;;    :report-to
;;    "{\"group\":\"gws\",\"max_age\":2592000,\"endpoints\":[{\"url\":\"https://csp.withgoogle.com/csp/report-to/gws/cdt1\"}]}",
;;    :x-frame-options "SAMEORIGIN",
;;    :strict-transport-security "max-age=31536000",
;;    :content-security-policy
;;    "object-src 'none';base-uri 'self';script-src 'nonce-xM7BqmSpeu5Zd6usKOP4JA' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/cdt1",
;;    :alt-svc
;;    "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
;;    :content-type "application/json; charset=UTF-8",
;;    :cross-origin-opener-policy "same-origin-allow-popups; report-to=\"gws\"",
;;    :content-encoding "br"},
;;   :mime "application/json",
;;   :remote-ip "142.251.41.36"},
;;  :done? true};; => nil

A typical pattern of get-ajax usage is the following. You’d like to check if a certain request has been fired to the server. So you press a button, wait for a while, and then read the requests made by your browser.

Having a list of requests, you search for the one you need (e.g. by its URL) and then check its state. The :state field has the same semantics of the XMLHttpRequest.readyState. It’s an integer from 1 to 4 with the same behavior.

To check if a request has been finished, done, or failed, use these predicates:

;; fill the logs
(e/go driver "https://google.com")
(e/wait 2) ;; give ajax requests a chance to finish

(def reqs (dev/get-ajax driver))
;; you'd search for what you are interested in here
(def req (last reqs))

(dev/request-done? req)
;; => true

(dev/request-failed? req)
;; => nil

(dev/request-success? req)
;; => true

Note that request-done? doesn’t mean the request has succeeded. It only means its pipeline has reached a final step.

Tip	When you read dev logs, you consume them from an internal buffer that gets flushed. The second call to `get-requests` or `get-ajax` will return an empty list.

Perhaps you want to collect these logs. The function dev/get-performance-logs returns a list of logs, so you accumulate them in an atom or whatever:

;; setup a collector
(def logs (atom []))

;; make requests
(e/refresh driver)

;; collect as needed
(do (swap! logs concat (dev/get-performance-logs driver))
    true)

(count @logs)
;; 136

The logs->requests and logs->ajax functions convert already fetched logs into requests. Unlike get-requests and get-ajax, they are pure functions and won’t flush anything.

;; convert our fetched requests from our collector atom
(dev/logs->requests @logs)
(last (dev/logs->requests @logs))
;;    {:state 4,
;;     :id "14535.162",
;;     :type :ping,
;;     :xhr? false,
;;     :url
;;     "https://www.google.com/gen_204?atyp=i&r=1&ei=Zd2aYsrzLozStQbzgbqIBQ&ct=slh&v=t1&m=HV&pv=0.48715273690818806&me=1:1654316389931,V,0,0,1200,1053:0,B,1053:0,N,1,Zd2aYsrzLozStQbzgbqIBQ:0,R,1,1,0,0,1200,1053:93,x:42832,e,U&zx=1654316432856",
;;     :with-data? true,
;;     :request
;;     {:method :post,
;;      :headers
;;      {:Referer "https://www.google.com/",
;;       :sec-ch-ua-full-version-list
;;       "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"102.0.5005.61\", \"Google Chrome\";v=\"102.0.5005.61\"",
;;       :sec-ch-viewport-width "1200",
;;       :sec-ch-ua-platform-version "\"10.15.7\"",
;;       :sec-ch-ua
;;       "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
;;       :sec-ch-ua-platform "\"macOS\"",
;;       :sec-ch-ua-full-version "\"102.0.5005.61\"",
;;       :sec-ch-ua-wow64 "?0",
;;       :sec-ch-ua-model "",
;;       :sec-ch-ua-bitness "\"64\"",
;;       :sec-ch-ua-mobile "?0",
;;       :sec-ch-dpr "1",
;;       :sec-ch-ua-arch "\"x86\"",
;;       :User-Agent
;;       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"}},
;;     :response
;;     {:status nil,
;;      :headers
;;      {:alt-svc
;;       "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"",
;;       :bfcache-opt-in "unload",
;;       :content-length "0",
;;       :content-type "text/html; charset=UTF-8",
;;       :date "Sat, 04 Jun 2022 04:20:32 GMT",
;;       :server "gws",
;;       :x-frame-options "SAMEORIGIN",
;;       :x-xss-protection "0"},
;;      :mime "text/html",
;;      :remote-ip "142.251.41.36"},
;;     :done? true}

When working with logs and requests, pay attention to their count and size. The maps have many keys, and the number of items in collections can be large. Printing a slew of events might freeze your editor. Consider using clojure.pprint/pprint as it relies on max level and length limits.

Postmortem: Auto-save Artifacts in Case of Exception

Sometimes, diagnosing what went wrong during a failed UI test run can be difficult. Use the with-postmortem to save useful data to disk before the exception was triggered:

a screenshot of the visible browser page
HTML code of the current browser page
JS console logs, if available for your browser

Example:

(try
  (e/with-postmortem driver {:dir "target/etaoin-play/postmortem"}
    (e/click driver :non-existing-element))
  (catch Exception _e
    "yup, we threw!"))
;; => "yup, we threw!"

An exception will occur. Under target/etaoin-postmortem you will find three postmortem files named like so: <browser>-<host>-<port>-<datetime>.<ext>, for example:

$ tree target
target
└── etaoin-postmortem
    ├── chrome-127.0.0.1-49766-2022-06-04-12-26-31.html
    ├── chrome-127.0.0.1-49766-2022-06-04-12-26-31.json
    └── chrome-127.0.0.1-49766-2022-06-04-12-26-31.png

The available with-postmortem options are:

{;; directory to save artifacts
 ;; will be created if it does not already exist, defaults to the current working directory
 :dir "/home/ivan/UI-tests"

 ;; directory to save screenshots; defaults to :dir
 :dir-img "/home/ivan/UI-tests/screenshots"

 ;; the same but for HTML sources
 :dir-src "/home/ivan/UI-tests/HTML"

 ;; the same but for console logs
 :dir-log "/home/ivan/UI-tests/console"

 ;; a string template to format a timestamp; See SimpleDateFormat Java class
 :date-format "yyyy-MM-dd-HH-mm-ss"}

Driver Options

When creating a WebDriver instance, you can optionally include an options map to tweak the WebDriver and web browser behaviour.

Here, for example, we set an explicit path to the chrome WebDriver binary:

(def driver (e/chrome {:path-driver "/Users/ivan/downloads/chromedriver"}))

`:host` for WebDriver

Default: <not set>

Example: :host "192.68.1.12"

When:

Specified, Etaoin attempts to connect to an existing running WebDriver process.
Omitted, Etaoin creates a new local WebDriver process (unless webdriver-url is specified).

Alternative: see webdriver-url.

`:port` for WebDriver

Default: Etaoin selects a random unused port when launching a local WebDriver process. When connecting to a remote WebDriver process, it varies by vendor:

chrome 9515
firefox 4444
safari 4445
edge 17556

Example: :port 9997

`:webdriver-url`

Default: <not set>

Example: :web-driver-url "https://chrome.browserless.io/webdriver"

When:

Specified, Etaoin attempts to connect to a pre-existing running WebDriver process.
Omitted, creates a new local WebDriver process (unless :host for WebDriver is specified).

See Connecting to an Existing Running WebDriver.

Alternative: see :host above.

`:path-driver` to WebDriver binary

Default: Varies by browser vendor:

chrome "chromedriver"
firefox "geckodriver"
safari "safaridriver"
edge "msedgedriver"

Example: :path-driver "/Users/ivan/Downloads/geckodriver"

Typically used if your WebDriver binary is not on your PATH.

`:args-driver` for WebDriver binary

Default: <not set>

Example: :args-driver ["--binary" "/path/to/firefox/binary"]
(geckodriver specific, you’d probably use :path-browser to web browser binary instead)

Specifies extra command line arguments to the WebDriver binary.

`:webdriver-failed-launch-retries`

Default:

:safari driver 4
all other drivers 0

Example: :webdriver-failed-launch-retries 3

Introduced to compensate for mysterious but recoverable failed launches of safaridriver.

`:path-browser` to web browser binary

Default: <not set>, The WebDriver implementation will make an attempt to find the web browser binary.

Example: :path-browser "/Users/ivan/Downloads/firefox/firefox

Typically used if your web browser binary is not on your PATH.

`:args` for web browser binary

Default: <not set>

Example: :args ["--incognito" "--app" "http://example.com"]

Specifies extra command line arguments for the web browser binary; see vendor docs for what is available.

`:log-level` for the web browser console

Default: :all

Example: :log-level :err

Web browser minimal console log level. Only messages with this level and above will be collected. From least to most verbose:

nil, :off or :none for no messages
:err, :error, :severe, :crit or :critical
:warn or :warning
:debug
:all for all messages.

Applies to Chrome and Edge only.

See Reading a Browser’s Console Logs

`:driver-log-level`

Default: <not set>

Example: :driver-log-level "INFO"

WebDriver minimal log level. Values vary by browser driver vendor:

chrome & edge "OFF" "SEVERE" "WARNING" "INFO" or "DEBUG"
firefox "fatal" "error" "warn" "info" "config" "debug" or "trace"
safari "debug" - safaridriver
- has only one detailed log level which we enable via its --diagnose option and abstract via "debug"
- only logs to a log file which Etaoin automatically discovers and populates as :driver-log-file in the driver map
- see :post-stop-fns for one way to dump this log file

`:log-stdout` & `:log-stderr` for WebDriver output

Default: no logging: /dev/null, on Windows NUL

Example:

  :log-stdout "target/chromedriver-out.log"
  :log-stderr "target/chromedriver-err.log"

Specify :inherit to have WebDriver process output destination inherit from its calling process (for example, the console or some existing redirection to a file).

`:driver-log-file` for discovered WebDriver log file

Default: <not set> Etaoin will set this value for you

Example: <n/a> Not set by user

Only populated for safaridriver when :driver-log-level set to "debug".

`:post-stop-fns` to hook in behaviour at driver stop

Default: <not set>

Example: One usage is to dump safaridriver :driver-log-file.

:post-stop-fns [(fn dump-discovered-log [driver]
                  (if-let [log (:driver-log-file driver)]
                    (do
                      (println "-[start]-safaridriver log file" log)
                      (with-open [in (io/input-stream log)]
                        (io/copy in *out*))
                      (println "-[end]-safaridriver log file" log))
                    (println "-no safaridriver log file discovered-")))]

`:profile` path to web browser profile

Default: <not set>

Example: :profile "/Users/ivan/Library/Application Support/Firefox/Profiles/iy4iitbg.Test"

Path to custom web browser profile, see Setting the Browser Profile.

`:env` variables for WebDriver process

Default: <not set>

Example: :env {:MOZ_CRASHREPORTER_URL "http://test.com"}

Map of extra environment variables to use when launching the Webdriver process.

`:size` of intial web browser window

Default: [1024 680]

Example: size: [640 480]

Initial web browser window width and height in pixels.

`:url` to open in web browser

Default: <not set>

Example: :url "https://clojure.org"

Only works with Firefox at this time.

`:user-agent` for web browser

Default: <not set>, Governed by WebDriver/web browser vendor

Example: :user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

Overrides the web browser returned User-Agent header.

`:download-dir` for web browser

Default: <not set>, Governed by web browser vendor

Example: :download-dir "target/chrome-downloads"

The directory where the web browser downloads files.

`:headless` web browser

Default: Normally false, but automatically true for driver creation functions like chrome-headless, with-chrome-headless, etc.

Example: :headless true

Run the web browser without a user interface. See Using Headless Drivers.

`:prefs` for web browser

Default: <not set>

Example: see one usage in File Download Directory.

Map of web browser specific preferences.

`:proxy` for web browser

Default: <not set>

Example: See HTTP Proxy

`:load-strategy`

Default: :normal

Example: :load-strategy :none

Controls how long the WebDriver should wait before interacting with a page. See Load Strategy.

`:capabilities`

Default: <not set>

Example: See HTTP Proxy for an example usage.

A WebDriver's capabilities can be vendor-specific and can specify preferred options. Read WebDriver vendor docs before setting anything here. While reading docs, note that Etaoin passes along :capabilities under firstMatch.

Using Headless Drivers

Google Chrome, Firefox, and Microsoft Edge can be run in headless mode. When headless, none of the UI windows appear on the screen. Running without a UI is helpful when:

running integration tests on servers that do not have a graphical output device
running local tests without having them take over your local UI

Ensure your browser supports headless mode by checking if it accepts --headless command-line argument when running it from the terminal.

When starting a driver, pass the :headless boolean flag to switch into headless mode. This flag is ignored for Safari, which, as of August 2024, still does not support headless mode.

(require '[etaoin.api :as e])

(def driver (e/chrome {:headless true})) ;; runs headless Chrome
;; do some stuff
(e/quit driver)

or

(def driver (e/firefox {:headless true})) ;; runs headless Firefox
;; you can also check if a driver is in headless mode:
(e/headless? driver)
;; => true
(e/wait 1) ;; seems to appease Firefox on Linux
(e/quit driver)

There are several shortcuts to run Chrome or Firefox in headless mode:

(def driver (e/chrome-headless))
;; do some stuff
(e/quit driver)

;; or

(def driver (e/firefox-headless {:log-level :all})) ;; with extra settings
;; do some stuff
(e/quit driver)

;; or

(require '[etaoin.api :as e])

(e/with-chrome-headless driver
  (e/go driver "https://clojure.org"))

(e/with-firefox-headless {:log-level :all} driver ;; notice extra settings
  (e/go driver "https://clojure.org"))

There are also the when-headless and when-not-headless macros that conditionally execute a block of commands:

(e/with-chrome driver
  (e/when-not-headless driver
    ;;... some actions that might not be available in headless mode
    )
  ;;... common actions for both versions
  )

File Download Directory

To specify a directory where the browser should download files, use the :download-dir option:

(def driver (e/chrome {:download-dir "target/etaoin-play/chrome-downloads"}))
;; do some downloading
(e/driver quit)

Now, when you click on a download link, the file will be saved to that folder. Currently, only Chrome, Edge and Firefox are supported.

Firefox requires specifying MIME-types of the files that should be downloaded without showing a system dialog. By default, when the :download-dir parameter is passed, the library adds the most common MIME-types: archives, media files, office documents, etc. If you need to add your own one, override that Firefox preference manually via the :prefs option:

(def driver (e/firefox {:download-dir "target/etaoin-play/firefox-downloads"
                        :prefs {:browser.helperApps.neverAsk.saveToDisk
                                "some-mime/type-1;other-mime/type-2"}}))
;; do some downloading
(e/driver quit)

To check whether a file was downloaded during UI tests, see Check Whether a File has been Downloaded.

Managing User-Agent

Set a custom User-Agent header with the :user-agent option when creating a driver, for example:

(e/with-firefox {:user-agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"}
                driver
  (e/wait 1) ;; seems to appease Firefox on Linux
  (e/get-user-agent driver))
;; => "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

Setting this header is important when using headless browsers as many websites implement some sort of blocking when the User-Agent includes the "headless" string. This can lead to 403 responses or some weird behavior on the site.

HTTP Proxy

To set proxy settings use environment variables HTTP_PROXY/HTTPS_PROXY or pass a map of the following type:

{:proxy {:http "some.proxy.com:8080"
         :ftp "some.proxy.com:8080"
         :ssl "some.proxy.com:8080"
         :socks {:host "myproxy:1080" :version 5}
         :bypass ["http://this.url" "http://that.url"]
         :pac-url "localhost:8888"}}

;; example
(e/chrome {:proxy {:http "some.proxy.com:8080"
                   :ssl "some.proxy.com:8080"}})

Note	A `:pac-url` is for a proxy autoconfiguration file. Used with Safari as other proxy options do not work in Safari.

To fine-tune the proxy you use the original object and pass it as capabilities:

(e/chrome {:capabilities
           {:proxy
            {:proxyType "manual"
             :proxyAutoconfigUrl "some.proxy.com:8080"
             :ftpProxy "some.proxy.com:8080"
             :httpProxy "some.proxy.com:8080"
             :noProxy ["http://this.url" "http://that.url"]
             :sslProxy "some.proxy.com:8080"
             :socksProxy "some.proxy.com:1080"
             :socksVersion 5}}})

Connecting to an Existing Running WebDriver

To connect to an existing WebDriver, specify the :host parameter.

Tip	When neither the `:host` nor the `:webdriver-url` parameter is specified, Etaoin will launch a new WebDriver process.

The :host can be a hostname (localhost, some.remote.host.net) or an IP address (127.0.0.1, 183.102.156.31). If the port is not specified, the default :port is assumed.

Both :host and :port are ignored if :webdriver-url is specified.

Example:

;; Connect to an existing chromedriver process on localhost on port 9515
(def driver (e/chrome {:host "127.0.0.1" :port 9515})) ;; for connection to driver on localhost on port 9515

;; Connect to an existing geckodriver process on remote most on default port
(def driver (e/firefox {:host "192.168.1.11"})) ;; the default port for firefox is 4444

;; Connect to a chrome instance on browserless.io via :webdriver-url
;; (replace YOUR-API-TOKEN with a valid browserless.io api token if you want to try this out)
(e/with-chrome {:webdriver-url "https://chrome.browserless.io/webdriver"
                :capabilities {"browserless:token" "YOUR-API-TOKEN"
                               "chromeOptions" {"args" ["--no-sandbox"]}}}
               driver
  (e/go driver "https://en.wikipedia.org/")
  (e/wait-visible driver [{:id :simpleSearch} {:tag :input :name :search}])
  (e/fill driver {:tag :input :name :search} "Clojure programming language")
  (e/fill driver {:tag :input :name :search} k/enter)
  (e/get-title driver))
;; => "Clojure programming language - Search results - Wikipedia"

Setting the Browser Profile

When running Chrome or Firefox, you may specify a special web browser profile made for test purposes. A profile is a folder that keeps browser settings, history, bookmarks, and other user-specific data.

Imagine, for example, that you’d like to run your integration tests against a user that turned off Javascript execution or image rendering.

Tip	This is a hypothetical example. Turning off JavaScript will affect/break certain WebDriver features. And it can affect certain WebDriver implementations, for example.

Create and Find a Profile in Chrome

In the top right corner of the main window, click on a user button.
In the dropdown, select "Manage People".
Click "Add person", submit a name, and press "Save".
The new browser window should appear. Now, setup the new profile as you want.
Open chrome://version/ page. Copy the file path that is beneath the Profile Path caption.

Create and Find a Profile in Firefox

Run Firefox with -P, -p, or -ProfileManager key as the official page describes.
Create a new profile and run the browser.
Setup the profile as needed.
Open about:support page. Near the Profile Folder caption, press the Show in Finder button. A new folder window should appear. Copy its path from there.

Running a Driver with a Profile

Once you’ve got a profile path, launch a driver with the :profile key as follows:

;; Chrome
(def chrome-profile
  "/Users/ivan/Library/Application Support/Google/Chrome/Profile 2/Default")

(def chrome-driver (e/chrome {:profile chrome-profile}))

;; Firefox
(def ff-profile
  "/Users/ivan/Library/Application Support/Firefox/Profiles/iy4iitbg.Test")

(def firefox-driver (e/firefox {:profile ff-profile}))

Writing and Running Integration Tests For Your Application

Headless Testing

It is not unusual for Continuous Integration services to have no display. This is especially true for Linux runners.

When running your tests on Linux with no display, you have 2 choices:

Run the WebDriver in headless mode
Use a virtual display

Note	Things being what they are, WebDrivers can behave differently when run headless.

The technologies we use for Etaoin’s CI testing on GitHub Actions for Linux are:

Xvfb - acts as an X virtual display
fluxbox - a lightweight windows manager (needed by geckodriver/Firefox to support window positioning operations)

You can see how we make use of these tools in the Etaoin test script, but in a nutshell:

To install:

sudo apt get install -y xvfb fluxbox

Tip	As of this writing Xvfb is pre-installed on the linux runner on GitHub Actions, but fluxbox is not.

Ensure DISPLAY env var is set:

export DISPLAY=:99.0

Launch the virtual display and fluxbox:

Xvfb :99 -screen 0 1024x768x24 &
fluxbox -display :99 &

Basic Fixture

It is desirable to have your tests be independent of one another. One way to achieve this is through the use of a test fixture. The fixture’s job is to, for each test:

create a new driver
run the test with the driver
shutdown the driver

A dynamic *driver* var might be used to hold the driver.

(ns project.test.integration
  "A module for integration tests"
  (:require [clojure.test :refer [deftest is use-fixtures]]
            [etaoin.api :as e]))

(def ^:dynamic *driver*)

(defn fixture-driver
  "Executes a test running a driver. Bounds a driver
   with the global *driver* variable."
  [f]
  (e/with-chrome [driver]
    (binding [*driver* driver]
      (f))))

(use-fixtures
  :each ;; start and stop driver for each test
  fixture-driver)

;; now declare your tests

(deftest ^:integration
  test-some-case
  (doto *driver*
    (e/go url-project)
    (e/click :some-button)
    (e/refresh)
    ...
    ))

If, for some reason, you want to reuse a single driver instance for all tests:

(ns project.test.integration
  "A module for integration tests"
  (:require [clojure.test :refer [deftest is use-fixtures]]
            [etaoin.api :as e]))

(def ^:dynamic *driver*)

(defn fixture-browser [f]
  (e/with-chrome-headless driver
    (e/disconnect-driver driver)
    (binding [*driver* driver]
      (f))
    (e/connect-driver driver)))

;; creating a session every time that automatically erases resources
(defn fixture-clear-browser [f]
  (e/connect-driver *driver*)
  (e/go *driver* "http://google.com")
  (f)
  (e/disconnect-driver *driver*))

;; this is run `once` before running the tests
(use-fixtures
  :once
  fixture-browser)

;; this is run `every` time before each test
(use-fixtures
  :each
  fixture-clear-browser)

...some tests

For faster testing, you can use this example:

.....

(defn fixture-browser [f]
  (e/with-chrome-headless driver
    (binding [*driver* driver]
      (f))))

;; note that resources, such as cookies, are deleted manually,
;; so this does not guarantee that the tests are clean
(defn fixture-clear-browser [f]
  (e/delete-cookies *driver*)
  (e/go *driver* "http://google.com")
  (f))

......

Multi-Driver Fixtures

In the example above, we examined a case when you run tests against a single type of driver. However, you may want to test your site on multiple drivers, say, Chrome and Firefox. In that case, your fixture may become a bit more complex:

(def driver-type [:firefox :chrome])

(defn fixture-drivers [f]
  (doseq [type driver-types]
    (e/with-driver type {} driver
      (binding [*driver* driver]
        (testing (format "Testing in %s browser" (name type))
          (f))))))

Now, each test will be run twice. Once for Firefox and then once for Chrome. Please note the test call is prepended with the testing macro that puts the driver name into the report. When a test fails, you’ll know what driver failed the test.

Tip	See also Etaoin’s API tests for an example of this strategy.

Postmortem Handler To Collect Artifacts

To save some artifacts in case of an exception, wrap the body of your test into the with-postmortem handler as follows:

(deftest test-user-login
  (e/with-postmortem *driver* {:dir "/path/to/folder"}
    (doto *driver*
      (e/go "http://127.0.0.1:8080")
      (e/click-visible :login)
      ;; any other actions...
      )))

If any exception occurs in that test, artifacts will be saved.

To avoid copying and pasting the options map, declare it at the top of the module. If you use Circle CI, it would be great to save the data into a special artifacts directory that might be downloaded as a zip file once the build has been finished:

(def pm-dir
  (or (System/getenv "CIRCLE_ARTIFACTS") ;; you are on CI
      "/some/local/path"))               ;; local machine

(def pm-opt
  {:dir pm-dir})

Now pass that map everywhere into PM handler:

  ;; test declaration
  (e/with-postmortem *driver* pm-opt
    ;; test body goes here
    )

When an error occurs, you will find a PNG image of your browser page at the time of the exception and an HTML dump.

See Postmortem: Auto-save Artifacts in Case of Exception.

Running Tests By Tag

Since UI tests may take lots of time to pass, it’s definitely a good practice to pass both server and UI tests independently from each other.

If you are using leiningen, here are a few tips.

First, add ^:integration tag to all the tests that are run under the browser as follows:

(deftest ^:integration
  test-password-reset-pipeline
  (doto *driver*
    (go url-password-reset)
    (click :reset-btn)
    ;; and so on...
  ))

Then, open your project.clj file and add test selectors:

:test-selectors {:default (complement :integration)
                 :integration :integration}

Now, when you launch lein test you will run all the tests except browser integration tests. To run integration tests, launch lein test :integration.

Check Whether a File has been Downloaded

Sometimes, a file downloads automatically when you click a link or visit a page. In tests, you might need to ensure a file has been downloaded successfully. A typical scenario would be:

Provide a custom empty download folder when running a browser (see File Download Directory).
Click on a link or perform any action needed to start the file download.
Wait for some time; for small files, 5-10 seconds would be enough.
Using files API, scan that directory and try to find a new file. Check if it matches a proper extension, name, creation date, etc.

Example:

(require '[clojure.java.io :as io]
         '[clojure.string :as str])

;; Local helper that checks whether it is really an Excel file.
(defn xlsx? [file]
  (-> file
      .getAbsolutePath
      (str/ends-with? ".xlsx")))

;; Top-level declarations
(def DL-DIR "/Users/ivan/Desktop")
(def driver (e/chrome {:download-dir DL-DIR}))

;; Later, in tests...
(e/click-visible driver :download-that-application)
(e/wait driver 7) ;; wait for a file has been downloaded

;; Now, scan the directory and try to find a file:
(let [files (file-seq (io/file DL-DIR))
      found (some xlsx? files)]
  (is found (format "No *.xlsx file found in %s directory." DL-DIR)))

Running Selenium IDE files

Etaoin can play the files produced by Selenium IDE. Selenium IDE allows you to record web interactions for later playback. It is installed as an optional extension in your web browser.

Once installed and activated, it records your actions to a JSON file with the .side extension. You can save that file and run it with Etaoin.

Let’s imagine you’ve installed the IDE and recorded some actions as per Selenium IDE documentation. Now that you have a test.side file, you could do something like this:

(require '[clojure.java.io :as io]
         '[etaoin.api :as e]
         '[etaoin.ide.flow :as flow])

(def driver (e/chrome))

(def ide-file (io/resource "ide/test.side"))

(def opt
    {;; The base URL redefines the one from the file.
     ;; For example, the file was written on the local machine
     ;; (http://localhost:8080), and we want to perform the scenario
     ;; on staging (https://preprod-001.company.com)
     :base-url "https://preprod-001.company.com"

     ;; keywords :test-.. and :suite-.. (id, ids, name, names)
     ;; are used to select specific tests. When not passed,
     ;; all tests get run. For example:

     :test-id "xxxx-xxxx..."         ;; a single test by its UUID
     :test-name "some-test"          ;; a single test by its name
     :test-ids ["xxxx-xxxx...", ...] ;; multiple tests by their ids
     :test-names ["some-test1", ...] ;; multiple tests by their names

     ;; the same for suites:

     :suite-id    ...
     :suite-name  ...
     :suite-ids   [...]
     :suite-names [...]})

(flow/run-ide-script driver ide-file opt)

Everything related to the IDE feature can be found under the etaoin.ide namespace.

CLI Arguments

You may also run a .side script from the command line. Here is a clojure example:

clojure -M -m etaoin.ide.main -d firefox -p '{:port 8888}' -r ide/test.side

As well as from an uberjar. In this case, Etaoin must be in the primary dependencies, not the :dev or :test related.

java -cp .../poject.jar -m etaoin.ide.main -d firefox -p '{:port 8888}' -f ide/test.side

We support the following arguments (check them out using the clojure -M -m etaoin.ide.main -h command):

  -d, --driver-name name   :chrome  The name of driver. The default is `:chrome`
  -p, --params params      {}       Parameters for the driver represented as an
                                    EDN string, e.g '{:port 8080}'
  -f, --file path                   Path to an IDE file on disk
  -r, --resource path               Path to an IDE resource
      --test-ids ids                Comma-separeted test ID(s)
      --suite-ids ids               Comma-separeted suite ID(s)
      --test-names names            Comma-separeted test name(s)
      --suite-names names           Comma-separeted suite name(s)
      --base-url url                Base URL for tests
  -h, --help

Pay attention to --params. This must be an EDN string representing a Clojure map. That’s the same map that you pass into a driver at creation time.

Please note the IDE support is still experimental. If you encounter unexpected behavior, feel free to open an issue. At the moment, we only support Chrome and Firefox for IDE files.

Webdriver in Docker

To work with the WebDrivers in Docker, you can take ready-made images:

Example for Chrome:

docker run --name chromedriver -p 9515:4444 -d -e CHROMEDRIVER_WHITELISTED_IPS='' robcherry/docker-chromedriver:latest

for Firefox:

docker run --name geckodriver -p 4444:4444 -d instrumentisto/geckodriver

To connect to an existing running WebDriver process, you need to specify the :host. In this example :host would be localhost or 127.0.0.1. The :port would be the appropriate port for the running WebDriver process as exposed by docker. If the port is not specified, the default port is set.

(def driver (e/chrome-headless {:host "localhost" :port 9515 :args ["--no-sandbox"]}))
(def driver (e/firefox-headless {:host "localhost"})) ;; will try to connect to port 4444

Troubleshooting

WebDriver Support Across Vendors is Inconsistent

Vendors don’t always do a great job of completely following or implementing the W3C WebDriver Spec.

The web-plaforms-tests dashboard is a great way to get a feel for vendor support of the spec.

Old Versions of WebDrivers can have Limitations

Reproduction	For example, `chromedriver` used to throw an error when calling `maximize`: (e/with-chrome driver (e/maximize driver)) ;; an exception with "cannot get automation extension" was thrown
Cause	This was a bug in `chromedriver` that was fixed in chromedriver v2.28.
Solution	Updating to the current WebDriver resolved the issue.

New Versions of WebDrivers can Introduce Bugs

Reproduction	For example, `chromedriver` v103 started sporadically failing with `unknown error: cannot determine loading status\nfrom unknown error: unexpected command response`.
Cause	Likely a bug in chromedriver
Solution	Upgrade to newer version that fixes bug. For this particular bug I simply suffered retrying failed tests while waiting for newer version.

Clicking on a Link Does not Work with Safari

Reproduction

An attempt to click on a link does nothing.

Given the following HTML

<div class="faculty_course_title" title="Course Title">
 <a href="https://clojure.org">Course Title</a>
</div>

An attempt to click on the link does nothing:

(e/click-single driver [{:tag :div :class "faculty_course_title"} {:tag :a}])

Cause

Odd as it may seem, this is a long-standing (maybe on-again-off-again?) bug in safaridriver.

Work-around

You can, if you wish, employ JavaScript to click the link:

(let [elem (e/query driver [{:tag :div :class "faculty_course_title"} {:tag :a}])]
  (e/js-execute driver "arguments[0].click()" (e/el->ref elem)))

XPath and Searching from Root vs Current node

Reproduction

;; we intend to find an element with the text 'some' under an element with id 'mishmash'
(e/get-element-text driver [{:id :mishmash} "//*[contains(text(),'some')]"])
;; => "A little sample page to illustrate some concepts described in the Etaoin user guide."
;; but we've found the first element with text 'some'

Cause

In a vector, every expression searches from the previous one in a loop. Without a leading dot, the XPath "//..." clause means to find an element from the root of the whole page. With a dot, it means to find from the current node, which is one from the previous query, and so forth.

Solution

Add the XPath dot.

(e/get-element-text driver [{:id :mishmash} ".//*[contains(text(),'some')]"])
;; => "some other paragraph"
;; that's what we were looking for!

Clicking On Non-Visible Element

Reproduction

(e/click driver :cantseeme)
;; as of this writing, on chrome throws an exception with message containing 'not interactable'

Cause

You cannot interact with an element that is not visible or is so small that a human could not click on it.

Selectors not Working

Symptom: Selectors for locating elements are not working, even though the elements are clearly available.
Possible cause: Your script may have clicked a link that opened a new tab or window. Even though the new window is in the foreground, the driver instance is still connected to the original window.
Solution: Call switch-window-next when a new tab or window is opened to point the driver to the new tab/window.

Unpredictable errors in Chrome when the window is not active

Reproduction: when you focus on another window, a WebDriver session that is run under Google Chrome fails.
Solution: Google Chrome may suspend a tab when it has been inactive for some time. When the page is suspended, no operation can be done on it. No clicks, Js execution, etc. So try to keep Chrome window active during test session.

Invalid argument: can’t kill an exited process for Firefox

Reproduction

When you try to start the driver you get an error:

(def driver (e/firefox {:headless true}))
;; throws an exception containing message with 'invalid argument: can't kill an exited process'

Possible Cause

Running Firefox as root in a regular user’s session is not supported

To Diagnose

Run the driver with the path to the log files and the "trace" log level and explore the output.

(def driver (firefox {:log-stdout "ffout.log" :log-stderr "fferr.log" :driver-log-level "trace"}))

Similar Problem

mozilla/geckodriver#1655

DevToolsActivePort file doesn’t exist error on Chrome

Reproduction: When you try to start the chromedriver you get an error:

(def driver (e/chrome))
;; throws an exception with message containing 'DevToolsActivePort file doesn't exist'

Possible Cause

A common cause for Chrome to crash during startup is running Chrome as root user (administrator) on Linux. While it is possible to work around this issue by passing --no-sandbox flag when creating your WebDriver session, such a configuration is unsupported and discouraged. Ideally you would configure your environment to run Chrome as a regular user instead.

Note	We noticed that the Selenium docker images always invoke chrome with `--no-sandbox`. Our test docker images don’t run from the root user, so we’ve not bothered with this.

Potential Solution

Run driver with argument --no-sandbox. Caution! This bypasses the OS security model.

(e/with-chrome {:args ["--no-sandbox"]} driver
  (e/go driver "https://clojure.org"))

Similiar Problem

A similar problem is described here

Files

01-user-guide.adoc

Latest commit

History

01-user-guide.adoc

File metadata and controls

User Guide

Introduction

History

Interesting Alternatives

Supported OSes & Browsers

Installation

Add the Etaoin Library Dependency

For Clojure Users

For Bababashka Users

Installing the Browser WebDrivers

Getting Started

Playing Along in your REPL

More Getting Started

Unit Tests as Docs

Creating and Quitting the Driver

Selecting Elements

Simple Queries, XPath, CSS

Map Syntax Queries

Vector Syntax Queries

Advanced Queries

Querying the nth Element Matched

Querying a Tree

Querying the Shadow DOM

Interacting with Queried Elements

Interactions

UNICODE and Emojis

Emulate how a Real Person Might Type

Mouse Clicks

Selecting an option from a dropdown

Keyboard Chords

File Uploading

Scrolling

Working with frames and iframes

Executing Javascript

Asynchronous Scripts

Wait Functions

Load Strategy

Actions

Capturing Screenshots

Screenshots for Specific Elements

Screenshots after each form

Print Page to PDF

Peeking deeper

Invoking WebDriver Implementation Specific Features

Reading a Browser’s Console Logs

DevTools: Tracking HTTP Requests, XHR (Ajax)

Postmortem: Auto-save Artifacts in Case of Exception

Driver Options

:host for WebDriver

:port for WebDriver

:webdriver-url

:path-driver to WebDriver binary

:args-driver for WebDriver binary

:webdriver-failed-launch-retries

:path-browser to web browser binary

:args for web browser binary

:log-level for the web browser console

:driver-log-level

:log-stdout & :log-stderr for WebDriver output

:driver-log-file for discovered WebDriver log file

:post-stop-fns to hook in behaviour at driver stop

:profile path to web browser profile

:env variables for WebDriver process

:size of intial web browser window

:url to open in web browser

:user-agent for web browser

:download-dir for web browser

:headless web browser

:prefs for web browser

:proxy for web browser

:load-strategy

:capabilities

Using Headless Drivers

File Download Directory

`:host` for WebDriver

`:port` for WebDriver

`:webdriver-url`

`:path-driver` to WebDriver binary

`:args-driver` for WebDriver binary

`:webdriver-failed-launch-retries`

`:path-browser` to web browser binary

`:args` for web browser binary

`:log-level` for the web browser console

`:driver-log-level`

`:log-stdout` & `:log-stderr` for WebDriver output

`:driver-log-file` for discovered WebDriver log file

`:post-stop-fns` to hook in behaviour at driver stop

`:profile` path to web browser profile

`:env` variables for WebDriver process

`:size` of intial web browser window

`:url` to open in web browser

`:user-agent` for web browser

`:download-dir` for web browser

`:headless` web browser

`:prefs` for web browser

`:proxy` for web browser

`:load-strategy`

`:capabilities`