Fetching pages with WebDriver

Javascript support / Fetching via WebDriver

The backend can be configured to fetch pages via Chrome using the built in WebDriver network interface, this is mainly used where the pages you are watching are using Javascript to render the page content (the basic fetcher does not execute any JS!), The easiest way is to enable it is to uncomment the following in your local docker-compose.yml and restart your docker-compose.

Note: RaspberryPi needs a different selenium/webdriver runner, please edit your docker-compose.yml and use the recommended RaspberryPi image more information here

    browser-chrome:
        hostname: browser-chrome
        image: selenium/standalone-chrome-debug:3.141.59
        volumes:
            # Workaround to avoid the browser crashing inside a docker container
            # See https://github.com/SeleniumHQ/docker-selenium#quick-start
            - /dev/shm:/dev/shm
        restart: unless-stopped

If using docker (instead of docker-compose) the following will get ChangeDetection.io and the chromium WebDriver up and running:

docker run -d \
  --name selenium \
  --restart unless-stopped \
  -p 4444:4444 \
  --shm-size="2g" \
  selenium/standalone-chrome-debug:3.141.59

docker run -d \
  --name changedetection.io \
  --restart unless-stopped \
  --link selenium \
  -p 5000:5000 \
  -e WEBDRIVER_URL="http://localhost:4444/wd/hub" \
  -v datastore-volume:/datastore \
  dgtlmoon/changedetection.io

Then visit /settings and [Fetching] tab and enable the WebDriver/Chrome option

The URL for the WebDriver interface is set with the WEBDRIVER_URL environment variable (http://browser-chrome:4444/wd/hub by default)

Raspberry Pi

Known to work with RaspberryPi-4 Use the seleniarm/standalone-chromium:4.0.0-20211213 as image:. Note that this currently only supports the 64-bit version of Raspbian OS.

• It's good to set the ENV var FETCH_WORKERS to a low value like 2 or 3, because maybe having 10 sessions of chrome open might be a bit too heavy on your rPi

There's a few things left todo

• Make tests use the full docker stack with the chromedriver + relevant settings
• Lower the number of workers (make it configurable per backend?) 10 is too many (is it?), but this is totally fine for requests/plaintext
• Some kind of dom-wait-until-fully loaded or other
• Handle failures better (better feedback so we know if its from chromedriver, or from the website)
• Save last screenshot? (This extends to saving an entire page screenshot, abstracting out the handler for diff-management, then supplying an image-diff handler)