The following is a collection of notes on how to automate any website with hopefully zero knowledge about programming but with some knowledge on web technologies such as HTML and CSS (XPath, CSS selectors, and so forth.). The goal is to generate a workflow visually, such as logging in to a website, or clicking some buttons on a website, and then automate the recorded process on a server.

• Selenium IDE browser addon, for Firefox the Selenium IDE can be downloaded on the Mozilla addons page and should be installed on the machine that will generate the automation workflow,
• selenium-side-runner and a webdriver of choice must be installed, for this example, the Google Chrome webdriver will be used.

Assuming that nodejs is installed on the machine that will playback the workflow, the following commands install the selenium side runner:

npm install -g selenium-side-runner
npm install -g --unsafe-perm=true chromedriver

The --unsafe-perm=true parameter is required in case an error is received when trying to install the Chrome driver.

• a browser, since this example uses the Google Chrome webdriver, the Google Chrome web-browser will be installed. Google Chrome can be obtained from Google. On Debian or Ubuntu, when attempting to install the package, an error about missing dependencies might be issued. Install the required dependencies, including X (unfortunately, even though it will not be used):

apt-get install xdg-utils

Now the following requirements should have been installed:

• Selenium IDE browser addon on your work machine (client),
• selenium-side-runner, chromedriver and Google Chrome on the machine that will execute the workflow (server)

This step is self-explanatory for the most part. Clicking the Selenium IDE addon button will launch a new window for the Selenium IDE. The principle is easy to grasp: you specify a website URL to automate, press the record button, perform a number of actions on that website (clicking buttons, filing in forms, etc.) and then play back the script to check that the script is performing all the actions properly.

In case the script fails to start at the very beginning, simply insert a new command at the very beginning of the script. Similarly, in case the Selenium IDE indicates an error mentioning CSP, on Firefox this error can be bypassed by going to about:config and setting security.csp.enable to false.

Here is a screenshot of an example script used to log-in to www.myanonamouse.net:

As can be observed, all the commands are listed as the script has been recording and if the play button is pressed, the script will perform all commands one after the other.

Once a script has been generated and tested to be working properly, the script can be saved by clicking on the diskette button to a file ending in .side (for example www.myanonamouse.net-login.side) that has then to be transferred to the machine that will perform the automation.

With the side script transferred to the machine that will run the automation, in this example, www.myanonamouse.net-login.side, the script can then be executed headlessly using the Google Chrome web driver. For example, issue:

selenium-side-runner -c "goog:chromeOptions.args=[--headless,--nogpu,--no-sandbox] browserName=chrome"  www.myanonamouse.net-login.side

where:

• www.myanonamouse.net-login.side is the script transferred from the previous step

Note that due to security restrictions, the Google Chrome webdriver will only run as a regular user on Linux. Running the script as the root user will generate errors that will lead to the Chrome webdriver to fail.

The script should now run and perform all the steps in the background without opening any visible browser windows. After running, selenium-side-runner will display some statistics such as whether the script ran successfully and the time it took for the script to run.

Assuming that a Selenium IDE side-script named www.myanonamouse.net-login.side exists at /opt/myanonamouse/www.myanonamouse.net-login.side, the following script can be placed under /etc/cron.weekly in order to run the Selenium IDE script once a week:

#!/bin/bash
 
# Restart the script as an user via su.
if [ "$(id -u)" -eq 0 ]; then
    # exec replaces current shell
    exec su myanonamouse "$0" -- "$@"
fi
 
# Run the Selenium side script.
cd /opt/myanonamouse
selenium-side-runner \
    -c "goog:chromeOptions.args=[--headless,--nogpu,--no-sandbox] browserName=chrome" \
    /opt/myanonamouse/www.myanonamouse.net-login.side 2>&1 2>/dev/null >/dev/null

Whilst "selenium-side-runner" is part of the Selenium project, it seems that installing the runner with NPM will result in x86 binaries being installed that will be incompatible with ARM architectures. Furthermore, the official runner has some issues with timeouts that cannot be configured and that might be relevant on busy systems. However, the side-runner does not depend on Selenium directly and anyone can use any programming language or framework to implement a runner. For example, there exists a project that implements the Selenium side runner in Java.

The Selenium grid is a testbed that can be accessed using the selenium runner remote driver and that is supposed to have various browsers available for testing. The easiest to use is the standalone grid version by running a Docker container provided by the official project repository. In our case, the image was started using SystemD via the command:

/usr/bin/docker run --name=selenium \
  --rm \
  --interactive \
  --hostname selenium \
  -p 4444:4444 \
  -p 7900:7900 \
  --shm-size="2g" \
  selenium/standalone-chromium:latest

After running the container, selenium-side-runner can be pointed at the container on port 4444 in order to run Selenium tests. For instance, using the Java Selenium side runner:

java -jar /usr/local/bin/selenese-runner.jar \
  --driver remote \
  --remote-browser chrome \
  --remote-url http://selenium:4444 \
  --headless \
  /tmp/test.side

where:

• --remote-browser chrome requests a Chrome driver to run the tests and assumes that either selenium/standalone-chromium:latest or selenium/standalone-chrome:latest images are running,
• http://selenium:4444 is the address of the Selenium standalone grid container,
• /tmp/test.side is the Selenium test script generated with the Selenium IDE to execute