Amador Pahim

Testing

Running Robot Framework tests with Avocado

apahim

Avocado Framework is a Testing Framework born to replace Autotest/Virttest. It’s extremely flexible, feature rich and has a powerful Test API.

Robot Framework is also a Testing Framework. It has easy-to-use tabular test data syntax and it utilizes the keyword-driven testing approach.

Avocado has a very sane architecture, well integrated with most popular CIs and providing all the output formats you will ever need. Its Test API is intended to be used by code writers. The full Test API is available for tests written in Python. And since any executable can serve a test for Avocado, some features are also available through environment variables and output analysers.

On the other hand, Robot has probably the most easy-to-use Test API, providing a fast learning experience for test writers.

This post shows how to take advantage of both worlds by running Robot tests with Avocado.


Installing

To install Avocado, run:

~$ sudo pip install avocado-framework

Here you can find more information about the Avocado installation methods: http://avocado-framework.readthedocs.io/en/47.0/GetStartedGuide.html

To install Robot, run:

~$ sudo pip install robotframework

Here you can find more information about the Robot installation methods:
https://github.com/robotframework/robotframework/blob/master/INSTALL.rst

selenium2library is arguably the most popular library of Robot. For the purpose of this post, let’s use selenium2library. To install it, run:

~$ sudo dnf install python-selenium
~$ sudo pip install robotframework-selenium2library

Web Testing Demo Package

To have a practical example, let’s download the Web Demo package (please check the current version download link):

$ cd ~/Downloads
~/Downloads$ wget https://bitbucket.org/robotframework/webdemo/downloads/WebDemo-20150901.zip
~/Downloads$ unzip WebDemo-20150901.zip
Archive: WebDemo-20150901.zip
inflating: WebDemo/README.rst
inflating: WebDemo/demoapp/server.py
inflating: WebDemo/demoapp/html/welcome.html
inflating: WebDemo/demoapp/html/error.html
inflating: WebDemo/demoapp/html/index.html
inflating: WebDemo/demoapp/html/demo.css
inflating: WebDemo/login_tests/resource.robot
inflating: WebDemo/login_tests/gherkin_login.robot
inflating: WebDemo/login_tests/invalid_login.robot
inflating: WebDemo/login_tests/valid_login.robot

Before running the Web Demo app tests, we have to start the app
server:

~$ python ~/Downloads/WebDemo/demoapp/server.py &
[1] 6210 Demo server starting on port 7272.

The standard usage of Robot would be:

~$ robot ~/Downloads/WebDemo/login_tests/

Avocado Basic Usage

Tests using the Avocado API are called INSTRUMENTED tests. For example, the Python test file below:

import avocado

class MyTest(avocado.Test):
    def test1(self):
        self.assertEqual(1, 1)

    def test2(self):
        self.assertEqual(2, 2)

Will be resolved by Avocado like this:

~$ avocado list ~/avocado/tests/test_demo.py
INSTRUMENTED /home/apahim/avocado/tests/test_demo.py:MyTest.test1
INSTRUMENTED /home/apahim/avocado/tests/test_demo.py:MyTest.test2

And executed like this:

~$ avocado run ~/avocado/tests/test_demo.py
JOB ID : bcea40d3b59737036057b974250aa8a674400409
JOB LOG : /home/apahim/avocado/job-results/job-2017-03-29T13.23-bcea40d/job.log
(1/2) /home/apahim/avocado/tests/test_demo.py:MyTest.test1: PASS (0.01 s)
(2/2) /home/apahim/avocado/tests/test_demo.py:MyTest.test2: PASS (0.01 s)
RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
TESTS TIME : 0.02 s
JOB HTML : /home/apahim/avocado/job-results/job-2017-03-29T13.23-bcea40d/html/results.html

But, as I said before, anything can serve as a test. Executables that are not INSTRUMENTED tests are SIMPLE tests. Example:

~$ avocado list /bin/true /bin/false
SIMPLE /bin/true
SIMPLE /bin/false

Running SIMPLE tests will provide similar output, only with a simpler test name:

~$ avocado run /bin/true /bin/false
JOB ID : b70b2905ece8f77e43f56bfec7640f440cacf8e1
JOB LOG : /home/apahim/avocado/job-results/job-2017-03-29T13.26-b70b290/job.log
(1/2) /bin/true: PASS (0.03 s)
(2/2) /bin/false: FAIL (0.06 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
TESTS TIME : 0.09 s
JOB HTML : /home/apahim/avocado/job-results/job-2017-03-29T13.26-b70b290/html/results.html

Avocado External Runner

A test runner is the component that knows how to run the test. Avocado has a built-in test runner. It knows how to execute the Avocado test types (INSTRUMENTED and SIMPLE). Avocado also supports external test runners to run tests that the built-in test runner doesn’t know how to run.

For example, If you tell Avocado to run a test called 5, Avocado will fail to run the test:

~$ avocado run 5
Unable to resolve reference(s) '5' with plugins(s) 'file', 'external',
try running 'avocado list -V 5' to see the details.

But the sleep command knows what to do with a 5, right? 🙂
Let’s tell Avocado to use sleep as an external runner:

$ avocado run 5 --external-runner /bin/sleep
JOB ID : 64e88026153e5d391579a67a6bad7d201f059485
JOB LOG : /home/apahim/avocado/job-results/job-2017-03-29T13.31-64e8802/job.log
(1/1) 5: PASS (5.03 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
TESTS TIME : 5.03 s
JOB HTML : /home/apahim/avocado/job-results/job-2017-03-29T13.31-64e8802/html/results.html

Avocado will basically call the /bin/sleep, passing the 5 as an argument, and use the command exit code to determine if test either PASSed or FAILed.

With that in mind, even Avocado not knowing what Robot tests are, we could easily run the robot tests in Avocado using an external runner.

Command below will fail:

~$ avocado run ~/Downloads/WebDemo/login_tests/
Unable to resolve reference(s) '~/Downloads/WebDemo/login_tests/'
with plugins(s) 'file', 'external', try running
'avocado list -V ~/Downloads/WebDemo/login_tests/' to see the details.

But using robot as the external runner will execute the test suite:

~$ avocado run ~/Downloads/WebDemo/login_tests/ --external-runner /usr/bin/robot
JOB ID : b31c4a312c8ad984a72370c8ea9762b9444eb073
JOB LOG : $HOME/avocado/job-results/job-2017-03-16T16.33-b31c4a3/job.log
(1/1) $HOME/Downloads/WebDemo/login_tests/: PASS (9.15 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
TESTS TIME : 9.15 s
JOB HTML : $HOME/avocado/job-results/job-2017-03-16T16.33-b31c4a3/html/results.html

The inconvenience with the execution above is that all Robot tests present in the suite will be executed as only one Avocado test. Of course you can inspect the Avocado job.log to see all tests results, but if you are expecting more granularity, making each Robot test to be a unique Avocado test, you have to use the Avocado Robot Plugin.


Avocado Robot Plugin

To make the integration with Robot more powerful, the Avocado v48 codename “Lost Boundaries” shipped the first experimental version of the Robot plugin. You can Install it with:

~$ sudo pip install avocado-framework-plugin-robot

With this new plugin, you can run Robot tests the same way you run Avocado INSTRUMENTED and SIMPLE tests.

List tests with:

~$ avocado list ~/Downloads/WebDemo/login_tests/
ROBOT /home/apahim/Downloads/WebDemo/login_tests/gherkin_login.robot:Gherkin Login.Valid Login
ROBOT /home/apahim/Downloads/WebDemo/login_tests/valid_login.robot:Valid Login.Valid Login
ROBOT /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Invalid Username
ROBOT /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Invalid Password
ROBOT /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Invalid Username And Password
ROBOT /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Empty Username
ROBOT /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Empty Password
ROBOT /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Empty Username And Password

You can also use the Avocado test references syntax to filter the tests that you want to list and/or run:

~$ avocado list ~/Downloads/WebDemo/login_tests/:Valid\ Login
ROBOT /home/apahim/Downloads/WebDemo/login_tests/gherkin_login.robot:Gherkin Login.Valid Login
ROBOT /home/apahim/Downloads/WebDemo/login_tests/valid_login.robot:Valid Login.Valid Login

To run the tests, just replace the list command by the run command:

~$ avocado run ~/Downloads/WebDemo/login_tests/
JOB ID : 023d41582548d3a504370c236678611c5e3eb12a
JOB LOG : /home/apahim/avocado/job-results/job-2017-03-29T13.41-023d415/job.log
(1/8) /home/apahim/Downloads/WebDemo/login_tests/gherkin_login.robot:Gherkin Login.Valid Login: PASS (2.67 s)
(2/8) /home/apahim/Downloads/WebDemo/login_tests/valid_login.robot:Valid Login.Valid Login: PASS (2.64 s)
(3/8) /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Invalid Username: PASS (2.75 s)
(4/8) /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Invalid Password: PASS (2.70 s)
(5/8) /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Invalid Username And Password: PASS (2.70 s)
(6/8) /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Empty Username: PASS (2.68 s)
(7/8) /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Empty Password: PASS (2.70 s)
(8/8) /home/apahim/Downloads/WebDemo/login_tests/invalid_login.robot:Invalid Login.Empty Username And Password: PASS (2.64 s)
RESULTS : PASS 8 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
TESTS TIME : 21.48 s
JOB HTML : /home/apahim/avocado/job-results/job-2017-03-29T13.41-023d415/html/results.html

Here you can watch the commands above in action:

As you can see, this plugin causes the individual Robot tests to become visible directly to Avocado. Now Avocado can run them one-by-one and report their individual status. You can now use Avocado advanced features, like the varianter, the job replay and so on, with your Robot tests.

Hope you give this plugin a try. Your feedback on both this post and the plugin is more than appreciated.

Cya.

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top