|Note: This tutorial assumes that you have completed the previous tutorials: basic usage of roslisp.|
|Please ask about problems and questions regarding this tutorial on answers.ros.org. Don't forget to include in your question the link to this page, the versions of your OS & ROS, and also add appropriate tags.|
Unit Testing with RTDescription: This explains how to create unit tests with RT, also for rostest
Tutorial Level: INTERMEDIATE
Next Tutorial: Using actionlib
Creating a unit test with RT
The COMMON LISP implementation used for roslisp is SBCL. SBCL ships with a minimalist testing framework called RT. The ROS package roslisp_testing extends this to provide functionality to write unit tests that are compliant with the rostest testing tool.
roslisp_testing is also used to test roslisp.
Tests can be created for interactive use using the RT macros.
As an example, we will define the fibonacci sequence function and unit test it:
CL-USER> (ros-load:load-system "roslisp_testing" "roslisp-testing") CL-USER> (in-package roslisp-testing) ROSLISP-TESTING> (defun fib-trec (n) "Tail-recursive Fibonacci number function" (labels ((calc-fib (n a b) (if (= n 0) a (calc-fib (- n 1) b (+ a b))))) (calc-fib n 0 1))) ROSLISP-TESTING> (deftest fibtrec-test (list (fib-trec 0) (fib-trec 1) (fib-trec 2) (fib-trec 3) (fib-trec 4) (fib-trec 5)) (1 1 1 2 3 5))
This defines the function and a single unit test for it. We can run the test using RT by calling:
(do-test 'fibtrec-test) Test FIBTREC-TEST failed Form: (LIST (FIB-TREC 0) (FIB-TREC 1) (FIB-TREC 2) (FIB-TREC 3) (FIB-TREC 4) (FIB-TREC 5)) Expected value: (1 1 1 2 3 5) Actual value : (0 1 1 2 3 5). NIL (FIBTREC-TEST 0 (LIST (FIB-TREC 0) (FIB-TREC 1) (FIB-TREC 2) (FIB-TREC 3) (FIB-TREC 4) (FIB-TREC 5)) ((1 1 1 2 3 5)) ((0 1 1 2 3 5)))
As you can see we messed up the test case, expecting the result for (FIB-TREC 0) to be 1, while it is defined as 0. The first return value NIL shows that there have been errors, the next result lists for each testcase the name, the time the test took, the tested function body, the expected result and the actual result.
So we redefine the test, and run it again. Note you can run the last test that was defined by just calling (do-test) without arguments.
ROSLISP-TESTING> (deftest fibtrec-test (list (fib-trec 0) (fib-trec 1) (fib-trec 2) (fib-trec 3) (fib-trec 4) (fib-trec 5)) (0 1 1 2 3 5)) ROSLISP-TESTING> (do-test) FIBTREC-TEST (FIBTREC-TEST 0)
So this time no error occured, all tests passed, which is why we get non-NIL as first result, and a list of all tests with the time they took.
Tests expecting errors
Now maybe we want our fibonacci funtion to be robust as well, so that it returns a SIMPLE-ERROR when called with negative numbers. So we define a unit test for that. RT does not provide explicit handling of errors as expected values, but roslisp-testing defines a test fixture for that.
ROSLISP-TESTING> (defun fib-trec (n) "Tail-recursive Fibonacci number function" (when (< n 0) (error "fibonacci numbers only defined for positive integers: ~a" n)) (labels ((calc-fib (n a b) (if (= n 0) a (calc-fib (- n 1) b (+ a b))))) (calc-fib n 0 1))) ROSLISP-TESTING> (deftest fibtrec-negative-test (type-of (with-fixture roslisp-testing:error-caught () (fib-trec -1))) simple-error)
Note we give the new unit test a different name. We can now call all unit tests:
ROSLISP-TESTING> (do-tests) Doing 2 pending tests of 2 tests total. FIBTREC-TEST FIBTREC-NEGATIVE-TEST No tests failed. T ((FIBTREC-NEGATIVE-TEST 12) (FIBTREC-TEST 0)) 12
RT remembers all tests that were loaded so far, to make it forget tests, call function (rem-all-tests)
Fixtures are pieces of code that are called before and after tests. in LISP, those are macros, and you can use the def-fixture and with-fixture macros to use fixtures in an explicit way. Note you can as well use defmacro and give your macro a name that makes it plain that it is a fixture.
Once you start keeping tests in files for later regression testing, RT reaches its limits in organizing many testcases. roslisp-testing extends RT with the concept of suites. This way, you can define multiple test cases in a file, store them in a suite, and load a different file with other test cases.
Testcase files should generally look like this:
;; mypackage-suite1.lisp (in-package :test-mypackage) (rem-all-tests) (deftest foo ...) (deftest bar ...) (create-suite "mypackage-suite1")
;; mypackage-suite2.lisp (in-package :test-mypackage) (rem-all-tests) (deftest floo ...) (deftest barl ...) (create-suite "mypackage-suite2")
The package declaration of such a test package should minimally contain this:
;; package.lisp (in-package :cl-user) (defpackage :test-mypackage (:documentation "tests for my package") (:use :cl #+sbcl :sb-rt #-sbcl :rtest :gtest-adapter ))
Also extend your ROS manifest.xml to include roslisp_testing.
Using suites with rostest
Test cases that are organized in suites as above can be used with rostest. All you need to do is to create an executable file that calls all your tests and transforms the results into the gtest format. For tests written with RT as in this tutorial, a wrapper function exists that does all that for you.
Given the example above:
;; mypackage-allsuites.lisp (in-package :test-mypackage) (defun mypackage-test-main () "runs the test suites and writes results to xml as gtest would" (rt-do-suites->gtest '("mypackage-suite1" "mypackage-suite2") sb-ext:*posix-argv*))
An executable file which calls mypackage-test-main is compatible with the rostest framework.
Creating suite scripts
The following is a simple example for testing using scripts. Create a file name simpletest in a ROS package, maybe neatly in a subfolder test.
#!/usr/bin/env sh "true";exec /usr/bin/env rosrun roslisp_runtime run-roslisp-script.sh --script "$0" "$@" (ros-load:load-system "roslisp_testing" "roslisp-testing") (in-package :roslisp-testing) ;; simple test to show ho to use rt-test ;; call this with (do-test 'addition) (deftest test-addition ;; the test (+ 21 21) ;; the expected result 42) ;; another simple test. Note the expected result argument does not get ;; evaluated, therefore (list 1 2) and '(1 2) do not work!!! (deftest test-append ;; the test (append '(1) '(2)) ;; the expected result (1 2)) (create-suite "simple-suite") (rt-do-suites->gtest '("simple-suite") sb-ext:*posix-argv*)
The first two lines are the lisp script shebang lines. We then load roslisp-testing to have access to the test functions. We define two simple tests, and create a suite. Finally the file does the gtest conversion if the script is called with a gtest command line argument.
Make this file executable and run it:
$ chmod u+x simpletest $ ./simpletest No --gtest_output=xml:filename given in args NIL Doing 2 pending tests of 2 tests total. TEST-ADDITION TEST-APPEND No tests failed.
As you can see, the function warns that no gtest output xml argument was given, but performs the tests anyway. You can also run the test and generate xml:
$ ./simpletest --gtest_output=xml:foo.xml ... $ cat foo.xml <testsuites> <testsuite name="simple-suite" tests="2" failures="0" errors="0" time="0.0"> <testcase classname="TEST-APPEND" name="TEST-APPEND" status="run" time="0.0"> </testcase> <testcase classname="TEST-ADDITION" name="TEST-ADDITION" status="run" time="0.0"> </testcase> <system-out><![CDATA[Doing 2 pending tests of 2 tests total. TEST-ADDITION TEST-APPEND No tests failed.]]></system-out> <system-err><![CDATA]></system-err> </testsuite> </testsuites>
As you can see an xml file was generated summarizing the results in gtest compatible format.
Suites in built executables
It is also possible to use LISP executables defined in clean .lisp files and build as explained in the roslisp tutorials. For that, instead of calling (rt-do-suites->gtest), define a main function in the test file:
... (defun simple-test-main () (rt-do-suites->gtest '("simple-suite") sb-ext:*posix-argv*))
And build your executable using this as main function.
You would also need the dependency to roslisp-testing in your manifest and .asd file in that case.
Invoke using rostest
With rostest you can automize invocations of tests. We will create an small package only for the purpose of demonstrating how it is done:
$ roscreate-pkg simpletest_pkg $ cd simpletest_pkg $ mkdir test
place the script file simpletest from the previous section inside test, and do not forget to make it executable.
Not we do not need to depend on roslisp or on roslisp-testing, as we just create a script and not a compiled executable.
Then create a file named simple-rostest.launch in the test folder:
<launch> <test test-name="simple_tests" pkg="simpletest_pkg" type="simpletest"/> </launch>
Rostest will manage to find the executable file within the package.
To run from the command line:
$ rostest simpletest_pkg simple-rostest.launch ... logging to ....log [ROSUNIT] Outputting test results to .../.ros/test_results/simpletest_pkg/TEST-rostest__test_simple-rostest.xml testsimple_tests ... ok [ROSTEST]----------------------------------------------------------------------- [simpletest_pkg.simple_tests/TEST-APPEND][passed] [simpletest_pkg.simple_tests/TEST-ADDITION][passed] SUMMARY * RESULT: SUCCESS * TESTS: 2 * ERRORS: 0 * FAILURES: 0 rostest log file is in ....log
And, as the rostest documentation suggests, you can also add this to your CMakelists.txt. Add the line:
and then call the test using make:
$ roscd simpletest_pkg $ make test
Other test frameworks
RT is a minimalist testing framework, there are other COMMON LISP testing frameworks available freely, such as FiveAM and stefil. Wrapping those for rostest just means to transform their test results to the gtest XML format.
Helper functions exist to make this somewhat easier. There are testsuite result structures defined in roslisp-testing.
(defstruct gtestfailure type ;; string message ;; string ) (defstruct gtestcase classname ;; string name ;; string time ;; float failures ;; list of gtestfailure ) (defstruct gtestsuite name ;; string time ;; float testcases ;; list of gtestcase sysout ;; string syserr ;; string )
The function (defun run-suites-write-gtest-file (posix-args suite-fun-list transform-fun &key (no-output nil))...) provides all the common code to create the gtest compatible xml. You need to provide a list of suite-functions, that is functions which will run your suite and return some result object of a type of your choice, and a transform function which will transform this result object into a structure of type gtestsuite.
The function run-suites-write-gtest-file will then parse the posix-arg list for the gtest parameter, run your suite functions and capture stdout and stderr messages, transform the results and serialize them into gtest-compatible xml.