lisp-unit is a Common Lisp library that supports unit testing. There is a long history of testing packages in Lisp, usually called "regression" testers. More recent packages in Lisp and other languages have been inspired by JUnit for Java. For more information on both unit testing and JUnit, visit junit.org.

This page has two parts:

Overview

My main goal for lisp-unit was to make it simple to use, particularly for beginning Lisp programmers. The advantages of lisp-unit are:

Written in portable Common Lisp.
Just one file to load.
Dead-simple to define and run tests. See example.
Supports redefining functions and even macros without reloading tests.
Supports test-first programming.
Supports testing return values, printed output, macro expansions, and error conditions.
Produces short readable output with a reasonable level of detail.
Groups tests by package for modularity.

How to Use lisp-unit

Use Quicklisp to compile and load lisp-unit.
Evaluate (use-package :lisp-unit).
Load a file of tests. See below for how to define tests. Example: exercise-tests.lisp.
Run the tests with run-tests.

Any test failures will be printed, along with a summary of how many tests were run, how many passed, and how many failed.

You define a test with define-test:

(define-test name exp₁ exp₂ ...)

This defines a test called name. The expressions can be anything, but typically most will be assertion forms.

Tests can be defined before the code they test, even if they're testing macros. This is to support test-first programming.

After defining your tests and the code they test, run the tests with

(run-tests)

This runs every test defined in the current package. To run just certain specific tests, use

(run-tests name₁ name₂ ...)

e.g., (run-tests greater summit).

The following example

defines some tests to see if pick-greater returns the larger of two arguments
defines a deliberately broken version of pick-greater
runs the tests

First, we define some tests.

> (in-package :cs325-user)
        #<PACKAGE CS325-USER>
        > (define-test pick-greater
          (assert-equal 5 (pick-greater 2 5))
          (assert-equal 5 (pick-greater 5 2))
          (assert-equal 10 (pick-greater 10 10))
          (assert-equal 0 (pick-greater -5 0))
          )
        PICK-GREATER

Following good test-first programming practice, we run these tests before writing any code.

> (run-tests pick-greater)
            PICK-GREATER: Undefined function PICK-GREATER called with arguments (2 5).

This shows that we need to do some work. So we define our broken version of pick-greater.

        > (defun pick-greater (x y) x)  ;; deliberately wrong
        PICK-GREATER

Now we run the tests again:

> (run-tests pick-greater)
        PICK-GREATER: (PICK-GREATER 2 5) failed: Expected 5 but saw 2
        PICK-GREATER: (PICK-GREATER -5 0) failed: Expected 0 but saw -5
        PICK-GREATER: 2 assertions passed, 2 failed.

This shows two failures. In both cases, the equality test returned NIL. In the first case it was because (pick-greater 2 5) returned 2 when 5 was expected, and in the second case, it was because (pick-greater -5 0) returned -5 when 0 was expected.

Assertion Forms

The most commonly used assertion form is

(assert-equal value form)

This tallies a failure if form returns a value not equal to value. Both value and test are evaluated in the local lexical environment. This means that you can use local variables in tests. In particular, you can write loops that run many tests at once:

> (define-test my-sqrt
          (dotimes (i 5)
            (assert-equal i (my-sqrt (* i i)))))
        MY-SQRT
        
        > (defun my-sqrt (n) (/ n 2))   ;; wrong!!
        
        > (run-tests my-sqrt)
        MY-SQRT: (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2
        MY-SQRT: (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2
        MY-SQRT: (MY-SQRT (* I I)) failed: Expected 4 but saw 8
        MY-SQRT: 2 assertions passed, 3 failed.

However, the above output doesn't tell us for which values of i the code failed. Fortunately, you can fix this by adding expressions at the end of the assert-equal. These expression and their values will be printed on failure.

> (define-test my-sqrt
          (dotimes (i 5)
            (assert-equal i (my-sqrt (* i i)) i)))  ;; added i at the end
        MY-SQRT
        > (run-tests my-sqrt)
        MY-SQRT: (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2
          I => 1
        MY-SQRT: (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2
          I => 3
        MY-SQRT: (MY-SQRT (* I I)) failed: Expected 4 but saw 8
          I => 4
        MY-SQRT: 2 assertions passed, 3 failed.

The next most useful assertion form is

(assert-true test)

This tallies a failure if test returns false. Again, if you need to print out extra information, just add expressions after test.

There are also assertion forms to test what code prints, what errors code returns, or what a macro expands into. A complete list of assertion forms is in the reference section.

Do not confuse assert-true with Common Lisp's assert macro. assert is used in code to guarantee that some condition is true. If it isn't, the code halts. assert has options you can use to let a user fix what's wrong and resume execution. A similar collision of names exists in JUnit and Java.

How to Organize Tests with Packages

Tests are grouped internally by the current package, so that a set of tests can be defined for one package of code without interfering with tests for other packages.

If your code is being defined in cl-user, which is common when learning Common Lisp, but not for production-level code, then you should define your tests in cl-user as well.

If your code is being defined in its own package, you should define your tests either in that same package, or in another package for test code. The latter approach has the advantage of making sure that your tests have access to only the exported symbols of your code package.

For example, if you were defining a date package, your date.lisp file would look like this:

(defpackage :date
          (:use :common-lisp)
          (:export #:date->string #:string->date))
          
        (in-package :date)
        
        (defun date->string (date) ...)
        (defun string->date (string) ...)

Your date-tests.lisp file would look like this:

(defpackage :date-tests
          (:use :common-lisp :lisp-unit :date))
        
        (in-package :date-tests)
        
        (define-test date->string
          (assert-true (string= ... (date->string ...)))
          ...)
        ...

You could then run all your date tests in the test package:

(in-package :date-tests)
        
        (run-tests)

Alternately, you could run all your date tests from any package with:

(lisp-unit:run-all-tests :date-tests)

Reference Section

Here is a list of the functions and macros exported by lisp-unit.

Test definition forms

(define-test name exp₁ exp₂ ...): This macro defines a test called name with the expressions specified, in the package specified by the value of *package* in effect when define-test is executed. The expresssions are assembled into runnable code whenever needed by run-tests. Hence you can define or redefine macros without reloading tests using those macros.
(get-tests [package): This function returns the names of all the tests that have been defined for the package. If no package is given, the value of *package* is used.
(get-test-code name [package): This function returns the body of the code stored for the test name under package. If no package is given, the value of *package* is used.
(remove-tests names [package]): This function removes the tests named for the given package. If no package is given, the value of *package* is used.
(remove-all-tests [package]): This function removes the tests for the given package. If no package is given, it removes all tests for the current package. If nil is given, it removes all tests for all packages.
(run-all-tests package): This macro runs all the tests defined in the specified package and reports the results.
(run-tests name₁ name₂ ...): This macro runs the tests named and reports the results. The package used is the value of *package* in effect when the macro is expanded. If no names are given, all tests for that package are run.
(use-debugger [flag]): By default, errors that occur while running tests are simply counted and ignored. You can change this behavior by calling use-debugger with one of three possible flag values: t (the default) means your Lisp's normal error handling routines will be invoked when errors occur; :ask means you will be asked what to do when an error occurs, and nil means errors are counted and ignored, i.e., the standard behavior.

Assertion forms

All of the assertion forms are macros. They tally a failure if the associated predication returns false. Assertions can be made about return values, printed output, macro expansions, and even expected errors. Assertion form arguments are evaluated in the local lexical environment.

All assertion forms allow you to include additional expressions at the end of the form. These expressions and their values will be printed only when the test fails.

Return values are unspecified for all assertion forms.

(assert-eq value form [form₁ form₂ ...])

(assert-eql value form [form₁ form₂ ...])

(assert-equal value form [form₁ form₂ ...])

(assert-equalp value form [form₁ form₂ ...])

(assert-equality predicate value form [form₁ form₂ ...])

These macros tally a failure if value is not equal to the result returned by form, using the specified equality predicate.

In general, assert-equal is used for most tests. But any binary predicate can be used, with assert-equality, e.g.,

(assert-equality unordered-equal '(a b c) (unique-atoms '((b c) a ((b a) c))))

(assert-true test [form₁ form₂ ...])

(assert-false test [form₁ form₂ ...])

assert-true tallies a failure if test returns false. assert-false tallies a failure if test returns true.

(assert-prints "output" form [form₁ form₂ ...])

This macro tallies a failure if form does not print to standard output stream output equal to the given string, ignoring differences in beginning and ending newlines.

(assert-expands expansion form [form₁ form₂ ...])

This macro tallies a failure if (macroexpand-1 form) does not produce a value equal to expansion.

(assert-error condition-type form [form₁ form₂ ...])

This macro tallies a failure if form does not signal an error that is equal to or a subtype of condition-type. Use error to refer to any kind of error. See condition types in the Common Lisp Hyperspec for other possible names. For example,

(assert-error 'arithmetic-error (foo 0))

would assert that foo is supposed to signal an arithmetic error when passed zero.

(fail format-string [form₁ form₂ ...])

Calling this function tallies a failure. A string describing the failure is constructed by calling (format nil format-string [form₁ form₂ ...]). For example,

(when (> (length queue) 100)
          (fail "Queue exceeded expected size by " (- (length queue) 100)))

Utility predicates

Several predicate functions are exported that are often useful in writing tests with assert-equality.

(logically-equal value₁ value₂)

This predicate returns true of the two values are either both true, i.e., non-NIL, or both false.

(set-equal list₁ list₂ [:test])

This predicate returns true the first list is a subset of the second and vice versa. For example, (set-equal '(a b a) '(a b a a)) is true.

:test can be used to specify an equality predicate. The default is equal.

(unordered-equal list₁ list₂ [:test])

This predicate returns true the first list is a permutation of the second. For example, (unordered-equal '(a b a) '(b a a)) is true, but (unordered-equal '(a b a) '(a b a a)) is false.

:test can be used to specify an equality predicate. The default is equal.

Experimental

The following facilities to control how results are reported is still under development. Comments welcome.

Listeners

lisp-unit calls three listener functions to report test results and summary statistics. These functions can be rebound using the facilities below.

Test Listener

The test listener is called after each assertion form in a test is executed. The listener is passed

the truth or falsity of the assertion
a keyword for type of assertion; current valid values for type are:
- :ERROR for ASSERT-ERROR
- :MACRO for ASSERT-EXPANDS
- :OUTPUT for ASSERT-PRINTS
- :RESULT for ASSERT-TRUE, ASSERT-FALSE
- :FAILURE for FAIL calls
- :EQUAL for all other assertions
the name of test in which assertion occurred
the form tested
the expected value
the actual value
any extra arguments passed to assertion form
how many assertions evaluated so far
how many assertions passed so far

Two test listener functions are exported:

SHOW-FAILURE-RESULT, the default, only prints when an assertion fails. It prints the form, expected and actual values, and the values of any extra forms.
SHOW-NO-RESULT never prints anything.

Error Listener

The error listener is called when an error occurs in running a test. The listener is passed

the error object
the name of the test

Three error listener functions are exported:

SHOW-ERROR, the default, prints the error message. Further execution of the test's forms is terminated.
COUNT-ERROR prints nothing but the error count is incremented. Further execution of the test's forms is terminated.
DEBUG-ERROR causes the Lisp debugger to be invoked so that the stack can be examined.

Summary Listener

The summary listener is called after

all forms in a test have been executed
all tests in a package have been run
all tests in all packages have been run

The listener is passed:

the name of the test or a package just finished
the number of assertions evaluated
the number that passed
the number of errors that occurred

Three summary listener functions are exported:

SHOW-SUMMARY, the default, prints the summaries at both the test and package level.
SHOW-PACKAGE-SUMMARY prints the summaries at only the package level.
SHOW-NO-SUMMARY never prints summaries.

Rebinding Listeners

The three listeners are stored in the exported global variables *test-listener*, *error-listener*, *summary-listener*. So one way to change listeners is with let. For example, to show only package-level summary counts:

(let ((*summary-listener* 'show-package-summary))
          (run-tests))

The above would still show failures and error messages. To hide those and just get the counts:

(let ((*error-listener* 'count-error)
              (*summary-listener* 'show-package-summary)
              (*test-listener* 'show-no-result))
          (run-tests))

To show no individual test results and only package summaries with failures, we need to define a function that checks the number of failures.

(defun show-failing-package (name test-count pass-count error-count)
          (when (or (< pass-count test-count) (> error-count 0))
            (show-summary name test-count pass-count error-count)))
        
        (let ((*error-listener* 'count-error)
              (*summary-listener* 'show-failing-package)
              (*test-listener* 'show-no-result))
          (run-tests))

`with-listeners`

A simpler way to rebind listeners is with with-listeners. The second example above could be done with:

(with-listeners (count-error show-package-summary show-no-result)
          (run-tests))

To use user-defined listeners, you first say what listener variable they're for with set-listener-variable. The third example above could be done with:

(defun show-failing-package (name test-count pass-count error-count)
          (when (or (< pass-count test-count) (> error-count 0))
            (show-summary name test-count pass-count error-count)))
        
        (set-listener-variable 'show-failing-package '*summary-listener*)
        
        (with-listeners (count-error show-failing-package show-no-result)
          (run-tests))

A Lisp Unit Tester