Testing with fiveam

Question

I can't figure out how to test a function with fiveam. I setup a project using cl-project. My project name is my-projects/proj1:

~/quicklisp% tree local-projects 
local-projects
├── dtrace
│   ├── dtrace.asd
│   └── dtrace.lisp
├── my-projects
│   └── proj1
│       ├── #proj1.asd#
│       ├── README.markdown
│       ├── README.org
│       ├── proj1.asd
│       ├── proj1.asd~
│       ├── src
│       │   ├── main.fasl
│       │   ├── main.lisp
│       │   └── main.lisp~
│       └── tests
│           ├── main.fasl
│           ├── main.lisp
│           └── main.lisp~
└── system-index.txt

And:

~/quicklisp/local-projects% cat system-index.txt 
dtrace/dtrace.asd
my-projects/proj1/proj1.asd

Here is proj1/proj1.asd:

(defsystem "proj1"
  :version "0.0.1"
  :author "7stud"
  :license "any"
  :depends-on ()
  :components ((:module "src"
                :components
                ((:file "main"))))
  :description ""
  :in-order-to ((test-op (test-op "proj1/tests"))))

(defsystem "proj1/tests"
  :author "7stud"
  :license "any"
  :depends-on ("proj1"
               "fiveam")
  :components ((:module "tests"
                :components
                ((:file "main"))))
  :description "Test system for proj1"
  :perform (test-op (op c) (symbol-call :fiveam '#:run! :proj1)))

proj1/src/main.lisp:

(defpackage proj1
  (:use :cl))
(in-package :proj1)

;; blah blah blah.

(defun anyoddp (number-list)
  (cond ((null number-list) nil)
        ((oddp (first number-list)) t)
        (t (anyoddp (rest number-list)))))

proj1/tests/main.lisp:

(defpackage proj1/tests/main
  (:use :cl
        :proj1
        :fiveam))
(in-package :proj1/tests/main)

;; Note: To run this test file, execute `(asdf:test-system :proj1)' in your Lisp.

;; (deftest test-target-1
;;   (testing "should (= 1 1) to be true"
;;            (ok (= 1 1))))

(def-suite master-suite
  :description "Test my system.")

(def-suite anyoddp-tests
  :description "Test anyoddp"
  :in master-suite)

(in-suite anyoddp-tests)

(test simple-maths
  (is (= 3 (+ 1 1))))

(test anyoddp-with-odds
  (let ((result (anyoddp '(1 3 4))))
    (is (equal result t))
    "True expected but got ~a" result))

I've tried every permutation of commands that I've read about in an attempt to test my proj1 "system", and I either get an error or Didn't run anything...huh?. Here's my latest attempt:

CL-USER> (ql:quickload "fiveam")
To load "fiveam":
  Load 1 ASDF system:
    fiveam
; Loading "fiveam"

("fiveam")
CL-USER> (asdf:load-system "proj1")
T
CL-USER> (asdf:test-system :proj1)
; compiling file "/Users/7stud/quicklisp/local-projects/my-projects/proj1/tests/main.lisp" (written 20 MAR 2024 10:04:40 AM):

; wrote /Users/7stud/.cache/common-lisp/sbcl-2.4.0-macosx-arm64/Users/7stud/quicklisp/local-projects/my-projects/proj1/tests/main-tmpUSHT8RIL.fasl
; compilation finished in 0:00:00.004
 Didn't run anything...huh?; in: ALEXANDRIA:NAMED-LAMBDA PROJ1/TESTS/MAIN::%TEST-ANYODDP-WITH-ODDS
;     (PROJ1/TESTS/MAIN::ANYODDP '(1 3 4))
; 
; caught STYLE-WARNING:
;   undefined function: PROJ1/TESTS/MAIN::ANYODDP
; 
; compilation unit finished
;   Undefined function:
;     PROJ1/TESTS/MAIN::ANYODDP
;   caught 1 STYLE-WARNING condition
T
CL-USER> 

Why am I getting

undefined function: PROJ1/TESTS/MAIN::ANYODDP

??? At the top of proj1/tests/main.lisp it says:

(defpackage proj1/tests/main
  (:use :cl
        :proj1
        :fiveam))

Isn't use :proj1 supposed to let me call the functions defined in proj1 without the package name? Okay, here is proj1/tests/main.lisp with the package name added to the function name anyoddp:

(defpackage proj1/tests/main
  (:use :cl
        :proj1
        :fiveam))
(in-package :proj1/tests/main)

(def-suite master-suite
  :description "Test my system.")

(def-suite anyoddp-tests
  :description "Test anyoddp"
  :in master-suite)

(in-suite anyoddp-tests)

(test simple-maths
  (is (= 3 (+ 1 1))))

(test anyoddp-with-odds
  (let ((result (proj1::anyoddp '(1 3 4))))   ;; **CHANGE HERE**
    (is (equal result t))
    "True expected but got ~a" result))

Then:

CL-USER> (ql:quickload "fiveam")
To load "fiveam":
  Load 1 ASDF system:
    fiveam
; Loading "fiveam"

("fiveam")
CL-USER> (asdf:load-system "proj1")
T
CL-USER> (asdf:test-system :proj1)
; compiling file "/Users/7stud/quicklisp/local-projects/my-projects/proj1/tests/main.lisp" (written 20 MAR 2024 10:33:11 AM):

; wrote /Users/7stud/.cache/common-lisp/sbcl-2.4.0-macosx-arm64/Users/7stud/quicklisp/local-projects/my-projects/proj1/tests/main-tmp7YCPD44Y.fasl
; compilation finished in 0:00:00.004
 Didn't run anything...huh?
T
CL-USER> 

I'm also wondering if there is a way to use fiveam to test a function in a file that is not in a project. Suppose I have this file:

;;  a.lisp

(defun anyoddp (number-list)
  (cond ((null number-list) nil)
        ((oddp (first number-list)) t)
        (t (anyoddp (rest number-list)))))

How can I add some tests to a.lisp, then run those tests?

Answer 1

I made some progress running tests in a project. I was able to execute my tests with the following code:

proj1/src/main.lisp:

(defpackage proj1
  (:use :cl)
  (:export :anyoddp))

(in-package :proj1)

;; blah blah blah.n

(defun anyoddp (number-list)
  (cond
    ((null number-list) nil)
    ((oddp (first number-list)) t)
    (t (anyoddp (rest number-list)))))

proj1/tests/main.lisp:

(defpackage proj1/tests/main
  (:use :cl
        :proj1
        :fiveam)
  (:export #:test-proj1))
(in-package :proj1/tests/main)

(defun test-proj1 ()
  (run! 'master-suite))

(def-suite master-suite
  :description "Test my system.")
(in-suite master-suite)

(test simple-maths
  (is (= 3 (+ 1 1))))


(def-suite anyoddp-tests
  :description "Test anyoddp"
  :in master-suite)
(in-suite anyoddp-tests)

(test anyoddp-with-odds
  (let ((result (anyoddp '(1 3 4))))   
    (is (equal result t))
    "True expected but got ~a" result))

(test anyoddp-with-evens
  (let ((result (anyoddp '(2 4 6))))
    (is (equal result nil))
    "Nil expected but got ~a" result))

Then in slime:

CL-USER> (ql:quickload 'proj1/tests)
; Evaluation aborted on #<QUICKLISP-CLIENT:SYSTEM-NOT-FOUND {7007278153}>.
CL-USER> (ql:register-local-projects)
NIL
CL-USER> (ql:quickload 'proj1/tests)
To load "proj1/tests":
  Load 1 ASDF system:
    proj1/tests
; Loading "proj1/tests"

(PROJ1/TESTS)
CL-USER> (proj1/tests/main:test-proj1)

Running test suite MASTER-SUITE
 Running test SIMPLE-MATHS f
 Running test suite ANYODDP-TESTS
  Running test ANYODDP-WITH-ODDS .
  Running test ANYODDP-WITH-EVENS .
 Did 3 checks.
    Pass: 2 (66%)
    Skip: 0 ( 0%)
    Fail: 1 (33%)

 Failure Details:
 --------------------------------
 SIMPLE-MATHS in MASTER-SUITE []: 
      
(+ 1 1)

 evaluated to 

2

 which is not 

=

 to 

3


 --------------------------------

NIL
(#<IT.BESE.FIVEAM::TEST-FAILURE {700832DF43}>)
NIL
CL-USER> 

I have no idea why I got this error:

CL-USER> (ql:quickload 'proj1/tests)
; Evaluation aborted on #<QUICKLISP-CLIENT:SYSTEM-NOT-FOUND {7007278153}>.

I, or rather cl-project, defined the "system" proj1/tests in the file ~/quicklisp/local-projects/my-projects/proj1/proj1.asd:

(defsystem "proj1"
  :version "0.0.1"
  :author "7stud"
  :license "any"
  :depends-on ()
  :components ((:module "src"
                :components
                ((:file "main"))))
  :description "Some functions"
  :in-order-to ((test-op (test-op "proj1/tests"))))

(defsystem "proj1/tests"
  :author ""
  :license ""
  :depends-on ("proj1"
               "fiveam")
  :components ((:module "tests"
                :components
                ((:file "main"))))
  :description "Test system for proj1"
  :perform (test-op (op c) (symbol-call :fiveam '#:run!)))

In any case, the command:

(ql:register-local-projects)

causes quickslip to look through the directories under ~/quicklisp/local-projects and write the paths to all the .asd files it finds into ~/quicklisp/local-projects/system-index.txt. After I evaluated that command, then (ql:quickload 'proj1/tests) was successful.

Essentially, I loaded a system, then I used a package name that was defined somewhere in the system to call a function defined in that package:

(proj1/tests/main:test-proj1)
 |              | |        |
 +-------+------+ +---+----+
         |            |
      package      function 
       name          name

And, here are the changes I made to proj1.asd to integrate with asdf:

(defsystem "proj1"
  :version "0.0.1"
  :author "7stud"
  :license "any"
  :depends-on ()
  :components ((:module "src"
                :components
                ((:file "main"))))
  :description "Some functions"
  :in-order-to ((test-op (test-op "proj1/tests"))))

(defsystem "proj1/tests"
  :author ""
  :license ""
  :depends-on ("proj1"
               "fiveam")
  :components ((:module "tests"
                :components
                ((:file "main"))))
  :description "Test system for proj1"
  :perform (test-op (o s)
       (symbol-call :fiveam '#:run!
          (find-symbol* '#:master-suite
                        :proj1/tests/main))))

Then:

CL-USER> (ql:quickload 'proj1/tests)
To load "proj1/tests":
  Load 1 ASDF system:
    proj1/tests
; Loading "proj1/tests"
[package proj1]...................................
[package proj1/tests/main]
(PROJ1/TESTS)
CL-USER> (asdf:test-system 'proj1/tests)

Running test suite MASTER-SUITE
 Running test SIMPLE-MATHS f
 Running test suite ANYODDP-TESTS
  Running test ANYODDP-WITH-ODDS .
  Running test ANYODDP-WITH-EVENS .
 Did 3 checks.
    Pass: 2 (66%)
    Skip: 0 ( 0%)
    Fail: 1 (33%)

 Failure Details:
 --------------------------------
 SIMPLE-MATHS in MASTER-SUITE []: 
      
(+ 1 1)

 evaluated to 

2

 which is not 

=

 to 

3


 --------------------------------

T

And:

CL-USER> (asdf:test-system 'proj1)

Running test suite MASTER-SUITE
 Running test SIMPLE-MATHS f
 Running test suite ANYODDP-TESTS
  Running test ANYODDP-WITH-ODDS .
  Running test ANYODDP-WITH-EVENS .
 Did 3 checks.
    Pass: 2 (66%)
    Skip: 0 ( 0%)
    Fail: 1 (33%)

 Failure Details:
 --------------------------------
 SIMPLE-MATHS in MASTER-SUITE []: 
      
(+ 1 1)

 evaluated to 

2

 which is not 

=

 to 

3


 --------------------------------

T
CL-USER> 

Whenever I made changes to the files in ~/quicklisp/local-projects/my-projects/proj1/, I usually tried to execute:

CL-USER> (ql:quickload 'proj1)

or

CL-USER> (ql:quickload 'proj1/tests)

in the slime repl. I found that the slime repl would often get out of sync with the current files in proj1, which would cause errors.

As far as I can figure out, this line:

  :perform (test-op (o s)
       (symbol-call :fiveam '#:run!
          (find-symbol* '#:master-suite
                        :proj1/tests/main))))

has to use some trickery because it is read before the packages/functions in proj1 are created. symbol-call is in the uiop package, and it is used to specify a function call that is read before the package containing the function definition exists. You specify the package name, the function name, and the args. In symbol-call above, the package name is :fiveam, the function name is '#:run! and the arg for the function is whatever is returned by:

(find-symbol* '#:master-suite :proj1/tests/main)

find-symbol* is a function in the uiop package which comes with asdf which comes with sbcl, which is the lisp implementation I'm using. find-symbol* takes a symbol and a package name as arguments and is used when the package isn't present.

The result is, I am specifying a function call something like:

(fiveam:run! proj1/tests/main:master_suite)

But, instead of dealing with all that crazy syntax, I found I can do this:

 :perform (test-op (o s)
       (symbol-call :proj1/tests/main :test-proj1)))

test-proj1 is this function:

(defun test-proj1 ()
  (run! 'master-suite))

which is defined in the package proj1/tests/main, which is defined in the file .../proj1/test/main.lisp. Then I only needed to export the function test-proj1:

...proj1/tests/main.lisp:

(defpackage proj1/tests/main
  (:use :cl
        :proj1
        :fiveam)
  (:export #:test-proj1))
(in-package :proj1/tests/main)

(defun test-proj1 ()
  (run! 'master-suite))

I found nothing about what test-op (o s) does.

Resources:

1. ASDF manual: test-op

2. Tutorial: Working with FiveAM

Answer 2

I'm also wondering if there is a way to use fiveam to test a function in a file that is not in a project.

Well you can define tests anywhere and run them by the test suite or the test name.

You can also put them behind a feature flag.

First possibility, a test for the developer to check manually:

#+(or)
;; or your own marker in *features*.
(equal t (anyoddp '(1 2 3)))

Just C-c C-c on it.

We can add fiveam tests.

(use-package :fiveam)

(def-suite oddtest)

(in-suite oddtest)

(test simple-maths
  (is (= 2 (+ 1 2))))

and run one:

CL-USER> (in-package :proj1)
PROJ1> (run! 'SIMPLE-MATHS)
Running test suite ODDTEST
 Running test SIMPLE-MATHS f
 Did 1 check.
    Pass: 0 ( 0%)
    Skip: 0 ( 0%)
    Fail: 1 (100%)

 Failure Details:

[…]

Running the test suite:

PROJ1> (run! 'oddtest)
Running test suite ODDTEST
Running test suite ODDTEST
[…]

and we could place everything under a #+5am feature flag.

You can use a couple parameters to make writing/running tests more interactive.

Running the test at compilation

  #+(or)
  (setf fiveam:*run-test-when-defined* t)

now C-c C-c runs the test on the REPL.

Getting the debugger on a test failure

  #+(or)
  (setf fiveam:*on-failure* :debug)

now we get a debugger on a test failure.

Answer 3

Here is how I got tests to run in a single file:

;; c.lisp

(defpackage my-list-functions
  (:use :cl
        :fiveam))
(in-package my-list-functions)

(defun anyoddp (alist)
  (cond ((null alist) nil)
        ((oddp (first alist)) t)
        (t (anyoddp (rest alist)))))

(def-suite master-suite
  :description "All tests")

(def-suite simple-tests
  :description "Some simple tests"
  :in master-suite)
(in-suite simple-tests)

(test test1
  (is (= 2 (+ 1 1))))

(test test2
      (is (= 3 (+ 4 3))))

(def-suite anyoddp-tests
  :description "Test anyoddp"
  :in master-suite)
(in-suite anyoddp-tests)

(test anyoddp-with-list-of-evens
  (is (equal (anyoddp '(2 4 6)) nil))
  (is (equal (anyoddp '(2)) nil))
  (is (equal (anyoddp '()) nil)))

(test anyoddp-with-list-containing-odds
  (is (equal (anyoddp '(2 3 4)) t))
  (is (equal (anyoddp '(2 3)) t))
  (is (equal (anyoddp '(1)) t)))

Here's what I tried:

C-x C-s   save c.lisp
C-c C-k   compile c.lisp

The name "FIVEAM" does not designate any package.
   [Condition of type PACKAGE-DOES-NOT-EXIST]

Then:

CL-USER> (ql:quickload "fiveam")
To load "fiveam":
  Load 1 ASDF system:
    fiveam
; Loading "fiveam"

("fiveam")
CL-USER>

Then C-c C-k again in the c.lisp buffer. slime said:

; compiling file "/Users/7stud/lisp_programs/c.lisp" (written 20 MAR 2024 03:08:06 PM):

; wrote /Users/7stud/lisp_programs/c.fasl
; compilation finished in 0:00:00.009

Then:

CL-USER> (run! 'master-suite)
; in: RUN! 'MASTER-SUITE
;     (RUN! 'MASTER-SUITE)
; 
; caught STYLE-WARNING:
;   undefined function: COMMON-LISP-USER::RUN!
; 
; compilation unit finished
;   Undefined function:
;     RUN!
;   caught 1 STYLE-WARNING condition
; Evaluation aborted on #<UNDEFINED-FUNCTION RUN! {7005A44223}>.
CL-USER> 

Then:

CL-USER> (fiveam:run! 'master-suite)
 Didn't run anything...huh?
T
NIL
NIL
CL-USER>

Then:

CL-USER> (in-package my-list-functions)
#<PACKAGE "MY-LIST-FUNCTIONS">

MY-LIST-FUNCTIONS> (run! 'master-suite)

Running test suite MASTER-SUITE
 Running test suite SIMPLE-TESTS
  Running test TEST1 .
  Running test TEST2 f
 Running test suite ANYODDP-TESTS
  Running test ANYODDP-WITH-LIST-OF-EVENS ...
  Running test ANYODDP-WITH-LIST-CONTAINING-ODDS ...
 Did 8 checks.
    Pass: 7 (87%)
    Skip: 0 ( 0%)
    Fail: 1 (12%)

 Failure Details:
 --------------------------------
 TEST2 in SIMPLE-TESTS []: 
      
(+ 4 3)

 evaluated to 

7

 which is not 

=

 to 

3


 --------------------------------

NIL
(#<IT.BESE.FIVEAM::TEST-FAILURE {7006ADD9F3}>)
NIL
MY-LIST-FUNCTIONS> 

And:

MY-LIST-FUNCTIONS> (run! 'anyoddp-tests)

Running test suite ANYODDP-TESTS
 Running test ANYODDP-WITH-LIST-OF-EVENS ...
 Running test ANYODDP-WITH-LIST-CONTAINING-ODDS ...
 Did 6 checks.
    Pass: 6 (100%)
    Skip: 0 ( 0%)
    Fail: 0 ( 0%)

T
NIL
NIL
MY-LIST-FUNCTIONS>