Dynamically generating rackunit test-suite: tests pass and also raise exception

Question

I am trying to dynamically generate a bunch of test cases.

Basically I want to run the same tests several times with different values. Maybe there's a better way to do that, if so let me know, I haven't found it yet.

I found make-test-suite which says you can pass it a list of test-case instances.

So I made a for/fold loop which collects the test cases generated for each set of values into a flat list.

What I don't understand is it seems to successfully run all the tests while also raising an exception:

tests->test-suite-action received #<void> in list of tests (#<void> #<void> #<void> #<void> #<void> #<void> #<void> #<void>), which is not a test.
  context...:
   /usr/share/racket/pkgs/rackunit-lib/rackunit/private/test-suite.rkt:139:7
   /usr/share/racket/pkgs/rackunit-lib/rackunit/private/test-suite.rkt:136:2
   /usr/share/racket/pkgs/rackunit-lib/rackunit/private/test-suite.rkt:61:0: apply-test-suite
   /usr/share/racket/pkgs/rackunit-lib/rackunit/text-ui.rkt:91:0: run-tests
   "/write-thru-hash/tests.rkt": [running body]
   temp35_0
   for-loop
   run-module-instance!
   /usr/share/racket/pkgs/compiler-lib/compiler/commands/test.rkt:179:16

...

1 1 write-thru-hash/tests.rkt
8 tests passed
rkt-write-thru-hash_tests_1 exited with code 1

I put a writeln in each of my test cases and I see those lines printed out where I abbreviated with ... above. So I know the tests are actually running.

(The tests also ran and worked fine before I multiplied them with the for/fold loop and just built them in the body of test-suite)

My tests.rkt code looks like:

(define test-cases-list
  (for/fold ([test-cases (list)])
            ([db-type (list 'sqlite 'postgres)])
    (append test-cases
      (list
        (test-case/fixture "basic operations" ... )
        (test-case/fixture "complex serializable keys and values" ... )
        (test-case/fixture "custom table-name" ... )
        (test-case/fixture "use initial src-hash" ... )))))

(define db-tests
  (make-test-suite "db-tests" test-cases-list))

(I'm using the fixture library https://docs.racket-lang.org/fixture/)

UPDATE:

Actually I think the writeln in my test cases are printing at definition time ... i.e. they are running too early because they are outside of the test-suite context.

I guess the tail check in each test-case returns #<void>, so I have filled a list with the results of the tests (voids) rather than the test cases themselves and given that to the test suite, hence the error.

But then I don't see how to actually use make-test-suite...?

Answer 1

Eventually found a simple way to do this.

My problem was in trying to have several variations of my test cases all dynamically generated into one test-suite. I did not find a way to get that to work, I suspect I would have to write a macro.

Instead the simple way is to dynamically define multiple test-suite instances and then loop over a list of them to run them:

(define db-test-suites
  (for/list ([db-type (list 'sqlite 'postgres)]
             [db-conn-fixture (list sqlite-connection-delete-after-use
                                    postgres-connection)])
    (define suite-name (format "db-tests: ~a" db-type))
    (test-suite suite-name
      (test-case/fixture "basic operations"
        #:fixture db-conn-fixture
        (define db-conn (fixture-value db-conn-fixture))
        ...)
      (test-case/fixture "complex serializable keys and values"
        #:fixture db-conn-fixture
        (define db-conn (fixture-value db-conn-fixture))
        ...))))

(for ([current-test-suite db-test-suites])
  (run-tests current-test-suite))

UPDATE:

I also found a way to actually use make-test-suite:

(define all-tests (make-test-suite "db-test-suites" db-test-suites))

(run-tests all-tests)

This takes advantage of the fact that a test-suite can contain other nested test-suites. I think this form is nicer (a single run-tests call) than the one above (multiple run-tests calls in a for loop).

All my problems in the original question were in trying to pre-define a bunch of test-case instances outside of a test-suite container. But here we pass make-test-suite a list of test-suite instances, allowing the tests within them to defer running until we later call run-tests.