Inconsistent results when testing a chan processing function

Question

The process-async function tested within the midje framework produces inconsistent results. Most of the time it checks as expected, but from time to time, it reads out.json at its initial state (""). I rely on the async-blocker function to wait on process-async before checking.

What's wrong with my approach?

(require '[[test-with-files.core :refer [with-files public-dir]])

(defn async-blocker [fun & args]
  (let [chan-test (chan)]
    (go (>! chan-test (apply fun args)))
    (<!! chan-test)))

(defn process-async
  [channel func]
  (go-loop  []
    (when-let  [response  (<! channel)]
      (func response)
      (recur))))

(with-files [["/out.json" ""]]
    (facts "About `process-async"
           (let [channel (chan)
                 file (io/resource (str public-dir "/out.json"))
                 write #(spit file (str % "\n") :append true)]
             (doseq [m ["m1" "m2"]] (>!! channel m))
             (async-blocker process-async channel write)
             (clojure.string/split-lines (slurp file)) => (just ["m1" "m2"] :in-any-order)
             )
           )
    )

Answer 1

The problem is that process-async returns immediately with "[...] a channel which will receive the result of the body when completed" (since go-loop is just syntactic sugar for (go (loop ...)) and go returns immediately).

This means that the blocking <!! in async-blocker will have a value almost immediately and the order in which the go blocks from process-async and async-blocker get executed is undetermined. It might be that most of the time the block from process-async executes first because it gets created first, but that is not much of a guarantee in a concurrent context.

According to the documentation for <!! it "Will return nil if closed. Will block if nothing is available." This means that if you can assume that the return value of (apply fun args) is a channel returned by go, you should be able to block by using <!! in the following way:

(defn async-blocker [fun & args]
  (<!! (apply fun args)))

This will unblock once there is a value in the channel (i.e. the return value from the go block).

There are other options to wait for the result of another go block. You could for example provide the original chan-test as an argument to fun and then put a value in chan-test when the go block created in fun terminates. But I think, given the code you showed, other approaches might be unnecessarily more complicated.