Serving Static Files with Common Lisp

Question

I'm writing a minimal web server in Common Lisp, and want it to be able to have some basic static file serving ability. The idea is for this to be used in development, testing and other low-traffic situations, so it doesn't have to be particularly efficient. It also won't have to deal with large files. It does, however, have to work, and this errors when attempting to serve images:

...
(with-open-file (s path :direction :input :element-type 'octet)
             (let ((buf (make-array (file-length s) :element-type 'octet)))
               (read-sequence buf s)
               (write! (make-instance 
                'response :content-type mime
                :body buf) 
                   sock))
             (socket-close sock))
...

Specifically, it gives me the error

The value 137 is not of type CHARACTER.
   [Condition of type TYPE-ERROR]
...

The relevant part of write! looks like

...
(write-string "Content-Length: " stream) (write (length body) :stream stream) (crlf stream)
(crlf stream)
(write-sequence body stream)
(crlf stream)
...

I've tried changing it to

...
(write-string "Content-Length: " stream) (write (length body) :stream stream) (crlf stream)
(crlf stream)
(write-string (flexi-streams:octets-to-string body) stream)
(crlf stream)
...

whereupon the error disappears, but the client gets a mangled version of the file. You can see more context around the above code here.

What am I doing wrong?

The system is built on top of :usocket and uses :flexi-streams for octet. The above is taking a very similar approach to Hunchentoot (see the static file snippet here; that code does chunking and a few other checks before-hand, but otherwise seems to be doing the same thing I am above). I'm running the latest stuff out of :quicklisp for everything, and running it on SBCL 1.0.57.0.debian on top of 64-bit Debian stable.

Answer 1

Thanks to jasom and Bike from #lisp for the answer:

If you want to serve static files from a usocket-based server, you need to

1. Include flexi-streams
2. Make sure to create binary streams when accepting
3. Make the streams bivalent before doing anything (otherwise character/string writes/reads are going to error on you)

The first point is easy enough. Either call (ql:quickload :flexi-streams), or add #:flexi-streams to the :depends-on line of your .asd.

The second point involves calling socket-accept with the additional argument:

(socket-accept ready :element-type 'octet)

Third, instead of passing around a socket and writing to (socket-stream socket) everywhere, you'll need to do a make-flexi-stream call.

(let ((stream (flex:make-flexi-stream (socket-stream sock) :external-format :utf-8)))
    ;;; stuff you want to do
    )

Depending on your specific situation, it might make sense either to wrap every write/read in the above, or do it once (right after you call socket-accept) then pass the flexi-stream around along with the socket.