Lisp Web Tips

This is a random collection of tips that may help you when developing JSON-based web services with Lisp.

Debugging web services

In most systems, it's a little tricky to debug web applications because they run as separate processes, and error messages and trace output are often sent to a console stream, not to your Lisp Listener window.

To see error and trace output that occur in web request handling

Use explicit JSON encoding in CL-JSON

CL-JSON, like other encoder/decoder tools for Lisp and other languages, has a standard set of rules for mapping from JSON objects to Lisp objects, and back. The atomic cases, like numbers and strings are easy, as are arrays. But there's a problem when going from Lisp lists to JSON. Should a list like 

((a 1) (b 2) (c 3))

be mapped to an array of pairs or a JSON object with the keys A, B and C? CL-JSON's table say that lists go to arrays, except for association lists. The documentation doesn't say how it decides when a list is an association list. If you look at the code, you'll see that it tests the first pair to see if the CDR is an atom. The above list is NOT an association list, by that rule.

CL-JSON has functions like encode-json-alist that can force a list to be treated as an association list but that doesn't help on nested lists.

So if you're writing code to construct nested data to be translated to JSON, what can you do? One alternative is to use the streaming encoder functions. But that makes for code that's harder to unit test.

Fortunately, study of the source code shows that there is an undocumented but exported macro with-explicit-encoder that lets you mark each list for how you want it treated. You use it like this:

(with-explicit-encoder (encode-json-to-string ...))

You mark the front of every list, to specify how you want it encoded. Put :array to map a list to an array, :alist to map a list of pairs to a JSON object, and :plist to map a plist to a JSON object. For example:

(:plist :test "exam-1"
      :scores (:array (:plist :name "John" :score 85)
                      (:plist :name "Mary" :score 92)))

will be encoded into this JSON:

{ "test": "exam-1",
      "scores": [ 
        { "name": "John", "score": 85 },
        { "name": "Mary", "score": 92}
      ]
    }

You need to be sure your code labels every nested list properly, or you will get an error from the JSON encoder.

If your lists use keywords only for JSON keys, a simple recursive function can convert any nested list into explicit plist form: 

(defun plistify (lst)
      (cond ((atom lst) lst)
            (t (cons (if (keywordp (car lst)) :plist :array)
                    (mapcar 'plistify lst)))))

Then this call:

(plistify '(:test "exam-1"
                :scores ((:name "John" :score 85)
                        (:name "Mary" :score 92))))

will generate the explicit plist above.

Check for JSON failure on the client

Because the now-standard JavaScript fetch() function returns a Promise, you might assume that you use a catch() clause to handle errors. This actually only catches JavaScript errors and network failure. Server failure messages, such as 404 (not found) and 500 (internal server error), are treated the same as server success responses. You need to check to see if the response is OK in your then() clause. To consolidate your error handling, you can throw an error to the catch() if the response is not OK, like this example:

const getJson = async (url) => {
      try {
        const response = await fetch(url);
        if (!response.ok) throw new Error(response.statusText);
        return response.json();
      } catch (error) {
        console.log(error)
      };
    };