Web Services Overview

A web service is just a machine -- the server -- connected to the Internet, running a computer program that listens to HTTP requests from client programs, such as browsers or other web servers.

For our purposes, the difference between a web service and a web server is that a web service sends data, while a web server sends HTML.

We'll limit ourselves here to web services that send data to clients using JSON (JavaScript Object Notation). JSON is a relatively readable, compact notation for data containing arbitrarily nested numbers, strings, arrays, and key-value tables, as shown in these examples.

Note that JSON is more restricted than JavaScript objects. For example, { name: "John", age: 10 } is fine in JavaScript, but in JSON you have to write { "name": "John", "age": 10 }.

Web services are, in my opinion, a great way to go, even when implementing a web site. That's the focus on this page.

See my web tips page for some technical notes on web services.

Web Sites with Web Services

When the web began, it had a very simple architecture: clients, e.g., browsers, sent URL's to servers and servers returned HTML files.

static html

A major factor in the expansion of the web into all kinds of areas was the fact that a client program doesn't need to where or how a server got the HTML it sent out. So the web soon expanded to include HTML dynamically constructed by applying HTML templates to data.

server-side dynamic html

In the past few years, there's been an additional decoupling, by having servers send data separately from the HTML. The client requests data via AJAX calls.

This turns the web servers into more general web services. It also allows web design to be done purely in web technologies, i.e., HTML, CSS, JavaScript, and the many modern frameworks, such as Angular, React, and Vue.

Lisp client-side technologies

The disadvantages of doing this include:

The advantages are many, however, including:

Lisp Technologies for Web Services

There are a number of Lisp web servers. One is Franz, Inc.'s AServe library for Allegro and Lispworks users. Students using Allegro already have AServe available and ready to load.

Another popular server is Hunchentoot. This is commonly used with SBCL and LispWorks.

These provide the basis for implementing Lisp-based web sites. But to avoid writing the same boilerplate code over and over, a web services framework can be helpful.

The CS325 Simple Server

Much of the documentation for AllegroServe, Hunchentoot, and other Lisp web servers focuses on how to generate HTML in Lisp. A simpler way to go, though, is to create the frontend with HTML, CSS, and JavaScript, and have it call the backend to get data and calculated results. This is the purpose of the Simple Server framework. It is implemented for AllegroServe and Hunchentoot. It supports static assets and uses JSON for data passing.

Installation

(ql:quickload "simple-server")

By default, Simple Server uses AllegroServe for Allegro, and Hunchentoot for SBCL and Lispworks. If you wish to change this, edit the file simple-server.asd.

Hunchentoot only: Hunchentoot tries to load the openSSL library. If you don't have it, loading Simple Server will get an error about not finding libSSL. You can either install openSSL, or just select the option in the error display to act as if the file cl+ssl loaded successfully. This will mean you can't run the server with HTTPS requests. SSL and HTTPS are beyond the scope of this course.

Starting and stopping

To start the server, 

(simple-server:start-server)

To stop the server

(simple-server:stop-server)
.

The default port is 8000. You can specify a different port, e.g., 

(simple-server:start-server :port 8001)

You can have servers running on several ports. To stop a specific server call simple-server with a :port argument. If no server is given, all servers are stopped.

Defining Simple Server apps

To create a Simple Server app, create an ASDF module in ~/quicklisp/local-projects/cs325/simple-server/apps/. Creating an ASDF module involves creating the files listed in the table below. Once those exist, you start Lisp and tell Quicklisp to look for and register the new project. Then, as needed, you can use Quicklisp to load the project with all the files and libraries that it needs.

FilePurposeExample
ASD file Specifies the files and libraries needed. numbers.asd
Package file Defines the Lisp package(s) needed. Typically there is a package for server-specific code, that uses in turn imports and uses other packages that handle the "business" logic. packages.lisp
Server code file Defines the code to process HTTP requests and return JSON numbers-server.lisp
Business logic code file Defines regular Lisp functions to do the actual work. numbers.lisp

For each service, you define a server package to hold the code to process requests. Typically the server package will use another package to do the logic. This second package is just Lisp, with no server-specific code. Each package

For example, here's how to build a service to convert numbers to Roman numerals or English text, using the Lisp format function:

Create a directory in local-projects, e.g,. ~/quicklisp/local-projects/numbers.

In that directory, create the files

See the JSON Demo app for an example of each file.

Update Quicklisp's registry of local projects with 

(ql:register-local-projects)

Now when you want to run this service, 

(ql:quickload "numbers"

If the server is already running, it is not necessary to restart it.

Writing server code

The server code is responsible for accepting HTTP requests, calling the appropriate Lisp functions, and sending back data, typically in JSON form.

Use defroute to map URLs to those wrapper functions.

(defroute http-method url function) [macro]

This macro specifies what function to call when a client sends url via http-method. The relevant HTTP methods are :get, :put, or :post. Url is a string path. If you want to run multiple apps, use URLs that include the the app name, e.g., "/numbers/to-roman".

Function should evaluate to a function object or a function name. The function will be called with two arguments: data and params. data will be a string with JSON data if a web page did an AJAX call with PUT or POST. params will be an alist of the request query parameters, if any, for a GET or form POST.

A handy utility function defined by SimpleServer is param-value.

(param-value key alist &optional default) [function]
This function returns the value for key in alist, if present, else default, if given, else NIL. The lookup uses a case-insensitive string match, so (param-value "number" ...) and (param-value :number ...) do the same thing.

Any library can be used to read and write JSON. A popular library is CL-JSON. There's a good comparison of them here.

In CL-JSON, you use decode-json-from-string to parse a JSON string into an alist. You use encode-json-alist-to-string to convert an a-list to a string that can be returned to the browser.

Example Web App: The JSON Demo Server

The JSON demo code shows how to build a JSON-based web app with Simple Server.

To try the demo:

> (ql:quickload "json-demo")
  ...
  > (in-package #:json-demo)
  ...
  > (start-server)

Assuming there were no errors, point your browser at http://localhost:8000/json-demo/demo.html. A form should appear with several simple examples that use JavaScript to communicate with the server.

If you get an error that start-server is not defined, you probably forgot to do the in-package.

Simple Server apps are defined in subfolders of ~/quicklisp/local-projects/cs325/simple-server/apps/. The JSON demo is in the subfolder json-demo. That subfolder has the app's home page, demo.html, at the top level, so the URL to get to the app is http://localhost:8000/json-demo/demo.html.

Some notes on the JavaScript and Lisp in the demo code in the file json-demo.lisp follow. The Lisp uses the CL-JSON library to map Lisp data to and from JSON. You can use any JSON library if you wish. There's a good comparison of them here.

Getting data example

The first form on the demo page is an example of getting JSON data using the HTTP GET method and information contained in the form. Td do this, the JavaScript code calls the method fetch() with a URL that is constructed using a URLSearchParams object.

On the back-end, to make this work, the Lisp code associates the route /exports with a Lisp function EXPORTS that returns a list of exported symbols: 

(defroute :get "/exports" 'exports)

The URL is expected to have the form /exports?name=...&package=.... The package is optional, and defaults to COMMON-LISP. The function EXPORTS looks like this: 

(defun exports (data params)
    (let ((name (param-value "name" params ""))
          (package (param-value "package" params "COMMON-LISP")))
      (encode-json-plist-to-string 
      (list :name name :package package
            :symbols (exports-list name package)))))

This is typical code for a function to handle requests. It collects data from URL request parameters, calls another Lisp function to do the work, then encodes the result in JSON form to return to the browser. It

The function EXPORTS-LIST -- see the Lisp file for the code -- is a normal Lisp function that knows nothing about JSON. It takes two strings and returns a list of those Lisp symbols that package exports that contain name as a substring. Separating the request handling from the code logic makes it easier to debug the logic in Lisp.

EXPORTS-LIST returns a plist with this data for encoding in the equivalent JSON object to return to the client. It could have just as easily created an alist.

Saving data example

Data should be saved using the HTTP method POST. The data to be saved can be passed either in search parameters, as with GET, but the example JavaScript instead sends the data in JSON in the body of the request. While not necessary for this simple case, sending JSON allows for more complicated data structures to be sent.

On the back-end, the desired route needs to be associated with a Lisp function. 

(defroute :post "/save-data" 'save-data)

Since the data is coming in the body of request, SAVE-DATA calls the CL-JSON function DECODE-JSON to get an alist equivalent of a JSON object from the request. Then it calls a normal Lisp function, UPDATE-A-LIST, to simulate storing the data, and then calls the CL-JSON functon ENCODE-JSON-ALIST-TO-STRING to create a JSON object of the return value to return to the browser. 

(defun save-data (data params)
    (encode-json-alist-to-string (update-alist (get-json-from-string data))))

  (defun get-json-from-string (str)
    (and str (stringp str) (> (length str) 0)
        (decode-json-from-string str)))

get-json is typical code for parsing a string to get a JSON object. You need to make sure the string is not empty, or a JSON parsing error will occur.

update-alist, defined in the demo code, merges an alist with an alist stored in an internal special variable. It returns the merged result. By design, update-alist is a normal Lisp function that knows nothing about JSON.