Making HTTP requests with Lisp

There are several libraries for making HTTP calls to web servers, to get HTML, XML, or JSON. We will use Drakma.

See this page for a quick intro, with links, to JSON, web services, and CL-JSON. Don't worry about the later parts on SimpleServer yet.

Install Open SSL

You need to have SSL working in Lisp in order to access HTTPS URLs. On Windows, that means you need to have OpenSSL installed. These instructions should work. Some notes:

Note: both the Win32 and Win64 versions can be installed if you want to run several Lisps.

If you get an error in Lisp about a missing SSL library, check your Windows directories for the files libssl32.dll and libeay32.dll.

Install Drakma

This should be easy.

(ql:quickload "drakma")

Using Drakma

Get data with URLs in Drakma is simple:

(drakma:http-request url)

url needs to be a string containing a complete URL. Drakma will send a request for that URL to the server indicated, just as a browser or other user client would.

drakma:http-request returns seven values, as Lisp multiple values. If you need to save or use values other than the first, use multiple-value-bind or a similar form. The values returned, in order, are

Examples Calls

If the URL is for an HTML file, like this:

(drakma:http-request "")

you will see a string containing the raw HTML.

If the URL is for data not labelled as text, including a call for JSON, like this:

(drakma:http-request "")

you will get an array of byte codes.

Allegro CL Express: If you get an "SSL not supported" error, replace two files in ~/quicklisp/dists/quicklisp/software/drakma-2.0.2/ with their current versions in the Drakma repository: drakma.asd and util.lisp.

An array of byte codes can be converted to a string using normal Lisp functions, e.g.,

(map 'string 'code-char (drakma:http-request ""))

A better way, that doesn't create an intermediate string, is used in the testing below.

Yet another approach is to tell Drakma that HTTP response content labeled application/json is plain text. You can do that with this:

(push (cons "application" "json") drakma:*text-content-types*)

Test getting and parsing HTML

If you are not using Allegro Lisp, get the open source version of the HTML parser using QuickLisp.

(ql:quickload "cl-html-parse")

Now do

(net.html.parser:parse-html (drakma:http-request ...))

to test parsing the HTML returned by the URL you tried before. You should get back a list version of the page, i.e., a version you can process with standard Lisp list functions.

If you want to have the HTTP client and HTML parser available whenever you start Lisp, put the quickload commands above into your cs325.lisp file.

Testing getting and parsing JSON

Get the CL-JSON parser.

(ql:quickload "cl-json")

Now test getting parsed JSON using the Drakma JSON example code. Change it to use the CL-JSON parser instead of the YASON parser.

Test on at least two different JSON sites. See here for some simple examples.

Test. You should get back a list version of the JSON data, similar to the one shown in the example using the YASON parser.