Cover Material, Copyright, and License

Copyright 2011-2020 Mark Watson. All rights reserved. This book may be shared using the Creative Commons “share and share alike, no modifications, no commercial reuse” license.

This eBook will be updated occasionally so please periodically check the leanpub.com web page for this book for updates.

This is the sixth edition released summer of 2020.

Please visit the author’s website.

If you found a copy of this book on the web and find it of value then please consider buying a copy at leanpub.com/lovinglisp to support the author and fund work for future updates.

Preface

Notes on the Sixth Edition Published June 2020

Two examples optionally use the CAPI user interface toolkit provided with LispWorks Common Lisp and work with the free personal edition. The first CAPI application is Knowledge Graph Navigator and the second CAPI example is Knowledge Graph Creator. Both of these examples build up utilities for working with Knowledge Graphs and the Semantic Web.

I expand the Plot Library chapter to generate either PNG graphics files or if you are using the free personal edition of LispWorks you can also direct plotting output to a new window in interactive programs.

I added a new chapter on using the py4cl library to embed Python libraries and application code into a Common Lisp system. I provide new examples for embedding spaCy and TensorFlow applications in Common Lisp applications. In earlier editions, I used a web services interface to wrap Python code using spaCy and TensorFlow. I am leaving that chapter intact, renaming it from “Using Python Deep Learning Models In Common Lisp” to “Using Python Deep Learning Models In Common Lisp With a Web Services Interface.” The new chapter for this edition is “Using the PY4CL Library to Embed Python in Common Lisp.”

Notes on the Fifth Edition Published September 2019

There were two chapters added:

A complete application for processing text to generate data for Knowledge Graphs (targeting the open source Neo4J graph database and also support RDF semantic web/linked data)

A library for accessing the state of the art spaCy natural language processing (NLP) library and also a state of the art deep learning model. These models are implemented in thin Python wrappers that use Python libraries like spaCy, PyTorch, and TensorFlow. These examples replace a simple hybrid Java and Common Lisp example in previous editions.

I have added text and explanations as appropriate throughout the book and I removed the CouchDB examples.

I have made large changes to how the code for this book is packaged. I have reorganized the example code on GitHub by providing the examples as multiple Quicklisp libraries or applications. I now do this with all of my Common Lisp code and it makes it easier to write smaller libraries that can be composed into larger applications. In my own workflow, I also like to use Makefile targets to build standalone applications that can be run on other computers without installing Lisp development environments. Please follow the directions at the end of the Preface for configuring Quicklisp for easy builds and use of the example software for this book.

Why Use Common Lisp?

Why Common Lisp? Isn’t Common Lisp an old language? Do many people still use Common Lisp?

I believe that using Lisp languages like Common Lisp, Clojure, Racket, and Scheme are all secret weapons useful in agile software development. An interactive development process and live production updates feel like a breath of fresh air if you have development on heavy weight like Java Enterprise Edition (JEE).

Yes, Common Lisp is an old language but with age comes stability and extremely good compiler technology. There is also a little inconsistency between different Common Lisp systems in such things as handling threads but with a little up front knowledge you can choose which Common Lisp systems will support your requirements.

A Request from the Author

I spent time writing this book to help you, dear reader. I release this book under the Creative Commons “share and share alike, no modifications, no commercial reuse” license and set the minimum purchase price to $5.00 in order to reach the most readers. Under this license you can share a PDF version of this book with your friends and coworkers and I encourage you to do so. If you found this book on the web (or it was given to you) and if it provides value to you then please consider doing one of the following to support my future writing efforts and also to support future updates to this book:

Purchase a copy of this book leanpub.com/lovinglisp/ or any other of my leanpub books at https://leanpub.com/u/markwatson

Hire me as a consultant

I enjoy writing and your support helps me write new editions and updates for my books and to develop new book projects. Thank you!

Older Book Editions

The fourth edition of this book was released in May 2017 and the major changes were:

Added an example application KGCreator that processes text data to automatically generate data for Knowledge Graphs. This example application supports the Neo4J graph database as well as semantic web/linked data systems. The major changes were:

Added a backpropagation neural network example

Added a deep learning example using the Java based Armed Bear Common Lisp with the popular DeepLearning4j library

Added a heuristic search example

Added two machine learning examples (K-Means clustering and SVM classification) using the CLML library

A few edits to the previous text

The third edition was released in October 2014. The major changes made in the 2014 edition are:

I reworked the chapter Common Lisp Basics .

. I added material to the chapter on using QuickLisp.

The second edition was released in 2013 and was derived from the version that I distributed on my web site and I moved production of the book to leanpub.com.

Acknowledgments

I would like to thank Jans Aasman for contributing as technical editor for the fourth edition of this book. Jans is CEO of Franz.com which sells Allegro Common Lisp as well as tools for semantic web and linked data applications.

I would like to thank the following people who made suggestions for improving previous editions of this book:

Sam Steingold, Andrew Philpot, Kenny Tilton, Mathew Villeneuve, Eli Draluk, Erik Winkels, Adam Shimali, and Paolo Amoroso.

I would like to also thank several people who pointed out typo errors in this book and for specific suggestions: Martin Lightheart, Tong-Kiat Tan, Rainer Joswig, Gerold Rupprecht, and David Cortesi. I would like to thanks the following Reddit /r/lisp readers who pointed out mistakes in the fifth edition of this book: arnulfslayer, rpiirp, and itmuckel. I would like to thank Ted Briscoe for pointing out a problem with the spacy web client example in the 6th edition.

I would like to thank Paul Graham for coining the phrase “The Secret Weapon” (in his excellent paper “Beating the Averages”) in discussing the advantages of Lisp and giving me permission to reuse his phrase.

I would especially like to thank my wife Carol Watson for her fine work in editing this book.

Setting Up Your Common Lisp Development System and Quicklisp

These instructions assume the use of SBCL. See comments for LispWorks, Franz Common Lisp, and Closure Common List at the end of this section. I assume that you have installed SBCL and Quicklisp by following the instructions at lisp-lang.org/learn/getting-started. These instructions also guide you through installing the Slime extensions for Emacs. I use both Emacs + Slime and VSCode with Common Lisp plugins for editing Common Lisp. If you like VSCode then I recommend Yasuhiro Matsumoto’s Lisp plugin for syntax highlighting. For both Emacs and VSCode I usually run a separate REPL in a terminal window and don’t run an editor-integrated REPL. I think that am in the minority in using a separate REPL running in a shell.

I have been using Common Lisp since about 1982 and Quicklisp has been the most revolutionary change in my Common Lisp development (even more so than getting a hardware Lisp Machine and the availability of Coral Common Lisp on the Macintosh). I am going to ask you, dear reader, to trust me and adopt the following advice that I have adopted from Zach Beane, the creator and maintainer of Quicklisp:

Create the file ~/.config/common-lisp/source-registry.conf.d/projects.conf if it does not exist on your system

Assuming that you have cloned the repository for this book (loving-common-lisp) in your home directory (if you have a special place where you clone git repos, adjust the following), edit this configuration file to look like this:

1 (:tree 2 (:home "loving-common-lisp/src/") 3 )

This will make subdirectories of loving-common-lisp/src/ load-able by using Quicklisp. For example, the subdirectory loving-common-lisp/src/spacy_client contains a package named spacy that can now be accessed from any directory on your system using:

$ sbcl (ql:quickload "spacy") * (spacy:spacy-client "My sister has a dog Henry. She loves him.") * (defvar x (spacy:spacy-client "President Bill Clinton went to Congress. He gave a speech on taxes and Mexico.")) * (spacy:spacy-data-entities x) * (spacy:spacy-data-tokens x)

This example uses the deep learning NLP models in spaCy.

List of Quicklisp Projects and Small Examples in this Book

The major example libraries and applications will be in their own packages. The function and data definitions for all short code snippets in this book are in a package loving-snippets in the subdirectory loving-common-lisp/src/loving_snippets. Whenever you work through the short examples in this book I will assume that you have opened a SBCL (or other Common Lisp) REPL and loaded this package:

$ sbcl * (ql:quickload "spacy_web_client")

On one of my Linux laptops, for reasons I haven’t discovered yet, using ~/.config/common-lisp/source-registry.conf.d/projects.conf to set a root directory for Quicklisp to look for packages does not work. If by small chance this does not work for you, you can set symbolic file links from the example book packages to your ~/quicklisp/local-projects directory. This is the directory where Quicklisp stores local copies of libraries that you install. For example:

$ cd ~/quicklisp/local-projects $ ln -s loving-common-lisp/src/loving_snippets . $ ln -s loving-common-lisp/src/kgcreator . $ ln -s loving-common-lisp/src/kbnlp . etc.

Hopefully you won’t have to bother doing this workaround.

While most of the longer examples in this book are Quicklisp projects, there are also many very short code snippets in the book that are found in the subdirectories src/code_snippets_for_book and a few short program examples not configured as Quicklisp projects in the src/loving_snippets subdirectory:

1 $ ls code_snippets_for_book 2 closure1.lisp nested.lisp read-test-1.lisp 3 readline-test.lisp do1.lisp read-from-string-test.lisp 4 read-test-2.lisp recursion1.lisp 5 Marks-MacBook:src $ ls loving_snippets 6 HTMLstream.lisp astar_search.lisp macro1.lisp 7 Hopfield_neural_network.lisp backprop_neural_network.lisp macro2.lisp README.md l \ 8 ambda1.lisp mongo_news.lisp

The longer examples packaged as Quicklisp projects in the src directory are:

fasttag: my part of speech tagger

solr_examples: client for open source solr search engine

categorize_summarize: my NLP code for categorizing text and generating summaries

coref_web_client: client for a spaCy based web service that performs anaphora resolution (i.e., replaces pronouns in text with the nouns that the pronouns refer to)

hunchentoot_examples : examples so web services and web clients

spacy_web_client: client for a general purpose web service using state of the art deep learning models for NLP

clml_examples: examples using the Common List Machine Learning library

kbnlp: My NLP code

myutils: miscelanious functions that are used in several other example libraries in this book

webscrape: demo for how to scrape web sites

clsql_examples : examples showing how to access relational databases using the CLSQL library

entities_dbpedia : use the public DbPedia (data from WikiPedia) public web interface to get information about people, companies, locations, etc.

kgcreator: my application for processing text, extracting entities, and generating data for Knowledge Graphs (supports Neo4J and RDF semantic web/linked data applications

kgn: the application Knowledge Graph Navigator

plotlib: a very simple plotting library that writes plots to PNG graphics files

I have used the SBCL implementation of Common Lisp in this book. There are many fine Common Lisp implementations from Franz, LispWorks, Clozure Common Lisp, etc. If you have any great difficulty adopting the examples to your choice of Common Lisp implementations and performing web search does not suggest a solution then you can reach me through my web site markwatson.com. The examples that may not be portable are creating a standalone executable for my KGCreator example and the examples using the Common Lisp Machine Learning library.

Introduction

This book is intended to get you, the reader, programming quickly in Common Lisp. Although the Lisp programming language is often associated with artificial intelligence, this introduction is on general Common Lisp programming techniques. Later we will look at general example applications and artificial intelligence examples.

The Common Lisp program examples are distributed on the github repo for this book.

Why Did I Write this Book?

Why the title “Loving Common Lisp”? Simple! I have been using Lisp for almost 40 years and seldom do I find a better match between a programming language and the programming job at hand. I am not a total fanatic on Lisp, however. I often use Python for deep learning. I like Ruby, Java and Javascript for server side programming, and the few years that I spent working on Nintendo video games and virtual reality systems for SAIC and Disney, I found C++ to be a good bet because of stringent runtime performance requirements. For some jobs, I find the logic-programming paradigm useful: I also enjoy the Prolog language.

In any case, I love programming in Lisp, especially the industry standard Common Lisp. As I wrote the second edition of this book over a decade ago, I had been using Common Lisp almost exclusively for an artificial intelligence project for a health care company and for commercial product development. While working on the third edition of this book, I was not using Common Lisp professionally but since the release of the Quicklisp Common Lisp package manager I have found myself enjoying using Common Lisp more for small side projects. I use Quicklisp throughout in the third edition example code so you can easily install required libraries. For the fourth and fifth editions of this book I have added more examples using neural networks and deep learning. In this new sixth edition I have added a complete application that uses CAP for the user interface.

As programmers, we all (hopefully) enjoy applying our experience and brains for tackling interesting problems. My wife and I recently watched a two-night 7-hour PBS special “Joseph Campbell, and the Power of Myths.” Campbell, a college professor for almost 40 years, said that he always advised his students to “follow their bliss” and not to settle for jobs and avocations that are not what they truly want to do. That said I always feel that when a job calls for using Java, Python or other languages besides Lisp, that even though I may get a lot of pleasure from the job I am not following my bliss.

My goal in this book is to introduce you to one of my favorite programming languages, Common Lisp. I assume that you already know how to program in another language but if you are a complete beginner you can still master the material in this book with some effort. I challenge you to make this effort.

There are several Common Lisp compilers and runtime tools available for free on the web:

CLISP – licensed under the GNU GPL and is available for Windows, Macintosh, and Linux/Unix

Clozure Common Lisp (CCL) – open source with good Mac OS X and Linux support

CMU Common Lisp – open source implementation

SBCL – derived from CMU Common Lisp

ECL – compiles using a separate C/C++ compiler

ABCL – Armed Bear Common Lisp for the JVM

There are also fine commercial Common Lisp products:

LispWorks – high quality and reasonably priced system for Windows and Linux. No charge for distributing compiled applications lispworks.com

Allegro Common Lisp - high quality, great support and higher cost. franz.com

MCL – Macintosh Common Lisp. I used this Lisp environment in the late 1980s. MCL was so good that I gave away my Xerox 1108 Lisp Machine and switched to a Mac and MCL for my development work. Now open source but only runs on the old MacOS

I currently (mostly) use SBCL, CCL, and LispWorks. The SBCL compiler produces very fast code and the compiler warning can be of great value in finding potential problems with your code. Like CCL because it compiles quickly so is often preferable for development.

For working through this book, I will assume that you are using SBCL or CCL. For the example in the last chapter you will need LispWorks and the free Personal edition is fine for the purposes of experimenting with the example application and the CAPI user interface library.

How is Lisp Different from Languages like Java and C++?

This is a trick question! Lisp is slightly more similar to Java than C++ because of automated memory management so we will start by comparing Lisp and Java.

In Java, variables are strongly typed while in Common Lisp values are strongly typed. For example, consider the Java code:

1 Float x = new Float(3.14f); 2 String s = "the cat ran" ; 3 Object any_object = null; 4 any_object = s; 5 x = s; // illegal: generates a 6 // compilation error

Here, in Java, variables are strongly typed so a variable x of type Float can’t legally be assigned a string value: the code in line 5 would generate a compilation error. Lisp code can assign a value to a variable and then reassign another value of a different type.

Java and Lisp both provide automatic memory management. In either language, you can create new data structures and not worry about freeing memory when the data is no longer used, or to be more precise, is no longer referenced.

Common Lisp is an ANSI standard language. Portability between different Common Lisp implementations and on different platforms is very good. I have used Clozure Common Lisp, SBCL, Allegro Lisp (from Franz Inc), LispWorks, and CLISP that all run well on Windows, Mac OS X, and Linux. As a Common Lisp developer you will have great flexibility in tools and platforms.

ANSI Common Lisp was the first object oriented language to become an ANSI standard language. The Common Lisp Object System (CLOS) is probably the best platform for object oriented programming.

In C++ programs, a common bug that affects a program’s efficiency is forgetting to free memory that is no longer used. In a virtual memory system, the effect of a program’s increasing memory usage is usually just poorer system performance but can lead to system crashes or failures if all available virtual memory is exhausted. A worse type of C++ error is to free memory and then try to use it. Can you say “program crash”? C programs suffer from the same types of memory related errors.

Since computer processing power is usually much less expensive than the costs of software development, it is almost always worth while to give up a few percent of runtime efficiency and let the programming environment of runtime libraries manage memory for you. Languages like Lisp, Ruby, Python, and Java are said to perform automatic garbage collection.

I have written six books on Java, and I have been quoted as saying that for me, programming in Java is about twice as efficient (in terms of my time) as programming in C++. I base this statement on approximately ten years of C++ experience on projects for SAIC, PacBell, Angel Studios, Nintendo, and Disney. I find Common Lisp and other Lisp languages like Clojure and Scheme to be about twice as efficient (again, in terms of my time) as Java. That is correct: I am claiming a four times increase in my programming productivity when using Common Lisp vs. C++.

What do I mean by programming productivity? Simple: for a given job, how long does it take me to design, code, debug, and later maintain the software for a given task.

Advantages of Working in a Lisp Environment

We will soon see that Lisp is not just a language; it is also a programming environment and runtime environment.

The beginning of this book introduces the basics of Lisp programming. In later chapters, we will develop interesting and non-trivial programs in Common Lisp that I argue would be more difficult to implement in other languages and programming environments.

The big win in programming in a Lisp environment is that you can set up an environment and interactively write new code and test new code in small pieces. We will cover programming with large amounts of data in the Chapter on Natural Language Processing, but let me share a a general use case for work that I do that is far more efficient in Lisp:

Much of my Lisp programming used to be writing commercial natural language processing (NLP) programs for my company www.knowledgebooks.com. My Lisp NLP code uses a large amount of memory resident data; for example: hash tables for different types of words, hash tables for text categorization, 200,000 proper nouns for place names (cities, counties, rivers, etc.), and about 40,000 common first and last names of various nationalities.

If I was writing my NLP products in C++, I would probably use a relational database to store this data because if I read all of this data into memory for each test run of a C++ program, I would wait 30 seconds every time that I ran a program test. When I start working in any Common Lisp environment, I do have to load the linguistic data into memory one time, but then can code/test/code/test… for hours with no startup overhead for reloading the data that my programs need to run. Because of the interactive nature of Lisp development, I can test small bits of code when tracking down errors and when writing new code.

It is a personal preference, but I find the combination of the stable Common Lisp language and an iterative Lisp programming environment to be much more productive than other languages and programming environments.

Common Lisp Basics

The material in this chapter will serve as an introduction to Common Lisp. I have attempted to make this book a self contained resource for learning Common Lisp and to provide code examples to perform common tasks. If you already know Common Lisp and bought this book for the code examples later in this book then you can probably skip this chapter.

For working through this chapter we will be using the interactive shell, or repl, built into SBCL and other Common Lisp systems. For this chapter it is sufficient for you to download and install SBCL. Please install SBCL right now, if you have not already done so.

Getting Started with SBCL

When we start SBCL, we see an introductory message and then an input prompt. We will start with a short tutorial, walking you through a session using SBCL repl (other Common LISP systems are very similar). A repl is an interactive console where you type expressions and see the results of evaluating these expressions. An expression can be a large block of code pasted into the repl, using the load function to load Lisp code into the repl, calling functions to test them, etc. Assuming that SBCL is installed on your system, start SBCL by running the SBCL program:

1 % sbcl 2 ( running SBCL from : / Users / markw / sbcl ) 3 This is SBCL 2.0 . 2 , an implementation of ANSI Common Lisp . 4 More information about SBCL is available at < http : // www . sbcl . org /> . 5 6 SBCL is free software , provided as is , with absolutely no warranty . 7 It is mostly in the public domain ; some portions are provided under 8 BSD - style licenses . See the CREDITS and COPYING files in the 9 distribution for more information . 10 11 * ( defvar x 1.0 ) 12 13 X 14 * x 15 16 1.0 17 * ( + x 1 ) 18 19 2.0 20 * x 21 22 1.0 23 * ( setq x ( + x 1 )) 24 25 2.0 26 * x 27 28 2.0 29 * ( setq x " the dog chased the cat " ) 30 31 " the dog chased the cat " 32 * x 33 34 " the dog chased the cat " 35 * ( quit )

We started by defining a new variable x in line 11. Notice how the value of the defvar macro is the symbol that is defined. The Lisp reader prints X capitalized because symbols are made upper case (we will look at the exception later).

In Lisp, a variable can reference any data type. We start by assigning a floating point value to the variable x, using the + function to add 1 to x in line 17, using the setq function to change the value of x in lines 23 and 29 first to another floating point value and finally setting x to a string value. One thing that you will have noticed: function names always occur first, then the arguments to a function. Also, parenthesis is used to separate expressions.

I learned to program Lisp in 1976 and my professor half-jokingly told us that Lisp was an acronym for “Lots-of Irritating Superfluous Parenthesis.” There may be some truth in this when you are just starting with Lisp programming, but you will quickly get used to the parenthesis, especially if you use an editor like Emacs that automatically indents Lisp code for you and highlights the opening parenthesis for every closing parenthesis that you type. Many other editors support coding in Lisp but I personally use Emacs or sometimes VScode (with Common Lisp plugins) to edit Lisp code.

Before you proceed to the next chapter, please take the time to install SBCL on your computer and try typing some expressions into the Lisp listener. If you get errors, or want to quit, try using the quit function:

1 * (+ 1 2 3 4) 2 3 10 4 * (quit) 5 Bye.

If you get an error you can enter help to get options for handling an error. When I get an error and have a good idea of what caused the error then I just enter :a: to abort out of the error).

As we discussed in the introduction, there are many different Lisp programming environments that you can choose from. I recommend a free set of tools: Emacs, Quicklisp, slime, and SBCL. Emacs is a fine text editor that is extensible to work well with many programming languages and document types (e.g., HTML and XML). Slime is an Emacs extension package that greatly facilitates Lisp development. SBCL is a robust Common Lisp compiler and runtime system that is often used in production.

We will cover the Quicklisp package manager and using Quicklisp to setup Slime and Emacs in a later chapter.

I will not spend much time covering the use of Emacs as a text editor in this book since you can try most of the example code snippets in the book text by copying and then pasting them into a SBCL repl and by loading the book example source files directly into a repl. If you already use Emacs then I recommend that you do set up Slime sooner rather than later and start using it for development. If you are not already an Emacs user and do not mind spending the effort to learn Emacs, then search the web first for an Emacs tutorial. That said, you will easily be able to use the example code from this book using any text editor you like with a SBCL repl. I don’t use the vi or vim editors but if vi is your weapon of choice for editing text then a web search for “common lisp vi vim repl” should get you going for developing Common Lisp code with vi or vim. If you are not already an Emacs or vi user then using VSCode with a Common Lisp plugin is recommended.

Here, we will assume that under Windows, Unix, Linux, or Mac OS X you will use one command window to run SBCL and a separate editor that can edit plain text files.

Making the repl Nicer using rlwrap

While reading the last section you (hopefully!) played with the SBCL interactive repl. If you haven’t played with the repl, I won’t get too judgmental except to say that if you do not play with the examples as you read you will not get the full benefit from this book.

Did you notice that the backspace key does not work in the SBCL repl? The way to fix this is to install the GNU rlwrap utility. On OS X, assuming that you have homebrew installed, install rlwrap with:

1 brew install rlwrap

If you are running Ubuntu Linux, install rlwrap using:

1 sudo apt-get install rlwrap

You can then create an alias for bash or zsh using something like the following to define a command rsbcl:

1 alias rsbcl='rlwrap sbcl'

This is fine, just remember to run sbcl if you don’t need rlwrap command line editing or run rsbcl when you do need command line editing. That said, I find that I always want to run SBCL with command line editing, so I redefine sbcl on my computers using:

1 -> ~ which sbcl 2 /Users/markw/sbcl/sbcl 3 -> ~ alias sbcl='rlwrap /Users/markw/sbcl/sbcl'

This alias is different on my laptops and servers, since I don’t usually install SBCL in the default installation directory. For each of my computers, I add an appropriate alias in my .zshrc file (if I am running zsh) or my .bashrc file (if I am running bash).

The Basics of Lisp Programming

Although we will use SBCL in this book, any Common Lisp environment will do fine. In previous sections, we saw the top-level Lisp prompt and how we could type any expression that would be evaluated:

1 * 1 2 1 3 * 3.14159 4 3.14159 5 * "the dog bit the cat" 6 "the dog bit the cat" 7 * (defun my-add-one (x) 8 (+ x 1)) 9 MY-ADD-ONE 10 * (my-add-one -10) 11 -9

Notice that when we defined the function my-add-one in lines 7 and 8, we split the definition over two lines and on line 8 you don’t see the “*” prompt from SBCL – this lets you know that you have not yet entered a complete expression. The top level Lisp evaluator counts parentheses and considers a form to be complete when the number of closing parentheses equals the number of opening parentheses and an expression is complete when the parentheses match. I tend to count in my head, adding one for every opening parentheses and subtracting one for every closing parentheses – when I get back down to zero then the expression is complete. When we evaluate a number (or a variable), there are no parentheses, so evaluation proceeds when we hit a new line (or carriage return).

The Lisp reader by default tries to evaluate any form that you enter. There is a reader macro ‘ that prevents the evaluation of an expression. You can either use the ‘ character or quote:

1 * (+ 1 2) 2 3 3 * '(+ 1 2) 4 (+ 1 2) 5 * (quote (+ 1 2)) 6 (+ 1 2) 7 *

Lisp supports both global and local variables. Global variables can be declared using defvar:

1 * (defvar *x* "cat") 2 *X* 3 * *x* 4 "cat" 5 * (setq *x* "dog") 6 "dog" 7 * *x* 8 "dog" 9 * (setq *x* 3.14159) 10 3.14159 11 * *x* 12 3.14159

One thing to be careful of when defining global variables with defvar: the declared global variable is dynamically scoped. We will discuss dynamic versus lexical scoping later, but for now a warning: if you define a global variable avoid redefining the same variable name inside functions. Lisp programmers usually use a global variable naming convention of beginning and ending dynamically scoped global variables with the * character. If you follow this naming convention and also do not use the * character in local variable names, you will stay out of trouble. For convenience, I do not always follow this convention in short examples in this book.

Lisp variables have no type. Rather, values assigned to variables have a type. In this last example, the variable x was set to a string, then to a floating-point number. Lisp types support inheritance and can be thought of as a hierarchical tree with the type t at the top. (Actually, the type hierarchy is a DAG, but we can ignore that for now.) Common Lisp also has powerful object oriented programming facilities in the Common Lisp Object System (CLOS) that we will discuss in a later chapter.

Here is a partial list of types (note that indentation denotes being a subtype of the preceding type):

1 t [top level type (all other types are a sub-type)] 2 sequence 3 list 4 array 5 vector 6 string 7 number 8 float 9 rational 10 integer 11 ratio 12 complex 13 character 14 symbol 15 structure 16 function 17 hash-table

We can use the typep function to test the type of value of any variable or expression or use type-of to get type information of any value):

1 * (setq x '(1 2 3)) 2 (1 2 3) 3 * (typep x 'list) 4 T 5 * (typep x 'sequence) 6 T 7 * (typep x 'number) 8 NIL 9 * (typep (+ 1 2 3) 'number) 10 T 11 * (type-of 3.14159) 12 single-float 13 * (type-of "the dog ran quickly") 14 (simple-array character (19)) 15 * (type-of 100193) 16 (integer 0 4611686018427387903)

A useful feature of all ANSI standard Common Lisp implementations’ top-level listener is that it sets * to the value of the last expression evaluated. For example:

1 * (+ 1 2 3 4 5) 2 15 3 * * 4 15 5 * (setq x *) 6 15 7 * x 8 15

All Common Lisp environments set * to the value of the last expression evaluated. This example may be slightly confusing because * is also the prompt character in the SBCL repl that indicates that you can enter a new expression for evaluation. For example in line 3, the first * character is the repl prompt and the second * we type in to see that value of the previous expression that we typed into the repl.

Frequently, when you are interactively testing new code, you will call a function that you just wrote with test arguments; it is useful to save intermediate results for later testing. It is the ability to create complex data structures and then experiment with code that uses or changes these data structures that makes Lisp programming environments so effective.

Common Lisp is a lexically scoped language that means that variable declarations and function definitions can be nested and that the same variable names can be used in nested let forms; when a variable is used, the current let form is searched for a definition of that variable and if it is not found, then the next outer let form is searched. Of course, this search for the correct declaration of a variable is done at compile time so there need not be extra runtime overhead. We can nest defun special form inside each other and inside let expressions but this defines the nested functions globally. We use the special forms flet and labels to define functions inside a scoped environment. Functions defined inside a labels special form can be recursive while functions defined inside a flet special form cannot be recursive. Consider the following example in the file nested.lisp (all example files are in the src directory):

1 ( flet (( add-one ( x ) 2 ( + x 1 )) 3 ( add-two ( x ) 4 ( + x 2 ))) 5 ( format t "redefined variables: ~A ~A~%" ( add-one 100 ) ( add-two 100 ))) 6 7 ( let (( a 3.14 )) 8 ( defun test2 ( x ) 9 ( print x )) 10 ( test2 a )) 11 12 ( test2 50 ) 13 14 ( let (( x 1 ) 15 ( y 2 )) 16 ;; define a test function nested inside a let statement: 17 ( flet (( test ( a b ) 18 ( let (( z ( + a b ))) 19 ;; define a helper function nested inside a let/function/let: 20 ( flet (( nested-function ( a ) 21 ( + a a ))) 22 ( nested-function z ))))) 23 ;; call nested function 'test': 24 ( format t "test result is ~A~%" ( test x y )))) 25 26 ( let (( z 10 )) 27 ( labels (( test-recursion ( a ) 28 ( format t "test-recursion ~A~%" ( + a z )) 29 ( if ( > a 0 ) 30 ( test-recursion ( - a 1 ))))) 31 ( test-recursion 5 )))

We define a top level flet special form in lines 1-5 that defines two nested functions add-one and add-two and then calls each nested function in the body of the flet special form. For many years I have used nested defun special forms inside let expressions for defining local functions and you will notice this use in a few later examples. However, functions defined inside defun special forms have global visibility so they are not hidden in the local context where they are defined. The example of a nested defun in lines 7-12 shows that the function test2 has global visibility inside the current package.

Functions defined inside of a flet special form have access to variables defined in the outer scope containing the flet (also applies to labels). We see this in lines 14-24 where the local variables x and y defined in the let expression are visible inside the function nested-function defined inside the flet.

The final example in lines 26-31 shows a recursive function defined inside a labels special form.

Assuming that we started SBCL in the src directory we can then use the Lisp load function to evaluate the contents of the file nested.lisp in the sub-directory code_snippets_for_book using the load function:

* (load "./code_snippets_for_book/nested.lisp") redefined variables: 101 102 3.14 50 test result is 6 test-recursion 15 test-recursion 14 test-recursion 13 test-recursion 12 test-recursion 11 test-recursion 10 T *

The function load returned a value of t (prints in upper case as T) after successfully loading the file.

We will use Common Lisp vectors and arrays frequently in later chapters, but will also briefly introduce them here. A singly dimensioned array is also called a vector. Although there are often more efficient functions for handling vectors, we will just look at generic functions that handle any type of array, including vectors. Common Lisp provides support for functions with the same name that take different argument types; we will discuss this in some detail when we cover this in the later chapter on CLOS. We will start by defining three vectors v1, v2, and v3:

1 * (setq v1 (make-array '(3))) 2 #(NIL NIL NIL) 3 * (setq v2 (make-array '(4) :initial-element "lisp is good")) 4 #("lisp is good" "lisp is good" "lisp is good" "lisp is good") 5 * (setq v3 #(1 2 3 4 "cat" '(99 100))) 6 #(1 2 3 4 "cat" '(99 100))

In line 1, we are defining a one-dimensional array, or vector, with three elements. In line 3 we specify the default value assigned to each element of the array v2. In line 5 I use the form for specifying array literals using the special character #. The function aref can be used to access any element in an array:

* (aref v3 3) 4 * (aref v3 5) '(99 100) *

Notice how indexing of arrays is zero-based; that is, indices start at zero for the first element of a sequence. Also notice that array elements can be any Lisp data type. So far, we have used the special operator setq to set the value of a variable. Common Lisp has a generalized version of setq called setf that can set any value in a list, array, hash table, etc. You can use setf instead of setq in all cases, but not vice-versa. Here is a simple example:

* v1 #(NIL NIL NIL) * (setf (aref v1 1) "this is a test") "this is a test" * v1 #(NIL "this is a test" NIL) *

When writing new code or doing quick programming experiments, it is often easiest (i.e., quickest to program) to use lists to build interesting data structures. However, as programs mature, it is common to modify them to use more efficient (at runtime) data structures like arrays and hash tables.

Symbols

We will discuss symbols in more detail the Chapter on Common Lisp Packages. For now, it is enough for you to understand that symbols can be names that refer to variables. For example:

> (defvar *cat* "bowser") *CAT* * *cat* "bowser" * (defvar *l* (list *cat*)) *L* * *l* ("bowser") *

Note that the first defvar returns the defined symbol as its value. Symbols are almost always converted to upper case. An exception to this “upper case rule” is when we define symbols that may contain white space using vertical bar characters:

* (defvar |a symbol with Space Characters| 3.14159) |a symbol with Space Characters| * |a symbol with Space Characters| 3.14159 *

Operations on Lists

Lists are a fundamental data structure of Common Lisp. In this section, we will look at some of the more commonly used functions that operate on lists. All of the functions described in this section have something in common: they do not modify their arguments.

In Lisp, a cons cell is a data structure containing two pointers. Usually, the first pointer in a cons cell will point to the first element in a list and the second pointer will point to another cons representing the start of the rest of the original list.

The function cons takes two arguments that it stores in the two pointers of a new cons data structure. For example:

* (cons 1 2) (1 . 2) * (cons 1 '(2 3 4)) (1 2 3 4) *

The first form evaluates to a cons data structure while the second evaluates to a cons data structure that is also a proper list. The difference is that in the second case the second pointer of the freshly created cons data structure points to another cons cell.

First, we will declare two global variables l1 and l2 that we will use in our examples. The list l1 contains five elements and the list l2 contains four elements:

* (defvar l1 '(1 2 (3) 4 (5 6))) L1 * (length l1) 5 * (defvar l2 '(the "dog" calculated 3.14159)) L2 * l1 (1 2 (3) 4 (5 6)) * l2 (THE "dog" CALCULATED 3.14159) >

You can also use the function list to create a new list; the arguments passed to function list are the elements of the created list:

* (list 1 2 3 'cat "dog") (1 2 3 CAT "dog") *

The function car returns the first element of a list and the function cdr returns a list with its first element removed (but does not modify its argument):

* (car l1) 1 * (cdr l1) (2 (3) 4 (5 6)) *

Using combinations of car and cdr calls can be used to extract any element of a list:

* (car (cdr l1)) 2 * (cadr l1) 2 *

Notice that we can combine calls to car and cdr into a single function call, in this case the function cadr. Common Lisp defines all functions of the form cXXr, cXXXr, and cXXXXr where X can be either a or d.

Suppose that we want to extract the value 5 from the nested list l1. Some experimentation with using combinations of car and cdr gets the job done:

* l1 (1 2 (3) 4 (5 6)) * (cadr l1) 2 * (caddr l1) (3) (car (caddr l1)) 3 * (caar (last l1)) 5 * (caar (cddddr l1)) 5 *

The function last returns the last cdr of a list (i.e., the last element, in a list):

* (last l1) ((5 6)) *

Common list supplies alternative functions to car and cdr that you might find more readable: first, second, third, fourth, and rest. Here are some examples:

* (defvar *x* '(1 2 3 4 5)) *X* * (first *x*) 1 * (rest *x*) (2 3 4 5) * (second *x*) 2 * (third *x*) 3 * (fourth *x*) 4

The function nth takes two arguments: an index of a top-level list element and a list. The first index argument is zero based:

* l1 (1 2 (3) 4 (5 6)) * (nth 0 l1) 1 * (nth 1 l1) 2 * (nth 2 l1) (3) *

The function cons adds an element to the beginning of a list and returns as its value a new list (it does not modify its arguments). An element added to the beginning of a list can be any Lisp data type, including another list:

* (cons 'first l1) (FIRST 1 2 (3) 4 (5 6)) * (cons '(1 2 3) '(11 22 33)) ((1 2 3) 11 22 33) *

The function append takes two lists as arguments and returns as its value the two lists appended together:

* l1 (1 2 (3) 4 (5 6)) * l2 ('THE "dog" 'CALCULATED 3.14159) * (append l1 l2) (1 2 (3) 4 (5 6) THE "dog" CALCULATED 3.14159) * (append '(first) l1) (FIRST 1 2 (3) 4 (5 6)) *

A frequent error that beginning Lisp programmers make is not understanding shared structures in lists. Consider the following example where we generate a list y by reusing three copies of the list x:

* (setq x '(0 0 0 0)) (0 0 0 0) * (setq y (list x x x)) ((0 0 0 0) (0 0 0 0) (0 0 0 0)) * (setf (nth 2 (nth 1 y)) 'x) X * x (0 0 X 0) * y ((0 0 X 0) (0 0 X 0) (0 0 X 0)) * (setq z '((0 0 0 0) (0 0 0 0) (0 0 0 0))) ((0 0 0 0) (0 0 0 0) (0 0 0 0)) * (setf (nth 2 (nth 1 z)) 'x) X * z ((0 0 0 0) (0 0 X 0) (0 0 0 0)) *

When we change the shared structure referenced by the variable x that change is reflected three times in the list y. When we create the list stored in the variable z we are not using a shared structure.

Using Arrays and Vectors

Using lists is easy but the time spent accessing a list element is proportional to the length of the list. Arrays and vectors are more efficient at runtime than long lists because list elements are kept on a linked-list that must be searched. Accessing any element of a short list is fast, but for sequences with thousands of elements, it is faster to use vectors and arrays.

By default, elements of arrays and vectors can be any Lisp data type. There are options when creating arrays to tell the Common Lisp compiler that a given array or vector will only contain a single data type (e.g., floating point numbers) but we will not use these options in this book.

Vectors are a specialization of arrays; vectors are arrays that only have one dimension. For efficiency, there are functions that only operate on vectors, but since array functions also work on vectors, we will concentrate on arrays. In the next section, we will look at character strings that are a specialization of vectors.

We could use the generalized make-sequence function to make a singularly dimensioned array (i.e., a vector). Restart sbcl and try:

* (defvar x (make-sequence 'vector 5 :initial-element 0)) X * x #(0 0 0 0 0) *

In this example, notice the print format for vectors that looks like a list with a proceeding # character. As seen in the last section, we use the function make-array to create arrays:

* (defvar y (make-array '(2 3) :initial-element 1)) Y * y #2A((1 1 1) (1 1 1)) >

Notice the print format of an array: it looks like a list proceeded by a # character and the integer number of dimensions.

Instead of using make-sequence to create vectors, we can pass an integer as the first argument of make-array instead of a list of dimension values. We can also create a vector by using the function vector and providing the vector contents as arguments:

* (make-array 10) #(NIL NIL NIL NIL NIL NIL NIL NIL NIL NIL) * (vector 1 2 3 'cat) #(1 2 3 CAT) *

The function aref is used to access sequence elements. The first argument is an array and the remaining argument(s) are array indices. For example:

* x #(0 0 0 0 0) * (aref x 2) 0 * (setf (aref x 2) "parrot") "parrot" * x #(0 0 "parrot" 0 0) * (aref x 2) "parrot" * y #2A((1 1 1) (1 1 1)) * (setf (aref y 1 2) 3.14159) 3.14159 * y #2A((1 1 1) (1 1 3.14159)) *

Using Strings

It is likely that even your first Lisp programs will involve the use of character strings. In this section, we will cover the basics: creating strings, concatenating strings to create new strings, for substrings in a string, and extracting substrings from longer strings. The string functions that we will look at here do not modify their arguments; rather, they return new strings as values. For efficiency, Common Lisp does include destructive string functions that do modify their arguments but we will not discuss these destructive functions here.

We saw earlier that a string is a type of vector, which in turn is a type of array (which in turn is a type of sequence). A full coverage of the Common Lisp type system is outside the scope of this tutorial introduction to Common Lisp; a very good treatment of Common Lisp types is in Guy Steele’s “Common Lisp, The Language” which is available both in print and for free on the web. Many of the built in functions for handling strings are actually more general because they are defined for the type sequence. The Common Lisp Hyperspec is another great free resource that you can find on the web. I suggest that you download an HTML version of Guy Steele’s excellent reference book and the Common Lisp Hyperspec and keep both on your computer. If you continue using Common Lisp, eventually you will want to read all of Steele’s book and use the Hyperspec for reference.

The following text was captured from input and output from a Common Lisp repl. First, we will declare two global variables s1 and space that contain string values:

* (defvar s1 "the cat ran up the tree") S1 * (defvar space " ") SPACE *

One of the most common operations on strings is to concatenate two or more strings into a new string:

* (concatenate 'string s1 space "up the tree") "the cat ran up the tree up the tree" *

Notice that the first argument of the function concatenate is the type of the sequence that the function should return; in this case, we want a string. Another common string operation is search for a substring:

* (search "ran" s1) 8 * (search "zzzz" s1) NIL *

If the search string (first argument to function search) is not found, function search returns nil, otherwise search returns an index into the second argument string. Function search takes several optional keyword arguments (see the next chapter for a discussion of keyword arguments):

( search search-string a-longer-string :from-end :test :test-not :key :start1 :start2 :end1 :end2 )

For our discussion, we will just use the keyword argument :start2 for specifying the starting search index in the second argument string and the :from-end flag to specify that search should start at the end of the second argument string and proceed backwards to the beginning of the string:

* (search " " s1) 3 * (search " " s1 :start2 5) 7 * (search " " s1 :from-end t) 18 *

The sequence function subseq can be used for strings to extract a substring from a longer string:

* (subseq s1 8) "ran up the tree" >

Here, the second argument specifies the starting index; the substring from the starting index to the end of the string is returned. An optional third index argument specifies one greater than the last character index that you want to extract:

* (subseq s1 8 11) "ran" *

It is frequently useful to remove white space (or other) characters from the beginning or end of a string:

* (string-trim '(#\space #\z #\a) " a boy said pez") "boy said pe" *

The character #\space is the space character. Other common characters that are trimmed are #\tab and #

ewline. There are also utility functions for making strings upper or lower case:

* (string-upcase "The dog bit the cat.") "THE DOG BIT THE CAT." * (string-downcase "The boy said WOW!") "the boy said wow!" >

We have not yet discussed equality of variables. The function eq returns true if two variables refer to the same data in memory. The function eql returns true if the arguments refer to the same data in memory or if they are equal numbers or characters. The function equal is more lenient: it returns true if two variables print the same when evaluated. More formally, function equal returns true if the car and cdr recursively equal to each other. An example will make this clearer:

* (defvar x '(1 2 3)) X * (defvar y '(1 2 3)) Y * (eql x y) NIL * (equal x y) T * x (1 2 3) * y (1 2 3) *

For strings, the function string= is slightly more efficient than using the function equal:

* (eql "cat" "cat") NIL * (equal "cat" "cat") T * (string= "cat" "cat") T *

Common Lisp strings are sequences of characters. The function char is used to extract individual characters from a string:

* s1 "the cat ran up the tree" * (char s1 0) #\t * (char s1 1) #\h *

Using Hash Tables

Hash tables are an extremely useful data type. While it is true that you can get the same effect by using lists and the assoc function, hash tables are much more efficient than lists if the lists contain many elements. For example:

* (defvar x '((1 2) ("animal" "dog"))) X * (assoc 1 x) (1 2) * (assoc "animal" x) NIL * (assoc "animal" x :test #'equal) ("animal" "dog") *

The second argument to function assoc is a list of cons cells. Function assoc searches for a sub-list (in the second argument) that has its car (i.e., first element) equal to the first argument to function assoc. The perhaps surprising thing about this example is that assoc seems to work with an integer as the first argument but not with a string. The reason for this is that by default the test for equality is done with eql that tests two variables to see if they refer to the same memory location or if they are identical if they are numbers. In the last call to assoc we used “:test #’equal” to make assoc use the function equal to test for equality.

The problem with using lists and assoc is that they are very inefficient for large lists. We will see that it is no more difficult to code with hash tables.

A hash table stores associations between key and value pairs, much like our last example using the assoc function. By default, hash tables use eql to test for equality when looking for a key match. We will duplicate the previous example using hash tables:

* (defvar h (make-hash-table)) H * (setf (gethash 1 h) 2) 2 * (setf (gethash "animal" h) "dog") "dog" * (gethash 1 h) 2 ; T * (gethash "animal" h) NIL ; NIL *

Notice that gethash returns multiple values: the first value is the value matching the key passed as the first argument to function gethash and the second returned value is true if the key was found and nil otherwise. The second returned value could be useful if hash values are nil.

Since we have not yet seen how to handle multiple returned values from a function, we will digress and do so here (there are many ways to handle multiple return values and we are just covering one of them):

* (multiple-value-setq (a b) (gethash 1 h)) 2 * a 2 * b T *

Assuming that variables a and b are already declared, the variable a will be set to the first returned value from gethash and the variable b will be set to the second returned value.

If we use symbols as hash table keys, then using eql for testing for equality with hash table keys is fine:

* (setf (gethash 'bb h) 'aa) AA * (gethash 'bb h) AA ; T *

However, we saw that eql will not match keys with character string values. The function make-hash-table has optional key arguments and one of them will allow us to use strings as hash key values:

(make-hash-table &key :test :size :rehash-size :rehash-threshold)

Here, we are only interested in the first optional key argument :test that allows us to use the function equal to test for equality when matching hash table keys. For example:

* (defvar h2 (make-hash-table :test #'equal)) H2 * (setf (gethash "animal" h2) "dog") "dog" * (setf (gethash "parrot" h2) "Brady") "Brady" * (gethash "parrot" h2) "Brady" ; T *

It is often useful to be able to enumerate all the key and value pairs in a hash table. Here is a simple example of doing this by first defining a function my-print that takes two arguments, a key and a value. We can then use the maphash function to call our new function my-print with every key and value pair in a hash table:

* (defun my-print (a-key a-value) (format t "key: ~A value: ~A~\%" a-key a-value)) MY-PRINT * (maphash #'my-print h2) key: parrot value: Brady key: animal value: dog NIL *

The function my-print is applied to each key/value pair in the hash table. There are a few other useful hash table functions that we demonstrate here:

* (hash-table-count h2) 2 * (remhash "animal" h2) T * (hash-table-count h2) 1 * (clrhash h2) #S(HASH-TABLE EQUAL) * (hash-table-count h2) 0 *

The function hash-table-count returns the number of key and value pairs in a hash table. The function remhash can be used to remove a single key and value pair from a hash table. The function clrhash clears out a hash table by removing all key and value pairs in a hash table.

It is interesting to note that clrhash and remhash are the first Common Lisp functions that we have seen so far that modify any of its arguments, except for setq and setf that are macros and not functions.

Using Eval to Evaluate Lisp Forms

We have seen how we can type arbitrary Lisp expressions in the Lisp repl listener and then they are evaluated. We will see in the Chapter on Input and Output that the Lisp function read evaluates lists (or forms) and indeed the Lisp repl uses function read.

In this section, we will use the function eval to evaluate arbitrary Lisp expressions inside a program. As a simple example:

* (defvar x '(+ 1 2 3 4 5)) X * x (+ 1 2 3 4 5) * (eval x) 15 *

Using the function eval, we can build lists containing Lisp code and evaluate generated code inside our own programs. We get the effect of “data is code”. A classic Lisp program, the OPS5 expert system tool, stored snippets of Lisp code in a network data structure and used the function eval to execute Lisp code stored in the network. A warning: the use of eval is likely to be inefficient in non-compiled code. For efficiency, the OPS5 program contained its own version of eval that only interpreted a subset of Lisp used in the network.

Using a Text Editor to Edit Lisp Source Files

I usually use Emacs, but we will briefly discuss the editor vi also. If you use vi (e.g., enter “vi nested.lisp”) the first thing that you should do is to configure vi to indicate matching opening parentheses whenever a closing parentheses is typed; you do this by typing “:set sm” after vi is running.

If you choose to learn Emacs, enter the following in your .emacs file (or your _emacs file in your home directory if you are running Windows):

1 ( set-default 'auto-mode-alist 2 ( append ' (( "\\.lisp$" . lisp-mode ) 3 ( "\\.lsp$" . lisp-mode ) 4 ( "\\.cl$" . lisp-mode )) 5 auto-mode-alist ))

Now, whenever you open a file with the extension of “lisp”, “lsp”, or “cl” (for “Common Lisp”) then Emacs will automatically use a Lisp editing mode. I recommend searching the web using keywords “Emacs tutorial” to learn how to use the basic Emacs editing commands - we will not repeat this information here.

I do my professional Lisp programming using free software tools: Emacs, SBCL, Clozure Common Lisp, and Clojure. I will show you how to configure Emacs and Slime in the last section of the Chapter on Quicklisp.

Recovering from Errors

When you enter forms (or expressions) in a Lisp repl listener, you will occasionally make a mistake and an error will be thrown. Here is an example where I am not showing all of the output when entering help when an error is thrown:

* (defun my-add-one (x) (+ x 1)) MY-ADD-ONE * (my-add-one 10) 11 * (my-add-one 3.14159) 4.14159 * (my-add-one "cat") debugger invoked on a SIMPLE-TYPE-ERROR: Argument X is not a NUMBER: "cat" Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [ABORT] Exit debugger, returning to top level. (SB-KERNEL:TWO-ARG-+ "cat" 1) 0] help The debug prompt is square brackets, with number(s) indicating the current control stack level and, if you've entered the debugger recursively, how deeply recursed you are. ... Getting in and out of the debugger: TOPLEVEL, TOP exits debugger and returns to top level REPL RESTART invokes restart numbered as shown (prompt if not given). ERROR prints the error condition and restart cases. ... Inspecting frames: BACKTRACE [n] shows n frames going down the stack. LIST-LOCALS, L lists locals in current frame. PRINT, P displays function call for current frame. SOURCE [n] displays frame's source form with n levels of enclosing forms. Stepping: START Selects the CONTINUE restart if one exists and starts single-stepping. Single stepping affects only code compiled with under high DEBUG optimization quality. See User Manual for details. STEP Steps into the current form. NEXT Steps over the current form. OUT Stops stepping temporarily, but resumes it when the topmost frame that was stepped into returns. STOP Stops single-stepping. ... 0] list-locals SB-DEBUG::ARG-0 = "cat" SB-DEBUG::ARG-1 = 1 0] backtrace 2 Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1002AC32F3}> 0: (SB-KERNEL:TWO-ARG-+ "cat" 1) 1: (MY-ADD-ONE "cat") 0] :0 *

Here, I first used the backtrace command :bt to print the sequence of function calls that caused the error. If it is obvious where the error is in the code that I am working on then I do not bother using the backtrace command. I then used the abort command :a to recover back to the top level Lisp listener (i.e., back to the greater than prompt). Sometimes, you must type :a more than once to fully recover to the top level greater than prompt.

Garbage Collection

Like other languages like Java and Python, Common Lisp provides garbage collection (GC) or automatic memory management.

In simple terms, GC occurs to free memory in a Lisp environment that is no longer accessible by any global variable (or function closure, which we will cover in the next chapter). If a global variable *variable-1* is first set to a list and then if we later then set *variable-1* to, for example nil, and if the data referenced in the original list is not referenced by any other accessible data, then this now unused data is subject to GC.

In practice, memory for Lisp data is allocated in time ordered batches and ephemeral or generational garbage collectors garbage collect recent memory allocations far more often than memory that has been allocated for a longer period of time.

Loading your Working Environment Quickly

When you start using Common Lisp for large projects, you will likely have many files to load into your Lisp environment when you start working. Most Common Lisp implementations have a function called defsystem that works somewhat like the Unix make utility. While I strongly recommend defsystem for large multi-person projects, I usually use a simpler scheme when working on my own: I place a file loadit.lisp in the top directory of each project that I work on. For any project, its loadit.lisp file loads all source files and initializes any global data for the project.

The last two chapters of this book provide example applications that are configured to work with Quicklisp, which we will study in the next chapter.

Another good technique is to create a Lisp image containing all the code and data for all your projects. There is an example of this in the first section of the Chapter on NLP. In this example, it takes a few minutes to load the code and data for my NLP (natural language processing) library so when I am working with it I like to be able to quickly load a SBCL Lisp image.

All Common Lisp implementations have a mechanism for dumping a working image containing code and data.

Functional Programming Concepts

There are two main styles for doing Common Lisp development. Object oriented programming is well supported (see the Chapter on CLOS) as is functional programming. In a nut shell, functional programming means that we should write functions with no side effects. First let me give you a non-functional example with side effects:

( defun non-functional-example ( car ) ( set-color car "red" ))

This example using CLOS is non-functional because we modify the value of an argument to the function. Some functional languages like the Lisp Clojure language and the Haskell language dissuade you from modifying arguments to functions. With Common Lisp you should make a decision on which approach you like to use.

Functional programming means that we avoid maintaining state inside of functions and treat data as immutable (i.e., once an object is created, it is never modified). We could modify the last example to be function by creating a new car object inside the function, copy the attributes of the car passed as an object, change the color to “red” of the new car object, and return the new car instance as the value of the function.

Functional programming prevents many types of programming errors, makes unit testing simpler, and makes programming for modern multi-core CPUs easier because read-only objects are inherently thread safe. Modern best practices for the Java language also prefer immutable data objects and a functional approach.

Quicklisp

For several decades managing packages and libraries was a manual process when developing Lisp systems. I used to package the source code for specific versions of libraries as part of my Common Lisp projects. Early package management systems mk-defsystem and ASDF were very useful, but I did not totally give up my practice keeping third party library source code with my projects until Zach Beane created the Quicklisp package system. You will need to have Quicklisp installed for many of the examples later in this book so please take the time to install it now as per the instructions on the Quicklisp web site.

Using Quicklisp to Find Packages

We will need the Common Lisp Hunchentoot library later in the Chapter on Network Programming so we will install it now using Quicklisp as an example for getting started with Quicklisp.

We already know the package name we want, but as an example of discovering packages let’s start by using Quicklisp to search for all packages with “hunchentoot” in the package name:

1 * (ql:system-apropos "hunchentoot") 2 #<SYSTEM clack-handler-hunchentoot / clack-20131111-git / quicklisp 2013-11-11> 3 #<SYSTEM hunchentoot / hunchentoot-1.2.21 / quicklisp 2013-11-11> 4 #<SYSTEM hunchentoot-auth / hunchentoot-auth-20101107-git / quicklisp 2013-11-11> 5 #<SYSTEM hunchentoot-cgi / hunchentoot-cgi-20121125-git / quicklisp 2013-11-11> 6 #<SYSTEM hunchentoot-dev / hunchentoot-1.2.21 / quicklisp 2013-11-11> 7 #<SYSTEM hunchentoot-single-signon / hunchentoot-single-signon-20131111-git / quickl\ 8 isp 2013-11-11> 9 #<SYSTEM hunchentoot-test / hunchentoot-1.2.21 / quicklisp 2013-11-11> 10 #<SYSTEM hunchentoot-vhost / hunchentoot-vhost-20110418-git / quicklisp 2013-11-11>

We want the base package seen in line 3 and we can install the base package as seen in the following example:

1 * (ql:quickload :hunchentoot) 2 To load "hunchentoot": 3 Load 1 ASDF system: 4 hunchentoot 5 ; Loading "hunchentoot" 6 ....... 7 (:HUNCHENTOOT)

In line 1, I refer to the package name using a symbol :hunchentoot but using the string “hunchentoot” would have worked the same. The first time you ql:quickload a library you may see additional printout and it takes longer to load because the source code is downloaded from the web and cached locally in the directory ~/quicklisp/local-projects. In most of the rest of this book, when I install or use a package by calling the ql:quickload function I do not show the output from this function in the repl listings.

Now, we can use the fantastically useful Common Lisp function apropos to see what was just installed:

1 * (apropos "hunchentoot") 2 3 HUNCHENTOOT::*CLOSE-HUNCHENTOOT-STREAM* (bound) 4 HUNCHENTOOT:*HUNCHENTOOT-DEFAULT-EXTERNAL-FORMAT* (bound) 5 HUNCHENTOOT::*HUNCHENTOOT-STREAM* 6 HUNCHENTOOT:*HUNCHENTOOT-VERSION* (bound) 7 HUNCHENTOOT:HUNCHENTOOT-CONDITION 8 HUNCHENTOOT:HUNCHENTOOT-ERROR (fbound) 9 HUNCHENTOOT::HUNCHENTOOT-OPERATION-NOT-IMPLEMENTED-OPERATION (fbound) 10 HUNCHENTOOT::HUNCHENTOOT-SIMPLE-ERROR 11 HUNCHENTOOT::HUNCHENTOOT-SIMPLE-WARNING 12 HUNCHENTOOT::HUNCHENTOOT-WARN (fbound) 13 HUNCHENTOOT:HUNCHENTOOT-WARNING 14 HUNCHENTOOT-ASD:*HUNCHENTOOT-VERSION* (bound) 15 HUNCHENTOOT-ASD::HUNCHENTOOT 16 :HUNCHENTOOT (bound) 17 :HUNCHENTOOT-ASD (bound) 18 :HUNCHENTOOT-DEV (bound) 19 :HUNCHENTOOT-NO-SSL (bound) 20 :HUNCHENTOOT-TEST (bound) 21 :HUNCHENTOOT-VERSION (bound) 22 *

As long as you are thinking about the new tool Quicklisp that is now in your tool chest, you should install most of the packages and libraries that you will need for working through the rest of this book. I will show the statements needed to load more libraries without showing the output printed in the repl as each package is loaded:

1 ( ql : quickload "clsql" ) 2 ( ql : quickload "clsql-postgresql" ) 3 ( ql : quickload "clsql-mysql" ) 4 ( ql : quickload "clsql-sqlite3" ) 5 ( ql : quickload : drakma ) 6 ( ql : quickload : hunchentoot ) 7 ( ql : quickload : cl-json ) 8 ( ql : quickload "clouchdb" ) ;; for CouchDB access 9 ( ql : quickload "sqlite" )

You need to have the Postgres and MySQL client developer libraries installed on your system for the clsql-postgresql and clsql-mysql installations to work. If you are unlikely to use relational databases with Common Lisp then you might skip the effort of installing Postgres and MySQL. The example in the Chapter on the Knowledge Graph Navigator uses the SQLite database for caching. You don’t need any extra dependencies for the sqlite package.

Using Quicklisp to Configure Emacs and Slime

I assume that you have Emacs installed on your system. In a repl you can setup the Slime package that allows Emacs to connect to a running Lisp environment:

( ql:quickload "quicklisp-slime-helper" )

Pay attention to the output in the repl. On my system the output contained the following:

1 [package quicklisp-slime-helper] 2 slime-helper.el installed in "/Users/markw/quicklisp/slime-helper.el" 3 4 To use, add this to your ~/.emacs: 5 6 (load (expand-file-name "~/quicklisp/slime-helper.el")) 7 ;; Replace "sbcl" with the path to your implementation 8 (setq inferior-lisp-program "sbcl")

If you installed rlwrap and defined an alias for running SBCL, make sure you set the inferior lisp program to the absolute path of the SBCL executable; on my system I set the following in my .emacs file:

1 (setq inferior-lisp-program "/Users/markw/sbcl/sbcl")

I am not going to cover using Emacs and Slime, there are many good tutorials on the web you can read.

In later chapters we will write libraries and applications as Quicklisp projects so that you will be able to load your own libraries, making it easier to write small libraries that you can compose into larger applications.

Defining Lisp Functions

In the previous chapter, we defined a few simple functions. In this chapter, we will discuss how to write functions that take a variable number of arguments, optional arguments, and keyword arguments.

The special form defun is used to define new functions either in Lisp source files or at the top level Lisp listener prompt. Usually, it is most convenient to place function definitions in a source file and use the function load to load them into our Lisp working environment.

In general, it is bad form to use global variables inside Lisp functions. Rather, we prefer to pass all required data into a function via its argument list and to get the results of the function as the value (or values) returned from a function. Note that if we do require global variables, it is customary to name them with beginning and ending * characters; for example:

1 ( defvar *lexical-hash-table* 2 ( make-hash-table :test #' equal :size 5000 ))

Then in this example, if you see the variable *lexical-hash-table* inside a function definition, you will know that at least by naming convention, that this is a global variable.

In Chapter 1, we saw an example of using lexically scoped local variables inside a function definition (in the example file nested.lisp).

There are several options for defining the arguments that a function can take. The fastest way to introduce the various options is with a few examples.

First, we can use the &aux keyword to declare local variables for use in a function definition:

1 * (defun test (x &aux y) 2 (setq y (list x x)) 3 y) 4 TEST 5 * (test 'cat) 6 (CAT CAT) 7 * (test 3.14159) 8 (3.14159 3.14159)

It is considered better coding style to use the let special operator for defining auxiliary local variables; for example:

1 * (defun test (x) 2 (let ((y (list x x))) 3 y)) 4 TEST 5 * (test "the dog bit the cat") 6 ("the dog bit the cat" "the dog bit the cat") 7 *

You will probably not use &aux very often, but there are two other options for specifying function arguments: &optional and &key.

The following code example shows how to use optional function arguments. Note that optional arguments must occur after required arguments.

1 * (defun test (a &optional b (c 123)) 2 (format t "a=~A b=~A c=~A~%" a b c)) 3 TEST 4 * (test 1) 5 a=1 b=NIL c=123 6 NIL 7 * (test 1 2) 8 a=1 b=2 c=123 9 NIL 10 * (test 1 2 3) 11 a=1 b=2 c=3 12 NIL 13 * (test 1 2 "Italian Greyhound") 14 a=1 b=2 c=Italian Greyhound 15 NIL 16 *

In this example, the optional argument b was not given a default value so if unspecified it will default to nil. The optional argument c is given a default value of 123.

We have already seen the use of keyword arguments in built-in Lisp functions. Here is an example of how to specify key word arguments in your functions:

1 * (defun test (a &key b c) 2 (format t "a=~A b=~A c=~A~%" a b c)) 3 TEST 4 * (test 1) 5 a=1 b=NIL c=NIL 6 NIL 7 * (test 1 :c 3.14159) 8 a=1 b=NIL c=3.14159 9 NIL 10 * (test "cat" :b "dog") 11 a=cat b=dog c=NIL 12 NIL 13 *

Using Lambda Forms

It is often useful to define unnamed functions. We can define an unnamed function using lambda; for example, let’s look at the example file src/lambda1.lisp. But first, we will introduce the Common Lisp function funcall that takes one or more arguments; the first argument is a function and any remaining arguments are passed to the function bound to the first argument. For example:

1 * (funcall 'print 'cat) 2 CAT 3 CAT 4 * (funcall '+ 1 2) 5 3 6 * (funcall #'- 2 3) 7 -1 8 *

In the first two calls to funcall here, we simply quote the function name that we want to call. In the third example, we use a better notation by quoting with #’. We use the #’ characters to quote a function name.

Consider the following repl listing where we will look at a primary difference between quoting a symbol using ‘ and with #’:

1 $ ccl 2 Clozure Common Lisp Version 1.12 DarwinX8664 3 ? 'barfoo531 4 BARFOO531 5 ? ( apropos "barfoo" ) 6 BARFOO531 7 ? #' bar987 8 > Error: Undefined function: BAR987

On line three we create a new symbol BARFOO531 that is interned as you can see from looking at all interned symbols containing the string “barfoo”. Line 7 throws an error because #’ does not intern a new symbol.

Here is the example file src/lambda1.lisp:

1 ( defun test () 2 ( let (( my-func 3 ( lambda ( x ) ( + x 1 )))) 4 ( funcall my-func 1 )))

Here, we define a function using lambda and set the value of the local variable my-func to the unnamed function’s value. Here is output from the function test:

1 * (test) 2 2 3 4 *

The ability to use functions as data is surprisingly useful. For now, we will look at a simple example:

1 * (defvar f1 #'(lambda (x) (+ x 1))) 2 3 F1 4 * (funcall f1 100) 5 6 101 7 * (funcall #'print 100) 8 9 100 10 100

Notice that the second call to function testfn prints “100” twice: the first time as a side effect of calling the function print and the second time as the returned value of testfn (the function print returns what it is printing as its value).

Using Recursion

Later, we will see how to use special Common Lisp macros for programming repetitive loops. In this section, we will use recursion for both coding simple loops and as an effective way to solve a variety of problems that can be expressed naturally using recursion.

As usual, the example programs for this section are found in the src directory. In the file src/recursion1.lisp, we see our first example of recursion:

1 ;; a simple loop using recursion 2 3 ( defun recursion1 ( value ) 4 ( format t "entering recursion1(~A)~\%" value ) 5 ( if ( < value 5 ) 6 ( recursion1 ( 1+ value ))))

This example is simple, but it is useful for discussing a few points. First, notice how the function recursion1 calls itself with an argument value of one greater than its own input argument only if the input argument “value” is less than 5. This test keeps the function from getting in an infinite loop. Here is some sample output:

1 * (load "recursion1.lisp") 2 ;; Loading file recursion1.lisp ... 3 ;; Loading of file recursion1.lisp is finished. 4 T 5 * (recursion1 0) 6 entering recursion1(0) 7 entering recursion1(1) 8 entering recursion1(2) 9 entering recursion1(3) 10 entering recursion1(4) 11 entering recursion1(5) 12 NIL 13 * (recursion1 -3) 14 entering recursion1(-3) 15 entering recursion1(-2) 16 entering recursion1(-1) 17 entering recursion1(0) 18 entering recursion1(1) 19 entering recursion1(2) 20 entering recursion1(3) 21 entering recursion1(4) 22 entering recursion1(5) 23 NIL 24 * (recursion1 20) 25 entering recursion1(20) 26 NIL 27 *

Why did the call on line 24 not loop via recursion? Because the input argument is not less than 5, no recursion occurs.

Closures

We have seen that functions can take other functions as arguments and return new functions as values. A function that references an outer lexically scoped variable is called a closure. The example file src/closure1.lisp contains a simple example:

1 ( let* (( fortunes 2 ' ( "You will become a great Lisp Programmer" 3 "The force will not be with you" 4 "Take time for meditation" )) 5 ( len ( length fortunes )) 6 ( index 0 )) 7 ( defun fortune () 8 ( let (( new-fortune ( nth index fortunes ))) 9 ( setq index ( 1+ index )) 10 ( if ( >= index len ) ( setq index 0 )) 11 new-fortune )))

Here the function fortune is defined inside a let form. Because the local variable fortunes is referenced inside the function fortune, the variable fortunes exists after the let form is evaluated. It is important to understand that usually a local variable defined inside a let form “goes out of scope” and can no longer be referenced after the let form is evaluated.

However, in this example, there is no way to access the contents of the variable fortunes except by calling the function fortune. At a minimum, closures are a great way to hide variables. Here is some output from loading the src/closure1.lisp file and calling the function fortune several times:

1 * (load "closure1.lisp") 2 ;; Loading file closure1.lisp ... 3 ;; Loading of file closure1.lisp is finished. 4 T 5 * (fortune) 6 "You will become a great Lisp Programmer" 7 * (fortune) 8 "The force will not be with you" 9 * (fortune) 10 "Take time for meditation" 11 * (fortune) 12 "You will become a great Lisp Programmer" 13 *

Using the Function eval

In Lisp languages we often say that code is data. The function eval can be used to execute code that is stored as Lisp data. Let’s look at an example:

1 $ ccl 2 Clozure Common Lisp Version 1.12 DarwinX8664 3 ? ' ( + 1 2.2 ) 4 ( + 1 2.2 ) 5 ? ( eval ' ( + 1 2.2 )) 6 3.2 7 ? ( eval ' ( defun foo2 ( x ) ( + x x ))) 8 FOO2 9 ? ( foo2 4 ) 10 8

I leave it up to you, dear reader, how often you are motivated to use eval. In forty years of using Lisp languages my principle use of eval has been in modifying the standard version of the Ops5 programming language for production systems to support things like multiple data worlds and new actions to spawn off new data worlds and to remove them. Ops5 works by finding common expressions in a set of production rules (also referred to as “expert systems”) and factoring them into a network (a Rete network if you want to look it up) with common expressions in rules stored in just a single place. eval is used a lot in Ops5 and I used it for my extensions to Ops5.

Defining Common Lisp Macros

We saw in the last chapter how the Lisp function eval could be used to evaluate arbitrary Lisp code stored in lists. Because eval is inefficient, a better way to generate Lisp code automatically is to define macro expressions that are expanded inline when they are used. In most Common Lisp systems, using eval requires the Lisp compiler to compile a form on-the-fly which is not very efficient. Some Lisp implementations use an interpreter for eval which is likely to be faster but might lead to obscure bugs if the interpreter and compiled code do not function identically.

The ability to add functionality and syntax to the Common Lisp language, to in effect extend the language as needed, is truly a super power of languages like Common Lisp and Scheme.

Example Macro

The file src/macro1.lisp contains both a simple macro and a function that uses the macro. This macro example is a bit contrived since it could be just a function definition, but it does show the process of creating and using a macro. We are using the gensym function to define a new unique symbol to reference a temporary variable:

1 ;; first simple macro example: 2 3 ( defmacro double-list ( a-list ) 4 ( let (( ret ( gensym ))) 5 ` ( let (( , ret nil )) 6 ( dolist ( x , a-list ) 7 ( setq , ret ( append , ret ( list x x )))) 8 , ret ))) 9 10 ;; use the macro: 11 12 ( defun test ( x ) 13 ( double-list x ))

The backquote character seen at the beginning of line 5 is used to quote a list in a special way: nothing in the list is evaluated during macro expansion unless it is immediately preceded by a comma character. In this case, we specify ,a-list because we want the value of the macro’s argument a-list to be substituted into the specially quoted list. We will look at dolist in some detail in the next chapter but for now it is sufficient to understand that dolist is used to iterate through the top-level elements of a list, for example:

1 * (dolist (x '("the" "cat" "bit" "the" "rat")) 2 (print x)) 3 "the" 4 "cat" 5 "bit" 6 "the" 7 "rat" 8 NIL 9 *

Notice that the example macro double-list itself uses the macro dolist. It is common to next macros in the same way functions can be nested.

Returning to our macro example in the file src/macro1.lisp, we will try the function test that uses the macro double-list:

1 * (load "macro1.lisp") 2 ;; Loading file macro1.lisp ... 3 ;; Loading of file macro1.lisp is finished. 4 T 5 * (test '(1 2 3)) 6 (1 1 2 2 3 3) 7 *

Using the Splicing Operator

Another similar example is in the file src/macro2.lisp:

1 ;; another macro example that uses ,@: 2 3 ( defmacro double-args ( &rest args ) 4 ` ( let (( ret nil )) 5 ( dolist ( x ,@ args ) 6 ( setq ret ( append ret ( list x x )))) 7 ret )) 8 9 ;; use the macro: 10 11 ( defun test ( &rest x ) 12 ( double-args x ))

Here, the splicing operator ,@ is used to substitute in the list args in the macro double-args.

Using macroexpand-1

The function macroexpand-1 is used to transform macros with arguments into new Lisp expressions. For example:

1 * (defmacro double (a-number) 2 (list '+ a-number a-number)) 3 DOUBLE 4 * (macroexpand-1 '(double n)) 5 (+ N N) ; 6 T 7 *

Writing macros is an effective way to extend the Lisp language because you can control the code passed to the Common Lisp compiler. In both macro example files, when the function test was defined, the macro expansion is done before the compiler processes the code. We will see in the next chapter several useful macros included in Common Lisp.

We have only “scratched the surface” looking at macros; the interested reader is encouraged to search the web using, for example, “Common Lisp macros.” There are two books in particular that I recommend that take a deep dive into Common Lisp macros: Paul Graham’s “On Lisp” and Doug Hoyte’s “Let Over Lambda.” Both are deep books and will change the way you experience software development. A good plan of study is spending a year absorbing “On Lisp” before tackling “Let Over Lambda.”

Using Common Lisp Loop Macros

In this chapter, we will discuss several useful macros for performing iteration (we saw how to use recursion for iteration in Chapter 2):

dolist – a simple way to process the elements of a list

dotimes – a simple way to iterate with an integer valued loop variable

do – the most general looping macro

loop – a complex looping macro that I almost never use in my own code because it does not look “Lisp like.” I don’t use the loop macro in this book. Many programmers do like the loop macro so you are likely to see it when reading other people’s code.

dolist

We saw a quick example of dolist in the last chapter. The arguments of the dolist macro are:

( dolist ( a-variable a-list [optional-result-value] ) ... body... )

Usually, the dolist macro returns nil as its value, but we can add a third optional argument which will be returned as the generated expression’s value; for example:

1 * (dolist (a '(1 2) 'done) (print a)) 2 1 3 2 4 DONE 5 * (dolist (a '(1 2)) (print a)) 6 1 7 2 8 NIL 9 *

The first argument to the dolist macro is a local lexically scoped variable. Once the code generated by the dolist macro finishes executing, this variable is undefined.

dotimes

The dotimes macro is used when you need a loop with an integer loop index. The arguments of the dotimes macro are:

( dotimes ( an-index-variable max-index-plus-one [optional-result-value] ) ... body... )

Usually, the dotimes macro returns nil as its value, but we can add a third optional argument that will be returned as the generated expression’s value; for example:

1 * (dotimes (i 3 "all-done-with-test-dotimes-loop") (print i)) 2 3 0 4 1 5 2 6 "all-done-with-test-dotimes-loop" 7 *

As with the dolist macro, you will often use a let form inside a dotimes macro to declare additional temporary (lexical) variables.

do

The do macro is more general purpose than either dotimes or dolist but it is more complicated to use. Here is the general form for using the do looping macro:

( do (( variable-1 variable-1-init-value variable-1-update-expression ) ( variable-2 variable-2-init-value variable-2-update-expression ) . . ( variable-N variable-N-init-value variable-N-update-expression )) ( loop-termination-test loop-return-value ) optional-variable-declarations expressions-to-be-executed-inside-the-loop )

There is a similar macro do* that is analogous to let* in that loop variable values can depend on the values or previously declared loop variable values.

As a simple example, here is a loop to print out the integers from 0 to 3. This example is in the file src/do1.lisp:

;; example do macro use

( do (( i 0 ( 1+ i ))) (( > i 3 ) "value-of-do-loop" ) ( print i ))

In this example, we only declare one loop variable so we might as well as used the simpler dotimes macro.

Here we load the file src/do1.lisp:

1 * (load "do1.lisp") 2 ;; Loading file do1.lisp ... 3 0 4 1 5 2 6 3 7 ;; Loading of file do1.lisp is finished. 8 T 9 *

You will notice that we do not see the return value of the do loop (i.e., the string “value-of-do-loop”) because the top-level form that we are evaluating is a call to the function load; we do see the return value of load printed. If we had manually typed this example loop in the Lisp listener, then you would see the final value value-of-do-loop printed.

Using the loop Special Form to Iterate Over Vectors or Arrays

We previousely used dolist to iterate over elements in lists. For efficiency we will often use vectors (one dimensional arrays) and we can use loop to similarly handle vectors:

( loop for td across testdata do ( print td ))))

where testdata is a one dimensional array (a vector) and inside the do block the local variable td is assigned to each element in the vector.

Common Lisp Package System

In later chapters we will see two complete applications that are defined as Quicklisp projects: the chapter on the Knowledge Graph Creator and the chapter on the Knowledge Graph Navigator. Another example for setting up a Quicklib project can be seen in the chapter Plotting Data.

While these later chapters provide practical examples for bundling up your own projects in packages, the material here will give you general background information that you should know.

In the simple examples that we have seen so far, all newly created Lisp symbols have been placed in the default package. You can always check the current package by evaluating the expression package:

> *package* #<PACKAGE COMMON-LISP-USER> >

As we will use in the following example, the package :cl is an alias for :common-lisp-user.

We will define a new package :my-new-package and two functions foo1 and foo2 inside the package. Externally to this package, assuming that it is loaded, we can access foo2 using my-new-package:foo2. foo1 is not exported so it cannot be accessed this way. However, we can always start a symbol name with a package name and two colon characters if we want to use a symbol defined in another package so we can use my-new-package::foo1. Using :: allows us access to symbols not explicitly exported.

When I leave package :my-new-package in line 22 and return to package :cl, and try to access my-new-package:foo1 notice that an error is thrown.

On line 3 we define the alias :p1 for the package :my-new-package and we use this alias in line 44. The main point of the following example is that we define two functions in a package but only export one of these functions. By default the other function is not visible outside of the new package.

1 * (defpackage "MY-NEW-PACKAGE" 2 (:use :cl) 3 (:nicknames "P1") 4 (:export :FOO2)) 5 6 #<PACKAGE "MY-NEW-PACKAGE"> 7 * (in-package my-new-package) 8 9 #<PACKAGE "MY-NEW-PACKAGE"> 10 * (defun foo1 () "foo1") 11 12 FOO1 13 * (defun foo2 () "foo2") 14 15 FOO2 16 * (foo1) 17 18 "foo1" 19 * (foo2) 20 21 "foo2" 22 * (in-package :cl) 23 24 #<PACKAGE "COMMON-LISP"> 25 * (my-new-package:foo2) 26 27 "foo2" 28 * (my-new-package:foo1) 29 30 debugger invoked on a SB-INT:SIMPLE-READER-PACKAGE-ERROR in thread 31 #<THREAD "main thread" RUNNING {1001F1ECE3}>: 32 The symbol "FOO1" is not external in the MY-NEW-PACKAGE package. 33 34 Stream: #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {100001C343}> 35 36 Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL. 37 38 restarts (invokable by number or by possibly-abbreviated name): 39 0: [CONTINUE] Use symbol anyway. 40 1: [ABORT ] Exit debugger, returning to top level. 41 42 * 1 43 44 * (p1:foo2) 45 46 "foo2"

Since we specified a nickname in the defpackage expression, Common Lisp allows the use of the nickname (in this case P1) in calling function foo2 that is exported from package :my-new-package.

Near the end of the last example, we switched back to the default package COMMON-LISP-USER so we had to specify the package name for the function foo2 on line 42.

What about the error on line 28 where my-new-package:foo1 is undefined because the function foo1 is not exported (see line 4)? It turns out that you can easily use symbols not exported from a package by using :: instead of a single :. Here, this would be defined: (my-new-package::foo1).

When you are writing very large Common Lisp programs, it is useful to be able to break up the program into different modules and place each module and all its required data in different name spaces by creating new packages. Remember that all symbols, including variables, generated symbols, CLOS methods, functions, and macros are in some package.

For small packages I sometimes put a defpackage expression at the top of the file immediately followed by an in-package expression to switch to the new package. In the general case, please properly use separate project and asdf files as I do in the later chapters Knowledge Graph Creator and Knowledge Graph Navigator.

Input and Output

We will see that the input and output of Lisp data is handled using streams. Streams are powerful abstractions that support common libraries of functions for writing to the terminal, files, sockets, and to strings.

In all cases, if an input or output function is called without specifying a stream, the default for input stream is *standard-input* and the default for output stream is *standard-output*. These default streams are connected to the Lisp listener that we discussed in Chapter 2. In the later chapter Knowledge Graph Navigator that supports a user interface, we will again use output streams bound to different scrolling output areas of the application window to write color-hilighted text. The stream formalism is general purpose, covering many common I/O use cases.

The Lisp read and read-line Functions

The function read is used to read one Lisp expression. Function read stops reading after reading one expression and ignores new line characters. We will look at a simple example of reading a file test.dat using the example Lisp program in the file read-test-1.lisp. Both of these files can be found in the directory src/code_snippets_for_book that came bundled with this web book. Start your Lisp program in the src directory. The contents of the file test.dat is:

1 1 2 3 2 4 "the cat bit the rat" 3 read with-open-file

In the function read-test-1, we use the macro with-open-file to read from a file. To write to a file (which we will do later), we can use the keyword arguments :direction :output. The first argument to the macro with-open-file is a symbol that is bound to a newly created input stream (or an output stream if we are writing a file); this symbol can then be used in calling any function that expects a stream argument.

Notice that we call the function read with three arguments: an input stream, a flag to indicate if an error should be thrown if there is an I/O error (e.g., reaching the end of a file), and the third argument is the value that function read should return if the end of the file (or stream) is reached. When calling read with these three arguments, either the next expression from the file test.dat will be returned, or the value nil will be returned when the end of the file is reached. If we do reach the end of the file, the local variable x will be assigned the value nil and the function return will break out of the dotimes loop. One big advantage of using the macro with-open-file over using the open function (which we will not cover) is that the file stream is automatically closed when leaving the code generated by the with-open-file macro. The contents of file read-test-1.lisp is:

( defun read-test-1 () "read a maximum of 1000 expressions from the file 'test.dat'" ( with-open-file ( input-stream "test.dat" :direction :input ) ( dotimes ( i 1000 ) ( let (( x ( read input-stream nil nil ))) ( if ( null x ) ( return )) ;; break out of the 'dotimes' loop ( format t "next expression in file: ~S~%" x )))))

Here is the output that you will see if you load the file read-test-1.lisp and execute the expression (read-test-1):

1 * (load "read-test-1.lisp") 2 ;; Loading file read-test-1.lisp ... 3 ;; Loading of file read-test-1.lisp is finished. 4 T 5 * (read-test-1) 6 next expression in file: 1 7 next expression in file: 2 8 next expression in file: 3 9 next expression in file: 4 10 next expression in file: "the cat bit the rat" 11 NIL

Note: the string “the cat bit the rat” prints as a string (with quotes) because we used a ~S instead of a ~A in the format string in the call to function format.

In this last example, we passed the file name as a string to the macro with-open-file. This is not generally portable across all operating systems. Instead, we could have created a pathname object and passed that instead. The pathname function can take eight different keyword arguments, but we will use only the two most common in the example in the file read-test-2.lisp in the src directory. The following listing shows just the differences between this example and the last:

( let (( a-path-name ( make-pathname :directory "testdata" :name "test.dat" ))) ( with-open-file ( input-stream a-path-name :direction :input )

Here, we are specifying that we want to use the file test.dat in the subdirectory testdata. Note: I almost never use pathnames. Instead, I specify files using a string and the character / as a directory delimiter. I find this to be portable for the Macintosh, Windows, and Linux operating systems using all Common Lisp implementations.

The file readline-test.lisp is identical to the file read-test-1.lisp except that we call function readline instead of the function read and we change the output format message to indicate that an entire line of text has been read

( defun readline-test () "read a maximum of 1000 expressions from the file 'test.dat'" ( with-open-file ( input-stream "test.dat" :direction :input ) ( dotimes ( i 1000 ) ( let (( x ( read-line input-stream nil nil ))) ( if ( null x ) ( return )) ;; break out of the 'dotimes' loop ( format t "next line in file: ~S~%" x )))))

When we execute the expression (readline-test), notice that the string contained in the second line of the input file has the quote characters escaped:

1 * (load "readline-test.lisp") 2 ;; Loading file readline-test.lisp ... 3 ;; Loading of file readline-test.lisp is finished. 4 T 5 * (readline-test) 6 next line in file: "1 2 3" 7 next line in file: "4 \"the cat bit the rat\"" 8 NIL 9 *

We can also create an input stream from the contents of a string. The file read-from-string-test.lisp is very similar to the example file read-test-1.lisp except that we use the macro with-input-from-string (notice how I escaped the quote characters used inside the test string):

( defun read-from-string-test () "read a maximum of 1000 expressions from a string" ( let (( str "1 2 \"My parrot is named Brady.\" (11 22)" )) ( with-input-from-string ( input-stream str ) ( dotimes ( i 1000 ) ( let (( x ( read input-stream nil nil ))) ( if ( null x ) ( return )) ;; break out of the 'dotimes' loop ( format t "next expression in string: ~S~%" x ))))))

We see the following output when we load the file read-from-string-test.lisp:

1 * (load "read-from-string-test.lisp") 2 ;; Loading file read-from-string-test.lisp ... 3 ;; Loading of file read-from-string-test.lisp is finished. 4 T 5 * (read-from-string-test) 6 next expression in string: 1 7 next expression in string: 2 8 next expression in string: "My parrot is named Brady." 9 next expression in string: (11 22) 10 NIL 11 *

We have seen how the stream abstraction is useful for allowing the same operations on a variety of stream data. In the next section, we will see that this generality also applies to the Lisp printing functions.

Lisp Printing Functions

All of the printing functions that we will look at in this section take an optional last argument that is an output stream. The exception is the format function that can take a stream value as its first argument (or t to indicate *standard-output*, or a nil value to indicate that format should return a string value).

Here is an example of specifying the optional stream argument:

1 * (print "testing") 2 3 "testing" 4 "testing" 5 * (print "testing" *standard-output*) 6 7 "testing" 8 "testing" 9 *

The function print prints Lisp objects so that they can be read back using function read. The corresponding function princ is used to print for “human consumption”. For example:

1 * (print "testing") 2 3 "testing" 4 "testing" 5 * (princ "testing") 6 testing 7 "testing" 8 *

Both print and princ return their first argument as their return value, which you see in the previous output. Notice that princ also does not print a new line character, so princ is often used with terpri (which also takes an optional stream argument).

We have also seen many examples in this book of using the format function. Here is a different use of format, building a string by specifying the value nil for the first argument:

1 * (let ((l1 '(1 2)) 2 (x 3.14159)) 3 (format nil "~A~A" l1 x)) 4 "(1 2)3.14159" 5 *

We have not yet seen an example of writing to a file. Here, we will use the with-open-file macro with options to write a file and to delete any existing file with the same name:

( with-open-file ( out-stream "test1.dat" :direction :output :if-exists :supersede ) ( print "the cat ran down the road" out-stream ) ( format out-stream "1 + 2 is: ~A~%" ( + 1 2 )) ( princ "Stoking!!" out-stream ) ( terpri out-stream ))

Here is the result of evaluating this expression (i.e., the contents of the newly created file test1.dat in the src directory):

1 % cat test1.dat 2 3 " the cat ran down the road " 1 + 2 is : 3 4 Stoking !!

Notice that print generates a new line character before printing its argument.

Plotting Data

We will use Zach Beane’s vecto library for plotting data with the results written to files. Ideally we would like to have interactive plotting capability but for the purposes of this book I need to support the combinations of all Common Lisp implementations on multiple operating systems. Interactive plotting libraries are usually implementation and OS dependent. We