License: MIT

Yapps (Yet Another Python Parser System) is an easy to use parser generator that is written in Python and generates Python code. Although there are several parser generators already available for Python, I had different goals, including learning about recursive descent parsers[1], and exploring new features, as my gut feeling back in the 1990s that parsing was not a solved problem, as widely believed[2]. Yapps is simple, is easy to use, and produces human-readable parsers. It is not the fastest, most powerful, or most flexible parser. Yapps is designed to be used when regular expressions are not enough and other parser systems are too much: situations where you may write your own recursive descent parser. On this page you can find both Yapps 1 and Yapps 2. Yapps 1 is more like a functional language (concise grammars of the form when you see this, return this), while Yapps 2 is more like an imperative language (more verbose grammars of the form if/while you see this, do this). Both are completely free (MIT license). For a quick demonstration of how easy it is to write a Yapps grammar, take a look at the 15 line expression evaluation example, which produces code to parse and evaluate expressions like 13-(1+5)*8/7 . You may also be interested in Michael Breen’s annotated calculator[3].

Features Some unusual features of Yapps that may be of interest are: Yapps produces human-readable recursive descent parsers. There are several heuristics used to keep the generated code simple. Yapps produces context-sensitive scanners that pick tokens based on the type of tokens accepted by the parser. In some situations, token matching is ambiguous unless the context is taken into account. Yapps rules can pass arguments down to subrules, so subrules can use information (such as declarations) that was parsed at higher levels in the parsing process. These are sometimes called attribute grammars. There are several disadvantages of using Yapps over another parser system: Yapps parsers are LL(1), which is less powerful in parsing power than LALR or SLR. There are some inconveniences having to do with LL(1) (such as the lack of operator precedence) that lead to more explicit grammars. The Yapps scanner can only read from strings, not from files, so it may not be useful if your input is large. However, it is possible to write a custom scanner for your application. (Note: the enhanced version of Yapps in Debian can read from files.) Yapps is not designed with efficiency in mind. It’s not the fastest parser available (nor is it the slowest). The Python.org site has a paper comparing Yapps to Spark and BisonGen[4]. The Python Wiki has a list of other Python parsers[5] you may want to try if Yapps isn’t right for your needs.

Yapps 2 vs. Yapps 1 # Yapps 2 is more flexible than Yapps 1 and is not backwards-compatible with Yapps 1. The main changes are: It allows statements and not only expressions to be embedded into the parser. This is wonderful for setting up loops. Here’s an example for parsing Lisp lists in Yapps 1: rule list: "(" seq ")" -> << seq >> rule seq: -> << [] >> | expr seq -> << [expr] + seq >> And here’s what is used in Yapps 2: rule list: r"\(" {{ e = [] }} # initialize the list ( # begin a loop expr {{ e.append(expr) }} # add each expr ) * # repeat as necessary r"\)" {{ return e }} # return the list Note that where Yapps 1 required you to set up a new recursive subrule, Yapps 2 allows you to express loops in a natural iterative style. In addition, the Yapps 2 style is more efficient (appending to an existing list). The Yapps 1 grammar is shorter, but harder to write, and less efficient.

It has convenience constructs like optional matching as well as the + and * operators from regular expressions. In the example above you can see that the ( expr ... ) * construct is used to repeat zero or more expressions. You can also mix choices and sequences together. For example, you can write A (B | C) D . These changes eliminate the need to use extra rules.

It uses Python 1.5-style (Perl-compatible) regular expressions. These regular expressions are now standard in the Python community, and are more powerful than the old-style regular expressions. In addition, you can use the Python r"string" syntax to avoid endless strings of backslashes.

You can choose to use the Yapps 2 run time library to make the generated parser smaller, or to make a standalone parser file.

Yapps 1: How it started # My desire for a parser generator began in late 1997 with a school project for which I used my own hand-written parser. At the same time, there was a thread on comp.lang.python about parsing e-mail addresses with nested <>’s. I was reading Stroustrup’s “The C++ Programming Language” and came across an expression parser. I was also reading Mark Lutz’s “Programming Python” and came across another expression parser. The combination of these four events inspired me to write a parser generator that could produce parsers similar to Lutz’s hand-written parser. Many of the features of Yapps are inspired by the ANTLR/PCCTS system written by Terrence Parr. ANTLR combines flexibility, speed, and parsing power into a system that produces readable recursive descent LL(k) parsers for C, C++, or Java. Download Yapps 1 is a single Python file, approximately 30k in length. It is likely that you will also want to see the documentation, available in these formats: HTML (browse online)

HTML (.zip) There are three tiny examples available: Name Grammar Parser Calculator expr.g expr.py Lisp Expressions lisp.g lisp.py Yapps Grammar parsedesc.g parsedesc.py