Although this text is written in Emacs, I strongly believe that a semantic-based, reliable, and fast support from tooling is a great boon to learnability and productivity. A great IDE support is a must for a modern parser generator, and this chapter talks mostly about IDE-related features.

The most important productivity boost of a parser generator is the ability to fiddle with grammar interactively. The UI for this might look as a three-pane view, where the grammar is on the first pane, example code to parse is in the second pane and the resulting parse tree is in the third one. Editing first two panes should reactively update the last one. This is difficult to implement with most yacc-like parser generators, I’ll talk more about it in the next section.

The second most important feature is inline tests: for complex grammars it could be really hard to map from a particular rule specification to actual code that is parsed by the rule. Having a test written alongside the rule is invaluable! The test should be just a snippet of code in the target language. The "gold" value of the parse tree for the snippet should be saved in the file alongside the grammar and should be updated automatically when the grammar changes. Having inline tests allows to fit the "three pane UI" from the previous into two panes because you can just use the test as your second pane.

Here’s a video that shows how it works in fall: https://youtu.be/gb1MJnTcvds.

Note that even if you write your parser by hand, you still should use such "inline tests". To do so, write them as comments with special markers, and write a small script which extracts such comments and turns them into tests proper. Here’s an example from one experimental hand-written parser of mine. Having such examples of "what does this if parses?" greatly simplifies reading of parser’s code!

Here’s the list of important misc IDE features, from super important to very important. They are not specific to parser generators, so, if you are using a parser generator to implement IDE support for your language, look into these first!

extend selection to the enclosing syntactic structure (and not just to a braced block). A super simple feature, but this combined with multiple cursors is arguably more powerful than vim’s text objects, and most definitely easier to use.

Fuzzy search of symbols in the current file/in the project: super handy for navigation, both more important and easier to implement than goto definition.

Precise syntax highlighting. Highlighting is not a super-important feature and actually works ok even with regex approximations, but if you already have the syntax tree, then why not use it?

Go to definition/find references.

Errors and warnings inline, with fixes if available.

Extract rule refactoring, pairs well with extend selection.

Code formatting.

Smart typing: indenting code on Enter , adding/removing trailing commas when joining/splitting lines, and in general auto magically fixing punctuation.

Code completion: although for parser generators dumb word-based completion tends to work OK.

Here’s a short demo of some of these features in fall: https://youtu.be/WRWmwfBLf7o.

I want to emphasize that most of these features are ridiculously easy to implement, if you have a parse tree for your language. Take, for example, "fuzzy search of symbols in the project". This is a super awesome feature for navigation. Basically, it is CTAGS done right: first, you parse each file (in parallel) and build a list of symbols for it. Then, as user types, you incrementally update the changed files. Using fall, I’ve implemented this feature for Rust, and it took me three small files:

* find_symbols.rs to extract symbols from a single file, 21(!) lines.

* indxr.rs, a generic infra to watch files for changes and recompute the index incrementally, 155 lines.

* symbol_index.rs glues the previous two together, and adds fst by ever-awesome BurntSushi on top for fuzzy search, 122 lines.

This is actually practical: initial indexing of rust-lang/rust repo takes about 30 seconds using a single core and fall’s ridiculously slow parser, and after that everything just works: