Jaunt - A friendly Clojure fork

TL;DR

I’ve started a fork of Clojure which I intend to operate in a community directed manner; Jaunt. Check out the demo script or just keep reading for more.

The Long Version

After several months of dinking with Oxlang on and off I came to a realization. Ox, while an annoyingly persistent daydream was doomed to be one of two things. Either it would be something akin to Haskell in S expressions, or it would be indistinguishable from Clojure with some tweaks under the hood.

I’d gotten Ox’s reader sort of working, and was starting to experiment with notations, explore notions of higher kinded types and generally was having a ball. However even getting that far was a highly educational exercise in how much Clojure already does fairly well. To continue down the road of Ox was to admit that I’d have to reinvent all of Clojure’s datastructures, most of the Clojure compiler and a whole boatload of the standard library. It was also to admit that due to limitations of the JVM as a target, I wouldn’t be able to play some of the many tricks which the GHC folks get to play even were I to go down the road of a pure language.

Why do all this? Because it’s hard? I’m no Heracles seeking labors or yaks for their own sake. I’d rather be learning something new or playing Dota or… a thousand other things.

YOUR FINEST YAK SIR — tail -f /dev/ardm0 (@arrdem) February 13, 2016

I swear I would.

Besides, even though Michiel Trimpe was kind enough to mention Ox alongside a number of other “conversational” programming languages in his ClojureX talk, I’m not convinced that it’s a good idea.

In looking back at that particular talk, one thing struck me. Sure in a pure functional programming language such as Haskell you could stick an arbitrary AST in a Merkle tree and have a globally distributed version control and distribution system with strong guarantees about tree uniqueness and reuse safety.

Unfortunately, Clojure is no such beast. We have no language level concept of effects (okay we have the io! macro but that’s hardly the same). While you could implement immutable namespaces (and in fact they are backed by immutable maps already) the concept doesn’t make a whole lot of sense to me.

The very notion of a conversational language is that you can reference any definition from any version of the program as if it were the current version. What does this look like at a REPL? Are namespaces (or rather full states) identified with a commit ID and compilation occurs in terms of a fully qualified “commit” and Var? Now we need notation for that. Maybe version/name/namespace or something. Some sort of explorer to browse older commits is in order… the list goes on and on. In short, I think you wind up reinventing a bunch of stuff that git already does pretty well.

This is not to say that Clojure’s existing mutable namespaces are ideal. For me, they work against the ns or file level code loading style I use when developing Clojure code. Old habits from C die hard as it were. A common failure mode I encounter is that I’ll rename a function but not all of its uses. IntelliJ and refactor-nrepl can be of some help here, but I’ve had them break down and I’m not always in a “full ide” configuration. Because namespaces are mutable, even if I were to perform a rename correctly, the old name is still present in the namespace. If I got it wrong, my code reloads, all symbols resolved then I get Really Cool Unexpected Behavior when parts of my code call the new function and parts call the old function. Or worse my code works fine until I restart my repl or the test server and everything explodes. Or the old name just sticks around cluttering up my autocompletion.

This becomes a real UX problem for me, because I rely heavily on refactoring. I start by sketching out what amount to types, or deriving operations from types I already have, and keep working renaming and moving code around until I get something I’m happy with.

@arrdem your problem is all the refactoring. Why isn't the code right from the start? — Chas Emerick (@cemerick) December 23, 2015

And Yet I Shave

The thing which came to me now a few weeks ago is that there’s an interesting halfway house between these two extremes of mutable and immutable namespaces. What if we gave Namespaces and Vars version numbers? When a Namespace is re-defined (the ns form is evaluated) then the Namespace’s version is incremented. When a def is evaluated, the version of the Var bound by the def is set to be the version of the Namespace in which that Var exists.

This allows traditional REPL style interaction with Namespaces. You may enter a Namespace, add bindings, aliases, imports and so forth as you will. However the version numbers give you important static information about the state of the Var in its Namespace.

If I have some def which I entered at the REPL, when I do something to ns backing file and reload it, that def persists. But, its version is now out of sync with the Namespace it exists within. When we reloaded the file, the Namespace’s version was incremented and then every definition was reevaluated. If my code is still using an old symbol which is not textually present in the file and was was not reloaded, its version doesn’t increase. Now the compiler can detect the version disparity and emit warnings that I’m using old code.

While this solves dependency freshness issues within a single def and a single Namespace, solving them globally requires more leg work. The compiler needs to be adapted to track the use and reach sets of each compiled expression, and to inter that information into the Var binding forms so that it’s possible for a newly compiled form to emit warnings about transitive use of stale vars.

These changes don’t alter the semantics of the Namespace at all outside of reloading. If you evaluate a new def or a single form in a Namespace it works just as before. However if you take this whole file reloading approach, the language can offer added support which papers over a rather painful pitfall.

A related change is that Namespaces can be made to purge their imports and aliases on reload. Right now, Namespaces persist imports and aliases until the VM is restarted. This leads to a nasty set of refactoring problems where you want to change the name of an alias, but can’t reuse a name because aliases are never cleared. If we consider a whole Namespace and its source file(s) as the unit of compilation we can again do better. This is another language level change which would meaningfully improve my day to day use of the language without breaking existing use patterns.

Both of these changes turn out to be quite simple to implement. In the course of only a few days I went from the concept of these changes to a build of Clojure (pr, build) which features Namespace clearing and Var versions. I’m using now daily because it better supports the way that I want to work with the language.

Waxing Political

Rich, Stuart Sierra, Stu Halloway, Alex and the rest of the Clojure/Core crew (see Clojure/Huh?) built Clojure and have shepherded it this far. I’m immensely grateful to them for the work they’ve done to date in building a tool which I’ve had great fun using. But I think our priorities diverge.

From what I’ve been able to gather by talking to people who’ve been around longer and what Rich has written, it should be clear that Clojure is not a community project in the same way Python or Rust is. Clojure is Rich’s personal project, which he is so kind as to share with the rest of us and in in its refinement Rich chooses to accept some amount of input from us as users. I’m somewhat ashamed to admit that it took me two years to realize this.

Open-source development problem in one sentence:

"Users assume they are customers" — Aleksey Shipilëv (@shipilev) January 20, 2016

Discussion of particulars, and the reasons for the status quo is I think a somewhat useless exercise. Rich has his project and administers it as he sees fit. As Clojure is Rich’s project, the priority of its governance is to conserve Rich’s time and it has repeatedly been made clear that there is no desire to change this. This is an eminently reasonable goal I can find no fault in. Rich doesn’t owe me or anyone else jack, let alone time. However the consequence of this structure and of not delegating is the slow and steady path which frustrates my desires for more rapid iteration.

A Fork in the Road

With loyalty the path of the last 18 months and clearly one of frustration, and voice ineffective due to the (reasonable!) priorities of Clojure/Core, The only course of action left to me is some form of exit.

I think that there’s a lot of possible improvements that can still be made to the language. Currently there doesn’t seem to be a lot of appetite for exploring that in Clojure itself. The namespace reloading stuff sketched above is just the first thing that popped to mind when I started thinking about how to improve my day to day experience and implementing that alone has shaken loose a number of other small but worthy improvements.

The consequence of this opinion is that I’ve started Jaunt.

Jaunt is my personal fork of Clojure which seeks to deal with the various organizational and technical limitations expressed above.

Technically, with Jaunt I want to explore a space of refinements to the language such as the Namespace reloading changes sketched above which add value without breaking compatibility with the ecosystem of libraries and tools like CIDER which I’ve come to rely on every day.

Organizationally, I want Jaunt to be a community driven affair. I don’t claim that my judgment is infallible, nor do I think I’ll have all the good ideas, nor will I have infinite time to consider changes even were these restrictions lifted. I’d love to involve contributors who want to bring improvements to the project within the stated bounds of preserving compatibility at least with the documented parts of Clojure.

Demo

Because otherwise how do you believe that any of this works, here’s a script that’ll build a empty leiningen project in a temporary directory and run a quick demo of some of the stale Var stuff.

No Promises

If you want a stable language. If you’re running code in production. If you don’t want to get down and dirty with the language implementation or the development process, stick with Clojure.

As the EPL states:

THE PROGRAM IS PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF … FITNESS FOR A PARTICULAR PURPOSE.

Jaunt should be a drop in replacement for Clojure. I want to keep it compatible. I foresee no reason to break the documented Clojure API. But some amount of drift is inevitable. I’ve already moved some stuff around inside the Java implementation. If you were depending on those undocumented yet public implementation details, all bets are off. I’ve changed how metadata behaves on vars (I think it’s a better approach but it does impose behavioral differences). I’ve added deprecated warnings. I added clojure.core/*line* and clojure.core/*column* (pr) which the compiler totally had internally and just didn’t expose forcing weirdness. And this list of differences (perhaps improvements) is just going to grow.

Whether Jaunt proves to be something I depend on in the future, whether I wander off into the land of types and leave it to someone else or whether it diverges too much from Clojure and dies as a result of the weight of its mutations, I’ve overheard enough interest in a project such as this that I think this is a worthwhile endeavor and I hope you’ll join me in this little adventure.

Update: Clojarr has been renamed to Jaunt, updated article accordingly.

^d