So I had this wild idea. Crypto in JS is slow (like, really, really slow) and also tends to be a lot more “incorrect” than your average C crypto library (NaCL, Nettle, etc). Why not do the crypto natively?

I also had this other wild idea. I love Common Lisp. That should be apparent by now. Common Lisp has an implementation called Embeddable Common Lisp (ECL for short) that lets you put lisp into another process via a nicely-defined C API. Any language or app that can call out to C (pretty much all of them) can have Common Lisp inside of it.

This series of posts will detail my travels as I put lisp in the desktop client, and hopefully in the near future, the mobile apps. Having Turtl’s core logic hosted in one language that runs everywhere will make it easy to port to lots of platforms. It means a bugfix or feature in one client will be available to all of them.

View the current “core” source on Turtl’s Github.

Using what’s known as an evented architecture (as well as message passing) Turtl will be able to run anywhere that has a crappy web browser (since there will no longer be any need for Uint32Array or window.crypto or any of that fun stuff) and a layer of glue to tie the two together.

So on to the journey! I compiled ECL from git, as well as libffi from git. I created a simple C wrapper that loads the main ECL library (in my case ecl.dll) and loads a very simple app. The idea is that in the next few days, Node-webkit will be able to call out to this wrapper and pass messages between lisp and Node.

First problem: making lisp self-contained

This is more of a requirement than an actual problem I ran into. ASDF, lisp’s de-facto package management system, tends to put compiled code for all packages into a user’s home directory. While this is fine on desktop, I don’t want wacky files getting blasted out in all directions on all platforms. So instead, I wanted to make sure the Turtl core keeps all of its compiled files inside its own directory.

To do this, I just run:

(setf asdf::*user-cache* (list (merge-pathnames #p"asdf/" (truename *default-pathname-defaults*)) "cache" :IMPLEMENTATION))

Which puts all cached files in asdf/cache/ . Perfect.

Note that this is a bit of a hack, but seems to work great. There is probably a better way.

Second problem: segfaults

My simple app loads everyone’s favorite threading library (bordeaux-threads) and everyone’s favorite non-blocking IO library for lisp (cl-async) and starts an event loop in a thread. Right?

Wrong. Directly after calling into the event loop, I got segfaults. Looking at the backtrace via GDB shows

#0 0x6b27e962 in ffi_closure_SYSV_inner () from c:\htdocs\turtl\core\ecl.dll #1 0x6b2813ee in ffi_closure_SYSV () from c:\htdocs\turtl\core\ecl.dll #2 0x62103add in event_process_active_single_queue () from c:\htdocs\turtl\core\libevent_core-2-0-5.dll #3 0x62103cdd in event_process_active () from c:\htdocs\turtl\core\libevent_core-2-0-5.dll #4 0x6210423f in event_base_loop () from c:\htdocs\turtl\core\libevent_core-2-0-5.dll #5 0x62103d70 in event_base_dispatch () from c:\htdocs\turtl\core\libevent_core-2-0-5.dll #6 0x6b2812e5 in ffi_call_win32 () from c:\htdocs\turtl\core\ecl.dll #7 0x6b27e856 in ffi_call () from c:\htdocs\turtl\core\ecl.dll #8 0x6b2415d6 in si_call_cfun (narg=4, fun=0x3c3aca0, return_type=0x6b29273c <cl_symbols+38332>, arg_types=0x3ed5c69, args=0x3ed5c61) at c:/src/ecl-git/src/c/ffi.d:949

Hmm. A libffi problem? Nope. After playing around with the command line options in vanilla ECL, I found my culprit. Turns out, I was running out of stack in my lisp image. I changed my main ECL init code from this:

res = si_safe_eval(3, c_string_to_object("\ (progn\ (handler-case (load \"src/loader\") (t (e) (format t \"error: ~a~%\" e))))\ "), Cnil, OBJNULL);

to this:

res = si_safe_eval(3, c_string_to_object("\ (progn\ (ext:set-limit 'ext:lisp-stack (expt 2 18))\ (handler-case (load \"src/loader\") (t (e) (format t \"error: ~a~%\" e))))\ "), Cnil, OBJNULL);

Notice the added (ext:set-limit 'ext:lisp-stack (expt 2 18)) . This tells ECL to give us a 256K lisp stack (as opposed to the default 32k). Also note that I’m setting this top-level in my progn , instead of setting it in src/loader.lisp . This is important, as ECL seems to ignore this directive when run from within a loaded file.

This cleared up my segfault issues.

Third problem: ECL not compiling some of my code

I wrote a futures library that works nicely with cl-async to eliminate the worst parts of continuation passing style (think, callbacks within callbacks within callbacks …). In every other lisp implementation, it runs fine. In ECL, my future-handler-case macro was barfing in certain instances.

The solution? Put on your goggles because we’re going into the time machine. About a year ago, I wrote to /r/lisp asking for help on a way to wrap my futures implementation with effective error handling. Two responses stuck out: the one I understood (and eventually ended up implementing) which is to temporarily save the attach macro’s transform function into a temporary variable and call it later through a macrolet that wraps attach in error handling, and the response suggested by Paul Khuong which involved saving macro state into a symbol macro which keeps a running tally of how many times attach has been wrapped (which I didn’t understand at all).

Well, turns out ECL doesn’t like the easy way. That’s ok, because I spent a good few hours figuring out exactly what was going in in Paul’s solution, and managed to crack it. The new code is now in cl-async-future (in a separate branch for now) and seems to compile fine on ECL without breaking anything yet.

Up next

In order to get ECL talking to Node-webkit, I want to get node-ffi built. This requires installing Visual C++ (gag) because gyp can’t figure out how to use MinGW (or so the node-gyp maintainers say).

Once Visual C++ is installed, I’m going to take a crack at installing node-ffi via nw-gyp, which is the Node-webkit equivalent of Node.js’ node-gyp.

Then it’s just a simple matter of getting Node-webkit to be able to send and receive messages to and from ECL.

Kid’s stuff!

See you next time.

- Andrew

UPDATE: Part 2!