Transcending Language Bias in Scriptable Applications

Posted 24 Sep 2002 at 20:36 UTC by beppu

Most people agree that scriptability in an application is a good thing. However, there is a problem that rears its head over and over again. The problem is that people tend to show a strong loyalty to their scripting language of choice. For example, if someone likes Perl, chances are their scriptable application will support just Perl, and all the Ruby, Python, Tcl, Scheme, etc. fans are left out in the cold. What many people don't realize is that it doesn't have to be this way. This article will compare and contrast the approaches to scriptability taken by Emacs, Vim, and the GIMP, discussing their individual strengths and weaknesses. Each one of these programs supports multiple scripting languages to varying degrees, but the implementation strategies are worlds apart. The GIMP, in particular, has an elegant approach that allows for C, Scheme, Perl, and Python code to interoperate in an unprecedented way. So powerful is the GIMP's approach that it might be a good idea to generalize it, and make it available to the programming community at large.

This article was originally published in the 8/2002 issue of Linux Magazine.





Comparing Different Strategies for Adding Scripting to an Application





Throughout the lifetime of this Power Tools column, scripting has been a recurring theme. It was definitely not planned to be this way, but the fact that every single article to date has had something to do with scripting is not a detail that should be overlooked. Could it be that the ability to script an application is what sets it above the rest? Many programs are merely used, but when a program is scriptable, you can start teaching it to perform complex actions that it couldn't do before. Being scriptable also means that the work of extending a program can be distributed and delegated. In a culture of openness such as the one most Unix users participate in, this results in a very creative and communal building process that blurs the line between user and programmer. For these reasons and many more, adding scripting to an application is a very powerful act. However, the way scripting is implemented in an application is not a very standardized process. Although the concept is not new, it has only recently come into vogue, and because of that, programmers in a rush to make their programs scriptable often don't research the prior art. Driven by their biases for this language or that, they limit their audience to those who have biases similar to their own. For example, a programmer's favorite language might be Python, so they make their application scriptable in Python only. This may not seem that harmful to some, but consider what could happen if these biases could be transcended? We could learn a lot by studying other cultures, so let's try to do that now.



Elisp-centric Emacs



Emacs has been around for so long, there's a good chance that it's older than you are. Its long life is a tribute to its forward-looking design which deliberately included the ability to be extended by writing programs in a dialect of Lisp called Elisp. Over the years, quite a few I<applications> have been built on top of Emacs using Elisp, and an active community of users and programmers has developed around this text editor. Why was Lisp chosen, though? RMS (Richard M. Stallman), the original author of Emacs, explains his rationale in an old interview by Linuxcare: Linuxcare: How did you come to choose LISP as the EMACS engine?



Richard: LISP is the most powerful programming language, and if you want an interpreter, LISP is the best. None of the other languages come anywhere near LISP in their power. The most exciting things about LISP are read, eval, and print. If you look at other languages, they have no equivalent for any of those.

And there's more, but space is limited in this column. In all seriousness, though, if you want to learn more about why Lisp is a powerful programming language, go find the Linuxcare interview with RMS and read it. RMS does a good job of explaining the unique feature that sets Lisp apart from all other programming languages. Relatively speaking, Lisp does not seem as powerful as it used to be, because abstractions like garbage collection and lexical closures are becoming readily available to programmers in the mainstream. However, it's sobering to think that we are just now becoming able to appreciate what Lisp had decades ago. Therefore, considering the timeframe in which Emacs was implemented, Lisp was an excellent choice. Also, in an essay titled, "Why you should not use Tcl", RMS retells a very important lesson that was learned while implementing Emacs. The principal lesson of Emacs is that a language for extensions should not be a mere "extension language". It should be a real programming language, designed for writing and maintaining substantial programs. Because people will want to do that! By "real", RMS means that a programming language should support a mature set of abstractions that allow the programmer to implement programs beautifully rather than in a contrived way. Lisp satisfies this criteria. However, Lisp is not without its faults. To people who have their first programming experience using a language like Basic or JavaScript, Lisp looks really weird. Right off the bat, there's a social barrier that Lisp must overcome. To partially solve this, John Tobey tried to bring Perl to Emacs. His perl module, called Emacs::EPL, creates a bridge between Emacs' Elisp interpeter and Perl. However, to anyone who has used this module, it is clear that Perl is a second-class citizen here. For one thing, the Perl interpreter is executed as a subprocess, and interprocess communications between between the two is accomplished via pipes. Secondly, you still need to have a good understanding of Lisp in order to use Emacs::EPL -- Perl alone is not enough. The bottom line is that Emacs was never meant to be programmed in anything other than Lisp, and this decision has had ramifications on just how extensible Emacs could be.



Vim and Diversity in Scripting



Vi is the other text editor in Unix land, and the philosophy embodied by vi and its derivatives is diametrically opposed to much of what Emacs stands for. Where Emacs is big, vi is small. Where Emacs tries to do everything by itself, vi focuses on the core task of text editing, and if you wanted to do a complex text manipulation, the use of Unix pipes and filters is encouraged. Where Emacs prefers a modeless style of editing, vi makes very explicit uses of modes (such as insert mode and command mode). In fact, this is one of vi's defining characterstics which sets it apart from almost every other editor in existence. Suffice it to say, Emacs and vi are very different from each other. However, scripting has added an interesting twist to this story. The most popular editor in the vi family is Vim, and it gained the ability to be scriptable in the late 1990s. It's funny, because vi users used to scoff at Emacs users for implementing all kinds of functionality not directly related to the task of text editing. For example, some Vim users might see Emacs' built-in psychotherapist (accessible via M-x doctor ) and feel a strange combination of amusement and disgust. They think the multiple mail and news clients that come bundled with Emacs are blasphemous. "Emacs isn't a text editor; it's an operating system!" What would these same people do if they found out that Tetris had been implemented on top of Vim? Then, what if they found out that there was a project called VINE that aims to make it possible for Vim to handle all one's mail, news and calendaring needs? In light of this, people who use Vim need to ask themselves whether Vim and Emacs are as philosophically different as they once believed. Of course, significant differences in the scripting models exist. For one thing, Vim's APIs are very minimal compared to Emacs'. On the other hand, Vim can support many scripting languages simultaneously. In addition to its own scripting language, called Vim Script, it can also be extended using Tcl, Perl, Python, and Ruby. This lets you extend Vim in a language that fits your personality, which is great in theory. However, in practice, Vim's approach is not quite perfect. Although the problems are partially technical, the social issues are much more significant. For one thing, it is very rare for someone to compile their Vim with support for more than one scripting language at a time. It's totally possible to do it, but people tend to show an amazing loyalty to their scripting language of choice. Unfortunately, this means that if a useful extension were created using Vim's Ruby interface but you only compiled in support for Vim's Perl interface, you'd be out of luck. This is such a serious problem that people who write extensions for Vim opt to use Vim Script instead of the more powerful alternatives, because this way they can be sure that their extension can be used as widely as possible. To improve this situation, extension writers need to be assured that any given scripting language that Vim supports will be enabled in a majority of Vim installations. Automatically, some of you scream, "BLOAT!," but you'd be surprised how small the impact of enabling Tcl, Python, Perl, and Ruby in a single vim binary actually is. Start-up time is not noticeably different, because the interpreters are started up on-demand rather than all at once in the beginning. File size is up by 60KB, and memory usage can be counted in 2 ways. If you count shared libraries, the memory consumption is up by 720KB, but if you don't then it's only up by 180KB. It's not that much by today's standards. (See The Efficiency of Vim for details.) And remember, this is for enabling 4 additional interpreters. If the benefits are judged to outweigh the costs, all the interpreters could be enabled by default in the configure script. Then, in a few years, extension authors could feel a lot more freedom to use the language they really want to use. Perhaps this should be a TODO item for Vim 7. After all, what good is supporting multiple scripting languages if hardly anyone uses them? Scripting in Vim also suffers from a variation on the same problem Emacs had when Emacs::EPL attempted to add Perl support. In Vim's case, the problem is that Vim Script has access to certain capabilities of Vim that neither the Tcl, Perl, Python, nor Ruby interfaces have direct access to, thus relegating these languages to 2nd class citizenship. What this means is that anyone who wants to create a non-trivial extension to Vim will have to know some Vim Script. This isn't that bad in practice, but it is not ideal, either. It is only mentioned here, because people should be aware that the GIMP has solved this problem in a very convincing fashion.



The GIMP and the Genius of the PDB



The GIMP (GNU Image Manipulation Program) is a free Photoshop clone that has put into question a lot of stereotypes about what Unix programs could and could not do. In addition to proving that it was realistic for a Unix program to be heavily graphical, one of its most touted features was its scriptability. At first, the GIMP could only be scripted in Scheme, but mature interfaces for Perl and Python have since been developed. What's more impressive is that all the extension languages stand on a level playing field. Therefore, although Scheme was first, it does not have any special benefits over Perl or Python. What makes this possible is the PDB (Procedural Database). Like its name implies, it is a database of procedures (or functions). A large portion of it is devoted to keeping track of C functions from libgimp. However, any scripting language that the GIMP supports can register its own functions in the PDB. Combine this with the ability of any of the supported languages to execute a function from this database, and you have the ultimate in interoperability. For example, a script written in Python can execute any function from the PDB regardless of whether it was written in C, Scheme, Python, or even Perl, and it's not at all contrived. Due to the good taste of the people who wrote the Scheme, Perl, and Python interfaces, calling a function from the PDB looks just like you're calling a native function! (See Listing 1.) [ LISTING 1 - Scheme, Perl, Python, and C Working Together ]

# Try this in the Gimp-Python Console. def peace(): # Scheme pdb.script_fu_sphere( "100", # radius "45.0", # lighting angle FALSE, # shadow? (255, 255, 255), # background color (192, 0, 0)) # sphere color # Perl img = pdb.perl_fu_yinyang( 256, # width 256, # height TRUE, # insert eyes? FALSE, # eyes are images? "", # top eye filename "", # bottom eye filename TRUE) # anti-aliasing? # Python pdb.python_fu_sphere( 100, # radius 45.0, # lighting angle FALSE, # shadow? (255, 255, 255), # background color ( 0, 0, 192)) # sphere color # C function from libgimp pdb.gimp_display_new(img) peace() This is probably one of the most important innovations in interoperability to come around in a while, and it's unfortunate that not many people are aware of it.



The Future of Scripting



We, as users and programmers, have to ask ourselves what we want in terms of scripting support. Hopefully, you have come to understand that extensibility is good and that having the potential to be extended in more than one language can be even better if done right. Also, what is right is not just a technical question but a social one as well. Some of you may think that implementing something like the PDB may require too much work, but remember that we work in a world of Free Software. What if the PDB could be separated from the GIMP, and then generalized and documented to the point where it could be distributed independently? Then every programmer who wanted to add scripting support to his or her C/C++ application could just reuse this modular component and have a solid foundation for supporting multiple scripting languages right from the start. If implemented, there is no question that the long-term benefits would be positive. The only question is how positive. "The best way to predict the future is to invent it."

-- Alan Kay, the creator of Smalltalk



SIDEBAR 1:

The Efficiency of Vim vim-6.1-rppt has Ruby, Perl, Python, and Tcl enabled, and vim-6.1 is a stock binary with no interpreters.

Time to Start and Stop Vim: /usr/bin/time -f "%E" ./vim-6.1-rppt -c :q /usr/bin/time -f "%E" ./vim-6.1 -c :q Write a script to do this repeatedly such that your sample set is sufficiently large, and calculate the average time for each vim binary. Then, calculate the difference between these two values. On a humble 566MHz Celeron, the difference was only:



0.053 seconds.

File Size: $ ls --sort=size -o vim-* -rwxrwxr-x 1 beppu 1106708 May 30 12:51 vim-6.1-rppt -rwxrwxr-x 1 beppu 1046356 May 30 12:56 vim-6.1 1106708 bytes - 1046356 bytes = 60352 bytes

Memory Consumption: PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 16892 beppu 9 0 4044 4032 2468 S 0.0 3.1 0:00 vim-6.1-rppt 16891 beppu 9 0 3312 3312 1928 S 0.0 2.6 0:00 vim-6.1 SIZE memory_pages_used * memory_page_size (4KB for i386) RSS total amount of memory used in kilobytes SHARE amount of shared memory used in kilobytes Not counting shared memory: (4032KB - 2468KB) - (3312KB - 1928KB) = 180KB Counting shared memory: 4032KB - 3312KB = 720KB go back



SIDEBAR 2:

Resources

Emacs: The Extensible, Customizable, Display Editor http://www.gnu.org/software/emacs/emacs-paper.html Emacs Wiki http://www.emacswiki.org/cgi-bin/wiki.pl Emacs::EPL http://john-edwin-tobey.org/perlmacs/ The Linuxcare Interview with RMS http://www.linuxcare.com/viewpoints/os-interviews/latest.epl Vim http://www.vim.org/ Vim at SourceForge http://vim.sf.net/ VINE - Vim Integrated News and Email http://www.mossbayeng.com/~ron/vim/vine.html The GIMP http://www.gimp.org/ Gimp-Perl http://www.goof.com/pcg/marc/gimp.html Gimp-Python http://www.daa.com.au/~james/pygimp/

Object Model, posted 24 Sep 2002 at 23:07 UTC by nymia » (Master)

This is an interesting article. Keeping the API of an application callable from the implementation of the grammar would probably lead to more variety of scripting languages. The GIMP example above shows the API has been designed to accomodate more than one, though. The interesting part is the PDB which can store and execute more than one language. Putting the PDB as a standard for scripting will definitely solve a lot of interface issues, though. Especially the part where one app written by an entirely independent group can suddenly become available for other non-related apps. But this is what CORBA is all about, isn't it?

... that's exactly why I've been working on it the whole time! :-) It's also what XPLC is all about. Currently, you have to describe interfaces (which are not unlike a group of function registered in the PDB) manually in C++, which kind of restricts it, but the goal is that any and all interfaces are to be available in any of the language bindings, without having to generate separate bindings for the functions (making the interface available for one language makes it available for all). And all of this without the overhead of CORBA (the minimally required part of XPLC is an 8k shared library that only the application needs to link with. The "full" thing is a 22k shared library. Having the lowest possible overhead is one of the main goals of the project.

Language based systems, posted 25 Sep 2002 at 01:30 UTC by mslicker » (Journeyer)

For those familer with language based systems, scripting is a rather clumsy approximation of what is native to these systems. In language based systems, an "application" is just a definition in the language. A library is just a collection of definitions. Writing modular code, it is possible to recombine definitions in new ways, creating new applications specific to a particular task. You mention Lisp and Smalltalk in your article. There have been systems based on each. Forth is important to mention. With Forth, the system and language are so tightly integrated that "Forth" commonly refers to both the system and the language. The future of scripting I believe is in the past. Language based systems have long existed, but have not come to widespread use. Perhaps people think there is more power with the muli-language aproach. From my experience a single expressive ubiquitous language is far more powerful than the multi-language approach currently in practice.

mslicker: the point of the article seems to be basically: 1) no group of random people will agree on the 'best' language, and people don't like changing languages. 2), exposing all the internal interfaces of an application is good. 3) it's really good when you can have access to all of the internals in the language of your choice. 4) gimp takes an approach like this. supporting more than one language is a matter of convienence: it allows people to use whatever language they're already familar with instead of having to learn whatever language the author of a program likes. it is intended to lower the barrier for users to customize applications. in general, making an application out of nice logically complete modules is just good design. in some ways you can think of it as expressing the application in a C like language with application-specific built-in functions. in fact i can readily think of several applications which use a scripting language to implement the high level functionality in terms of primitives in libraries (written in C generally for speed). generally though you want a scripting interface not to build an entirely new application, but just to change a few things or add a feature to an existing application. the point is though, that choice of language is to make it more convienent for folks to customise applications, because people will disagree on what a 'single expressive ubiquitous language' is.

..., posted 25 Sep 2002 at 07:02 UTC by tk » (Observer)

1. The future of scripting I believe is in the past. Language based systems have long existed, but have not come to widespread use. Perhaps people think there is more power with the muli-language aproach. From my experience a single expressive ubiquitous language is far more powerful than the multi-language approach currently in practice. A natural question to ask is why these ostensibly `better' systems haven't come to widespread use. In the context of scripting, the answer seems quite clear: The whole point of scripting languages is to allow people to extend functionality in a quick-and-dirty manner. Allowing different languages to be used means not only that a script-writer can work in a language he's familiar with, but that he can immediately leverage on any code libraries for that language. If one needs to spend 10 years to learn an esoteric language just so that he can write that filter plug-in for his mail client, even when all the code's already there in CPAN, then I think it defeats the whole purpose of scripting. 2. (Actually I have another idea. The "Unix philosophy" is to write programs which can be joined together by piping on stdin and stdout. Unfortunately this technique is becoming less and less popular, mainly due to the overhead needed for piping. Will it be feasible/worthwhile to use nefarious techniques to speed up the piping mechanism, to achieve extensibility?)

Runtime compatibility, posted 25 Sep 2002 at 11:16 UTC by jonkare » (Journeyer)

This is a fascinating approach. However, I'd like to raise the issue of runtime compatibility. Most scripting languages include some form of automatic memory management, typically garbage collection or reference counting. They're all different, although a number use the Boehm conservative collector. There's no problem if the user enables only one extension language - but then he can't use scripts made elsewhere in other languages. But if two runtimes with two different memory management systems are enabled, what can then happen? I don't know the details, but I would personally fear the worst.

There is a big advantage of limiting the number of languages used in integrated applications like Emacs: It makes them easier debug and extend. For example, if you want to see the differences between the CVS versions of a C++ file, you use many different Emacs packages, the version control interface (VC), the difference and merge tool (ediff) and the C++ mode (cc-mode). These again may be based on other packages, but ignore that for now. If there is a problem, say a bug or a missing feature in the interactin you want to remidy, you may have to study and perohaps extend the code for any or all of these packages. To do that today, you have to learn a one or two languages, namely Emacs Lisp and C (if the problem goes really deep). But imagine Emacs had supported plug in languages, and these packages had been written in Python, Perl and TCL. In that case, you would have to learn up to 5 languages (the three above, as well as C and Emacs Lisp) to fully understand the interaction. For applications where extensions rarely interact, it may be an advantage to support a choice of languages. But for integrated applications this just raise the bar for new developers unreasonable, as he will have to learn all the supported languages.

Reply, posted 25 Sep 2002 at 15:20 UTC by mslicker » (Journeyer)

A natural question to ask is why these ostensibly `better' systems haven't come to widespread use. tk writes: Perhaps a natural question, but not useful a one. What comes into widespread in computing is the result of economic forces (why do so many use Windows?), or unknown forces (why did C become so popular?). If it really takes 10 years to learn the general purpose language, then there is really something wrong with that language. I would expect a general purpose language to take more time than a language used for scripting. As abraham points out the barier is much lower when there is only one language to learn. Take a typical Unix environment, there are several languages you must learn to be productive in this environment. Unless you really want to get in over your head, these languages are fixed and must be dealt with. In my experience scripting can not be done in a "quick-and-dirty manner". In fact this is probably the main reason I've never scripted any application, the barrier is much to high. You must learn the details of these quite expansive applications and how this functionality maps onto the scripting language of your choice. And you must do this for each application for each scripting language. In a language based system, the skills you learn are continually reused for every task you might want to perform. As far as the effort expended goes, a language based system makes far better use of your time. (Actually I have another idea. The "Unix philosophy" is to write programs which can be joined together by piping on stdin and stdout. Unfortunately this technique is becoming less and less popular, mainly due to the overhead needed for piping. Will it be feasible/worthwhile to use nefarious techniques to speed up the piping mechanism, to achieve extensibility?) Piping is quite limited. It really doesn't make sense at all for interactive applications, that is why you don't see it's use. Instead far more horrible methods are used for interactive Unix applications. Piping is quite limited. It really doesn't make sense at all for interactive applications, that is why you don't see it's use. Instead far more horrible methods are used for interactive Unix applications. async writes: the point is though, that choice of language is to make it more convienent for folks to customise applications, because people will disagree on what a 'single expressive ubiquitous language' is. People also disagree on what is convienent. For me language based systems are far more convienent to extend than these mutli-language scriptable applications. When you program in any system there is a whole host of things you must agree to (or at least put up with), language based systems are no different. People also disagree on what is convienent. For me language based systems are far more convienent to extend than these mutli-language scriptable applications. When you program in any system there is a whole host of things you must agree to (or at least put up with), language based systems are no different.

So give me PDB, liberally licensed (at least the LGPL). If this was a separate package that did the bulk of heavy-lifting to make Python, Perl (5 & 6), and Scheme available to a scriptable application, I might use it. It still would be a ./configure option, so one could fall back to the native scripting language if looking for a lighterweight implementation.

I stand in the middle with XPLC. You only need to understand the interfaces, which it will be possible to have output in the language of your choice (still missing an IDL). So in a sense, XPLC acts as the "single expressive ubiquitous language". At the same time, those interfaces can be implemented in different languages. So people will disagree on the single language to use, but that's okay too. Where XPLC breaks down is if you need to delve in the innards of each of the components (following abraham's comment about Emacs extensions), the components might be written in whatever language their author fancied!

Touchee!, posted 26 Sep 2002 at 07:48 UTC by shlomif » (Master)

I wanted to write such an article myself. I do hope someone will take initiative and form a common extensibility layers with various languages as plug-ins. Of course, some languages are very different which makes it very hard. For instance, in Tcl, strings are null-terminated, which is a very bad decision.

I draw people's attention to the set of perl modules under the Inline heirarchy. I've used the C and Tcl interfaces before now to create language-agnostic applications in perl, and I'm fairly sure the Python one is also useable (although I've never used it). It's simple, effective, and powerful.