tech-userlevel archive

Lua in NetBSD

To : NetBSD User-Level Technical Discussion List <tech-userlevel%netbsd.org@localhost>

: Subject : Lua in NetBSD

: From : Marc Balmer <marc%msys.ch@localhost>

: Date: Sat, 24 Oct 2009 10:48:41 +0200

About one week ago, I suggested to the NetBSD community to add the Lua

language to NetBSD as a language for scripting applications and as a

general scripting language.

My original post is here: http://marc.info/?l=netbsd-tech-userlevel&m=125577239909997&w=2

I am overwehelmed by the number of positive reactions I got to my

proposal, it looks like I have been suggesting something that has been

waited for by many. While I suggested to import Lua into base, much

to my surprise it was even suggested to use it in the kernel, an idea

I like the longer I think of it.

There was some criticism, too, especially since I did not state what I

am going use Lua for, if it was in base; I wanted to discuss the

general case first, so I was vague in this regard on purpose.

I am currently working on a scheme to interface all kind of external

reference clocks to the system; not to get a timing base for

timecounters, but to get absolute time information. This is something

different. For timecounters you need a regular pulse, but a radio

clock might only give the correct time once a day. Maybe some people

know about my work in OpenBSD in this area. My current work builds on

top of that previous work, but provides better abstractions and better

interface (and thus more usuable information).

One family of external reference clocks provides timing information

using bytestream protocols, the NMEA 0183 protocol used by most GPS

receivers is one of the best known protocols that falls into this

category. The decoding of the NMEA protocol has been done using a tty

line discipline.

Using line disciplines for this kind of decoding comes at a price:

Line disciplines are a vehicle not to many programmers are familar

with (I could be wrong here, this is just an assumption), they are

somewhat hard to get right and to maintain. It means to do a lot of

string processing in the kernel, god beware if we have a serious,

undetected bug in that code... But the main problem is that there are

dozens of such protocols, most GPS vendors have their own proprietary

binary protocol, Meinberg has it's own serial protocol, endrun comes

to my mind and so on. So for every protocol we need a tty line

discipline, need to define a name for the line discipline, have a

program that can attach the line discipline to an open filedescriptor

and so on (for the latter I wrote ldattach(8) in OpenBSD). One hast

to touch many parts of the system to support a new protocol with a

line discipline.

Time sources are kernel entities (drivers) that provide time

information in a uniform way. They need to be drivers, since many

clocks are attached to a PCI bus or use USB etc. To avoid the line

discipline mess, I have a timesource that can be fed the time

externally. So there is a daemon that decodes the time information

and feeds it to the system. And this is were Lua comes into play.

The decoders for the protocols are written in Lua, and new decoders

can thus be added with ease and without requiring a kernel/userland

rebuild. Besides of that I use Lua to write software with graphical

user interfaces using X11/Xt and OpenMotif and to provide access to

system functionality like sysctls, gpio etc. from Lua.

Lua, being a small language that interfaces exceptionally well with C

programs, could be of good use in various other places, during the

discussion, the following uses have been proposed or mentioned

- extending DHCP clients for options processing - extending the bootloader - extending sysinst - uses in pkgsrc - scripting nvi

- providing access to NetBSD from Lua (though for this, the Lua in

pkgsrc is just fine)

- general scriptiing - maybe stuff that I have forgotten to list...

Thor proposed to add it to the kernel as well. He would be willing to

devote time and manpower to this project, so would I, and Lourival

Vieira Neto from PUC Rio (where Lua is being maintained) offered to

help, since he does a similar project in Linux ("Lunatik"). Having

Lua in the kernel would actually allow to use NetBSD in total novel

ways or for new fields of application:

- rapid prototyping of device drivers, or device drivers in Lua - experimentation with new network protocols - kauth(9) security models in Lua

- controlling the hardware environment (cpu freq etc) based on

evironmental measurements (the Lunatik project at PUC Rio goes

somewhat in this direction it seems)

- accept filters - filter rules

Of course some implementation issues with Lua in the kernel remain to

be solved: The number format. How to integrate it in the kernel.

Should the compiler be included or only the bytecode interpreter

(which means Lua scripts would be compiled to bytecode in userland and

the bytecode then be loaded into the kernel), etc.

A quick recap of why Lua is a language well suited for this kind of

applications:

- small memory footprint - fast execution speed - superb integration with C - code can be prcompiled to bytecode for faster loading - easy to learn with a "natural" syntax, not statically typed - extensible in useful ways using the metatables mechanism - very good data handling capabilities - no bloated libraries

Quite some developers told me in private that they would use Lua for

their work, if it was in the system and I think this all speaks for

the addition of Lua to NetBSD. If there are not to many objections to

this, I will ask core to give me a thumbs-up to initiate two

subprojects:

- Adding Lua to NetBSD userland - Adding Lua to the NetBSD kernel Thanks for your time && let's do it! - Marc Balmer