NetBSD-SoC: Provide support for dynamic NetBSD kernel extensions using the Lua language - Lunatik/NetBSD

What is it?

Process schedulers - this extension could allow users to define new algorithms for global process scheduling and/or to create different scheduling polices for independent sets of processes or threads at run-time. This could also help to implement quality of service (QoS).

- this extension could allow users to define new algorithms for global process scheduling and/or to create different scheduling polices for independent sets of processes or threads at run-time. This could also help to implement quality of service (QoS). Packet filters - this extension could allow the creation of more elaborated rules to process and filter the network packet traffic, instead of using plain rules tables.

- this extension could allow the creation of more elaborated rules to process and filter the network packet traffic, instead of using plain rules tables. Power management - this extension could allow users to define their own methods of governance of the system power usage. For example, users could define their own algorithm to scaling the central processing unit (CPU) frequency and voltage in order to save power or prevent overheating.

Status

April 5, 2010 : Started to study the NetBSD kernel and to write the GSoC proposal.

: Started to study the NetBSD kernel and to write the GSoC proposal. April 9, 2010 : Submitted the GSoC proposal.

: Submitted the GSoC proposal. April 21, 2010 : Community Bonding Period -- Students get to know mentors, read documentation, get up to speed to begin working on their projects.

: Community Bonding Period -- Students get to know mentors, read documentation, get up to speed to begin working on their projects. April 21, 2010 : Started to code.

: Started to code. May 24, 2010 : Students begin coding for their GSoC projects; Google begins issuing initial student payments

: Students begin coding for their GSoC projects; Google begins issuing initial student payments May 25, 2010 : Submitted the first patch, containing the Lua interpreter ported to NetBSD kernel.

: Submitted the first patch, containing the Lua interpreter ported to NetBSD kernel. July 12, 2010 : Mentors and students can begin submitting mid-term evaluations.

: Mentors and students can begin submitting mid-term evaluations. July 13, 2010 : Submitted a patch, containing the first version of the loading and execution mechanism. This version works with only one Lua state running in a kernel thread and serializes all requests to this state through a work queue.

: Submitted a patch, containing the first version of the loading and execution mechanism. This version works with only one Lua state running in a kernel thread and serializes all requests to this state through a work queue. July 16, 2010 : Mid-term evaluation deadline; Google begins issuing mid-term student payments provided passing student survey is on file.

: Mid-term evaluation deadline; Google begins issuing mid-term student payments provided passing student survey is on file. August 3, 2010 : Submitted a patch, containing the second version of the loading and execution mechanism. This version works with multiple Lua states and uses mutexes to isolate the access to them.

: Submitted a patch, containing the second version of the loading and execution mechanism. This version works with multiple Lua states and uses mutexes to isolate the access to them. August 9, 2010 : Suggested 'pencils down' date. Take a week to scrub code, write tests, improve documentation, etc.

: Suggested 'pencils down' date. Take a week to scrub code, write tests, improve documentation, etc. August 16, 2010 : Firm 'pencils down' date. Mentors, students and organization administrators can begin submitting final evaluations to Google.

: Firm 'pencils down' date. Mentors, students and organization administrators can begin submitting final evaluations to Google. August 17, 2010 : Submitted the latest patch, containing the third version of the loading and execution mechanism, a usage example, a user-space tool to load scripts into the kernel, a test module and a man page draft. This version also defines a kernel programming interface (KPI) to help kernel developers to extend their modules.

: Submitted the latest patch, containing the third version of the loading and execution mechanism, a usage example, a user-space tool to load scripts into the kernel, a test module and a man page draft. This version also defines a kernel programming interface (KPI) to help kernel developers to extend their modules. August 20, 2010: Final evaluation deadline; Google begins issuing student and mentoring organization payments provided forms and evaluations are on file.

Deliverables

Lua port - a port of the Lua interpreter to the NetBSD kernel. Lua works just like a regular ANSI C library, in fact Lua is a regular ANSI C library. However, since the kernel does not provide the entire C standard library, this project intends to deliver a modified Lua interpreter suited to work within the NetBSD kernel.

- a port of the Lua interpreter to the NetBSD kernel. Lua works just like a regular ANSI C library, in fact Lua is a regular ANSI C library. However, since the kernel does not provide the entire C standard library, this project intends to deliver a modified Lua interpreter suited to work within the NetBSD kernel. Load and execution mechanisms - mechanisms to load and execute Lua scripts in the kernel. This project intends to deliver infrastructure to isolate the Lua interpreter in its own kernel thread and to serialize the access to it, a set of functions to facilitate the access to the Lua interpreter and a mechanism to allow the code to be loaded into the kernel from the user-space.

Optional (would-be-nice) components:

Documentation - documentation of what was necessary to change in the Lua interpreter to port it to the NetBSD kernel and how to use the mechanisms created to load and execute Lua scripts in kernel.

- documentation of what was necessary to change in the Lua interpreter to port it to the NetBSD kernel and how to use the mechanisms created to load and execute Lua scripts in kernel. Usage example - a simple example that illustrates how to use the created kernel subsystem.

Documentation

Prerequisites

1. Building Lunatik

cd /usr/src/sys/lunatik # make # make install # modload lunatik lunatik: cmajor = 198 cd /usr/src/sys/lunatik/userspace # make # sh mknodlunatik.sh 198

2. Extending a kernel module

#include <sys/lunatik.h> static kmutex_t kmutex; static lunatik_State *Lk; (...) mutex_init(&kmutex, MUTEX_DEFAULT, IPL_NONE); Lk = lunatikL_newstate(KM_SLEEP, &kmutex, MUTEX_DEFAULT); if (Lk == NULL) return ENOMEM;

After creating a new Lunatik state, we need to register it to allow the Lunatik user-space command to dynamically load scripts.

extern lunatik_State *lunatik_registry[]; (...) /* we use this array to register Lunatik states. */ lunatik_registry[ 0 ] = Lk;

Then, we can define a function to bind the user-defined Fibonacci function, using the Lua C API:

static int fibo(lua_State *L) { int *n = (int *) lua_touserdata(L, -1); lua_getglobal(L, "fibo"); lua_pushnumber(L, *n); lua_pcall(L, 1, 1, 0); *n = lua_tonumber(L, -1); return 0; }

...and call the bind through the Lunatik run CFunction function (to handle the mutual exclusion to our Lua state).

int n = 10; int ret = -1; int size = -1; char *result = NULL; ret = lunatik_runcfunction(Lk, fibo, &n, &result, &size); if (ret != 0){ printf("error: %s

", result); lunatik_free(Lk, result, size); } else printf("fibo(10) = %d

", n);

3. Loading scripts from the user-space

# cd /usr/src/sys/lunatik/userspace # ./lunatik 0 Lunatik 0.1 > function fibo(n) if (n == 0) then return 0 elseif (n == 1) then return 1 else return fibo(n-1) + fibo(n-2) end end

Technical Details

NAME Lunatik -- kernel scripting with Lua SYNOPSIS #include <sys/lunatik.h> lunatik_State * lunatik_newstate(lua_Alloc, void *, void *, lunatik_lock_t, lunatik_unlock_t); void lunatik_close(lunatik_State *); inline void lunatik_lock(lunatik_State *); inline void lunatik_unlock(lunatik_State *); inline void * lunatik_alloc(lunatik_State *, size_t); inline void lunatik_free(lunatik_State *, void *, size_t); inline lua_State * lunatik_getluastate(lunatik_State *); int lunatik_runstring(lunatik_State *, const char *, char **, size_t *); int lunatik_runcfunction(lunatik_State *, lua_CFunction, void *, char **, size_t *); int lunatik_registerstate(lunatik_State *, const char *); #include <sys/kmem.h> #include <sys/mutex.h> lunatik_State *lunatikL_newstate(km_flag_t, kmutex_t *, kmutex_type_t); DESCRIPTION Lunatik is a subsystem designed to provide support for scripting the kernel with Lua. typedef void (*lunatik_lock_t)(void *); typedef void (*lunatik_unlock_t)(void *); typedef struct lunatik_State lunatik_State; FUNCTIONS - lunatik_newstate(allocf, ud, mutex, lockf, unlockf) Creates a new instance of the Lua interpreter in-kernel. This instance is called Lunatik State and holds a Lua state descriptor, and the givens data structures and functions. Returns the new Lunatik state allocated by the allocf function. If there is no memory available it returns NULL. - lunatik_close(Lk) Closes a given Lunatik state. The Lunatik state descriptor are freed using the registered allocf function. - lunatik_lock(Lk) Locks a Lunatik state using the registered lock function. - lunatik_unlock(Lk) Unlocks a Lunatik state using the registered unlock function. - lunatik_alloc(Lk, size) Allocates memory using the registered allocator function. If there is no memory available it returns NULL. - lunatik_free(Lk, ptr, size_t) Frees memory using the registered allocator function. - lunatik_getluastate(Lk) Returns the Lua state encapsulated in the given Lunatik state. - lunatik_runstring(Lk, string, resultp, sizep) Runs a string containing Lua code. Returns the execution result on resultp pointer if the pointer is not null. Returns the size of the result on sizep pointer if the pointer is not null. The result is allocated using the allocf registered function, so it must be freed using the lunatik_free call. - lunatik_runcfunction(Lk, cfunction, lightuserdata, resultp, sizep); Runs a lua_CFuntion on the encapsulated Lua state. Returns the execution result on resultp pointer if the pointer is not null. Returns the size of the result on sizep pointer if the pointer is not null. The result is allocated using the allocf registered function, so it must be freed using the lunatik_free call. The lightuserdata argument is put in the Lua stack. - lunatikL_newstate(km_flags, kmutex, kmutex_type) Creates a new Lunatik state using kmem allocator and a initialized given kmutex. The km_flags argument is given to kmem_alloc on each call. The kmutex_type argument determines the lock/unlock functions. Returns the new Lunatik state allocated by the kmem_alloc function. If there is no memory available it returns NULL.