Experiences of a Confessed LuaJIT Binder – The all in one

It seems I’ve been writing binders, wrappers, frameworks, and the like for the entirety of my 35 years of programming. Why is that? Well, because the tools I use to program are often times not the same as those used to create various libraries, frameworks, and the like. Sometimes it’s just that there is no universal bond between all those disparate libraries that you want to use, so you end up writing some wrappers, to make things look and behave similar.

I’ve written quite a lot about LuaJIT bindings, and produced rather a lot of them myself. Most recently, I was writing a binding for the D-Bus library on Linux. D-Bus is a communications protocol and mechanism meant to support Interprocess Communications. As such, it’s about low latency, speed on read and write, and relative simplicity. You can do everything from pop-up and alert, to logging commands in a persistent log.

Here I want to show an approach that I’ve slowly grown and evolved over the past few years. This approach to writing bindings is written with a few constraints and aspirations in mind:

Provide a low level interface that is as true to the underlying framework as possible.

Porting typical C code to the wrapping should be straight forward and obvious

Provide an interface that supports and leverages Lua idioms and practices

Provide a wrapper that does not require a separate compilation step

That list of constraints is brief, but can cause enough trouble depending on how seriously you take each one of them. So, let’s get to it.

Here I will use the LJIT2dbus project, because it gives a chance to exhibit all of the constraints listed above. Here’s a bit of code:

--dbus.lua local ffi = require("ffi") if not DBUS_INCLUDED then local C = {} require ("dbus-arch-deps"); require ("dbus-address"); require ("dbus-bus"); require ("dbus-connection"); require ("dbus-errors"); require ("dbus-macros"); require ("dbus-message"); require ("dbus-misc"); require ("dbus-pending-call"); require ("dbus-protocol"); require ("dbus-server"); require ("dbus-shared"); require ("dbus-signature"); require ("dbus-syntax"); require ("dbus-threads"); require ("dbus-types"); C.TRUE = 1; C.FALSE = 0;

Yah, ok, not so exciting, just a bunch of ‘require()’ statements pulling in other modules. What’s in one of these modules?

-- dbus-misc.lua local ffi = require("ffi") require("dbus-types") require("dbus-errors") ffi.cdef[[ char* dbus_get_local_machine_id (void); void dbus_get_version (int *major_version_p, int *minor_version_p, int *micro_version_p); ]]

Yes, again, more require() statements. But then, there are those couple of C functions that are defined. The other files have similar stuff in them. This is the basis of the constraint related to making the code familiar to a C programmer.

Let’s look at another file which might turn out to be more illustrative:

local ffi = require("ffi") require("dbus-macros") require("dbus-types") require("dbus-protocol") ffi.cdef[[ /** Mostly-opaque type representing an error that occurred */ typedef struct DBusError DBusError; ]] ffi.cdef[[ /** * Object representing an exception. */ struct DBusError { const char *name; /**< public error name field */ const char *message; /**< public error message field */ unsigned int dummy1 : 1; /**< placeholder */ unsigned int dummy2 : 1; /**< placeholder */ unsigned int dummy3 : 1; /**< placeholder */ unsigned int dummy4 : 1; /**< placeholder */ unsigned int dummy5 : 1; /**< placeholder */ void *padding1; /**< placeholder */ }; ]] local function DBUS_ERROR_INIT() return ffi.new("struct DBusError", { NULL, NULL, TRUE, 0, 0, 0, 0, NULL }); end ffi.cdef[[ void dbus_error_init (DBusError *error); void dbus_error_free (DBusError *error); void dbus_set_error (DBusError *error, const char *name, const char *message, ...); void dbus_set_error_const (DBusError *error, const char *name, const char *message); void dbus_move_error (DBusError *src, DBusError *dest); dbus_bool_t dbus_error_has_name (const DBusError *error, const char *name); dbus_bool_t dbus_error_is_set (const DBusError *error); ]]

So more require() statements, a data structure, and some C functions. And how to use it? For that, let’s look at the bottom of the dbus.lua file.

local Lib_dbus = ffi.load("dbus-1") local exports = { Lib_dbus = Lib_dbus; } setmetatable(exports, { __index = function(self, key) local value = nil; local success = false; -- try looking in table of constants value = C[key] if value then rawset(self, key, value) return value; end -- try looking in the library for a function success, value = pcall(function() return Lib_dbus[key] end) if success then rawset(self, key, value); return value; end -- try looking in the ffi.C namespace, for constants -- and enums success, value = pcall(function() return ffi.C[key] end) --print("looking for constant/enum: ", key, success, value) if success then rawset(self, key, value); return value; end -- Or maybe it's a type success, value = pcall(function() return ffi.typeof(key) end) if success then rawset(self, key, value); return value; end return nil; end, }) DBUS_INCLUDED = exports end return DBUS_INCLUDED

And just for reference, a typical usage of the same:

local dbus = require("dbus") local err = dbus.DBusError(); dbus.dbus_error_init(err); local bus = dbus.dbus_bus_get(dbus.DBUS_BUS_SESSION, err); if (dbus.dbus_bus_name_has_owner(bus, SYSNOTE_NAME, err) == 0) then io.stderr:write("Name has no owner on the bus!

"); return EXIT_FAILURE; end

First case is the creation of the ‘local err’ value.

local err = dbus.DBusError(); dbus.dbus_error_init(err);

The dbus.lua file does not have a function called ‘DBusError()’. All it has done is load a bunch of type and function declarations, to be used by the LuaJIT ffi mechanism. So, how do we get a function of that name? It doesn’t even exist in one of the required modules.

The trick here is in the ‘__index’ function of the dbus.lua table. The way the Lua language works, any time you make what looks like an access to the member of a table, if it can’t be found in the table, and if the __index function is implemented, it will get called, with the key passed in as a parameter.

In this case, the ‘__index’ function implements a series of lookups, trying to find the value that is associated with the specified key. First it tries looking in a table of constants. Then it tries looking in the actual library, for a function with the specified name. If it doesn’t find the value as a function, it will try to find it as a constant in the C namespace. This will find any enum, or static const int values that have been defined in ffi.cdef[[]] blocks. Finally, if it doesn’t find the key as one of the constants, it tries to figure out if maybe it’s a type (‘ffi.typeof’). This is in the case of DBusError, this one will succeed, and we’ll get back a type.

As it turns out, the types returned from the LuaJIT ffi can be used as constructors.

So, what we really have is this:

local errType = dbus.DBusError; local err = errType();

The fact that you can just do it all on one line is a short hand.

This is a great convenience. Also, once the value is found, it is stuck into the dbus table itself, so that the next time it’s used, this long lookup won’t occur. The type is already known, and will just be retrieved from the dbus (exports) table directly.

Well, this works for the other kinds of types as well. For example, the ‘dbus_error_init()’ function is defined in a ffi.cdef[[]] block, and that’s about all. So, when re reference it through dbus.dbus_error_init(), we’re going to look it up in the library, and try to use the function found there. And again, once it’s found, it will be stuffed into the dbus (exports) table, for future reference.

This works out great, in that it’s fairly minimal amount of work to get all interesting things defined in ffi.cdef blocks, then just use this __index lookup trick to actually return the values.

I’ve come to this style because otherwise you end up doing a lot more work trying to make the constants, types, enums, and functions universally accessible from anywhere within your program, without resorting to global values. Of course you can make the __index lookup as complex as you like. You can lazily load in modules, for example.

That’s it for this round. An all in one interface that gives you access to constants, enums, and functions in a ffi.cdef wrapped library.