In this article I want to explore Dart’s native C API. I didn’t notice an SQLite package for Dart so this might be an interesting (and hopefully not too difficult) example. As a company developing Flutter applications, we’re also interested in using Dart for server-side development, so the result might be actual useful.

Therefore, this is another instalment in my “how hard can it be” series of experiments. Last time I refrained from using Dart’s native C (or C++) API, but this time, let’s dive in!

Here is the plan:

Because my knowledge of C is rusty and my knowledge of the SQLite C-API non-existent, I will start by creating a minimal C program to learn how to access a database file and perform a query. Then, I will create a minimal native Dart library following the instructions of this article. This is mainly to learn how to setup everything. Lastly, I will combine the learnings of both parts and actually create the sqlite native library. Perhaps, I will also write some Dart code to provide an API similar to that of the great sqflite package.

The C-API

Skimming over this API documentation, it seems that I only need to implement three function calls:

sqlite3_open to open an existing database or creating a new one

to open an existing database or creating a new one sqlite3_close when I want to stop using the database

when I want to stop using the database sqlite3_exec to prepare an SQL statement, execute it and step through the result, gathering all column values for each row and freeing all allocated resources again

This is great news. I thought it might be more difficult to interface SQLite.

The C program

Let’s Start Simple

Here is a minimal C program to open and close a database called test.db . I rather unimaginative call this file test.c .

#include <stdio.h>

#include <sqlite3.h> #define CHECK(x) printf("%d

", (x) == SQLITE_OK) int main(int argc, char **argv) {

sqlite3 *db; CHECK(sqlite3_open("test.db", &db));

CHECK(sqlite3_close(db));

}

There are actually three variants of sqlite3_open but I figure, the most basic one is fine. There are also two variants of sqlite3_close and here, it looks like that sqlite3_close_v2 might actually be the better choice.

Make It So

To automate the build process, I create a simple Makefile . I’m on macOS but I think, it should work on Unix-like systems, too. Unfortunately, I’m not familiar with Windows so those readers have to figure it out on their own.

test: test.c

cc -Wall -lsqlite3 -o $@ $<

This will make test from test.c by running cc (the default C compiler) with all warnings enabled and with linking against the (preinstalled) SQLite library. The $@ variable, so much make file magic I remember, stands for the target name — that is test — and the variable $< stands for the first source file provided after the : . Without the -o option cc would call the executable a.out .

On my Mac, the SQLite database library is preinstalled and I don’t have to deal with include paths or library paths and my program builds (rather unexpected) cleanly on the first attempt.

Running test prints 1 two times which is expected as this is C telling me, that both calls succeeded. I also see a new test.db file in my current directory.

Let’s add another rule to the Makefile to clean up:

test: test.c

cc -Wall -lsqlite3 -o $@ $< clean:

rm -f test test.db

Once More With Functionality

So far, nothing really happened. Therefore, let’s use sqlite3_exec to execute some SQL statements. Used to Dart, I actually failed to see my error — adding a trailing comma after &error — for quite some time. After I removed it, the following code eventually compiled successfully:

int callback(void *ctx, int ncols, char **vals, char **cols) {

return 0;

} int main(int argc, char **argv) {

sqlite3 *db; CHECK(sqlite3_open("test.db", &db)); char *error;

CHECK(sqlite3_exec(

db,

"select foo from bar",

callback,

NULL,

&error

));

if (error) {

printf("Error: %s

", error);

sqlite3_free(error);

}



CHECK(sqlite3_close(db));

}

According to the documentation, this will prepare and execute the hardcoded select statement, calling callback for each row of the result. The ctx parameter will get whatever I pass instead of NULL . The error variable is used to tell the program if an error occurred, which is a heap-allocated C string I have to deallocate.

Running the program after compiling it, prints

1

0

Error: no such table: bar

1

which of course is expected as I have to such table yet. Of course, I can create it using the command line sqlite3 tool which is what I do next:

$ sqlite3 test.db

SQLite version 3.24.0 2018-06-04 14:10:15

Enter ".help" for usage hints.

sqlite> create table bar (

id integer primary key autoincrement,

foo text);

sqlite> insert into bar (foo) values ('hello');

sqlite> insert into bar (foo) values ('world');

sqlite> select * from bar;

1|hello

2|world

sqlite>

Running test again will now print three 1 . The error is gone.

Don’t Call Us, We Call You

Let’s implement the callback function which is called by sqlite3_exec for each row of the result generated by the select statement. For now, I’m just printing the column names and associated values.

int callback(void *ctx, int ncols, char **vals, char **cols) {

for (int i = 0; i < ncols; i++) {

printf("%s=%s

", cols[i], vals[i]);

}

return 0;

}

This prints (as expected) the values I inserted manually just a minute ago:

1

foo=hello

foo=world

1

1

The documentation mentions that if the callback returns something other than zero, the sqlite3_exec call would be aborted. Good to know, but right now, not needed.

The Dart Side

I feel, I now know how to use SQLite from C. So let’s switch gears and concentrate on how to implement native C libraries for Dart.

First, I setup a Dart project called dartsqlite like so:

$ mkdir dartsqlite

$ cd dartsqlite

$ mkdir bin lib src

$ touch bin/main.dart lib/sqlite.dart src/sqlite.c

$ echo name: dartsqlite >pubspec.yaml

Then, in sqlite.dart , I declare a single open function as a native function with the same name. According to the documentation, I have use an import statement to import a library from dart-ext which I have to compile as a shared (or dynamic) library, again called sqlite . It will be created from a source file called sqlite.c in just a moment.

library sqlite; import 'dart-ext:sqlite'; int open(String name) native "open";

In main.dart I will call open as follows:

import 'package:dartsqlite/sqlite.dart' as sqlite; void main() {

print(sqlite.open('test.db'));

}

Running this throws an error (which I think has a spelling error because of a missing whitespace, but perhaps I’m the only one who finds this ironic).

dlopen(libsqlite.dylib, 1): image not founderror: library handler failed

And now I know how the library must be called — at least on macOS. For Unix-like systems it’s probably libsqlite.so and I’d guess on Windows it’s something like sqlite.dll .

The most important piece of the puzzle is sqlite.c which contains the following boilerplate code. For a detailed explanation, see below.

#include <string.h>

#include "dart_api.h" Dart_NativeFunction ResolveName(Dart_Handle name, int argc, bool *auto_setup_scope); DART_EXPORT Dart_Handle sqlite_Init(Dart_Handle parent_library) {

if (Dart_IsError(parent_library)) {

return parent_library;

} Dart_Handle result_code =

Dart_SetNativeResolver(parent_library, ResolveName, NULL);

if (Dart_IsError(result_code)) {

return result_code;

} return Dart_Null();

} Dart_Handle HandleError(Dart_Handle handle) {

if (Dart_IsError(handle)) {

Dart_PropagateError(handle);

}

return handle;

} // --------------------------------------------------------------- void NativeOpen(Dart_NativeArguments arguments) {

Dart_SetReturnValue(arguments, Dart_NewInteger(42));

} // --------------------------------------------------------------- Dart_NativeFunction ResolveName(Dart_Handle name, int argc, bool *auto_setup_scope) {

if (!Dart_IsString(name)) {

return NULL;

}

if (auto_setup_scope == NULL) {

return NULL;

}



Dart_EnterScope(); const char *cname;

HandleError(Dart_StringToCString(name, &cname)); Dart_NativeFunction result = NULL; if (strcmp("open", cname) == 0) result = NativeOpen; Dart_ExitScope();

return result;

}

Some comments:

I include string.h because I’m using strcmp , a standard C function to compare strings and otherwise my -Wall setting would cause compile errors.

because I’m using , a standard C function to compare strings and otherwise my setting would cause compile errors. I include dart_api.h which defines the native Dart API.

which defines the native Dart API. The next line is a forward declaration of ResolveName , the function that will be registered with Dart to resolve the native functions, that is look them up by name. In my case, Dart will look for "open" because that is how I called my open function.

, the function that will be registered with Dart to resolve the native functions, that is look them up by name. In my case, Dart will look for because that is how I called my function. sqlite_Init is the entry point which is called by the Dart VM after the native library is loaded. It must have the same name as the library (as given in the Dart import 'dart-ext:…' statement followed by _Init .

is the entry point which is called by the Dart VM after the native library is loaded. It must have the same name as the library (as given in the Dart statement followed by . The initialisation function performs a few sanity checks and then returns null on success after registering the resolver function.

on success after registering the resolver function. HandleError is a utility function which checks whether a given Dart object (represented on the C side as a Dart_Handle thingy) is an error. If this is the case, that error is thrown and the execution of that function is aborted (probably doing some kind of long jump I guess). Otherwise, the value is returned unchanged and execution can proceed. All calls into the Dart API should be wrapped by this function because all calls can cause exception that must then be propagated.

is a utility function which checks whether a given Dart object (represented on the C side as a thingy) is an error. If this is the case, that error is thrown and the execution of that function is aborted (probably doing some kind of long jump I guess). Otherwise, the value is returned unchanged and execution can proceed. All calls into the Dart API should be wrapped by this function because all calls can cause exception that must then be propagated. NativeOpen is my implementation and the only part of this file that isn’t boilerplate code. Each native function must have the same signature as defined by Dart_NativeFunction . Such a function returns nothing and takes a Dart_NativeArguments object as its only argument. To return a value to the Dart side, Dart_SetReturnValue can be used. In my case, I’m returning the simplest thing I could come up with: 42 .

is my implementation and the only part of this file that isn’t boilerplate code. Each native function must have the same signature as defined by . Such a function returns nothing and takes a object as its only argument. To return a value to the Dart side, can be used. In my case, I’m returning the simplest thing I could come up with: . Last but not least there is ResolveName , which is called once for every unresolved native function and is supposed to return a pointer to a native function. There are again sanity checks I copied from the example. The given name, which is a Dart string, is then converted into a C string and compared to "open" . If the comparison is successful, I return a pointer to my function, otherwise NULL is returned and the Dart VM will throw an exception on the Dart side.

, which is called once for every unresolved native function and is supposed to return a pointer to a native function. There are again sanity checks I copied from the example. The given name, which is a Dart string, is then converted into a C string and compared to . If the comparison is successful, I return a pointer to my function, otherwise is returned and the Dart VM will throw an exception on the Dart side. I didn’t understand the auto_setup_scope variable so I ignored it. I’m also not 100% sure when to use scopes and when not. I’d welcome some additional explanations.

To compile everything, I create the following Makefile as before. I had to search for dart_api.h in my Dart implementation (installed via Homebrew according to this setup instructions). You propably have to adapt that path to your system. The documentation also asked me to define a DART_SHARED_LIB variable.

INCLUDE=/usr/local/Cellar/dart/2.2.0/libexec/include lib/libsqlite.dylib: src/sqlite.c

cc -Wall -I${INCLUDE} -DDART_SHARED_LIB -c $<

After I made this run successfully (which took a few attempts), I get a .o object file but not yet a shared (or dynamic) library. It’s probably possible to do this in a single line, but I simply added another compiler call to my make file like so:

INCLUDE=/usr/local/Cellar/dart/2.2.0/libexec/include lib/libsqlite.dylib: src/sqlite.c

cc -Wall -I${INCLUDE} -DDART_SHARED_LIB -c $<

cc -dynamiclib -undefined dynamic_lookup -arch x86_64\

-o $@ sqlite.o

This line looks different on Unix-like systems and you must certainly adapt it for creating a DLL on Windows. For macOS, I have to pass -dynamiclib to the linker and add -undefined dynamic_lookup because the documentation says so. I also specify the architecture (I have a 64-bit machine) and last but not least the name (and location) of my library. It should go into the same folder as the associated dart library file sqlite.dart .

Finally, if the stars are right, calling make shall create the dynamic library and calling dart bin/main.dart shall print 42 . Time to celebrate. I successfully extended my Dart VM to return the answer to life, the universe and everything.

$ make

cc -Wall -I/usr/local/Cellar/dart/2.2.0/libexec/include -DDART_SHARED_LIB -c src/sqlite.c

cc -dynamiclib -undefined dynamic_lookup -arch x86_64 -o lib/libsqlite.dylib sqlite.o

$ dart bin/main.dart

42

Calling SQLite

The rest is easy – I hope.

I will add Dart functions to not only open a database file but also to close it again and to execute SQL statements and then implement them exactly as shown in the first C example test.c .

I didn’t find an official way to pass a foreign C pointer in the Dart world, so I will simply use an integer, hoping that the Dart VM will not change 64-bit values. To return the result of a query, I will use a List<List<String>> . The first element should be a list of column names, all other elements are rows with column values.

Here is the definition in Dart:

library sqlite; import 'dart-ext:sqlite'; int open(String name) native "open";

void close(int db) native "close";

List<List<String>> exec(int db, String sql) native "exec";

open

To actually open the database, I need to extract the name argument and convert it into a C string and call sqlite3_open . If the call is successful, I return the pointer to the sqlite3 struct as a 64 bit unsigned integer back to the Dart side. Otherwise null is returned. I wrap everything in a new scope because the official example code does the same.

void NativeOpen(Dart_NativeArguments arguments) {

Dart_EnterScope(); Dart_Handle name = HandleError(Dart_GetNativeArgument(arguments, 0));

const char *cname;

HandleError(Dart_StringToCString(name, &cname)); sqlite3 *db; Dart_SetReturnValue(arguments, Dart_Null()); if (sqlite3_open(cname, &db) == SQLITE_OK) {

Dart_SetReturnValue(arguments,

Dart_NewIntegerFromUint64((uint64_t)db));

} Dart_ExitScope();

}

close

To call sqlite3_close , I extract the db argument and convert it from an 64-bit unsigned integer back into a sqlite3 struct pointer. For now, I don’t check whether the SQLite API call was successful or not because I’m not really sure what to do if this isn’t successful. Raising an exception? I also noticed that I probably should check for a passed null value.

void NativeClose(Dart_NativeArguments arguments) {

Dart_EnterScope(); Dart_Handle handle = HandleError(Dart_GetNativeArgument(arguments, 0));



sqlite3 *db;

HandleError(Dart_IntegerToUint64(handle, (uint64_t *)&db));



sqlite3_close_v2(db); Dart_ExitScope();

}

exec

To execute an SQL statement, I extract the db and the sql arguments as before. I could now call the callback function and create a Dart List object and populate it with column names or column values. There’s an API for that. But I didn’t find a way to create an expandable list, though, which I need to collect all the rows. I think, there’s no way to know in advance how many rows the result will hold. The Dart API allows me only to create fixed-length lists. After struggling with the native API for some time the simplest way I found was to simply pass a List instance to the function. This means, I change the Dart function as follows

List exec(int db, String sql, List result) native "exec";

and then use this C implementation:

int callback(void *ctx, int ncols, char **vals, char **cols) {

Dart_Handle list = ctx; Dart_Handle row = HandleError(Dart_NewList(ncols)); intptr_t length;

HandleError(Dart_ListLength(list, &length));

if (!length) {

for (int i = 0; i < ncols; i++) {

Dart_Handle val =

HandleError(Dart_NewStringFromCString(cols[i]));

HandleError(Dart_ListSetAt(row, i, val));

} Dart_Handle name =

HandleError(Dart_NewStringFromCString("add"));

HandleError(Dart_Invoke(list, name, 1, &row)); row = HandleError(Dart_NewList(ncols));

} for (int i = 0; i < ncols; i++) {

Dart_Handle val =

HandleError(Dart_NewStringFromCString(vals[i]));

HandleError(Dart_ListSetAt(row, i, val));

} Dart_Handle name =

HandleError(Dart_NewStringFromCString("add"));

HandleError(Dart_Invoke(list, name, 1, &row)); return 0;

} void NativeExec(Dart_NativeArguments arguments) {

Dart_EnterScope(); Dart_Handle handle = Dart_GetNativeArgument(arguments, 0); sqlite3 *db;

HandleError(Dart_IntegerToUint64(handle, (uint64_t *)&db)); Dart_Handle sql = Dart_GetNativeArgument(arguments, 1);

const char *csql;

HandleError(Dart_StringToCString(sql, &csql)); Dart_Handle list = Dart_GetNativeArgument(arguments, 2); char *error;

sqlite3_exec(db, csql, callback, list, &error);

if (error) {

Dart_Handle exception =

HandleError(Dart_NewStringFromCString(error));

sqlite3_free(error);

HandleError(Dart_ThrowException(exception));

} Dart_SetReturnValue(arguments, list); Dart_ExitScope();

}

There are two interesting points:

I’m calling the add method for List using Dart_Invoke which was surprisingly easy. I hope, I don’t have to somehow free my add string.

method for using which was surprisingly easy. I hope, I don’t have to somehow free my string. If there’s an SQL error, I’m converting it to a Dart string and raise it as an exception. Again, I hope this doesn’t mess with the required manual memory management.

I’m not sure what will happen, if a HandleError call inside the callback function will decide to throw an exception but hopefully, this will never happen and the VM will always have enough memory to allocate all those Dart strings.

Run, Dart, Run

But let’s stop worrying and test the finished application, after I have added these lines to the ResolveName function:

if (strcmp("open", cname) == 0) result = NativeOpen;

if (strcmp("close", cname) == 0) result = NativeClose;

if (strcmp("exec", cname) == 0) result = NativeExec;

Here is my new implementation of main.dart :

import 'package:dartsqlite/sqlite.dart' as sqlite; void main() {

int db = sqlite.open('test.db');

print(sqlite.exec(db, 'select * from bar', []));

sqlite.close(db);

}

I make the library and run the Dart application and after fixing List<List<String>> to List because I again stumbled about Darts static types and I don’t want to handle this problem right now, I see:

[[id, foo], [1, hello], [2, world]]

I feel great because I wrote my first native Dart library — and it actually works!

How to Proceed from Here

There’s room for improvement, though. Running an insert statement should somehow return the autoincrement value but this seems not possible with the simplified SQLite API I’m using. Also, I can’t pass arguments to SQL statement and all returned values are of type string. It’s probably time to drive deeper into the SQLite API.

I also wonder whether using a Dart closure would be a better implementation of callback because then, I could write more Dart code which is always a good thing. I would then probably pass a Map of column names to column values to the closure.

I would also love to try out Dart’s asynchronous API variant. Right now, all calls are performed synchronously and this might be a problem with larger queries if I actually want to use this library for server-side development.

These are all interesting topics for another article, though.

Thanks for reading this article and please check out my other articles, too.