Intro

November 14, 2010

This blog’s purpose is to describe the process of development of Virgil, Doors and LDX, which are my attempts to make Common Lisp more Windows-friendly.

* What purpose do these libraries serve?

Well, in short – Doors is intended to become a complete binding to the basic Windows API(including COM and OLE), LDX is an attempt to wrap DirectX into lisper-friendly form, and Virgil is an advanced FFI, which Doors and LDX use as a back-end.

* Why invent a new FFI instead of just using CFFI?

CFFI, in my opinion, suffers from several significant design flaws, which cannot be fixed rather easily:

(1) It does not handle translations of aggregate data types, such as structures and arrays, and instead force users to directly manipulate foreign memory. Moreover, there are no “true” aggregate types at all – all of the existing ones are represented as pointers, and because of this, the semantics of mem-ref and (setf mem-ref) operations are highly vague.

(2) And unfortunately, CFFI isn’t extensible enough to allow users of the library to fix this issue. For example, you can not define your own aggregate types and there is no way to extend existing ones.

There are also other minor issues: all unions are treated as aggregates, string translators do not insert BOM while encoding unicode strings and always allocate new lisp string while decoding foreign one, and so on.

However, CFFI has one very significant advantage – it is highly portable.

Virgil uses its primitive types, such as :int, and its basic macros, such as foreign-funcall, and implements an advanced and extensible FFI on top of it. A FFI that is quite similar to CFFI, but strives to address its issues.

“Advanced” means in particular that it can handle translation of lisp aggregate types, such as arrays and structures into foreign memory and back(and can even handle circular references). And “extensible” means that you can extend it in any desirable way, particulary you can even define your own structure and array translators if you are not satisfied with built-in ones.

* Doors

Doors is a binding to MS Windows API.

It strives to provide as lispy interface as possible. Therefore it uses Virgil meaning that most of the winapi function wrappers operate on lisp data – lisp structures, lisp arrays, strings etc. Also, windows error codes(those which are obtained through GetLastError function, or those which are represented by HRESULT type) are translated into lisp conditions.

Also, for the reasons described above i do not use any grovelers and/or binding generators like SWIG, and write all the code by hand.

The bindings are not yet complete, moreover they are far from completion, given how huge winapi is.

Parts of windows api, bindings to which are implemented so far:

Windows console: http://msdn.microsoft.com/en-us/library/ms682010%28v=VS.85%29.aspx

DLLs: http://msdn.microsoft.com/en-us/library/ms682589%28v=VS.85%29.aspx

PSAPI: http://msdn.microsoft.com/en-us/library/ms684884%28v=VS.85%29.aspx

Handles’ stuff: http://msdn.microsoft.com/en-us/library/ms724457%28v=VS.85%29.aspx

And stuff associated with processes: http://msdn.microsoft.com/en-us/library/ms684841%28v=VS.85%29.aspx

Here’s an example. Using PSAPI to find Firefox process, and empty its working set:

( let ((pids (make-array 1024 :element-type 'dword))) ( loop :for i :below (enum-processes pids) :for process = ( ignore-errors ;; System may deny access to some processes. ;; In this case, condition of type WINDOWS-ERROR will be signaled. (open-process '( :query-information :vm-read :set-quota ) (aref pids i))) :when process :do ( when (equalp ( ignore-errors ;; see above (module-base-name process)) "firefox.exe" ) (format t "~&Fat Firefox(PID ~a) uses ~:d KB of physical memory~%" (aref pids i) (floor (process-mc-working-set-size (process-memory-info process)) 1024)) (empty-working-set process) (write-line "...Used." )) (close-handle process))) ;; ==> Fat Firefox(PID 1476) uses 145,356 KB of physical memory ;; ...Used.



Few other examples can be found in my blog on livejournal(which is in Russian, fyi)

* What about COM?

Yes, Doors provides binding to Component Object Model.

The implementation is based on the paper “Modern languages and Microsoft’s component object model”, which describes interoperability with COM in Harlequin Dylan.

In short, both COM-interface wrappers and COM-objects are implemented as CLOS objects. And while interfaces are simple wrappers around com-pointers, objects are ordinary CLOS instances, with the exception that their classes must inherit from COM-OBJECT class. Both interface and object methods are implemented through generic functions, and the method tables for interfaces which are implemented by lisp-side COM-objects contain trampolines which call corresponding generic functions.

Interface wrappers, which classes inherit from UNKNOWN, may be finalized. If so, IUnknown::Release method is called inside the finalizer. But this feature is optional, because we can not determine whether the interface wrapper should be finalized in the general case. Well, at least i doubt it could be determined. If someone has any ideas on this subject, i will gladly accept them.

* LDX

LDX is a binding to Microsoft’s DirectX api. Its development has not yet started, but will begin soon.