Copyright 2004 (c) Rich Hickey. All rights reserved.

Updates copyright 2008 (c) Nick Levine. All rights reserved.

The use and distribution terms for this software are covered by the Common Public License 1.0, which can be found in the file CPL.TXT at the root of this distribution. By using this software in any fashion, you are agreeing to be bound by the terms of this license. You must not remove this notice, or any other, from this software.

Contents

Introduction

My objective was to provide comprehensive, safe, dynamic and Lisp-y access to Java and Java libraries as if they were Lisp libraries, for use in Lisp programs, i.e. with an emphasis on working in Lisp rather than in Java.

The approach I took was to embed a JVM instance in the Lisp process using JNI. I was able to do this using LispWorks' own FLI and no C (or Java! *) code, which is a tribute to the LW FLI. On top of the JNI layer (essentially a wrapper around the entire JNI API), I built this user-level API using Java Reflection. This first version was built with, and contains code specific to, LispWorks.

jfli ("jay fly") provides:

Automatic function generation for constructors, fields and methods, either by named class, or entire package (sub)trees given a jar file.

Java -> Lisp package and name mapping with an eye towards lack of surprise, lack of conflict, and useful editor completion.

setf-able setter generation for fields as well as for methods that follow the JavaBeans property protocol.

Java array creation and aref-like access to Java arrays.

A 'new' macro that allows for keyword-style field and property initialization.

Typed references to Java objects with an inheritance hierarchy on the Lisp side mirroring that on the Java side - allowing for Lisp methods specialized on Java class and interface types.

Implementation of arbitrary Java interfaces in Lisp, and callbacks from Java to Lisp via those interfaces. (* this required a single 5-line dummy Java proxy stub, provided with jfli)

Automatic lifetime maintenance of Lisp-referenced Java objects, boxing/unboxing of primitive args/returns, string conversions, Java exception handling, overload resolution etc.

jfli was built using LWM and LWW (using Apple's and Sun's JVMs respectively), and it works fine on both. More recently (2007) it's been tested extensively on LWL. It should be a trivial port to other LispWorks, and a possible port to any Common Lisp with a robust FLI. It should also work with any JVM with a conformant JNI implementation.

Download

jfli is hosted on SourceForge.

Setup and Configuration

jfli comprises: a small system of three Lisp files (and an accompanying defsys file), a system building utility (build-java-classes.lisp), and an optional Java .jar file. The first Lisp file, jni.lisp, defines a low-level API to the Java Native Interface, and is not documented here. The second, jfli.lisp, depends upon jni.lisp, and provides the user API documented here. Finally process.lisp provides recyclable processes.

To get started:

(load "defsys.lisp") (compile-system 'jfli :load t) (use-package :jfli) prior to creating the JVM you must tell the library how to find the Java JNI library by setting *jni-lib-path* .

If you wish to allow for callbacks from Java to Lisp, you must place jfli.jar in your classpath when creating the JVM.

Quick Start

This sample session presumes you have already compiled the jfli system.

CL-USER 4 > (load "/lisp/defsys") ; Loading lisp file C:\lisp\defsys.lisp #P"C:/lisp/defsys.lisp" CL-USER 5 > (load-system 'jfli) ; Loading fasl file c:\lisp\jni.ofasl ; Loading fasl file c:\lisp\jfli.ofasl ; Loading fasl file c:\lisp\process.ofasl (JFLI) ;The user API is entirely in the jfli package CL-USER 6 > (use-package :jfli) T ;tell the library where Java is located CL-USER 7 > (setf *jni-lib-path* "/j2sdk1.4.2_01/jre/bin/client/jvm.dll") "/j2sdk1.4.2_01/jre/bin/client/jvm.dll" ;this starts the VM and sets the class path (to the default plus ;jfli.jar, in this case) CL-USER 8 > (connect-jvm ()) 0 #<Pointer: JNI:PVM = #x081022A0> #<Pointer: JNI:PENV = #x0086A858> ;define wrappers for the members of Object CL-USER 9 > (def-java-class "java.lang.Object") NIL ;and of Properties, a Hashtable-like class CL-USER 10 > (def-java-class "java.util.Properties") #<STANDARD-CLASS |java.util|:PROPERTIES. 2066B964> ;the above will create these packages if they do not already exist ;use the packages for easy name access CL-USER 11 > (use-package "java.lang") T CL-USER 12 > (use-package "java.util") T ;create a Properties instance, note keyword-style member inits, string conversion etc ;also note typed return value CL-USER 13 > (setf p (new properties. :getproperty "fred" "ethel")) #<PROPERTIES. 20664A94> ;virtual functions work as normal CL-USER 14 > (object.tostring p) "{fred=ethel}" ;setter was generated for member function because it follows the JavaBeans property protocol CL-USER 15 > (setf (properties.getproperty p "ricky") "lucy") "lucy" CL-USER 16 > (object.tostring p) "{ricky=lucy, fred=ethel}" CL-USER 17 > (properties.size p) 2 ;totally dynamic access, create wrappers as you need CL-USER 18 > (def-java-class "java.lang.Class") #<STANDARD-CLASS CLASS. 20680EC4> CL-USER 19 > (class.getname (object.getclass p)) "java.util.Properties" CL-USER 20 > (def-java-class "java.util.Enumeration") #<STANDARD-CLASS ENUMERATION. 20669274> ;no need to wait for the vendor to enhance the language - you use Lisp! CL-USER 21 > (defmacro doenum ((e enum) &body body) (let ((genum (gensym))) `(let ((,genum ,enum)) (do () ((not (enumeration.hasmoreelements ,genum))) (let ((,e (enumeration.nextelement ,genum))) ,@body))))) DOENUM ;can't do this in Java yet CL-USER 22 > (doenum (prop (properties.elements p)) (print (object.tostring prop))) "lucy" "ethel" NIL ;doc strings are created giving original Java signatures and indicating overloads CL-USER 23 > (documentation 'properties.getproperty 'function) "java.lang.String getProperty(java.lang.String,java.lang.String) java.lang.String getProperty(java.lang.String) " CL-USER 24 > (documentation 'properties.new 'function) "java.util.Properties() java.util.Properties(java.util.Properties) "

API Reference

JVM Creation and Initialization

*jni-lib-path* Set this to point to your jvm dll prior to calling create-jvm. (setf *jni-lib-path* "C:/Program Files/Java/jre1.6.0_05/bin/client/jvm.dll")

Function (create-jvm &rest option-strings) -> unspecified Creates/starts the JVM. You must call this prior to calling any other jfli function, and you cannot subsequently make a further call to either create-jvm or connect-jvm (a Java limitation). The option strings can be used to control the JVM, especially the classpath: (create-jvm "-Djava.class.path=/Lisp/jfli.jar") See the JNI documentation for other initialization options.

Function (connect-jvm class-paths &optional option-strings) -> unspecified Alternative, simpler interface to creating/starting the JVM. class-paths is a list of pathnames (to which jfli.jar will be added, and then each path will each be merged with the jfli source location). The option strings can be used to control other aspects of the JVM. You must call this prior to calling any other jfli function, and you cannot subsequently make a further call to either create-jvm or connect-jvm (a Java limitation). (connect-jvm '("/Lisp/jfli.jar")) See the JNI documentation for other initialization options.

Function (enable-java-proxies) -> unspecified Sets up the Java->Lisp callback support. Must be called (once) before any calls to new-proxy, and requires jfli.jar be in the classpath.

Wrapper Generation

Macro (def-java-class full-class-name) -> unspecified Given the package-qualified, case-correct name of a Java class as a string, will generate wrapper functions for its public constructors, fields and methods. The core API for generation interfaces to Java is the def-java-class macro. This macro will, at expansion time, use Java reflection to find all of the public constructors, fields and methods of the given class and generate functions to access them. The Generated API When you e.g. (def-java-class "java.lang.ClassName") you get several symbols/functions: A package named |java.lang| (note case)

from which the following are exported: A class-symbol: classname. (note the dot is part of the name)

which can usually be used where a typename is required. It also serves as the name of the Lisp typed reference class. Every non-interface class with a public constructor will get; A constructor, (classname.new &rest args) -> typed-reference , which returns a typed reference to the newly created object A method defined on make-new , ultimately calling classname.new , specialized on (the value of) the class-symbol Note that if the constructor is overloaded, there is just one function generated, which handles overload resolution. The function documentation string describes the constructor signature(s) from the Java perspective. The same argument conversions are performed as are for fields (see below). All public fields will get a getter function:

(classname.fieldname [instance]) -> field value

and a setter:

(setf classname.fieldname [instance])

Instance field wrappers take a first arg which is the instance. Static fields get a symbol-macro *classname.fieldname* If the type of the field is primitive, the field value will be converted to a native Lisp value. If it is a Java String, it will be converted to a Lisp string. Otherwise, a generic reference to the Java object is returned. Similarly, when setting, Lisp values will be accepted for primitives, Lisp strings for Strings, or (generic or typed) references for reference types. Every public method will get a wrapper function:

(classname.methodname &rest args) -> return-value

As with constructors, if a method is overloaded a single wrapper is created that handles overload resolution. If a method follows the JavaBeans property protocol (i.e. it is called getSomething or isSomething and there is a corresponding setSomething ), then a (setf classname.methodname) will be defined that calls the latter. The same argument and return value conversions are performed as are for fields. The function documentation string describes the method signature(s) from the Java perspective. A Lisp class with the class-symbol as its name. It will have as its superclasses other Lisp classes corresponding to the Java superclass/superinterfaces, some of which may be forward-referenced-classes. An instance of this class will be returned by classname.new/make-new/new, at which point the entire hierarchy will consist of finalized standard-classes. Note that, due to the need to reference other Java types during the definition of a class wrapper, symbols, classes, and packages relating to those other types may also be created. In all cases they will be created with names and packages as described above.

When you e.g. you get several symbols/functions: Function (get-jar-classnames jar-file-name &rest packages) -> list-of-strings Returns a list of class name strings. Packages should be strings of the form "java/lang " for recursive lookup and "java/util/" (note trailing slash) for non-recursive.

Function (dump-wrapper-defs-to-file filename classnames) -> unspecified Given a list of classnames (say from get-jar-classnames ), writes calls to def-java-class to a file: (dump-wrapper-defs-to-file "/lisp/java-lang.lisp" (get-jar-classnames "/j2sdk1.4.2_01/jre/lib/rt.jar " "java/lang/")) (compile-file "/lisp/java-lang") (load "/lisp/java-lang") (use-package "java.lang") ;Wrappers for all of java.lang are now available

Function Function (cl-user:build-java-classes output-file &optional class-paths option-strings) -> unspecified Macroexpands def-java-class forms to sufficient extent that the results can be compiled and loaded into an image which has not yet been connected to the JVM. The utility connects to the JVM, generates source code and writes this to output-file which for example could be one of the members of your application's system. You can subsequently build that system in a fresh lisp, save the image and only connect that lisp to the JVM when the image is restarted. This function is defined in the utility file build-java-classes.lisp which is not part of the JFLI defsystem. To use it to generate forms and write them to output-file: Place your def-java-class forms in the file whose name is that of output-file with ".src" tagged onto the end Start a fresh image and load build-java-classes.lisp (interpreted) Call cl-user:build-java-classes ; class-paths and option-strings will be passed to connect-jvm The utility does string searches for "(def-java-class " and then calls read-from-string ; the rest of the file - comments, in-package forms, etc - is output as-is. If you call build-java-classes again (and you may), class-paths and option-strings will be ignored. Note that loading build-java-classes.lisp loads the JFLI system and then redefines parts of it; the consequences of using this image for any JFLI use other than calls to build-java-classes are undefined. CL-USER 1 > (load "/home/nick/p4/project/frob/master/code/jfli/build-java-classes.lisp") ; Loading text file /home/nick/p4/project/frob/master/code/jfli/build-java-classes.lisp ; Loading text file /home/nick/p4/project/frob/master/code/jfli/defsys.lisp ;; Creating system JFLI ; Loading fasl file /home/nick/p4/project/frob/master/code/jfli/jni.ufasl ; Loading fasl file /home/nick/p4/project/frob/master/code/jfli/jfli.ufasl ; Loading fasl file /home/nick/p4/project/frob/master/code/jfli/process.ufasl #P"/home/nick/p4/project/frob/master/code/jfli/build-java-classes.lisp" CL-USER 2 > (build-java-classes "../wombat/java-classes.lisp" (directory "../wombat/*.jar")) ;; Reading /home/nick/p4/project/frob/master/code/wombat/java-classes.lisp.src ;; 12 def-java-class forms processed "../wombat/java-classes.lisp" CL-USER 3 >



Object Creation

Generic Function (make-new class-symbol &rest args) -> typed-reference Allows for definition of before/after methods on constructors. Calls classname.new . The new macro expands into a call to this.

Macro (new class-spec &rest args) -> typed-reference

class-spec -> class-name | (class-name this-name)

class-name -> "package.qualified.ClassName" | classname.

args -> [actual-arg]* [init-arg-spec]*

init-arg-spec -> init-arg | (init-arg)

init-arg -> :settable-field-or-method [params]* value (note keyword)

| .method-name [args]* (note leading dot)

Creates a new instance of class-name, by expanding into a call to the make-new generic function, then initializes it by setting fields or accessors and/or calling member functions. If this-name is supplied, it will be bound to the newly-allocated object and available to the init-args: (new (button. this) shell *SWT.CENTER* ;the actual args :gettext "Call Lisp" ;a javabean property (.addlistener *swt.selection* ;a method call (new-proxy (listener. (handleevent (event) (declare (ignore event)) (setf (button.gettext this) ;this is bound to new instance (format nil "~A ~A" (lisp-implementation-type) (lisp-implementation-version))) nil)))) .setsize 200 100 ;can omit parens (.setlocation 40 40)) Expands into: (LET* ((#:G598 (MAKE-NEW BUTTON. SHELL *SWT.CENTER*)) (THIS #:G598)) (SETF (BUTTON.GETTEXT #:G598) "Call Lisp") (BUTTON.ADDLISTENER #:G598 *SWT.SELECTION* (NEW-PROXY (LISTENER. (HANDLEEVENT (EVENT) (DECLARE (IGNORE EVENT)) (SETF (BUTTON.GETTEXT THIS) (FORMAT NIL "~A ~A" (LISP-IMPLEMENTATION-TYPE) (LISP-IMPLEMENTATION-VERSION))) NIL)))) (BUTTON.SETSIZE #:G598 200 100) (BUTTON.SETLOCATION #:G598 40 40) #:G598)

Array Support

Generic Function (make-new-array type &rest dimensions) -> reference to new array Generic function with methods defined for all Java class designators: A "package.qualified.ClassName" string (the value of) A class-symbol - classname. A primitive designator keyword - :boolean|:byte|:char|:double|:float|:int|:long|:short Creates a Java array of the requested type with the requested dimensions.

Function (jlength array) -> integer Like length, for Java arrays

Function (jref array &rest subscripts) -> reference Like aref, for Java arrays of non-primitive (reference) types, settable.

Function (jref-xxx array &rest subscripts) -> value Where xxx = boolean|byte|char|double|float|int|long|short. Like jref, for Java arrays of primitive types, settable.

Proxies - Java calling back to Lisp

Proxies allow the creation of Java objects that implement one or more interfaces in Lisp, and thus callbacks from Java to Lisp. You must call enable-java-proxies before using this proxy API. A significant limitation is that LispWorks appears to not support calls back into Lisp other than from threads initiated by Lisp, so you must ensure that the proxy will not be called from an arbitrary Java thread! Macro (new-proxy &rest interface-defs) -> reference interface-def -> (interface-name method-defs+)

interface-name -> "package.qualified.ClassName" | classname. (must name a Java interface type)

method-def -> (method-name arg-defs* body)

arg-def -> arg-name | (arg-name arg-type) arg-type -> "package.qualified.ClassName " | classname. | :primitive

method-name -> symbol | string (matched case-insensitively) Creates, registers and returns a Java object that implements the supplied interfaces

Function (unregister-proxy proxy) -> unspecified Stops handling for the proxy (which must have been created by new-proxy ) and removes references from the Lisp side. Make sure it is no longer referenced from Java first!

Utilities

Function (jeq obj1 obj2) -> boolean Are the 2 java objects the same object? Note that this is not the same as Object.equals()

Function (find-java-class class-sym-or-string) -> reference to Java Class object Given a Java class designator, returns the Java Class object. Use this in preference to Class.forName() when using jfli.

Function (make-typed-ref java-ref) -> typed-reference Given a generic Java reference, determines the full type of the object and returns an instance of a typed reference wrapper. classname.new/make-new/new always return typed references, but since Java methods might return Object or some interface type, and we don't want to always incur the cost of type determination, field and method wrapper functions return generic references. Use this function to create a typed reference corresponding to the full actual type of the object when desired.

Function (box-xxx value) -> reference to Java primitive wrapper class Where xxx = boolean|byte|char|double|float|int|long|short|string. Given a compatible Lisp value, creates an instance of the corresponding Java primitive wrapper class, e.g. Integer. This should rarely be needed, but can be used to force overloading resolution.

Function (unbox-xxx ref) -> Lisp value Where xxx = boolean|byte|char|double|float|int|long|short|string. Given an instance of a Java primitive wrapper class, creates an instance of the corresponding compatible Lisp value. This should rarely be needed, but can be used to unbox values returned by Java Object-based APIs.

Threading

This is one area of jfli which needs more attention. The JNI call DetachCurrentThread has proved most problematic (SEGVs in foriegn code, random corruption in Java internals and heaven only knows what else) and so we need another solution to prevent used threads from "leaking". The solution proposed, to recycle lisp threads rather than letting them go, is implemented in the file process.lisp which can be used as a standalone facility indepenently of the jfli.

Function (mp:cached-process-run-function name keywords function &rest args) -> mp:process As for mp:process-run-function except that when the process terminates it is cached for later reuse (by a subsequent call to mp:cached-process-run-function) rather than dropped. It is recommended that all threads which are going to communicate with the JVM should be created using mp:cached-process-run-function. This applies in particular if your application uses lots of short-lived threads to talk to Java.

Error Handling

Condition Type java-exception An instance of this subtype of error will be signalled if an exception occurs in the JVM.

Function (java-exception-exception java-exception) -> reference Reader returning the Java exception whose occurance signalled java-exception .

Generic Function (describe-exception java-exception) -> string This function is called when the java-exception is signalled to generate a text description of the exception which is then stored in the exception and used by its print-object method (if *print-escape* is off). In other words, this is how the exception princ s. A default method is defined on java-exception ; you might want to define others.

Summary

I hope you find jfli useful. It is my sincere intent that it enhance the utility and interoperability of Common Lisp, a language with which I am still becoming familiar, and grow to appreciate more every day. I welcome comments and code contributions.

Rich Hickey, July 2004

Updated by Nick Levine, April 2008