Please Visit http://glyph.twistedmatrix.com/ - This Blog Is Closed. Sorry about the ads, I can't turn them off. glyf



Fullmetal Asynchronous



A very small part of this code was "serious work", for establishing a bridge between a Java server and a Python client; however, the actual protocol being spoken was not AMP, I merely needed to decode some AMP data in Java. The rest of it was written over the weekend on a lark, and to see how much Java I remembered from the days of my youth, to see what it's like to write a Java application these days.



It may seem pretty odd that I'm doing hacking in Java for fun, especially after being a



AMP, the Asynchronous Messaging Protocol, is a new protocol that will be first released in the upcoming Twisted version 2.5. The idea behind AMP is fairly simple. In some ways, it's the same idea behind XML-RPC. The gist of the idea behind XML-RPC is that there are a large variety of applications which have need of a protocol library with some properties: extremely simple - a simple protocol means you need to think about what you expose, which means compact, easy-to-understand interfaces.

language neutral - the protocol should have a reasonable set of data types, but not be tied to any specific programming language, to gather the widest possible adoption (and thereby provide the greatest value to implementors, as there will be more services they can access)

easy to deploy - you don't need to allocate a new server, you can just drop new XML-RPC responders in by creating a new file in most cases These properties, it turns out, are amazingly useful for a huge class of applications, and XML-RPC services have cropped up all over the place.



Twisted's main audience is often substantially different from the originally target audience of XML-RPC. Although superficially both are building network services, traditional network services are well-served by ... well, traditional infrastructure, like Apache, MySQL and the plethora of web frameworks available to Python, Perl, PHP, Ruby, Lisp, and so on. Applications that are quirky, hard to classify, and have a bit of a "twist" need Twisted (no pun intended -- okay maybe a little). In fact, much as I wish it were not so, "normal" applications will often hit roadblocks in Twisted where they won't on other platforms, because so much more effort has been spent on edge cases and behavior in exceptional situations than a smooth transition for "average" usage.



Although, for the reasons given above, Twisted applications defy genre to some extent, they do have a few features in common. They are network applications, after all. Many are message-routing or custom control panel applications, which require a two-way monitoring, streaming, or control protocol.

bit-for-bit correct - you can send chunks of data without error-prone encoding or escaping them in a format like base64. Typical applications exchange small chunks of text-based data. Many Twisted applications are exchanging media chunks or encrypted packets, where a botched newline conversion or mismatched codec is a catastrophic failure rather than an intermittent annoyance. Anyone who has used an XML library in the real world can attest to the difficulty of getting this right.



two-way - in many cases, notifications need to be generated and pushed down an active connection, rather than the request/response model offered by XMLRPC. In principle, of course, this is possible with an HTTP-based protocol as well as with a connection-oriented one, but in practice the presence of NATs and firewalls will prevent incoming connections in many situations where outgoing connections are easy.



asynchronicity - the ability to send and receive messages while outstanding requests are pending. The nature of Twisted's event loop is, of course, suited to this, but not all protocols provide this. IMAP4, for example, has several states where the client must implicitly synchronize with the server and halt sending further requests until all of its outstanding requests have been answered, regardless of whether it's possible to send those messages programmatically or not. (This is why I had to write a Reactor and a Deferred for Java - it's very hard to get real two-way communication going when you have to manage large groups of threads.) For quite a while now, PB, or "Perspective Broker" has been providing Twisted users with the latter triad of properties, and more. PB has lots of fun features: distributed garbage collection, string table compression, a pluggable object marshalling framework, dynamic proxies, authentication support, and remote error reporting.



In case that buzzword stew didn't clue you in, PB doesn't exactly have the "extremely simple" property. Unfortunately, nor is it language neutral in a practical way. There are Emacs Lisp, Scheme, and Java implementations of the protocol's encoding, and it was possible to write cross-language applications, but in practice it is prohibitively difficult to declare or replicate all the implicit nuances of Python objects serialized via PB. As an example, most Python objects are dictionaries, and the PB protocol reflects this - it is possible to have an object's state be a dictionary that refers to itself . That configuration is an impossibility in most languages. In some ways, this is a strength; you can simply assemble your objects and ship them over the network, which saves a lot of time during development, especially in protocols which need to communicate about a wide variety of different types.



In short, PB is great for the applications where you need it. It works especially well for prototyping games, and could probably be optimized into an awesome protocol even in production, and it can serve in the cases where you don't really need all of its power. However, over years of working with PB and discussing it with others who use Twisted, I've discovered that there are many applications that need a protocol with both the first, and the second set of properties. There's nothing inherent about wanting to send asynchronous notifications which requires exact, implicit over-the-wire copying of any type of object. It seems to me that there's a fairly widespread need - at least widespread among Twisted users - for a protocol that falls somewhere between PB's incredible power and XML-RPC's militant simplicity.



AMP is an attempt to build that protocol. It is a simple protocol that bridges both sets of requirements. It has some features of PB. Messages are tagged with request/response headers, so that you can easily encapsulate one into a Deferred. It has some features of XML-RPC. Only a few, very simple parameter types are described in the protocol itself, and there's an extremely simple low-level encoding (far simpler than XML) so that you don't have to implement very much at all to get something useful.



A mediocre programmer like yours truly can implement AMP in an hour or two in a language that they are familiar with. It was designed to be language neutral - and this brings me back to the Java implementation - I think that the Java implementation is a good proof of concept that an AMP protocol can be equally idiomatic in non-Python languages. In Python, this is what defining and sending a command to increase the volume on a media player might look like:

class IncreaseVolume:

commandName = 'IncreaseVolume'

arguments = [('howMuch', Integer()]

response = [('currentVolume', Integer())]

# ...

D = ampInstance.callRemote(IncreaseVolume, howMuch=4)

def showVol(result):

print 'Current volume:', result['currentVolume']

D.addCallback(showVol)

in Java, this is a bit more verbose, but I don't feel that Java is a second-class citizen in AMPtown, merely that the language itself is more verbose:

class IncreaseVolume {

class Arguments { public int howMuch; }

class Response { public int currentVolume; }

}

// ...

IncreaseVolume.Arguments ae = new IncreaseVolume.Arguments();

ae.howMuch = 7;

ampInstance.callRemote("IncreaseVolume", ae, new IncreaseVolume.Response())

.addCallback(new Deferred.Callback(Object response) {

System.out.println("Current volume: " +

((IncreaseVolume.Response) response).currentVolume);

});

I hope that in the coming year, AMP will enable a new generation of networked applications. I know that Divmod will be using it for a few. It's dead simple to write a custom chat protocol with it. For example, a whiteboarding application which wanted to mix user text messages with application control messages. Other two-way, asynchronous applications include custom GUIs that update when data is changed, lightweight notification of "push" software or RSS-feed updates, and monitoring software that aggregates and reports on the status of large numbers of devices in real time. It's as hard to give a complete survey of the list of possible AMP applications as it would be to give a complete list of possible socket applications, and I'm sure that the really interesting ones are the ones that haven't occurred to me yet.



In the nearer term, the code that spawned this post can be used for a lightweight bridge between Java and Python applications which need to exchange a small set of messages. The few initial users who are looking at this will probably be using it in a configuration where a Java application needs a feature from Twisted which is not very well supported in Java-land. Virtualizable IMAP4 server support is one such example, but there are others. Rather than try to convince an SVN snapshot of Jython to run Twisted's mainloop, you can run the IMAP4 code in a subprocess from Java and communicate with it over a socket, exchanging only the necessary messages. Since there's a reactor in Java as well as one in Python, the code looks fairly similar at both ends of the wire and the impedance mismatch is minimized. The java reactor can happily run in a thread, and need not interfere with existing Java code. The same is true if you need to make use of a Java library from a Twisted application.



As more AMP implementations are done, these kinds of integrations can be used for multi-language plugin systems. Rather than defining a thick API and loading various scripting systems as shared libraries, your application can define a simple AMP protocol and run subprocesses which are expected to speak it. I believe that would substantially reduce the complexity of most application-scripting systems that want to support multiple languages. That would also let you turn an existing Python scriptability system into one which could connect to an arbitrary language, by spawning that process from python code.



Finally, AMP in more languages means a way to spread the asynchronous Twisted gospel beyond the confines of the Python implementation itself. Python is great, but there are a lot of programmers out there writing a lot of code in a lot of other languages. I think that all of it could be improved by moving away from bundles of threads and back towards a simple, unified, event-driven model. Last week I checked in a branch which implements the kernel of Twisted in Java , including the core and TCP server portions of the reactor API and an AMP IProtocol implementation.A very small part of this code was "serious work", for establishing a bridge between a Java server and a Python client; however, the actual protocol being spoken was not AMP, I merely needed to decode some AMP data in Java. The rest of it was written over the weekend on a lark, and to see how much Java I remembered from the days of my youth, to see what it's like to write a Java application these days.It may seem pretty odd that I'm doing hacking in Java for fun, especially after being a semiprofessional java-hater some years ago. Although I must concede that Java has made some strides in the last few years, my pythonista fans need not worry. I still find Java far more tedious to write than Python, and the development process more cumbersome. I'm not doing this work because I want to switch, but rather, because I want to give AMP some cross-language street cred.AMP, the Asynchronous Messaging Protocol, is a new protocol that will be first released in the upcoming Twisted version 2.5. The idea behind AMP is fairly simple. In some ways, it's the same idea behind XML-RPC. The gist of the idea behind XML-RPC is that there are a large variety of applications which have need of a protocol library with some properties:These properties, it turns out, are amazingly useful for a huge class of applications, and XML-RPC services have cropped up all over the place.Twisted's main audience is often substantially different from the originally target audience of XML-RPC. Although superficially both are building network services, traditional network services are well-served by ... well, traditional infrastructure, like Apache, MySQL and the plethora of web frameworks available to Python, Perl, PHP, Ruby, Lisp, and so on. Applications that are quirky, hard to classify, and have a bit of a "twist" need Twisted (no pun intended -- okay maybe a little). In fact, much as I wish it were not so, "normal" applications will often hit roadblocks in Twisted where they won't on other platforms, because so much more effort has been spent on edge cases and behavior in exceptional situations than a smooth transition for "average" usage.Although, for the reasons given above, Twisted applications defy genre to some extent, they do have a few features in common. They are network applications, after all. Many are message-routing or custom control panel applications, which require a two-way monitoring, streaming, or control protocol.For quite a while now, PB, or "Perspective Broker" has been providing Twisted users with the latter triad of properties, and more. PB has lots of fun features: distributed garbage collection, string table compression, a pluggable object marshalling framework, dynamic proxies, authentication support, and remote error reporting.In case that buzzword stew didn't clue you in, PB doesn't exactly have the "extremely simple" property. Unfortunately, nor is it language neutral in a practical way. There are Emacs Lisp, Scheme, and Java implementations of the protocol's encoding, and it was possible to write cross-language applications, but in practice it is prohibitively difficult to declare or replicate all the implicit nuances of Python objects serialized via PB. As an example, most Python objects are dictionaries, and the PB protocol reflects this - it is possible to have an object's state be a dictionary that refers to itself . That configuration is an impossibility in most languages. In some ways, this is a strength; you can simply assemble your objects and ship them over the network, which saves a lot of time during development, especially in protocols which need to communicate about a wide variety of different types.In short, PB is great for the applications where you need it. It works especially well for prototyping games, and could probably be optimized into an awesome protocol even in production, and it can serve in the cases where you don't really need all of its power. However, over years of working with PB and discussing it with others who use Twisted, I've discovered that there are many applications that need a protocol with both the first, and the second set of properties. There's nothing inherent about wanting to send asynchronous notifications which requires exact, implicit over-the-wire copying of any type of object. It seems to me that there's a fairly widespread need - at least widespread among Twisted users - for a protocol that falls somewhere between PB's incredible power and XML-RPC's militant simplicity.AMP is an attempt to build that protocol. It is a simple protocol that bridges both sets of requirements. It has some features of PB. Messages are tagged with request/response headers, so that you can easily encapsulate one into a Deferred. It has some features of XML-RPC. Only a few, very simple parameter types are described in the protocol itself, and there's an extremely simple low-level encoding (far simpler than XML) so that you don't have to implement very much at all to get something useful.A mediocre programmer like yours truly can implement AMP in an hour or two in a language that they are familiar with. It was designed to be language neutral - and this brings me back to the Java implementation - I think that the Java implementation is a good proof of concept that an AMP protocol can be equally idiomatic in non-Python languages. In Python, this is what defining and sending a command to increase the volume on a media player might look like:in Java, this is a bit more verbose, but I don't feel that Java is a second-class citizen in AMPtown, merely that the language itself is more verbose:I hope that in the coming year, AMP will enable a new generation of networked applications. I know that Divmod will be using it for a few. It's dead simple to write a custom chat protocol with it. For example, a whiteboarding application which wanted to mix user text messages with application control messages. Other two-way, asynchronous applications include custom GUIs that update when data is changed, lightweight notification of "push" software or RSS-feed updates, and monitoring software that aggregates and reports on the status of large numbers of devices in real time. It's as hard to give a complete survey of the list of possible AMP applications as it would be to give a complete list of possible socket applications, and I'm sure that the really interesting ones are the ones that haven't occurred to me yet.In the nearer term, the code that spawned this post can be used for a lightweight bridge between Java and Python applications which need to exchange a small set of messages. The few initial users who are looking at this will probably be using it in a configuration where a Java application needs a feature from Twisted which is not very well supported in Java-land. Virtualizable IMAP4 server support is one such example, but there are others. Rather than try to convince an SVN snapshot of Jython to run Twisted's mainloop, you can run the IMAP4 code in a subprocess from Java and communicate with it over a socket, exchanging only the necessary messages. Since there's a reactor in Java as well as one in Python, the code looks fairly similar at both ends of the wire and the impedance mismatch is minimized. The java reactor can happily run in a thread, and need not interfere with existing Java code. The same is true if you need to make use of a Java library from a Twisted application.As more AMP implementations are done, these kinds of integrations can be used for multi-language plugin systems. Rather than defining a thick API and loading various scripting systems as shared libraries, your application can define a simple AMP protocol and run subprocesses which are expected to speak it. I believe that would substantially reduce the complexity of most application-scripting systems that want to support multiple languages. That would also let you turn an existing Python scriptability system into one which could connect to an arbitrary language, by spawning that process from python code.Finally, AMP in more languages means a way to spread the asynchronous Twisted gospel beyond the confines of the Python implementation itself. Python is great, but there are a lot of programmers out there writing a lot of code in a lot of other languages. I think that all of it could be improved by moving away from bundles of threads and back towards a simple, unified, event-driven model. From: micktwomey Date: September 21st, 2006 09:44 am (UTC) (Link) Interesting I can definitely see a use for this. I've been fishing for something better than XML-RPC but not as er, awkward to use as SOAP for my python <-> .net interop. Hmm, there's a thought, I should really look at writing a C# implementation.



Out of curiosity, did you look at BEEP when you were writing this? There seem to be a lot of similarities.

From: glyf Date: September 21st, 2006 02:57 pm (UTC) (Link) C# implementation Yes! The point of this post was really to get other implementors interested and excited about the protocol. I don't have the time or inclination to do anything with C# in the forseeable future, so if you have a use-case for it, the contribution of an implementation would be much appreciated! There have been a few abortive attempts to port Twisted concepts to C#, but nothing useful yet.



I spent a while trying to think of something that was not mean to say about BEEP, because I do not like it very much at all. So here is the nicest thing I can say about it: BEEP is a protocol for transporting XML. Part of the point of AMP is that (while you can certainly shuttle around XML payloads if you want to) the protocol's basics are not XML, or anything text-based. The protocol is binary, and allows transportation of arbitrary data. You can send PNG or MP3 fragments as easily as you can send XML fragments.



Then there's the complexity issue. BEEP's "core" specification, the very minimal thing you need to implement to get a functioning implementation, is over 50 pages long. The full-featured Python implementation of AMP, which contains copious documentation internally, both API and protocol documentation, is less than half that long.



So, uh, yeah. Forget BEEP! Python/.NET/Java interoperability is only one simple network hack away from reality... you could be an asynchronous integration hero :). There's no compliance suite yet, although there is someone working on one, so if you have any question about the way things work please feel free to ask me. From: micktwomey Date: September 22nd, 2006 10:10 am (UTC) (Link) Re: C# implementation So, uh, yeah. Forget BEEP!



That's what I suspected, I was just curious :)



The small set of working BEEP implementations always struck me a a dead give away that something was amiss.



I can't make any promises, but there is definite interest in good C# <-> python interoperability around here, so I could probably spend some time writing a C# implementation.

From: djfroofy Date: September 21st, 2006 06:06 pm (UTC) (Link) Very nice

// XXX could autoboxing help here? // return new Integer(((Integer)o).intValue() + 1); // Answer: only to a certain degree ... return ((Integer) o).intValue() + 1; Just looking over the source code ... You did a great job using 1.5 features to make java code almost as simple as Python - haha, like that will really ever happen ;) From: glyf Date: September 21st, 2006 09:13 pm (UTC) (Link) Re: Very nice Well, I'm glad you appreciated the effort, at least :).



I don't really have much time to maintain the java implementation. Right now it's a serious struggle to find any time to even get it into a state where I could commit it into Twisted trunk. If you're interested in the code, would you be interested in helping out on that front? From: djfroofy Date: September 22nd, 2006 02:36 pm (UTC) (Link) Re: Very nice Sure. I'd least like to take a crack at it - time permitting. The Java implementation is something I could possibly use at work if it matures enough ... that is, if my coup doesn't go according to plan.