Writing plug-ins in Python

Learn to write your first plug-in by extending a command-line tool in Python

What is a plug-in? One of the more popular examples of the use of plug-ins to extend an application is the Firefox plug-in community. There are Firefox plug-ins for Flash players, Web development, managing RSS feeds, and more. One of the more common uses of plug-ins is to enable third-party developers to extend an application. This is a very powerful idea, as it allows the original developers to harness the power of the unknown to extend their product in ways they may have not even imagined. Please see the Resources section for more information on plug-ins.

In a previous article for IBM developerWorks I wrote about the joy of creating command-line tools with Python. This article takes the command-line tools to the next level by creating plug-ins to extend them. Both plug-ins and command-line tools offer a convenient way to extend the functionality of existing code. Used together, they can be a very powerful tool.

To get started writing the plug-in, you are going to use an open source Python package I wrote called pathtool, which uses generators to walk a filesystem and yield a file object. This library was specifically written to allow developers to extend it by writing their own filters that do something to a file object, and then return the result.

The actual Python module code is a bit larger than you would like to see for an article, so I will only post a snippit of the API that you will actually use:

Listing 1. pathtool API

def path(fullpath, pattern="*", action=(lambda rec: print_rec(rec))): """This takes a path, a shell pattern, and an action callback This function uses the slower pathattr function which calculates checksums """ for rec in pathattr(fullpath): for new_record in match(pattern, rec): #applies filter action(new_record) #Applies lambda callback to generator object

Looking at this example, you can tell that the path function takes a mandatory path-positional argument along with a optional pattern keyword argument, and an optional action keyword argument called lambda callback. The default callback for path just print out the filename as an example. A developer would just need to easy_install pathtool. See the resources section for information on using the easy_install command, and then import the module and call the function as follows:

from pathtool import path path("/tmp", pattern="*.mp3", action=(lambda rec: print_rec(rec)))

Note: I have included the source code for pathtool with this article for added convenience. One thing to point out about this example is the use of lambda. You can read a link to a Python tutorial entry in the resources section about them, but in a nutshell, a lambda is a convenient way to tell a function to "call" another function.

Writing an pluggable command-line tool

Now that we have a general idea of how to use a path-walking library that includes a callback, it is time to jump into actually writing a command-line tool that is extensible using plug-ins. Look at the finished version first, and then I'll break it down into smaller pieces:

Listing 2. Command-line tool with plug-ins

#!/usr/bin/env python # encoding: utf-8 """ pathtool-cli.py 0.1 A commandline tool for walking a filesystem. Takes Action callback plugins in a plugin directory action=(lambda rec: print_rec(rec)) """ from pathtool import path import optparse import re import os import sys try: plugin_available = True from plugin import * from plugin import __all__ #note this is the registered plugin list except ImportError: plugin_available = False def path_controller(): descriptionMessage = """ A command line tool for walking a filesystem.\ Takes callback 'Action' functions as plugins.\ example: pathtool_cli /tmp print_path_ext """ p = optparse.OptionParser(description=descriptionMessage, prog='pathtool', version='pathtool 0.1.1', usage= '%prog [starting directory][action]') p.add_option('--pattern', '-p', help='Pattern Match Examples: *.txt, *.iso, music[0-5].mp3\ plain number defaults to * or match all. \ Uses UNIX standard wildcard syntax.', default='*') p.add_option('--list', '-l', action="store_true", help='lists available action plugins', default=False) options, arguments = p.parse_args() if options.list: try: print "Action Plugins Available:" if plugin_available: for p in __all__: print p finally: sys.exit(0) if len(arguments) == 2: fullpath = arguments[0] try: action_plugin = eval(arguments[1]) #note we expect the plugin author to write a method with our naming convention #path(fullpath,options.pattern,action=(lambda rec: move_to_tmp.plugin(rec))) path(fullpath, options.pattern,action=(lambda rec: action_plugin.plugin(rec))) except NameError: sys.stderr.write("Plugin Not Found") sys.exit(1) else: print p.print_help() def main(): path_controller() if __name__ == '__main__': main()

Running this example results in the following output:

# python pathtool_cli.py Usage: pathtool [starting directory][action] A command line tool for walking a filesystem. Takes callback 'Action' functions as plugins. example: pathtool_cli /tmp print_path_ext Options: --version show program's version number and exit -h, --help show this help message and exit -p PATTERN, --pattern=PATTERN Pattern Match Examples: *.txt, *.iso, music[0-5].mp3 plain number defaults to * or match all. Uses UNIX standard wildcard syntax. -l, --list lists available action plugins

From the output of the command, you can see that the tool expects to take a fullpath, and then an "action." The action is a plug-in that a developer creates. I have added a command-line option list that allows the user of the command-line tool to see what plug-ins are available. Look at the output of this:

# python pathtool_cli.py -l Action Plugins Available: move_to_tmp print_file_path_ext

Without knowing too much about how the tool works, someone can probably guess what an action does by the name of it. Because the print_file_path_ext action I have written just prints the file, path, ext, go ahead and run that and see what it looks like:

# python pathtool_cli.py /tmp print_file_path_ext /tmp/foo0.txt | foo0.txt | .txt /tmp/foo1.txt | foo1.txt | .txt /tmp/foo10.txt | foo10.txt | .txt /tmp/foo2.txt | foo2.txt | .txt /tmp/foo3.txt | foo3.txt | .txt /tmp/foo4.txt | foo4.txt | .txt /tmp/foo5.txt | foo5.txt | .txt /tmp/foo6.txt | foo6.txt | .txt /tmp/foo7.txt | foo7.txt | .txt /tmp/foo8.txt | foo8.txt | .txt /tmp/foo9.txt | foo9.txt | .txt

I created ten temp files using touch foo{0..10}.txt earlier, and now the command-line tool used a plug-in that it found to print the fullpath, the filename, and the extension, all separated by a "|" character.

Simple plug-in architecture explained

Up until now, I have been expecting you to trust me, without telling you where these "magic" plug-in actions are coming from. The first thing to look at is the import statement at the top of the module. Try the following:

plugin_available = True from plugin import * from plugin import __all__ #note this is the registered plugin list except ImportError: plugin_available = False

This import statement gives away the secret to the surprisingly simple plug-in architecture. The official Python documentation generally discourages the use of the syntax "from package import *", but if there is a good reason for it, such as writing plug-ins, then the plug-in author is responsible for creating an entry in the __init__.py file located in the plug-in directory. It should look like this:

"""Lists all of the importable plugins""" __all__ = ["move_to_tmp", "print_file_path_ext"]

By setting this, it allows all of the modules inside of the package, or directory, to be imported as *. Next I import the actual __all__ list to use as a way of showing the user what plug-ins are available. Finally, there is one more small bit of magic. Because the command-line tool doesn't know until runtime what plug-in action to use, use eval to convert the action string on the command-line into a callable function. This is the magic line:

action_plugin = eval(arguments[1])

Generally, the use of eval should be used with extreme caution, but in this case, it is reasonable to tell our tool to use plug-in methods.

A look at a plug-in

Now that you understand how the plug-in architecture is supposed to work, look at an actual plug-in. Note that in order for this architecture to work, there needs to be a plug-in directory in the current working directory or the Python site-packages directory. This particular plug-in is called print_file_path_ext.py, and it has a method called plug-in. This is an expected API that a plug-in developer must conform to:

Listing 3. Example plug-in

#!/usr/bin/env python # encoding: utf-8 """ prints path, name, ext, plugin """ def plugin(rec, verbose=True): """Moves matched files to tmp directory""" path = rec["path"] filename = rec["filename"] ext = rec["ext"] print "%s | %s | %s" % (path, filename, ext)

This plug-in is a very simple function, but it takes a rec parameter that is a dictionary that the pathtool module generates. That dictionary includes the following API:

{"path": path, "filename": file, "ext": ext, "size": size, "unique_id": unique_id, "mtime": mtime, "ctime": ctime}

In this example, I use the keys of the dictionary to print the values for that particular file object each time it is called. A plug-in author could write many other useful actions that could convert files, rename files, archive files, and more.

Summary

This article demonstrated a reasonably simple plug-in architecture that can be a useful way to extend command-line tools in Python. There are a few things that should be noted, though. First, there is a more sophisticated plug-in system available with easy_install, which is included in the references. This plug-in system allows a user to create "entry points" that define plug-ins for a particular tool. Second, the way our command-line tools is written only allows for one "action" plug-in. I will leave it as an exercise to the reader to modify the command-line tool such that it could receive a limitless amount of "chainable" callback actions.

One potential gotcha with creating chainable plug-ins is that the design must take into account the nature of the API it is using. In our case we are building on a foundation of generators that yields. In order for our tool to continue "chaining" plug-ins together, they must do their work, yet still yield the dictionary record back. I hope this article inspired you to write your own plug-ins for command-line tools, as well.

Downloadable resources

Related topics