Decorated Concurrency

A simplified parallel computing model for Python. DECO automatically parallelizes Python programs, and requires minimal modifcations to existing serial programs.

Install using pip:

pip install deco

General Usage

Using DECO is as simple as finding, or creating, two functions in your Python program. The first function is the one we want to run in parallel, and is decorated with @concurrent . The second function is the function which calls the @concurrent function and is decorated with @synchronized . Decorating the second function is optional, but provides some very cool benefits. Let's take a look at an example.

@concurrent # We add this for the concurrent function def process_lat_lon ( lat , lon , data ): #Does some work return result @synchronized # And we add this for the function which calls the concurrent function def process_data_set ( data ): results = defaultdict( dict ) for lat in range ( ... ): for lon in range ( ... ): results[lat][lon] = process_lat_lon(lat, lon, data) return results

That's it, two lines of changes is all we need in order to parallelize this program. Now this program will make use of all the cores on the machine it's running on, allowing it to run significantly faster.

What it does

The @concurrent decorator use multiprocessing.pool to parallelize calls to the target function

decorator use multiprocessing.pool to parallelize calls to the target function Indexed based mutation of function arguments is handled automatically, which pool cannot do

The @synchronized decorator automatically inserts synchronization events

decorator automatically inserts synchronization events It also automatically refactors assignments of the results of @concurrent function calls to happen during synchronization events

How it works

For an in depth discussion of the mechanisms at work, we wrote a paper for a class which can be found here.