asyncoro: Asynchronous, Concurrent, Distributed Programming with Python¶

Note asyncoro has been renamed pycos to better reflect its functionality and to avoid confusion with asyncore module.

asyncoro is a Python framework for asynchronous, concurrent, network, distributed programming and distributed computing, using generator functions, asynchronous completions and message passing. asyncoro can be used to create coroutines with generator functions, similar to the way threads are created with functions with Python’s threading module. Programs developed with asyncoro have same logic and structure as programs with threads, except for a few syntactic changes - mostly using yield with asynchronous completions that give control to asyncoro’s scheduler, which interleaves executions of generators, similar to the way an operating system executes multiple processes.

Unlike threads, creating processes (coroutines) with asyncoro is very efficient. Moreover, with asyncoro context switch occurs only when coroutines use yield (typically with an asychronous call), so there is no need for locking and there is no overhead of unnecessary context switches.

asyncoro features include:

For reference purposes, asyncoro with Python 2.7 on Raspberry Pi Model B+ (compliment by James Philips of pyeq2 project) running the program:

import time import asyncoro def coro_proc ( coro = None ): yield coro . suspend () coros = [ asyncoro . Coro ( coro_proc ) for i in xrange ( 10000 )] time . sleep ( 5 ) # wait for coroutines to run (and suspend) for coro in coros : coro . resume () for line in open ( '/proc/self/status' ): if line . startswith ( 'VmPeak' ): print ( 'VM Peak: %d MB' % ( int ( line . split ()[ 1 ]) / 1024 )) elif line . startswith ( 'VmHWM' ): print ( 'VM Max: %d MB' % ( int ( line . split ()[ 1 ]) / 1024 )) elif line . startswith ( 'VmStk' ): print ( 'VM Stack: %d MB' % ( int ( line . split ()[ 1 ]) / 1024 ))

shows that running 10,000 processes takes about 23 MB of memory (with about 10 MB taken by modules used in asyncoro). Memory and time scale linearly: For 100,000 processes memory used is 116 MB, for 200,000 processes it is 218 MB, for 300,000 processes it is 315 MB.

asyncoro works with Python 2.7+ and Python 3.1+ and tested on Linux, Mac OS X and Windows; it may work on other platforms too. asyncoro works with PyPy as well.

Dependencies¶ asyncoro is implemented with standard modules in Python. If psutil is available on nodes, node availability status (CPU, memory and disk) is sent in status messages, and shown in web browser so cluster/application performance can be monitored. Under Windows efficient polling notifier I/O Completion Ports is supported if pywin32 is available; otherwise, inefficient ‘select’ notifier is used. Node / Servers sends node availability status (availability of CPU as percent, memory in bytes and disk space in bytes) at pulse_interval frequency if psutil module is installed. This information may be useful to clients, for example, to analyze application performance, to filter nodes based on available resources etc. For IPv6, netifaces module is required with OS X. netifaces module is strongly recommended with Linux and Windows as well (for both IPv4 and IPv6). For IPv6 with Windows, win_inet_pton module is required with Python 2.7 (but not required with Python 3).

Download/Installation¶ asyncoro package is available in Python Package Index (PyPI) so it can be installed with: python -m pip install asyncoro

Docker Container describes how to build Docker image and run asyncoro modules in containers.

asyncoro can also be downloaded from Sourceforge Files. ‘py2’ directory contains files to be used with Python 2.7+ and ‘py3’ contains files to be used with Python 3.1+. To use asyncoro without installing, rename ‘py2’ to ‘asyncoro’ (to use with Python 2.7+); it can then be used from parent directory of ‘asyncoro’.