tl;dr: If handling interrupts is important, use a SyncManager (not multiprocessing.Manager) to handle shared state

I just hit the learning curve pretty hard with python’s multiprocessing — but I came through it and wanted to share my learnings.

Preliminary Thoughts

The bulk of this post is going to be around using the multiprocess library, but a few preliminary thoughts:

Multiprocessing and Threading is hard (especially in python):

Its starts off all hunky-dory — but trust me, factor in time to hit the wall … hard.

pstree is your friend.

Stop what you are doing, and pick your favorite package manager and install pstree : (for me: brew install pstree )

Its super-useful for dealing with sub-processes, and see what is going on.

In particular pstree -s <string> which searches your branches containing processes that contain the string in the command line. So much better than ps

Output looks like this:

1 2 3 4 5 6 7 8 9 10 11 12 ➜ pstree -s python -+ = 00001 root /sbin/launchd \- + = 00221 jtushman /sbin/launchd \- + = 00446 jtushman /Applications/iTerm.app/Contents/MacOS/iTerm -psn_0_188462 \- + = 07770 root login -fp jtushman \- + = 07771 jtushman -zsh \- + = 46662 jtushman python multi3.py |--- 46663 jtushman python multi3.py |--- 46664 jtushman python multi3.py |--- 46665 jtushman python multi3.py |--- 46666 jtushman python multi3.py \- -- 46667 jtushman python multi3.py

Know your options

There are more then one paralyzation framework / paradigms out there for python. Make sure you pick the right one for you before you dive-in. To name a few:

Important: Unless you are a ninja — do not mix paradigms. For example if you are using the multiprocessing library — do not use threading.locals

The Main Story

Axiom One: All child processes get SIG-INT

Note: I will use SIG_INT, Keyboard Interrupt, and Ctr-C interchangeably

Consider the following:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 from multiprocessing import Process , Manager from time import sleep def f ( process_number ): try : print "starting thread: " , process_number while True : print process_number sleep ( 3 ) except KeyboardInterrupt : print "Keyboard interrupt in process: " , process_number finally : print "cleaning up thread" , process_number if __name__ == '__main__' : processes = [] manager = Manager () for i in xrange ( 4 ): p = Process ( target = f , args = ( i ,)) p . start () processes . append ( p ) try : for process in processes : process . join () except KeyboardInterrupt : print "Keyboard interrupt in main" finally : print "Cleaning up Main"

The abbreviated output you get is as follows:

1 2 3 4 5 6 7 8 9 10 11 ^C Keyboard interrupt in process: 3 Keyboard interrupt in process: 0 Keyboard interrupt in process: 2 cleaning up thread 3 cleaning up thread 0 cleaning up thread 2 Keyboard interrupt in process: 1 cleaning up thread 1 Keyboard interrupt in main Cleaning up Main

The main take aways are:

Keyboard interrupt gets send to each sub process and main execution

the order in which the run is non-determanistic

Axiom Two: Beware multiprocessing.Manager (time to share memory between processes)

If it is possible in your stack to rely on a database, such as redis for keeping track of shared state — I recommend it. But if you need a pure python solution read on:

multiprocessing.Manager bill themselves as:

Managers provide a way to create data which can be shared between different processes. A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.

The key take away there is that the Manager actually kicks off a server process to manage state. Its like it is firing up your own little (not battle tested) private database. And if you Ctr-C your python process the manager will get the signal and shut it self down causing all sorts of weirdness.

Consider the following:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 from multiprocessing import Process , Manager from time import sleep def f ( process_number , shared_array ): try : print "starting thread: " , process_number while True : shared_array . append ( process_number ) sleep ( 3 ) except KeyboardInterrupt : print "Keyboard interrupt in process: " , process_number finally : print "cleaning up thread" , process_number if __name__ == '__main__' : processes = [] manager = Manager () shared_array = manager . list () for i in xrange ( 4 ): p = Process ( target = f , args = ( i , shared_array )) p . start () processes . append ( p ) try : for process in processes : process . join () except KeyboardInterrupt : print "Keyboard interrupt in main" for item in shared_array : # raises "socket.error: [Errno 2] No such file or directory" print item

Try running that and interrupting it was a Ctr-C, you will get a weird error:

You will get a socket.error: [Errno 2] No such file or directory when trying to access the shared_array. And thats because the Manager process has been interrupted.

There is a solution!

Axiom Two: Explicitly use multiprocessing.manangers.SyncManager to share state

and use the signals library to have the SyncManager ignore the interrupt signal (SIG_INT)

Consider the following code:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 from multiprocessing import Process from multiprocessing.managers import SyncManager import signal from time import sleep # initializer for SyncManager def mgr_init (): signal . signal ( signal . SIGINT , signal . SIG_IGN ) print 'initialized manager' def f ( process_number , shared_array ): try : print "starting thread: " , process_number while True : shared_array . append ( process_number ) sleep ( 3 ) except KeyboardInterrupt : print "Keyboard interrupt in process: " , process_number finally : print "cleaning up thread" , process_number if __name__ == '__main__' : processes = [] # now using SyncManager vs a Manager manager = SyncManager () # explicitly starting the manager, and telling it to ignore the interrupt signal manager . start ( mgr_init ) try : shared_array = manager . list () for i in xrange ( 4 ): p = Process ( target = f , args = ( i , shared_array )) p . start () processes . append ( p ) try : for process in processes : process . join () except KeyboardInterrupt : print "Keyboard interrupt in main" for item in shared_array : # we still have access to it! Yay! print item finally : # to be safe -- explicitly shutting down the manager manager . shutdown ()

Main take aways here are:

Explicitly using and starting a SyncManager (instead of Manager)

on its initialization having it ignore the interrupt

I will do a future post on gracefully shutting down child threads (once I figure that out ;–)

Thanks to @armsteady, who showed me the like on StackOverflow (link)