Please Visit http://glyph.twistedmatrix.com/ - This Blog Is Closed. Sorry about the ads, I can't turn them off. glyf



Functional Functions and the Python Singleton Unpattern

subscribers = []



def addSubscriber(subscriber):

subscribers.append(subscriber)



def publish(message):

for subscriber in subscribers:

subscriber.notify(message)

And then used it like this?

from publisher import publish



class worker:

def work(self):

publish(self)

I've done this many times myself.



I used to think that this was the "right" way to implement Singletons in Python. Other languages had static members and synchronized static accessors and factory methods; all kinds of rigamarole to achieve this effect, but Python simply had modules.



Now, however, I realize that there is no "right" way to implement Singleton in Python, because singletons are simply a bad thing to have. As



The module above is brittle, and as a result, unpleasant to test and extend.



It's difficult to test because the call to "publish" cannot be indirected without monkeying around with the module's globals - generally recognized to be poor style, and prone to errors which will corrupt later, unrelated tests.



It makes code that interacts with it difficult to test, because while you can temporary mangle global variables in the most egregious of whitebox tests, tests for code that is further away shouldn't need to know about the implementation detail of "publish". Furthermore, code which adds subscribers to the global list will destructively change the behavior of later tests (or later code, if you try to invoke your tests in a running environment, since we all know running environments are where the interesting bugs occur).



It's difficult to extend because there is no explicit integration point with 'publish', and all instances share the same look-up. If you want to override the behavior of "work" and send it to a different publisher, you can't call to the superclass's implementation.



Unfortunately, this probably doesn't seem particularly bad, because bad examples abound. It's just the status quo. Twisted's twisted.python.log module is used everywhere like this. The standard library's sys.path, sys.stdin/out/err, warnings.warn_explicit, and probably a dozen examples I can't think of off the top of my head, all work like this.



And there's a good reason that this keeps happening. Sometimes, you feel as though your program really does need a "global" registry for some reason; you find yourself wanting access to the same central object in a variety of different places. It seems convenient to have it available, and it basically works.



Here's a technique for implementing that convenience, while still allowing for a clean point of integration with other code.



First, make your "global" thing be a class.

class Publisher:

def __init__(self):

self.subscribers = []



def addSubscriber(self, subscriber):

self.subscribers.append(subscriber)



def publish(self, message):

for subscriber in self.subscribers:

subscriber.notify(message)



thePublisher = Publisher() Second, decide and document how "global" you mean. Is it global to your process? Global to a particular group of objects? Global to a certain kind of class? Document that, and make sure it is clear who should use the singleton you've created. At some point in the future, someone will almost certainly come along with a surprising requirement which makes them want a different, or wrapped version of your global thing, Documentation is always important, but it is particularly important when dealing with globals, because there's really no such thing as completely global, and it is difficult to determine from context just how global you intend for something to be.



Third, and finally, encourage using your singleton by using it as a default, rather than accessing it directly. For example:

from publisher import thePublisher



class Worker:

publisher = thePublisher



def work(self):

self.publisher.publish(self)

In this example, you now have a clean point of integration for testing and extending this code. You can make a single Worker instance, and change its "publisher" attribute before calling "work". Of course, if you're willing to burn a whole extra two lines of code, you can make it an optional argument to the constructor of Worker. If you decide that in fact, your publisher isn't global at all, but system-specific, this vastly decreases the amount of code you have to change.



Does this mean you should make everything into objects, and never use free functions? No. Free functions are fine, but functions in Python are for functional programming. The hint is right there in the name. If you are performing computations which return values, and calling other functions which do the same thing, it makes perfect sense to use free functions and not bog yourself down with useless object allocations and 'self' arguments.



Once you've started adding mutable state into the mix, you're into object territory. If you're appending to a global list, if you're setting a global "state" variable, even if you're writing to a global file, it's time to make a class and give it some methods. Have you ever written a module that looked like this?And then used it like this?I've done this many times myself.I used to think that this was the "right" way to implement Singletons in Python. Other languages had static members and synchronized static accessors and factory methods; all kinds of rigamarole to achieve this effect, but Python simply had modules.Now, however, I realize that there is no "right" way to implement Singleton in Python, because singletons are simply a bad thing to have. As Wikipedia points out , "It is also considered an anti-pattern since it is often used as a euphemism for global variable ."The module above is brittle, and as a result, unpleasant to test and extend.It's difficult to test because the call to "publish" cannot be indirected without monkeying around with the module's globals - generally recognized to be poor style, and prone to errors which will corrupt later, unrelated tests.It makes code that interacts with it difficult to test, because while you can temporary mangle global variables in the most egregious of whitebox tests, tests for code that is further away shouldn't need to know about the implementation detail of "publish". Furthermore, code which adds subscribers to the global list will destructively change the behavior of later tests (or later code, if you try to invoke your tests in a running environment, since we all know running environments are where the interesting bugs occur).It's difficult to extend because there is no explicit integration point with 'publish', and all instances share the same look-up. If you want to override the behavior of "work" and send it to a different publisher, you can't call to the superclass's implementation.Unfortunately, this probably doesn't seem particularly bad, because bad examples abound. It's just the status quo. Twisted's twisted.python.log module is used everywhere like this. The standard library's sys.path, sys.stdin/out/err, warnings.warn_explicit, and probably a dozen examples I can't think of off the top of my head, all work like this.And there's a good reason that this keeps happening. Sometimes, you feel as though your program really does need a "global" registry for some reason; you find yourself wanting access to the same central object in a variety of different places. It seems convenient to have it available, and it basically works.Here's a technique for implementing that convenience, while still allowing for a clean point of integration with other code.First, make your "global" thing be a class.Second, decide and document how "global" you mean. Is it global to your process? Global to a particular group of objects? Global to a certain kind of class? Document that, and make sure it is clear who should use the singleton you've created. At some point in the future, someone will almost certainly come along with a surprising requirement which makes them want a different, or wrapped version of your global thing, Documentation is always important, but it is particularly important when dealing with globals, because there's really no such thing as completely global, and it is difficult to determine from context just how global you intend for something to be.Third, and finally, encourage using your singleton by using it as a default, rather than accessing it directly. For example:In this example, you now have a clean point of integration for testing and extending this code. You can make a single Worker instance, and change its "publisher" attribute before calling "work". Of course, if you're willing to burn a whole extra two lines of code, you can make it an optional argument to the constructor of Worker. If you decide that in fact, your publisher isn't global at all, but system-specific, this vastly decreases the amount of code you have to change.Does this mean you should make everything into objects, and never use free functions? No. Free functions are fine, but functions in Python are for functional programming. The hint is right there in the name. If you are performing computations which return values, and calling other functions which do the same thing, it makes perfect sense to use free functions and not bog yourself down with useless object allocations and 'self' arguments.Once you've started adding mutable state into the mix, you're into object territory. If you're appending to a global list, if you're setting a global "state" variable, even if you're writing to a global file, it's time to make a class and give it some methods. (Deleted comment) From: glyf Date: July 8th, 2007 09:13 am (UTC) (Link) I'm curious, how did you test the Singleton classes before using this pattern? At work we actually ask people during interviews to describe their experiences with singletons, and people have consistently reported that testing is highly problematic, and control of scope even more so. It's Python. Some things are horrible, but everything is testable. For example, consider this little gem: class Thing: stuffed = False def stuff(self): self.stuffed = True globalThing = Thing() def frob(): globalThing.stuff() # Oh no! A global! What ever will we do! from types import FunctionType def testable(thunk, **newGlobals): "This'll fix that global's wagon but good." globs = {} globs.update(thunk.func_globals) globs.update(newGlobals) testable = FunctionType(thunk.func_code, globs, thunk.func_name, thunk.func_defaults, thunk.func_closure) return testable def test_frob(): localThing = Thing() frobPrime = testable(frob, globalThing=localThing) frobPrime() print localThing.stuffed test_frob()

Coming up with crimes against nature like this takes a little creativity and a little knowledge of the Python runtime. The real trick is figuring out which one is appropriate in a particular nasty situation, and avoiding nasty situations (mostly by avoiding globals). It's Python. Some things are horrible, but everything is testable. For example, consider this little gem:Coming up with crimes against nature like this takes a little creativity and a little knowledge of the Python runtime. The real trick is figuring out which one is appropriate in a particular nasty situation, and avoiding nasty situations (mostly by avoiding globals). (Deleted comment) From: glyf Date: July 8th, 2007 08:54 pm (UTC) (Link)



For example, libraries like JMock achieve similar, if not identical, results by using



(And who says Oh, sure, it's tedious and unpleasant in Java, but hey - you are using Java, clearly you like tedious and unpleasant things :). But it's often possible even without bytecode hacking.For example, libraries like JMock achieve similar, if not identical, results by using java.lang.reflect.Proxy and friends.(And who says bytecode manipulation has to be hard .) From: glyf Date: July 8th, 2007 09:21 am (UTC) (Link) I use Spring (an IoC container on the Java side) for much of the configuration I use, since it encapsulates the scope (instance per VM/session/etc) and insertion into the appropriate objects. What do you do with the apps you build in python? I have to confess I don't really understand this question. I have heard it before, and I keep my ear to the ground in the Java-verse, so I hear periodic murmurs about something or other totally revolutionizing Java programming by providing an even newer, even improveder way to initialize configuration and state and manage the lightweight dependency injection of control configuration inversion rama rama ding dong.



I would say "I don't think it's that big of a problem" if I even understood what the problem was that these massive configuration and initialization "container" frameworks were trying to solve. I assume it's not that big of a problem because I seem to be able to solve something similar by spending 30 seconds every six months writing a tac file, or these days maybe a whole ten minutes writing tests and docs and a twistd plugin.



The other possible answer is that I use Axiom, since Axiom applications are completely self-contained universes that have all the application data and "configuration" information and initialization bootstrapping bundled together in a convenient database format. I have to confess I don't really understand this question. I have heard it before, and I keep my ear to the ground in the Java-verse, so I hear periodic murmurs about something or other totally revolutionizing Java programming by providing an even newer, even improveder way to initialize configuration and state and manage the lightweight dependency injection of control configuration inversion rama rama ding dong.I would say "I don't think it's that big of a problem" if I even understood what the problem was that these massive configuration and initialization "container" frameworks were trying to solve. I assume it's not that big of a problem because I seem to be able to solve something similar by spending 30 seconds every six months writing a tac file, or these days maybe a whole ten minutes writing tests and docs and a twistd plugin.The other possible answer is that I use Axiom, since Axiom applications are completely self-contained universes that have all the application data and "configuration" information and initialization bootstrapping bundled together in a convenient database format. (Deleted comment) From: glyf Date: July 8th, 2007 09:07 pm (UTC) (Link) I'll assume you know what IoC is and skip to the "Why I use it" part. Maybe these aren't issues in Python/Twisted (...) I understand what the design pattern is, I've read fowler's paper, and it's a nice idea when it's appropriate, but it seems necessary so often in the Java universe that something appears distorted to me.

*Why* this situation occurs is probably more interesting than the solution I believe that's the thing that I'm missing, so yes, I'm curious about it.

So, I can write a simpler class that just "magically" has a DB connection, rather than figuring out how to get the connection in the class itself (usually a bit of work in Java, and inflexibly changed). Basically, I try to avoid this kind of magic. Pass parameters where you need 'em, when you are passing too many parameters at once, group them and turn them into objects. Nevow has a "magic" object, the "context", which you can get all kinds of juicy awfulness out of, but we are desperately trying to phase it out in favor of people just constructing objects that have all the things they need passed to their constructor. From what I glean, this isn't quite the same kind of magic though, because IoC containers seem to focus very much on the "top" object rather than random components throughout the page. I ended up writing configuration *in Python* and sourcing it, which was generally fairly easy and elegant (and fairly impossible in Java). The first Java program I wrote, ever, (I was 15) was a custom classloader which allowed you to load objects to serve as configuration. It might not be quite as spiffy as Python, but with judicious use of object array literal syntax, you can do a surprising number of things. I actually took this idea from java to python, but it sufficied in both areas. Does that explain the need/want for those sorts of features, and what the original question was getting at? Kind of. I mean, not really, I think I would need to feel the actual pain of building a real J2EE application to really understand this, and uh no thanks say no to drugs kids stay in school.

As a side note, I've never thought of a db format as "convenient", ... SQLite changes the game just a little bit, but in many cases that tiny change is enough to make a big difference. Axiom adds quite a bit of magic on top though. Observe:

glyph@illidan:~$ axiomatic mantissa Use database 'mantissa.axiom'? (Y/n) Enter Divmod™ Mantissa™ password for 'admin@localhost': Confirm Divmod™ Mantissa™ password for 'admin@localhost': glyph@illidan:~$ axiomatic offering install Quotient Use database 'mantissa.axiom'? (Y/n) glyph@illidan:~$ axiomatic userbase list Use database 'mantissa.axiom'? (Y/n) admin@localhost mantissa-base Quotient glyph@illidan:~$ axiomatic web --list Use database 'mantissa.axiom'? (Y/n) The hostname is not set. Configured to use HTTP port 8080. Configured to use HTTPS port 8443 with certificate /home/glyph/mantissa.axiom/files/server.pem Sessionless plugins: /static/webadmin => item(DeveloperSite) (prio. -255) Sessioned plugins: / => item(FrontPage) (prio. -257) glyph@illidan:~$ axiomatic web --port 8880 Use database 'mantissa.axiom'? (Y/n) glyph@illidan:~$ axiomatic web --list Use database 'mantissa.axiom'? (Y/n) The hostname is not set. Configured to use HTTP port 8880. Configured to use HTTPS port 8443 with certificate /home/glyph/mantissa.axiom/files/server.pem Sessionless plugins: /static/webadmin => item(DeveloperSite) (prio. -255) Sessioned plugins: / => item(FrontPage) (prio. -257) glyph@illidan:~$ axiomatic start -n Use database 'mantissa.axiom'? (Y/n) 2007-07-08 17:05:20-0400 [-] Log opened. 2007-07-08 17:05:20-0400 [-] twistd 2.5.0+rUnknown (/usr/bin/python 2.5.1) starting up 2007-07-08 17:05:20-0400 [-] reactor class: <class 'twisted.internet.selectreactor.SelectReactor'> 2007-07-08 17:05:20-0400 [-] xmantissa.website.AxiomSite starting on 8880 2007-07-08 17:05:20-0400 [-] Starting factory <xmantissa.website.AxiomSite instance at 0x8b48dcc> 2007-07-08 17:05:20-0400 [-] xmantissa.website.AxiomSite starting on 8443 ... I understand what the design pattern is, I've read fowler's paper, and it's a nice idea when it's appropriate, but it seems necessary so often in the Java universe that something appears distorted to me.I believe that's the thing that I'm missing, so yes, I'm curious about it.Basically, I try to avoid this kind of magic. Pass parameters where you need 'em, when you are passing too many parameters at once, group them and turn them into objects. Nevow has a "magic" object, the "context", which you can get all kinds of juicy awfulness out of, but we are desperately trying to phase it out in favor of people just constructing objects that have all the things they need passed to their constructor. From what I glean, this isn't quite the same kind of magic though, because IoC containers seem to focus very much on the "top" object rather than random components throughout the page.The first Java program I wrote, ever, (I was 15) was a custom classloader which allowed you to load objects to serve as configuration. It might not be quite as spiffy as Python, but with judicious use of object array literal syntax, you can do a surprising number of things. I actually took this idea from java to python, but it sufficied in both areas.Kind of. I mean, not really, I think I would need to feel the actual pain of building a real J2EE application to really understand this, and uh no thanks say no to drugs kids stay in school.SQLite changes the game just a little bit, but in many cases that tiny change is enough to make a big difference. Axiom adds quite a bit of magic on top though. Observe: