Start your Python project with optparse and logging

Python continues to be my favorite language to hack in. It’s useful for tasks big and small and has the advantage of being more readable and maintainable than a lot of other scripting languages. I remember when, having learned the basics of Python, I decided to rewrite my home-built CD-to-MP3 script suite from its perl incarnation (perl was my previous favorite language for system programming projects). I knocked that project out in only about 2 hours, including a major redesign made easy by Python’s object syntax and built in “pickle” for serialization.

Since then, I’ve usually chosen Python for a variety of system programming tasks. This includes things like making backups, deploying software, configuration file templating, web site scraping, health monitoring, and some bigger data crunching/graphing. Time after time, these quick projects (often at work) turned into something bigger. Eventually they get rolled into a product or adopted by the production operations group as standard kit.

At some point I realized that there was something that I was doing that made this transition from “quick hack” to “standard tool” easy: I always start my projects with logging and optparse in place from day zero. During development, this means that I don’t have to scatter print -style debugging statements throughout the code, I can just use logging.debug() and turn them on and off at will (via command line flags). Once deployed or passed on, it means other people using it can immediately start interacting with my script just like other familiar Unix utilities.

So, here is a variation on what my typical python script starts out with.

#!/usr/bin/python import logging def foo (): """These will only get output if you turn up verbosity.""" logging . debug ( "This is debug." ) logging . info ( "This is info." ) def bar (): """These will all be output a default logging levels.""" logging . warn ( "Warning! Things are getting scary." ) logging . error ( "Uh-oh, something is wrong." ) try : raise Exception ( "ZOMG tacos." ) except : logging . exception ( "Just like error, but with a traceback." ) if '__main__' == __name__ : # Late import, in case this project becomes a library, never to be run as main again. import optparse # Populate our options, -h/--help is already there for you. optp = optparse . OptionParser () optp . add_option ( '-v' , '--verbose' , dest = 'verbose' , action = 'count' , help = "Increase verbosity (specify multiple times for more)" ) # Parse the arguments (defaults to parsing sys.argv). opts , args = optp . parse_args () # Here would be a good place to check what came in on the command line and # call optp.error("Useful message") to exit if all it not well. log_level = logging . WARNING # default if opts . verbose == 1 : log_level = logging . INFO elif opts . verbose >= 2 : log_level = logging . DEBUG # Set up basic configuration, out to stderr with a reasonable default format. logging . basicConfig ( level = log_level ) # Do some actual work. foo () bar ()

This is obviously a pretty minimal setup, but it achieves what most people need. You get option parsing and checking along with usage information. You get warnings and errors spat out to sys.stderr (leaving sys.stdout for actual program output). You can specify -v to crank up verbosity.