If request information from a remote web server, you should make sure that your program can handle network problems and server fails appropriately. In case of some errors–e.g. connection timeout or HTTP error 503 (Service Temporarily Unavailable)–it makes sense to retry a few times if you think the error was intermittent.

Here I discus one of the ways to approach this kind of problems using a for – else construct. The for loop in Python has a little known else clause, which is executed if the for loop terminates successfully. Here is some boilerplate code:

for i in range(max_retries): try: #do stuff except SomeParticularException: continue # retrying else: break else: # network is down, act accordingly

In this example when SomeParticularException is caught, the for loop continues to execute, and after all max_retries iterations we reach the else clause. Note that it is really important to catch only the exceptions that indicate that you need to retry doing what you’re doing. If you catch everything with except Exception , it will catch, well, everything — including, for example, KeyboardInterrupt, which is definitely not what we want to achieve here.

If you send requests in many places across your program using different functions, it is reasonable to create a decorator which will make it easier to define different functions with the same “retrying behaviour”. Here’s an example of how it may be implemented if you use requests library. In case of requests, the underlying library urllib3 can also handle the retries. However, it doesn’t sleep between retries, and applies only to failed DNS lookups, socket connections and connection timeouts; if you want to retry on anything else, you’ll need to roll on your own. This is how I roll:

import requests import time class NetworkError(RuntimeError): pass def retryer(func): retry_on_exceptions = ( requests.exceptions.Timeout, requests.exceptions.ConnectionError, requests.exceptions.HTTPError ) max_retries = 10, timeout=5 def inner(*args, **kwargs): for i in range(max_retries): try: result = func(*args, **kwargs) except retry_on_exceptions: time.sleep(timeout) continue else: return result else: raise NetworkError return inner

Now every time you create a function which you want to have this behaviour, just decorate it:

@retryer def foo(stuff): #do stuff

You can catch the NetworkError and handle the case when network problems are not intermittent by retrying again after a longer timeout, notifying the user, or performing some other task. If you use a different library to make requests (for instance, various libraries to connect to some services via their REST api), just modify the retry_on_exceptions to include the relevant errors.

You can upgrade this example by creating a decorator which accepts arguments, which makes possible to set different number of retries and timeout for different functions:

import requests import time class NetworkError(RuntimeError): pass def retryer(max_retries=10, timeout=5): def wraps(func): request_exceptions = ( requests.exceptions.Timeout, requests.exceptions.ConnectionError, requests.exceptions.HTTPError ) def inner(*args, **kwargs): for i in range(max_retries): try: result = func(*args, **kwargs) except request_exceptions: time.sleep(timeout) continue else: return result else: raise NetworkError return inner return wraps

Now the decorator accepts arguments, which allows us to customise the behaviour of different functions:

@retryer(max_retries=7, timeout=12) def foo(stuff): #do stuff