What Try Again in a Few Seconds

The library presented in this article is becoming obsolete and un-maintained. I recommend you to read this post about tenacity instead.

I don't oft write about tools I employ when for my daily software development tasks. I recently realized that I really should start to share more than oft my workflows and weapons of choice.

One thing that I have a hard fourth dimension enduring while doing Python code reviews, is people writing utility code that is not directly tied to the cadre of their business. This looks to me every bit wasted time maintaining code that should exist reused from elsewhere.

So today I'd like to start with retrying, a Python package that you can use to… retry annihilation.

It's OK to fail

Often in computing, you take to bargain with external resources. That means accessing resources you don't control. Resource that tin can fail, become flapping, unreachable or unavailable.

Most applications don't deal with that at all, and explode in flight, leaving a skeptical user in front of the estimator. A lot of software engineers pass up to deal with failure, and don't bother handling this kind of scenario in their code.

In the best case, applications ordinarily handle simply the example where the external reached system is out of lodge. They log something, and inform the user that it should endeavour again afterward.

In this cloud calculating area, we tend to design software components with service-oriented architecture in mind. That ways having a lot of different services talking to each others over the network. And nosotros all know that networks tend to fail, and distributed systems too. Writing software with failing being role of normal operation is a terrific idea.

Retrying

In order to help applications with the treatment of these potential failures, you need a plan. Leaving to the user the burden to "endeavor again later" is rarely a expert choice. Therefore, most of the fourth dimension yous want your application to retry.

Retrying an action is a total strategy on its own, with a lot of options. You lot tin can retry only on certain condition, and with the number of tries based on time (e.g. every second), based on a number of tentative (e.g. retry three times and abort), based on the problem encountered, or even on all of those.

For all of that, I use the retrying library that y'all can call up hands on PyPI.

retrying provides a decorator called retry that you tin use on top of whatsoever function or method in Python to brand it retry in case of failure. By default, retry calls your role incessantly until it returns rather than raising an mistake.

              import random from retrying import retry  @retry def pick_one():     if random.randint(0, x) != 1:         enhance Exception("one was not picked")                          

This will execute the function pick_one until ane is returned past random.randint.

retry accepts a few arguments, such as the minimum and maximum delays to use, which as well tin can exist randomized. Randomizing delay is a expert strategy to avoid detectable pattern or congestion. Simply more over, information technology supports exponential delay, which can be used to implement exponential backoff, a good solution for retrying tasks while really avoiding congestion. It's especially handy for background tasks.

              @retry(wait_exponential_multiplier=m, wait_exponential_max=10000) def wait_exponential_1000():     print "Wait ii^x * thou milliseconds between each retry, up to ten seconds, and then ten seconds later"     raise Exception("Retry!")                          

You lot can mix that with a maximum delay, which can give you a good strategy to retry for a while, and then fail anyway:

              # Stop retrying later on 30 seconds anyway >>> @retry(wait_exponential_multiplier=chiliad, wait_exponential_max=10000, stop_max_delay=30000) ... def wait_exponential_1000(): ...     print "Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds later" ...     enhance Exception("Retry!") ... >>> wait_exponential_1000() Look 2^ten * g milliseconds betwixt each retry, upwards to ten seconds, and then ten seconds afterwards Wait ii^ten * 1000 milliseconds between each retry, upwardly to 10 seconds, and so ten seconds afterward Look ii^x * thousand milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards Expect 2^x * 1000 milliseconds betwixt each retry, up to x seconds, then 10 seconds afterwards Wait 2^10 * 1000 milliseconds between each retry, up to ten seconds, then 10 seconds afterwards Wait 2^ten * 1000 milliseconds between each retry, upwardly to ten seconds, then 10 seconds afterwards Traceback (most recent telephone call terminal):   File "<stdin>", line 1, in <module>   File "/usr/local/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f     return Retrying(*dargs, **dkw).call(f, *args, **kw)   File "/usr/local/lib/python2.7/site-packages/retrying.py", line 212, in call     raise endeavor.get()   File "/usr/local/lib/python2.7/site-packages/retrying.py", line 247, in become     six.reraise(self.value[0], cocky.value[1], self.value[2])   File "/usr/local/lib/python2.7/site-packages/retrying.py", line 200, in call     endeavor = Effort(fn(*args, **kwargs), attempt_number, False)   File "<stdin>", line iv, in wait_exponential_1000   Exception: Retry!                          

A design I use very oft, is the ability to retry merely based on some exception blazon. Y'all can specify a function to filter out exception you lot want to ignore or the one you want to use to retry.

              def retry_on_ioerror(exc):     return isinstance(exc, IOError)  @retry(retry_on_exception=retry_on_ioerror) def read_file():     with open("myfile", "r") as f:         return f.read()                          

retry will phone call the function passed as retry_on_exception with the exception raised equally first argument. It'south upward to the function to then render a boolean indicating if a retry should be performed or not. In the example above, this will only retry to read the file if an IOError occurs; if whatever other exception type is raised, no retry will be performed.

The same blueprint can exist implemented using the keyword argument retry_on_result, where you tin provide a function that analyses the outcome and retry based on it.

              def retry_if_file_empty(result):     return len(effect) <= 0  @retry(retry_on_result=retry_if_file_empty) def read_file():     with open("myfile", "r") as f:         return f.read()                          

This case will read the file until information technology stops existence empty. If the file does not be, an IOError is raised, and the default behavior which triggers retry on all exceptions kicks-in – the retry is therefore performed.

That's it! retry is really a good and modest library that you should leverage rather than implementing your own half-baked solution!

joneswhippercunch.blogspot.com

Source: https://julien.danjou.info/python-retrying/

Share this post