Highly distributed applications that consist of lots of small services talking among themselves are getting more and more popular, and that, in my opinion, is a good thing. But this architectural style brings with it a new class of problems that are less common in monolithic applications. Consider what happens when a service needs to send a request to another service, and this second service happens to be temporarily offline, or too busy to respond. If one little service goes offline at the wrong time, that can create a domino effect that can, potentially, take your entire application down.
In this article I'm going to show you techniques that can give your application some degree of tolerance for failures in dependent services. The basic concept is simple: we make the assumption that in most cases these failures are transient, so then when an operation fails, we just repeat it a few times, until it hopefully succeeds. Sounds easy, right? But as with most things, the devil is in the details, so keep reading if you want to learn how to implement a robust retry strategy.