Robust API communication with exponential backoff

Every API fails at random points of time and that’s unavoidable. Sadly, it’s not taken care correctly during integrating third party API’s. I see it very often. „Hey there! But I’m using try-catch and handle errors, sometimes even I log them to the file…” one might say. Well, so what? What happens when it fails and you miss data which needed to be fetched during the daily ETL process? Or your business partner misses information if you send data to their API and for some reason, it fails. What then? As long as you use cron and have output emailed to some mailbox, which is being monitored – you’ll notice. Maybe you use Sentry or any other application monitoring/error tracking software and you’ll spot some anomaly. But imagine having dozens of such jobs running on a daily basis – it’s easy to lose track.

I think you get my point now. API errors occur quite often. Most of them are due to temporary service unavailability, caused mainly by having too much traffic at the moment. The simple solution is to retry. In this post, I’ll show how to easily implement efficient retry mechanism.

According to Wikipedia

Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate.

In case, which I’ll explore, it’s a way to retry a piece of code, in case of an exception occurrence. It will delay every attempt with exponential pause, according to the equation (2^attempt) * baseTime

Below example uses PHP’s backoff library. Similar libs can be found in Python, node.js and probably any language of choice.

use GuzzleHttp\Exception\RequestException;
use Monolog\Logger;
use Monolog\Handler\StreamHandler;
use Monolog\Processor\UidProcessor;
use STS\Backoff\Backoff; # nothing to do with sts.pl ;)

$log = new Logger('x');
$log->pushHandler(new StreamHandler('php://stdout', Logger::DEBUG));
$log->pushProcessor(new UidProcessor());

$client = new GuzzleHttp\Client();
// below API fails 60% of time
// always fail first call in 5 minutes timespan
$url = 'https://sznapka.pl/fakeapi.php';
$log->debug(sprintf('Fetching from %s', $url));

$backoff = new Backoff(10, 'exponential', 10000, true);
$result = $backoff->run(function() use ($client, $url, $log) {
    try {
        $res = $client->request('GET', $url);
        $data = json_decode($res->getBody(), true);
        $log->info(sprintf('Got response, %d items', count($data)));
    } catch (RequestException $e) {
        $log->error($e->getResponse()->getBody());
        throw $e; // causes backoff lib to retry
    }
});
$log->debug('All done');

When you run it with php ./console.php you’ll get:

[22:29:15.649] x.DEBUG: Fetching from https://sznapka.pl/fakeapi.php [] {"uid":"6b3d944"}
[22:29:15.759] x.ERROR: API failed, what a surprise! [] {"uid":"6b3d944"}
[22:29:15.832] x.ERROR: API failed, what a surprise! [] {"uid":"6b3d944"}
[22:29:16.268] x.INFO: Got response, 5 rows [] {"uid":"6b3d944"}
[22:29:16.269] x.DEBUG: All done [] {"uid":"6b3d944"}

As you can see, the backoff library retries with exponential intervals until our wrapped closure doesn’t throw an exception. It has set 10 retries and waitCap 10 seconds, so will stop processing whenever one of those conditions appear. I’ve also defined jitter parameter to true to spread out retries and minimize collisions.

Last but not least – always log your external API usage. It will help you a lot during the debugging phase. Also, it’s very handy to use UidProcessor which puts an uid (per session) into logger’s context. It allows filtering logs from given invocation, which is especially helpful with overlapping calls or concurrent usage.