Necessity is the mother of invention.
Until yesterday we were proud of our smart infrastructure. Every site is running not on one, but on at least two independent servers. CloudFlare provides excellent service for DNS service. You can specify for each domain multiple origin IPs. In the process of testing - everything was working as intended. In production, the infrastructure was robust and able to dynamically scale up and down. With winter holidays and parties in Russia - one our server became unavailable. Things do happen, that is why we are hosting in Russia servers at independent providers with independent Internet "backbones". This accident highlighted passive nature of our failover design with CloudFlare. It is time for action.
What we needed is proactive DNS failover mechanism. CloudFlare is working on offering that as a product, but as of the time of writing this blog - they so far only register interest. At the same time - CloudFlare provides excellent API to build this functionality yourself.
Starting point is here. They are giving steps to implement the mechanism.
In the same article, they point to great work of lyoshenka. The code is clean and nice. The problem is that it assumes that code is running on the same server where the origin is. In case the site is offline - it cannot switch to the backup.
We have built upon the work of Lyoshenka and provided our version. This code requires explicitly setting MAIN and BACKUP IPs. The logic is following:
- Get origin IP from CloudFlare for specified SUBDOMAIN.
- Check whether it is accessible. If not - go to step 6.
- Check whether this IP is MAIN. If yes - no change - exit.
- Check whether MAIN IP is accessible. If not - no change - exit.
- Change in CloudFlare origin IP to MAIN IP - exit.
- Check whether origin IP is BACKUP. If yes - set BACKUP IP to MAIN IP.
- Check whether BACKUP IP is accessible. If not - no change - exit.
- Change in CloudFlare origin IP to BACKUP IP - exit.
You can find our code here.
in Sydney, January 06, 2017
Timur Yusupov, PhD