DB2University.com wasn't impacted by this AWS disaster, thanks to a DB2 feature known as High Availability and Disaster Recovery (HADR) which, being asynchronous, works exceptionally well for long distances. Essentially, the main server runs on US-East while a failover server runs in a different region. The exact second DB2 detects an issue with US-East, it switches over to the standby server running in a different region. All automated, and without downtimes.
> The exact second DB2 detects an issue with US-East, it switches over to the standby server
How does it do this? This is actually a very difficult problem because the monitoring system has to determine whether the primary site is down or whether its own network is experiencing trouble. And once you perform the failover, the old master might not learn that it has lost its master role, and may continue to serve requests to clients. Systems with automated failover usually use a lock service like Google's Chubby.
For most folks, it's better to have a manual failover script that the oncall engineer can run after diagnosing the issue. Automated failover requires a lot of extra complexity in your systems. There's the real risk of total service failure when the lock service goes down. And there are lots of interesting failure modes in the failover process. For a startup on a tight budget, it's probably not worth it just to change 30 minutes of downtime into 1 minute.
Exactly right. Network partition is a hard problem in automatic failover of replicated system. You don't want the standby to become master unless it can be sure the primary master is absolutely down. It's difficult to unwind the mess if two masters are active and taking changes.
In high availability system design, the secondary node literally has to shut down the primary's power (called the Shoot-At-The-Head technique) to ensure it's really down when it's not responding via network.
Of course over long distance cross-datacenter replication, shutting down power remotely is not reliable. In the last HA clusters I built, the failover between datacenters is done via manual decision. It means there could be a 15 minutes to 30 minutes window to do the manual failover, but it's an acceptable risk since datacenter failure is rare, like AWS failure once in a blue moon.
I remember when I first used HA-Linux in a project being highly amused when I came across the acronym STONITH and discovering that it meant "Shoot The Other Node In The Head" :-) http://www.linux-ha.org/wiki/STONITH
All jokes aside, it is indeed a very important concept when dealing with high availability.
Async replication doesn't produce inconsistency it produces uncommitted transactions. Every database produces them when it goes down whether it replicates or not.
When the master comes up all it has to do is reverse the transactions that the slave didn't receive. Voila, consistent database.
What about inconsistencies with data outside the database? Things like credit card transactions or other external API calls that were recorded on the master but not the slave will be inconsistent with your slaves view of the world. Is there a standard way of dealing with those kind of things or is that usually handled manually?
You use the logs from the slave to commit the 2nd portion of a two-phase commit, and immediately stop processing new transactions when you only have the slave up.
Your hypothetical API does support two-phase commit correct? Because if it doesn't you have lots of solutions for losing data/creating inconsistent data anyway.
How does it handle the possible inconsistency due to the asynchronous replication?
That would be a big question for me. I can sync databases easily and instantaneously within an availability zone, and nearly instantaneously across availability zones within a region. But once I have to replicate across regions, I add latency to the mix. If replication from US-East to US-West is asynchronous, how would I reconcile the two? I suppose with a catastrophic failure of the primary database in US-East, I could just write off any data that wasn't replicated before the failure, but that doesn't seem like a solid solution. Would it be better to write a transaction layer into the app, so that data isn't considered committed until it has been written and replicated across multiple regions?
There are all kinds of solutions available to you in that scenario. Two-phase commit, on DBs that support it, would probably go a long way towards enabling a transaction layer like you describe.
Usually, though, discussions of DR should start with determining what kinds of RPO and RTO you're willing to pay for and then evaluating which of the available solutions will get you there.
