Multi-data-centre and disaster recovery

Why it matters

A data centre can be lost to a power failure, a flood, a fire or a network cut. For a bank, losing its only site means no payments, no ATMs and no access to money until it is fixed. A second site, kept ready to take over, turns that disaster into an interruption. How ready the second site is, and how much recent data it holds, are design decisions with real costs.

The model explained

Two measures: RPO and RTO

The recovery point objective (RPORecovery point objective. The most data, measured in time, that an organisation is prepared to lose when a system fails.) is how much data, measured in time, the bank can afford to lose. An RPO of zero means no confirmed transaction may be lost. The recovery time objective (RTORecovery time objective. How quickly a system must be working again after a failure.) is how quickly service must be back. Both are business decisions first and technical designs second: a lower RPO or RTO costs more.

Active-passive and active-active

In active-passive, one site serves all traffic and the other waits, receiving a copy of the data. Failover means promoting the standby. It is simpler, but the standby sits idle, and a failover is a rare, stressful event. In active-active, both sites serve traffic at the same time. Losing one halves capacity instead of stopping service, but keeping one set of balances consistent across two live sites is hard.

Synchronous and asynchronous replication

With synchronous replication, a transaction is confirmed only after both sites have it. That gives an RPO of zero but adds the network round trip to every transaction, which limits how far apart the sites can be. With asynchronous replication, the primary confirms first and the copy follows a moment later. It is faster and allows distant sites, but the last few transactions can be lost in a failure and must be reconciled.

Split-brain

If the link between two sites fails, each may think the other is down and start acting as the primary. Two sites then accept conflicting updates to the same accounts: a split-brain. Designs prevent it with a third, independent witness that decides which site may act, and with rules that a site which cannot reach the witness stops accepting writes.

Traffic shifting and drills

Moving traffic from one site to another uses DNS, global load balancers or routing changes, each with its own delay. A failover that has never been practised is a hope, not a plan, so banks run drills: planned switches to the second site under real load.

What RBI expects

RBIReserve Bank of India. India's central bank. It regulates banks and payment systems, runs RTGS and NEFT, and holds the accounts in which banks settle with one another.’s IT governance Direction does not set a single number for every bank. It requires banks to define their own targets, aim high for critical systems and prove them in practice:

• RBI requires regulated entities to define recovery metrics, including a recovery point objective (RPO) and recovery time objective (RTO), for all critical information systems.^{1Reserve Bank of India. Reserve Bank of India (Information Technology Governance, Risk, Controls and Assurance Practices) Directions, 2023 (full text). As of 7 November 2023.}
• For critical information systems, regulated entities should prioritise a minimal RTO, approved by their IT Strategy Committee, and a near-zero RPO; with a non-zero RPO they need a documented method for reconciling data when resuming from the alternate site.^{1Reserve Bank of India. Reserve Bank of India (Information Technology Governance, Risk, Controls and Assurance Practices) Directions, 2023 (full text). As of 7 November 2023.}
• DR drills for critical information systems must be held at least every six months, and major issues found in a drill must be fixed and tested again before the next cycle.^{1Reserve Bank of India. Reserve Bank of India (Information Technology Governance, Risk, Controls and Assurance Practices) Directions, 2023 (full text). As of 7 November 2023.}
• A DR test must switch over to the DR or alternate site and run it as the primary site long enough to cover at least a full working day, from beginning of day to end of day.^{1Reserve Bank of India. Reserve Bank of India (Information Technology Governance, Risk, Controls and Assurance Practices) Directions, 2023 (full text). As of 7 November 2023.}
• Configurations and security patches must be identical at the primary data centre and the DR site.^{1Reserve Bank of India. Reserve Bank of India (Information Technology Governance, Risk, Controls and Assurance Practices) Directions, 2023 (full text). As of 7 November 2023.}
• Regulated entities must back up data and periodically restore it to check that it is usable.^{1Reserve Bank of India. Reserve Bank of India (Information Technology Governance, Risk, Controls and Assurance Practices) Directions, 2023 (full text). As of 7 November 2023.}

Trade-offs

Failure modes

• Drift between sites, where configuration or patches differ and the standby fails when it is needed.
• Split-brain after a network partition, leaving two conflicting sets of balances.
• Failover that was never really tested under full load.
• Shared dependencies, such as both sites relying on the same identity service, key store or network provider.
• Backups that cannot be restored, discovered during the incident.

When to use it, and when to avoid it

Every system that moves money or holds balances needs a tested second site. Choose active-active for the systems where even a short outage is unacceptable and the consistency problem can be solved, for example by partitioning customers between sites. Choose active-passive where simplicity matters more than instant recovery. Avoid claiming an RPO or RTO that the design cannot demonstrate in a drill.

Questions for a design review

• What RPO and RTO has the business approved for each critical system, and can the design meet them?
• Is replication synchronous or asynchronous, and how are any lost transactions reconciled?
• What stops both sites acting as the primary after a network partition?
• When was the last full switch-over drill, and did it run a complete business day at the second site?
• Which dependencies, such as identity, keys or network, are shared by both sites?