Multi-Region Failover Manager
Automated regional disaster recovery that treats failover as two problems on two clocks: DNS shifts traffic in about a minute while orchestration promotes the database behind it, demonstrated live against real AWS and torn down the same night.
/01Problem
Stateless failover is a DNS feature. Stateful failover is not: a cross-region read replica keeps the data close, but something still has to decide the primary is gone and promote the standby to writable. Those two mechanisms run on different clocks, and gating traffic on the slower one extends the outage for no reason.
The goal was a system where traffic shifts in seconds, the database promotes itself in minutes, and the window in between is honest and observable rather than hidden.
/02Approach
- An identical serverless order API (API Gateway v2 and Lambda in private-subnet-only VPCs, no IGW or NAT) runs in us-east-1 and us-west-2 over an RDS PostgreSQL primary with an encrypted cross-region read replica.
- A Route 53 health check probes the primary /health endpoint every 10 seconds, and failover routing answers with the standby endpoint the moment it fails, with no automation in the traffic path.
- A CloudWatch alarm on the health check metric emits a state change that EventBridge forwards cross-region to the standby bus, where a failover Lambda verifies the replica is promotable, calls PromoteReadReplica idempotently, and publishes what it did to SNS.
- Until promotion completes, the standby serves reads and returns an honest 409 on writes, which makes the promotion itself observable as a behavior change rather than a log line.
/03Architecture
Every response component lives in the standby region on purpose: the machinery that answers a regional failure must not depend on the failing region. The health check metric only exists in us-east-1, which is exactly why the alarm's consequences are shipped out of that region immediately.
Credentials live in Secrets Manager with cross-region replication reached through interface endpoints, TLS to PostgreSQL is verified against the RDS CA bundle, and the failover role can promote exactly one ARN. CI gates on Bandit, Checkov, and Trivy, with an OIDC-authenticated plan job and zero static keys.
/04Outcome
Demonstrated live: the primary was broken deliberately, DNS answered with the standby region in about 60 seconds, and the alarm-to-promotion automation fired about 100 seconds after the failure began. The promoted standby then accepted the same write it had correctly refused minutes earlier.
All 70 resources were destroyed the same night, with every leftover check across both regions coming back empty. Total session cost was about a quarter.