HOWTO: Validate DNS Cutover During a DR Failover¶
Purpose: Execute and validate DNS cutover during a DR failover, measuring propagation and confirming application reconnection with structured run records.
Difficulty: Intermediate
Track: Disaster Recovery Automation
Overview¶
DNS cutover is not an instantaneous operation: it is a sequence with timing constraints, observable propagation behaviour, and application-level effects that must all be confirmed. This HOWTO teaches the complete cutover validation workflow used in every HybridOps DR drill.
1. Pre-Cutover Preparation¶
- Confirming the TTL on the target record is sufficiently low.
- Pre-staging the DR target IP in DNS (where possible).
- Resolver path inventory: which resolvers does the application use?
2. Executing the Record Update¶
- PowerDNS API record update command.
- Confirming the update is accepted by the authoritative nameserver.
- Timestamp capture for the propagation window start.
3. Measuring Propagation¶
- Querying each resolver path with
dig. - Expected propagation timeline based on TTL.
- Handling resolvers with extended caching behaviour.
4. Application Reconnection Verification¶
- Connection pool drain and reconnect cycle.
- Probe: successful connection to the DR target endpoint.
- Error rate check: confirming no application-visible errors persist after propagation.
5. Run-record capture¶
- Full cutover timeline: update timestamp, propagation observations, reconnection time.
- Storing under
<runtime-root>/logs/dr/dns-cutover/. - Linking to the DR drill run record as a required supporting record.
References¶
- ADR-0501 – PostgreSQL on Dedicated VM with DR Replication
- HOWTO: Set Up PowerDNS Split-Horizon
- HOWTO: Execute a PostgreSQL Failover
License: MIT-0 for code, CC-BY-4.0 for documentation unless otherwise stated.