Cleanup the PostgreSQL App-Data DR Proof Lanes

Purpose: Tear down the temporary proof clusters created by the PostgreSQL app-data DR drill without damaging the live primary lane, the source drill lane, shared DNS, or shared networking state.
Owner: Platform engineering / SRE
Trigger: Successful completion of the proof drill, failed proof run that must be reset, or cost-control teardown after acceptance.
Impact: Deletes the isolated GCP proof VMs and isolated on-prem failback proof VMs, plus their attached HyOps module state lanes.
Severity: P3
Pre-reqs: The proof drill logs and state have already been captured, and operators have confirmed no one still needs the temporary proof clusters.
Rollback strategy: None for the proof lanes themselves. If teardown is accidental, rerun the proof drill to recreate them.

Context

This cleanup runbook is intentionally narrow.

It destroys only these proof lanes:

• platform/gcp/platform-vm#gcp_pg_vms_app_proof
• platform/postgresql-ha#postgresql_restore_gcp_app_proof
• platform/postgresql-ha-backup#postgresql_backup_config_gcp_app_proof
• platform/onprem/platform-vm#postgres_ha_vms_app_proof_drill
• platform/postgresql-ha#postgresql_restore_onprem_app_proof_drill
• platform/postgresql-ha-backup#postgresql_backup_config_onprem_app_proof_drill

It does not destroy:

• the live primary dev PostgreSQL lane
• the source drill lane at 10.12.0.41
• shared networking state such as org/gcp/wan-cloud-nat#gcp_pg_egress
• shared PowerDNS, NetBox, WAN, or runner state

Preconditions and safety checks

1. Confirm the proof outputs are already captured.

Recommended minimum:

• verification shell transcripts
• proof state JSON files

• relevant runner and module logs under ~/.hybridops/envs/dev/logs/ and ~/.hybridops/envs/drill/logs/

• Confirm you are destroying only the proof lanes.

Check the exact inputs files:

ls -1 "$HOME/.hybridops/envs/dev/config/modules/platform__gcp__platform-vm/instances/gcp_pg_vms_app_proof.inputs.yml"
ls -1 "$HOME/.hybridops/envs/dev/config/modules/platform__postgresql-ha/instances/postgresql_restore_gcp_app_proof.inputs.yml"
ls -1 "$HOME/.hybridops/envs/dev/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_gcp_app_proof.inputs.yml"

ls -1 "$HOME/.hybridops/envs/drill/config/modules/platform__onprem__platform-vm/instances/postgres_ha_vms_app_proof_drill.inputs.yml"
ls -1 "$HOME/.hybridops/envs/drill/config/modules/platform__postgresql-ha/instances/postgresql_restore_onprem_app_proof_drill.inputs.yml"
ls -1 "$HOME/.hybridops/envs/drill/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_onprem_app_proof_drill.inputs.yml"

1. Confirm the source drill lane still points at the original source leader.

jq '.outputs.db_host' \
  "$HOME/.hybridops/envs/drill/state/modules/platform__postgresql-ha/latest.json"

Expected result:

• 10.12.0.41

Steps

1. Destroy the on-prem failback proof backup config

cd /home/user/hybridops-studio/hybridops-core

./.venv/bin/hyops destroy --env drill \
  --module platform/postgresql-ha-backup \
  --state-instance postgresql_backup_config_onprem_app_proof_drill \
  --inputs "$HOME/.hybridops/envs/drill/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_onprem_app_proof_drill.inputs.yml"

Expected result:

• backup-config proof lane is removed

• Destroy the on-prem failback proof restore lane

cd /home/user/hybridops-studio/hybridops-core

./.venv/bin/hyops destroy --env drill \
  --module platform/postgresql-ha \
  --state-instance postgresql_restore_onprem_app_proof_drill \
  --inputs "$HOME/.hybridops/envs/drill/config/modules/platform__postgresql-ha/instances/postgresql_restore_onprem_app_proof_drill.inputs.yml"

Expected result:

• on-prem proof PostgreSQL HA state is removed

• Destroy the on-prem failback proof VMs

cd /home/user/hybridops-studio/hybridops-core

./.venv/bin/hyops destroy --env drill \
  --module platform/onprem/platform-vm \
  --state-instance postgres_ha_vms_app_proof_drill \
  --inputs "$HOME/.hybridops/envs/drill/config/modules/platform__onprem__platform-vm/instances/postgres_ha_vms_app_proof_drill.inputs.yml"

Expected result:

• pgfb-01, pgfb-02, and pgfb-03 are removed

• Destroy the GCP proof backup config lane

cd /home/user/hybridops-studio/hybridops-core

./.venv/bin/hyops destroy --env dev \
  --module platform/postgresql-ha-backup \
  --state-instance postgresql_backup_config_gcp_app_proof \
  --inputs "$HOME/.hybridops/envs/dev/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_gcp_app_proof.inputs.yml"

Expected result:

• GCP backup-config proof lane is removed

• Destroy the GCP proof restore lane

cd /home/user/hybridops-studio/hybridops-core

./.venv/bin/hyops destroy --env dev \
  --module platform/postgresql-ha \
  --state-instance postgresql_restore_gcp_app_proof \
  --inputs "$HOME/.hybridops/envs/dev/config/modules/platform__postgresql-ha/instances/postgresql_restore_gcp_app_proof.inputs.yml"

Expected result:

• GCP proof PostgreSQL HA state is removed

• Destroy the GCP proof VMs

cd /home/user/hybridops-studio/hybridops-core

./.venv/bin/hyops destroy --env dev \
  --module platform/gcp/platform-vm \
  --state-instance gcp_pg_vms_app_proof \
  --inputs "$HOME/.hybridops/envs/dev/config/modules/platform__gcp__platform-vm/instances/gcp_pg_vms_app_proof.inputs.yml"

Expected result:

• platform-proof-pgapp-01, platform-proof-pgapp-02, and platform-proof-pgapp-03 are removed

• Leave the backup-run proof record in place unless you explicitly want to clear audit state

The state instance platform/postgresql-ha-backup#postgresql_backup_run_gcp_app_proof is an execution record, not a standing infrastructure lane.

Default guidance:

• keep it as an audit anchor for the drill
• remove it only if you intentionally want a clean proof-state slate

Verification

Confirm the proof states are gone or no longer ok:

ls -1 "$HOME/.hybridops/envs/dev/state/modules/platform__gcp__platform-vm/instances/" | rg 'gcp_pg_vms_app_proof' || true
ls -1 "$HOME/.hybridops/envs/dev/state/modules/platform__postgresql-ha/instances/" | rg 'postgresql_restore_gcp_app_proof' || true
ls -1 "$HOME/.hybridops/envs/drill/state/modules/platform__onprem__platform-vm/instances/" | rg 'postgres_ha_vms_app_proof_drill' || true
ls -1 "$HOME/.hybridops/envs/drill/state/modules/platform__postgresql-ha/instances/" | rg 'postgresql_restore_onprem_app_proof_drill' || true

Also confirm the source drill lane is still intact:

jq '.status, .outputs.db_host' \
  "$HOME/.hybridops/envs/drill/state/modules/platform__postgresql-ha/latest.json"

Expected result:

• source drill lane remains ok
• source drill leader remains 10.12.0.41

Post-actions and clean-up

• Keep the proof SQL files under ~/.hybridops/envs/drill/config/drproof/.
• Preserve accepted drill logs and state JSONs in your review artefacts before pruning runtime history.
• Do not destroy gcp_pg_egress from this cleanup unless you have separately verified that no other GCP proof or DR flows depend on it.

Related

Related reading

• Repeatable PostgreSQL App-Data DR Drill
• Failover PostgreSQL HA to GCP (HyOps Blueprint)
• Failback PostgreSQL HA to On-Prem (HyOps Blueprint)
• Showcase – PostgreSQL HA DR Cycle

References

• ~/.hybridops/envs/dev/config/modules/platform__gcp__platform-vm/instances/gcp_pg_vms_app_proof.inputs.yml
• ~/.hybridops/envs/dev/config/modules/platform__postgresql-ha/instances/postgresql_restore_gcp_app_proof.inputs.yml
• ~/.hybridops/envs/dev/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_gcp_app_proof.inputs.yml
• ~/.hybridops/envs/drill/config/modules/platform__onprem__platform-vm/instances/postgres_ha_vms_app_proof_drill.inputs.yml
• ~/.hybridops/envs/drill/config/modules/platform__postgresql-ha/instances/postgresql_restore_onprem_app_proof_drill.inputs.yml
• ~/.hybridops/envs/drill/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_onprem_app_proof_drill.inputs.yml

Maintainer: HybridOps
License: MIT-0 for code, CC-BY-4.0 for documentation