Observability Terraform State Migration Runbook

One-time procedure to migrate the observability server from the production Terraform state to the new observability environment state.

CAUTION

Run steps in order. Skipping the state rm step will cause the production environment to destroy the observability server on the next terraform apply.

Prerequisites

1. Create workspace busflow-observability (CLI-driven, Local execution) at app.terraform.io
2. Create GitHub Actions environment observability in repo settings
3. Set the following repository-level secrets so the Terraform CI/CD can pass them to the observability environment:

Secret	Source	Description
PRODUCTION_MANAGER_IP	Already exists as production env secret — copy value	Production swarm manager IP
PRODUCTION_NETWORK_ID	cd environments/production && terraform output network_id	Hetzner network ID
PRODUCTION_SSH_PRIVATE_KEY	cd environments/production && terraform output -raw swarm_ssh_private_key	SSH key to access production manager
PRODUCTION_HOST_PUBLIC_KEY	cd environments/production && terraform output -raw manager_host_public_key	SSH host key for known_hosts

Step 1 — Remove observability resources from production state

cd infrastructure/terraform/environments/production

# List the resources to remove
terraform state list | grep -E 'obs|observability'

# Remove each resource (Terraform forgets them without destroying them)
terraform state rm 'module.swarm.hcloud_server.observability_worker[0]'
terraform state rm 'module.swarm.null_resource.swarm_join_observability[0]'
terraform state rm 'module.swarm.null_resource.label_observability_node[0]'
terraform state rm 'module.swarm.null_resource.install_lazydocker_observability[0]'
terraform state rm 'module.swarm.null_resource.observability_network[0]'
terraform state rm 'module.swarm.null_resource.observability_secrets[0]'
terraform state rm 'random_password.obs_grafana_admin'
terraform state rm 'random_password.obs_grafana_sso'
terraform state rm 'random_password.obs_minio_access'
terraform state rm 'random_password.obs_minio_secret'

Step 2 — Verify production plan is clean

terraform plan
# Expected: no changes (or only DNS record removals for grafana/telemetry)

Step 3 — Apply the production changes

terraform apply
# Removes grafana/telemetry DNS records from production management

Step 4 — Initialize and import the observability environment

cd ../observability
terraform init

# Import the existing server (get ID from Hetzner Cloud console or API)
terraform import 'module.observability_node.hcloud_server.observability' <HETZNER_SERVER_ID>

terraform plan
# Expected: some resources to create (SSH key, firewall, GitHub secrets, DNS)
# but the server itself should show no changes

terraform apply

Step 5 — Verify the deploy workflow

1. Go to GitHub Actions → "Deploy Observability Stack"
2. Run the workflow manually
3. Confirm it connects via OBSERVABILITY_MANAGER_IP and deploys successfully