ROSA HCP Disaster Recovery with ACM and OpenShift GitOps

This guide demonstrates how to set up an active/passive disaster recovery pattern for applications running on ROSA HCP clusters using Red Hat Advanced Cluster Management (ACM) and OpenShift GitOps (ArgoCD). ACM handles cluster health monitoring, while ArgoCD deploys the application to the active cluster.

The pattern works as follows:

• ArgoCD deploys the application to both clusters simultaneously via an ApplicationSet
• ACM monitors cluster health via klusterlet heartbeats
• A Placement resource detects when a cluster becomes unreachable
• During failover, only DNS needs to be switched to point to the healthy cluster
• The application is already running on the DR cluster, so there is no deployment delay

Prerequisites

Before starting this guide, complete the Create ROSA HCP Disaster Recovery Infrastructure guide. That guide sets up:

• EFS CSI Driver on both clusters
• S3 Cross-Region Replication for application data and backup buckets
• EFS replication from the primary to the DR Region

You need the environment variables from that guide still set in your shell:

• PRIMARY_CLUSTER_NAME, DR_CLUSTER_NAME
• PRIMARY_REGION, DR_REGION
• APP_BUCKET_PRIMARY, APP_BUCKET_DR
• APP_S3_ROLE_ARN_PRIMARY, APP_S3_ROLE_ARN_DR
• PRIMARY_EFS, DR_EFS

If you are starting a new shell session, re-run the environment variable steps from the DR infrastructure guide .

In addition to the shared infrastructure, this guide requires:

• A third ROSA HCP cluster for the ACM hub ($CLUSTER_ACM)
• AWS CLI, oc CLI, and rosa CLI configured
• For DNS-based failover:
  • A Route 53 public hosted zone
  • A custom domain or hostname where you are allowed to create DNS records
  • certbot
  • The certbot-dns-route53 plugin

If you follow the DNS failover or custom certificate workflow, verify the Route 53 Certbot plugin before starting those steps:

Environment Variables

The following variables carry over from the DR infrastructure guide : PRIMARY_CLUSTER_NAME, DR_CLUSTER_NAME, PRIMARY_REGION, DR_REGION, APP_BUCKET_PRIMARY, APP_BUCKET_DR, APP_S3_ROLE_ARN_PRIMARY, APP_S3_ROLE_ARN_DR, PRIMARY_EFS, DR_EFS, AWS_ACCOUNT_ID. Set the additional variables needed for this guide:

Update S3 IRSA Trust Policies

The DR infrastructure guide creates S3 IRSA roles with trust policies scoped to the dr-demo namespace. This guide deploys the application in the acm-demo namespace, so the trust policies must be updated to allow service accounts from both namespaces:

Log into the ACM Hub Cluster

All resources in this guide are created on the ACM hub cluster unless otherwise noted.

Install ACM on the Hub Cluster

Install the Red Hat Advanced Cluster Management operator. Create the namespace, OperatorGroup, and Subscription:

Wait for the operator CSV to reach Succeeded:

Create the MultiClusterHub to deploy ACM components:

Wait for the MultiClusterHub to be ready. This can take several minutes:

Wait until the output shows Running.

Import Managed Clusters into ACM

Import each regional cluster so ACM can monitor and manage them.

Import the Primary Cluster

Create the ManagedCluster resource:

Log in to the primary cluster and get a token:

Log back in to the ACM hub, create the namespace, then create the auto-import secret:

Wait for the cluster to be imported and available:

Wait until AVAILABLE shows True:

Import the DR Cluster

Create the ManagedCluster resource:

Log in to the DR cluster and get a token:

Log back in to the ACM hub, create the namespace, then create the auto-import secret:

Wait for the cluster to be imported and available:

Verify both clusters are imported:

Create a ManagedClusterSet

Group the regional clusters into a ManagedClusterSet so they can be referenced as a single pool for placement decisions.

Create the ManagedClusterSet:

Add both clusters to the set:

Install OpenShift GitOps on the Hub

Install the OpenShift GitOps operator which provides ArgoCD:

Wait for the operator to install:

Wait until the PHASE shows Succeeded.

Grant the ArgoCD service accounts cluster-admin privileges. The application controller needs this to deploy resources to managed clusters, and the ApplicationSet controller needs it to read cluster secrets:

Get the ArgoCD admin password:

Custom Deployment Health Check

ArgoCD’s built-in Deployment health check evaluates the Progressing condition before Available. When a Deployment’s rollout completed successfully but the worker nodes later go down, Progressing stays True (with reason NewReplicaSetAvailable) even though Available is False. ArgoCD reports Healthy because it returns early on the Progressing check without examining Available.

Add a custom health check that evaluates Available first, so ArgoCD correctly reports Degraded when a Deployment has zero available replicas:

Bind the ManagedClusterSet to GitOps

Create a ManagedClusterSetBinding in the openshift-gitops namespace to allow ArgoCD to use the cluster set:

Register Managed Clusters with ArgoCD

Enable the application-manager addon on both managed clusters. This addon creates the ArgoCD cluster secrets that allow ArgoCD to deploy to managed clusters via the ACM cluster-proxy:

Wait for the addons to become available:

Wait until both clusters show True in the AVAILABLE column.

Create a Placement to select all clusters in the DR cluster set. This is used by the GitOpsCluster to register both clusters as ArgoCD deployment targets:

Create the GitOpsCluster resource:

Verify the clusters appear as ArgoCD cluster secrets:

You should see secrets for both managed clusters. The application-manager addon copies ManagedCluster labels to the cluster secrets, including the cluster.open-cluster-management.io/clusterset label that the ApplicationSet uses to select clusters.

Configure Direct API Access for Health Monitoring

By default, the application-manager addon configures ArgoCD to reach managed clusters through ACM’s cluster-proxy. The proxy agent runs on the managed cluster’s worker nodes. When worker nodes go down during a failure, the proxy agent becomes unavailable and ArgoCD cannot re-evaluate application health, leaving it at the last cached state.

To allow ArgoCD to detect unhealthy applications during a failure, update the cluster secrets to use the direct HCP API URL. With ROSA HCP, the hosted control plane stays running even when all worker nodes are stopped, so ArgoCD can still query the Kubernetes API and detect that Deployments have zero available replicas.

Configure ACM Placement for Failover

Create the Placement that monitors cluster health. This Placement selects exactly one cluster from the dr-clusters set. The Steady prioritizer keeps the selection on the current cluster unless it becomes unreachable. The tolerations allow 30 seconds after a cluster is tainted as unreachable before the placement moves.

This Placement is used for health monitoring. To see which cluster ACM considers healthy:

Tune Lease Duration for Faster Failover Detection

By default, ACM checks the klusterlet heartbeat lease every 5 minutes. For a faster demo, reduce the lease duration to 10 seconds on both managed clusters.

Obtain a TLS Certificate (Optional)

If you want to serve the application on a custom domain with a valid TLS certificate, obtain one using Let’s Encrypt with the certbot-dns-route53 plugin. Certbot uses your AWS credentials to create a temporary TXT record in Route 53 for domain validation.

Important: Replace your-email@example.com with your actual email address.

Set the certificate directory:

Create the ArgoCD ApplicationSet

The ApplicationSet uses a merge generator that combines two sources:

• clusters: reads ArgoCD cluster secrets, selecting all clusters in the dr-clusters ManagedClusterSet so the application is deployed to both clusters
• list: provides per-cluster configuration (region, S3 bucket, IRSA role ARN, EFS file system ID)

The merge generator joins them by cluster name. Because the application is deployed to both clusters, failover only requires switching DNS to the DR cluster.

The YAML is built in segments to cleanly embed the multi-line PEM certificate and key. The first heredoc writes everything up to certificate: |, then sed appends the indented PEM content directly from the cert files, and a final heredoc closes the YAML.

Create the ApplicationSet:

Verify the Application was created and is syncing:

Wait until both applications show Synced and Healthy:

Prepare DR Cluster for EFS Data Continuity

When ArgoCD deploys the application to the DR cluster, the Helm chart dynamically provisions new EFS access points for each PVC. These new access points create fresh, empty subdirectories – the replicated data from the primary EFS lives under different paths. To ensure the DR application sees the replicated data, pre-create static PersistentVolumes on the DR cluster that point to the original data paths.

The demo application uses 3 EFS-backed PVCs:

• shared-flight-data – shared volume mounted by the dashboard and flight recorder
• flight-data-flight-recorder-0 – StatefulSet replica 0
• flight-data-flight-recorder-1 – StatefulSet replica 1

Record the PVC-to-path mapping

Log in to the primary cluster, then map each PVC to its EFS access point path. The PV volumeHandle format is <efs-id>::<access-point-id>, and each access point has a root directory path where the PVC’s data is stored:

Pre-stage static PersistentVolumes on the DR cluster

Log in to the DR cluster, then create static PVs with claimRef pre-binding. The claimRef reserves each PV for a specific PVC so that when ArgoCD deploys the application, the PVCs bind to these PVs instead of dynamically provisioning new access points:

Log back in to the ACM hub cluster.

Set Up DNS

Create a Route 53 A record pointing to the router of the active cluster.

Get the router IP for each cluster by extracting the apps domain from the console URL:

Create the DNS record pointing to the primary cluster:

Verify the application is accessible:

Failover Test

Simulate a region failure by stopping the worker instances on the primary cluster.

Delete EFS replication to promote the DR replica to read-write:

Disable auto-repair on the primary cluster’s machine pools so ROSA does not replace the stopped workers, then stop the instances:

Watch for ACM to detect the failure and ArgoCD to report the primary application as degraded. With the tuned lease duration (10s) and toleration (30s), ACM detection takes approximately 40 seconds. ArgoCD detects the degraded Deployments on the next health evaluation cycle:

Wait until the primary cluster shows Available: Unknown, the PlacementDecision shows only the DR cluster, and ArgoCD shows the primary application as Degraded while the DR application is Healthy:

Because the application is already deployed to both clusters, no ArgoCD redeployment is needed. The DR cluster’s application is already running.

Switch DNS to the DR cluster:

Flush local DNS cache and verify:

Failback

Failing back is a manual process. The Steady prioritizer in the health-monitoring Placement keeps the selection on the DR cluster even after the primary recovers, preventing unnecessary flip-flopping.

Start the primary worker instances:

Wait for the primary cluster to rejoin ACM and for the nodes to be replaced. This typically takes 3-5 minutes:

Wait until the primary cluster shows Available: True and acm-demo-${PRIMARY_CLUSTER_NAME} shows Synced and Healthy.

Re-establish EFS replication from primary to DR. Before allowing production traffic to return to the primary cluster, verify that the application is healthy and that any required DR-side EFS and S3 data has been reconciled. Application health alone is not sufficient to determine that a stateful workload is ready for failback.

Re-establish EFS replication so it is in place for future failovers. First, disable the overwrite protection that AWS enables on the replica after replication is deleted:

Switch DNS back to the primary cluster:

Flush local DNS cache and verify:

Failover Timeline Summary

Production Considerations

• Resource overhead: The application runs on both clusters simultaneously. For resource-intensive applications, consider whether the cost of running on both clusters is acceptable. The trade-off is faster failover (DNS-only, no deployment delay) versus higher steady-state resource consumption.
• EFS path mapping: Record and maintain the PVC-to-EFS access point path mapping as part of your DR runbook. In a real disaster, the primary cluster API might not be available to query. Update this mapping whenever PVCs are recreated.
• Data reconciliation before failback: Both EFS and S3 replication are one-directional (primary to DR). Data written during failover must be manually synced or merged back to the primary before re-establishing replication. See the Disaster Recovery with OADP on ROSA HCP guide for detailed failback data reconciliation steps.
• ACM hub availability: The ACM hub is a single point of failure for failover detection. In production, deploy the hub with high availability or consider an active-passive hub configuration.
• DNS automation: Replace the manual DNS switch with Route 53 health checks and failover routing policies for fully automated DR.
• Lease duration tuning: The 10-second lease used in this guide is aggressive. For production, balance detection speed against the risk of false positives from transient network issues. A 60-second lease is a reasonable starting point.
• EFS mount targets: Ensure the DR cluster has EFS mount targets in all worker subnets before a disaster occurs. Creating mount targets during a failover adds delay to the recovery process.

Cleanup

Delete the ApplicationSet:

Delete the Placements and GitOpsCluster:

Delete the ManagedClusterSetBinding and ManagedClusterSet:

Detach the managed clusters:

Delete the DNS record: