Published on

API Failover Across Independent Istio Clusters with an Existing NGINX Plus Ingress

Authors
  • avatar
    Name
    Wenzhuo Zhao
    Twitter

Running the same API in two Kubernetes clusters gives us a backup deployment. It still leaves an important question: who decides when a browser request should use that backup?

Consider two independent Istio meshes. Each has an Istio ingress gateway and several APIs, while a separate F5 NGINX Plus Ingress Controller serves the frontend. The browser currently calls the API gateway in Cluster A directly.

The solution explored here is to put the existing NGINX Plus ingress in the API request path. It probes each API through both gateways, prefers Cluster A, and uses Cluster B when that API's primary becomes unavailable. Istio continues routing requests inside each cluster.

This is a deployment template based on the original design discussion and the official references linked below. It has not been exercised against a live cluster; validate it against your installed controller and CRD versions before rollout.

Start with the actual request path

Here is the starting point:

ComponentCluster ACluster B
Frontendhttps://example.a.com/app-ahttps://example.b.com/app-a
API gatewayhttps://example.asm.comhttps://example.bsm.com
App A API/api/app-a/api/app-a
App B API/api/app-b/api/app-b

Both API applications listen on port 8000. For this example, their Kubernetes Services also expose port 8000.

A frontend being served through NGINX does not mean its API calls pass through NGINX. When JavaScript calls https://example.asm.com/api/app-a/users, the browser opens a separate connection to that domain. A failover rule on the frontend ingress cannot affect that connection.

Change the API base URL to https://api.example.com, and point that new name at the existing NGINX Plus ingress:

Browser requests enter the existing NGINX Plus ingress, which selects a primary or backup Istio gateway across two independent meshes.

Figure 1. The existing ingress gains an API hostname. The two Istio meshes stay independent.

NGINX connects to the gateways over ordinary HTTPS. There is no special connector between NGINX and Istio, and no cross-cluster service discovery is needed for this design.

Give each API its own health decision

The target behavior is deliberately specific:

Per-API failover sends App A to Cluster B during an App A failure in Cluster A, while App B stays on Cluster A; App A returns after recovery.

Figure 2. An App A failure changes App A's upstream selection while App B stays on Cluster A.

A successful TCP connection to a gateway says very little about an application behind it. Instead, each API should expose a readiness endpoint through its normal gateway route:

GET /api/app-a/health/ready

Return 200 when the application can serve its core workload and 503 when it cannot. Avoid returning 200 with a JSON body that merely says DOWN. Keep the check inexpensive, include essential dependencies, and exclude optional dependencies whose failure should not trigger a regional move.

NGINX Plus supports periodic HTTP probes and configurable failure/recovery thresholds. Probing the real API path also exercises gateway routing. See the NGINX HTTP health-check guide.

Kubernetes readiness probes remain useful for removing individual unhealthy Pods from their local Service. They do not select another cluster. Also, two clusters depending on the same failed database may both become unready; traffic routing cannot repair that shared dependency.

Prerequisites and example names

This walkthrough assumes F5 NGINX Ingress Controller running NGINX Plus, with custom resources enabled. It does not apply directly to the community ingress-nginx controller or an NGINX Open Source deployment.

Use these names consistently, replacing them with your actual resources:

SettingExample
Kubernetes contextscluster-a, cluster-b
NGINX resource namespaceapi-routing
Existing NGINX controller namespace / ConfigMapnginx-ingress / nginx-config
Existing controller Deployment / containernginx-ingress / nginx-ingress
IngressClassnginx
Istio Gatewayistio-system/api-gateway
API Serviceapp-a.backend.svc.cluster.local:8000

The NGINX Pods must be able to resolve and reach both gateway addresses on TCP 443. Verify routing, firewalls, NetworkPolicies and any load-balancer source restrictions. A gateway address reachable from your laptop may still be unreachable from those Pods.

Check the installed CRDs before using the examples:

kubectl --context cluster-a get ingressclass nginx
kubectl --context cluster-a explain virtualserver.spec.upstreams \
  --api-version=k8s.nginx.org/v1
kubectl --context cluster-a explain policy.spec.egressMTLS \
  --api-version=k8s.nginx.org/v1
kubectl --context cluster-a create namespace api-routing

If the namespace already exists, skip the last command. Confirm that the existing controller watches it and that no other Ingress or VirtualServer owns api.example.com on the same controller.

1. Align the gateway address, TLS SNI and HTTP Host

There are three separate pieces of an upstream HTTPS connection:

NGINX resolves a cluster-specific gateway address, then uses api.example.com for TLS SNI, certificate verification and the HTTP Host header.

Figure 3. Resolve a cluster-specific address, but use a common TLS and HTTP identity.

For example, NGINX resolves example.bsm.com to reach Gateway B, while sending api.example.com as both TLS SNI and HTTP Host. Gateway B's certificate must cover api.example.com, and its routing configuration must accept that host.

Keep example.asm.com and example.bsm.com pointing directly at their respective Istio gateways. They are the upstream addresses. If either resolves back to the NGINX API frontend, the proxy can call itself in a loop.

In each cluster, add a dedicated HTTPS server for the shared hostname to the existing Gateway. The following is an additional spec.servers entry, not a replacement for its current servers:

- port:
    number: 443
    name: https-api-shared
    protocol: HTTPS
  hosts:
    - api.example.com
  tls:
    mode: SIMPLE
    credentialName: api-shared-tls

Provision api-shared-tls using your normal certificate process in the namespace where the selected gateway workload loads credentials, typically istio-system. Each gateway needs a valid certificate for the shared name; the certificates need not have identical private keys. Keep the existing cluster-specific hosts and certificates intact. The Istio Gateway reference describes server, host and TLS configuration.

Update App A's existing VirtualService in Cluster A to accept the new hostname. The target shape is:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: app-a
  namespace: backend
spec:
  hosts:
    - example.asm.com
    - api.example.com
  gateways:
    - istio-system/api-gateway
  http:
    - match:
        - uri:
            exact: /api/app-a
        - uri:
            prefix: /api/app-a/
      route:
        - destination:
            host: app-a.backend.svc.cluster.local
            port:
              number: 8000

In Cluster B, use example.bsm.com in place of example.asm.com. Merge these changes into existing resources, retaining their authentication, CORS, rewrites and other routes. If the Kubernetes Service exposes 80 with targetPort: 8000, use 80 in the destination. See the Istio VirtualService reference.

2. Name the two external gateways

Save the following as external-services.yaml in your deployment workspace:

apiVersion: v1
kind: Service
metadata:
  name: istio-primary
  namespace: api-routing
spec:
  type: ExternalName
  externalName: example.asm.com
  ports:
    - name: https
      port: 443
---
apiVersion: v1
kind: Service
metadata:
  name: istio-backup
  namespace: api-routing
spec:
  type: ExternalName
  externalName: example.bsm.com
  ports:
    - name: https
      port: 443

These objects create no proxy Pods. They give the controller names for the two external destinations. Every API can reuse these Services while maintaining its own upstream health state.

ExternalName resolution also requires a resolver in the controller's active ConfigMap. Merge these keys into the existing ConfigMap, preserving its other settings:

data:
  resolver-addresses: '10.96.0.10'
  resolver-valid: '10s'
  resolver-timeout: '5s'

10.96.0.10 is illustrative. Use the resolver reachable from your NGINX Pods and capable of resolving both gateway names. Confirm the active ConfigMap from the controller's deployment arguments rather than assuming its name. Refer to the F5 ConfigMap reference.

3. Verify upstream certificates

There are two TLS hops: browser → NGINX, and NGINX → Istio. Provision the browser-facing certificate as api-public-tls in api-routing through your usual certificate management process.

For the second hop, put the CA bundle that trusts both gateway certificate chains into a Secret:

kubectl --context cluster-a -n api-routing \
  create secret generic istio-upstream-ca \
  --from-file=ca.crt=./trusted-ca.pem

Save this as upstream-tls-policy.yaml:

apiVersion: k8s.nginx.org/v1
kind: Policy
metadata:
  name: istio-upstream-tls
  namespace: api-routing
spec:
  egressMTLS:
    trustedCertSecret: istio-upstream-ca
    verifyServer: true
    verifyDepth: 2
    serverName: true
    sslName: api.example.com

Despite the policy's name, this example uses server-authenticated TLS: no client certificate is configured. Set verifyDepth for your actual certificate chains. The policy enables verification and SNI using the shared hostname. See the F5 EgressMTLS reference.

4. Configure App A's primary and backup

Save this as virtualserver.yaml:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: api-failover
  namespace: api-routing
spec:
  ingressClassName: nginx
  host: api.example.com
  tls:
    secret: api-public-tls
    redirect:
      enable: true
  policies:
    - name: istio-upstream-tls
  upstreams:
    - name: app-a
      service: istio-primary
      port: 443
      backup: istio-backup
      backupPort: 443
      lb-method: round_robin
      tls:
        enable: true
      connect-timeout: 3s
      read-timeout: 30s
      send-timeout: 30s
      next-upstream: 'off'
      healthCheck:
        enable: true
        path: /api/app-a/health/ready
        interval: 5s
        fails: 3
        passes: 3
        mandatory: true
        connect-timeout: 2s
        read-timeout: 2s
        send-timeout: 2s
        statusMatch: '200'
        headers:
          - name: Host
            value: api.example.com
  routes:
    - path: '~ ^/api/app-a(/|$)'
      action:
        proxy:
          upstream: app-a
          requestHeaders:
            set:
              - name: Host
                value: api.example.com

The backup is an ExternalName Service, and round_robin is compatible with backup selection. The route boundary excludes names such as /api/app-admin; no URI rewrite is configured. Probes require exactly 200, and new upstream peers must pass checks before receiving traffic. Field definitions are in the F5 VirtualServer reference.

Failover and retry are separate decisions. This example explicitly disables retrying a failed request with next-upstream: "off". Once health selection excludes A, subsequent requests can use B, but a request already sent to A can fail. This conservative starting point avoids proxy replays of ambiguous writes. Review retry behavior separately for your API; the NGINX proxy retry documentation explains its conditions and limitations.

To add App B, create another upstream named app-b, reuse the same two Services, set its probe to /api/app-b/health/ready, and add its own boundary-matched route. Sharing gateway addresses does not require sharing the upstream group.

5. Validate before changing DNS

Apply the Istio changes in both clusters first, then validate and apply the NGINX resources in Cluster A:

kubectl --context cluster-a apply --dry-run=server -f external-services.yaml
kubectl --context cluster-a apply --dry-run=server -f upstream-tls-policy.yaml
kubectl --context cluster-a apply --dry-run=server -f virtualserver.yaml

kubectl --context cluster-a apply -f external-services.yaml
kubectl --context cluster-a apply -f upstream-tls-policy.yaml
kubectl --context cluster-a apply -f virtualserver.yaml

kubectl --context cluster-a -n api-routing \
  get virtualserver api-failover -o yaml
kubectl --context cluster-a -n api-routing \
  describe virtualserver api-failover

Schema acceptance alone does not prove the controller accepted the configuration. Check status and events for the VirtualServer and Policy, then inspect the generated configuration in each controller Pod. For example, using the example Deployment and container names:

kubectl --context cluster-a -n nginx-ingress \
  exec deployment/nginx-ingress -c nginx-ingress -- nginx -T

Confirm that the upstream contains the primary and backup peers, that active probes cover both, and that the resulting locations include HTTPS, SNI, CA verification and the intended Host header. Check that the generated probe location also inherits the required TLS settings. Treat any mismatch as a rollout blocker.

Before publishing DNS, test against the ingress address directly:

# Replace the documentation address with the actual NGINX ingress IP.
NGINX_INGRESS_IP=203.0.113.10
curl --fail-with-body --resolve "api.example.com:443:${NGINX_INGRESS_IP}" \
  https://api.example.com/api/app-a/health/ready

Do not use -k: certificate verification is part of this test. After this works, point api.example.com at the existing ingress and update both frontend deployments to use it:

const API_BASE_URL = 'https://api.example.com'
fetch(`${API_BASE_URL}/api/app-a/users`)

Because the frontend and API still have different origins, retain CORS support for https://example.a.com and https://example.b.com. Verify preflight requests, authorization headers, cookies and redirects. Assign CORS to one layer to avoid conflicting headers. Check that the new path still enforces the intended authentication and authorization rules.

6. Run a controlled failure drill

Use a staging environment and an application response header such as X-Served-Cluster: A or B to identify who actually handled each request. A 200 response alone cannot prove failover.

TestExpected observation
Both App A deployments healthyRequests prefer A
App A becomes unready in ANew App A requests use B after detection
App B stays healthy in AApp B continues using A
App A in A recoversNew requests prefer A after recovery checks
Gateway A becomes unreachable; NGINX remains reachableAPIs with healthy backups use B
App A fails in both clustersApp A returns an error; it does not route to another app
Login, OPTIONS and a write during transitionAuth works; writes are not silently duplicated

With five-second probes and three failures, allow several probe cycles for detection. Timeouts, scheduling and passive failure handling affect observed timing; this is not a strict fifteen-second availability guarantee. Existing streams are not migrated, and failback applies to subsequent upstream selection.

Prefer a controlled readiness failure over blindly scaling production Deployments. Record the original configuration and coordinate any drill with HPA and GitOps reconciliation.

Where this solution stops

If the only NGINX ingress serving api.example.com is inside Cluster A, a complete Cluster A outage removes the component that makes the failover decision. A healthy API in B cannot help a browser reach an unavailable ingress.

For cluster-wide resilience, configure the existing NGINX Plus controllers in both clusters to serve the API hostname and use an existing global load balancer or health-aware DNS service to select a reachable ingress. Browser and resolver caching still affect DNS-based recovery. Merely publishing two ingress addresses does not establish health-aware entry-point failover.

Application recovery also depends on data: shared or replicated state, compatible token validation, session availability, file storage and sufficient standby capacity. A readiness check should reflect whether B can safely handle its workload, including writes.

The useful boundary is clear: NGINX Plus chooses a healthy cluster for each API; Istio routes to the application inside that cluster. Reusing the existing ingress gives us a small, testable change to the request path, while leaving entry-point resilience and data recovery as explicit parts of the wider design.