It's 2 AM, the Outage Spans Three Regions, and You Have Eleven Kibana Tabs

Part of the SoftClient4ES launch series. Previously: Stop ETL’ing Elasticsearch Into Your Warehouse Just to JOIN It and The JOIN Matrix: How Cross-Index JOIN Actually Works. This one is a story.

02:14

The page comes in worded the way the worst ones always are: “Elevated checkout failures. Multiple regions.”

Multiple regions. That word does something to your stomach at 2 AM, because it means the comfortable single-pane-of-glass mental model you use for normal incidents doesn’t apply. Your logs live in prod_us. Your metrics — request rates, p99 latencies, error counters — live in prod_eu, because that’s where the observability team consolidated them after the last reorg. Your distributed traces live in prod_ap, close to the service mesh that emits them.

Three Elasticsearch clusters. Three Kibana instances. Three SSO logins. And one question you need answered right now: which service, in which region, started failing first, and is the failure following the traffic or following a dependency?

So you start opening tabs.

The Tab-Hopping Tax

Tab one: prod_us Kibana. You filter logs to the last fifteen minutes, level ERROR, and eyeball the service breakdown. Checkout-api is lit up. You copy the spike timestamp into a sticky note, because you’re going to need it in the other two clusters and there is no way to carry it across.

Tab two: prod_eu Kibana. Different login. You pull up the latency metrics for checkout-api around that timestamp. p99 is through the roof — but is that cause or effect? You can’t tell from metrics alone. You need to line the latency spike up against the error spike against the traces, and right now those three things live in three browsers that have never heard of each other.

Tab three: prod_ap Kibana. Yet another login. You grab a trace ID from one of the failing requests — except you copied it from the US logs and now you’re hand-pasting it into an AP query box, hoping you didn’t fat-finger a character.

By 02:31 you have eleven tabs open, three timestamps on a sticky note, and a growing certainty that the correlation — the actual answer — only exists in your head, assembled by hand, and will evaporate the moment you blink. This is not analysis. This is clerical work performed under stress.

The data to answer the question exists. It’s just scattered across three clusters that refuse to be joined.

What If the Query Crossed Regions Instead of You?

Here’s the same triage, after this release.

You don’t open Kibana. You open the SQL surface you already trust for incidents — the REPL, or DBeaver, or whatever speaks to the Federation coordinator — and you write the correlation as one query, with each cluster named by its catalog prefix. That prefix is just the name you gave the cluster when you registered it, so in the chart’s three-region example these read us-east-1, eu-west-1, ap-south-1. The table and field names below are illustrative — swap in your own log, metric and trace indices; what matters is that each lives in a different registered catalog:

-- Illustrative schema: logs / metrics / traces across three registered Federation catalogs.
-- Adapt the index and field names to your own telemetry.
SELECT  l.service,
        l.region,
        COUNT(DISTINCT l.trace_id) AS failing_requests,
        MAX(m.p99_latency_ms)      AS worst_p99,
        MAX(t.duration_ms)         AS slowest_span,
        MIN(l.ts)                  AS first_error,
        MAX(l.ts)                  AS last_error
FROM    `prod_us`.logs    AS l
JOIN    `prod_eu`.metrics AS m
        ON l.service = m.service
       AND l.region  = m.region
JOIN    `prod_ap`.traces  AS t
        ON l.trace_id = t.trace_id
WHERE   l.level = 'ERROR'
  AND   l.ts >= NOW() - INTERVAL 30 MINUTE
  AND   m.ts >= NOW() - INTERVAL 30 MINUTE
GROUP BY l.service, l.region
HAVING  COUNT(DISTINCT l.trace_id) > 50
ORDER BY failing_requests DESC;

Two details in there are load-bearing. It counts DISTINCT l.trace_id, not rows: joining on a non-unique key multiplies rows, so a plain COUNT(*) would count joined tuples rather than failures — while MIN and MAX would be unaffected, so the wrong number would sit next to four correct ones and look right. And the 30-minute window is applied to both the logs and the metrics leg, so worst_p99 is the incident’s worst latency rather than the worst since the index was created.

The bare-alias ORDER BY in that query needs REPL bundle 0.20.4 (or JDBC / ADBC / Flight SQL driver 0.2.5) — on earlier builds it comes back as Ambiguous column. Repeat the expression (ORDER BY COUNT(DISTINCT l.trace_id) DESC) if you are not upgraded yet.

This is a Row 3 multi-source JOIN from the JOIN matrix: logs from the US cluster, metrics from the EU cluster, and traces from the AP cluster, joined coordinator-local. Each leg is fetched from its own region in parallel, gathered on the coordinator, and joined in one place. The result is a single table that finally puts the failing requests, the latency spike, and the slowest span on the same row:

| service       | region    | failing_requests | worst_p99 | slowest_span | first_error          | last_error           |
|---------------|-----------|------------------|-----------|--------------|----------------------|----------------------|
| checkout-api  | us-east-1 | 412              | 4,910     | 4,612        | 2026-08-12T02:09:12Z | 2026-08-12T02:31:40Z |
| checkout-api  | eu-west-1 | 266              | 4,880     | 4,180        | 2026-08-12T02:11:55Z | 2026-08-12T02:31:38Z |
| checkout-api  | ap-south-1|  61              |   220     |   210        | 2026-08-12T02:29:02Z | 2026-08-12T02:31:31Z |

Read that table for three seconds and the incident reshapes itself. us-east-1 failed first (02:09), eu-west-1 followed two minutes later (02:11), and ap-south-1 only started wobbling at 02:29 — and its p99 and its slowest span are both normal, an order of magnitude below the other two. This isn’t traffic-following. It’s a dependency propagating outward from us-east-1, and ap-south-1 is barely touched. You have a direction. You have a primary region. You have it in one query instead of eleven tabs — and it spanned all three clusters at once: logs in prod_us, metrics in prod_eu, traces in prod_ap.

One caveat worth knowing before you paste this into your own incident channel: this is an INNER JOIN, so an error whose trace was never sampled does not appear. Reach for LEFT JOIN when the left side is the population you actually mean — the JOIN cardinality notes cover both fan-out and dropped rows.

Drilling Into the Spans

The aggregate already told you the AP traces cluster is in play. Now you want the individual trace evidence — the specific slow spans behind that slowest_span number — so you drill the same prod_ap.traces leg down to the row level:

-- Illustrative schema: logs / metrics / traces across three registered Federation catalogs.
-- Adapt the index and field names to your own telemetry.
SELECT  l.service,
        l.region,
        t.trace_id,
        t.span_name,
        t.duration_ms
FROM    `prod_us`.logs   AS l
JOIN    `prod_ap`.traces AS t ON l.trace_id = t.trace_id
WHERE   l.service = 'checkout-api'
  AND   l.region  = 'us-east-1'
  AND   l.level   = 'ERROR'
  AND   l.ts >= NOW() - INTERVAL 30 MINUTE
ORDER BY t.duration_ms DESC
LIMIT 25;

The trace IDs never leave the query. No sticky note. No fat-fingered paste between two browsers. The slowest span on the failing requests comes back at the top, and it’s the same downstream dependency every time. You have your root cause, and you have it because the query crossed the region boundary so you didn’t have to.

Total wall-clock from page to root cause: the time it took to type two SQL statements and read two tables.

How It’s Wired

You don’t get cross-cluster JOIN by accident — it’s a deployment shape. The Federation coordinator is the piece that knows about all three clusters, runs each leg on its home cluster in parallel, gathers the results, and does the join coordinator-local. You stand it up the same way you stand up anything else in your cluster: with a Helm chart.

The public softclient4es-helm chart ships a three-region topology example with the same bones as this story — three registered clusters behind one coordinator. It splits by geography rather than by signal type (and runs mixed ES versions, 8 and 9, to show a migration window); the catalog names are yours to choose either way. There’s a single-cluster example to start with — the only one that runs on Community — and a heterogeneous-ready example that previews heterogeneous sources: Postgres, MySQL and Snowflake as JOIN legs, arriving in a later release. The operator guide that ships alongside the chart walks through registering clusters, wiring credentials, and the readiness model.

One operational detail worth knowing before you page yourself at 2 AM with this. By default the coordinator’s readiness probe is all-or-nothing: if any one regional sidecar is unreachable, the coordinator goes NotReady and every query fails — including the ones that only touch healthy regions. That is deliberate fail-closed routing, and it is the right default for correctness. But for incident work, where partial availability beats no availability, set federation.probes.useGrpc: false and the coordinator stays up and degrades per-query instead. The chart documents both modes; pick the one that matches how you want to be woken up.

A note on tiers, because it’s the honest part: Community has Federation, metered to one cluster. Single-cluster federation is free. The second cluster is what moves you to Pro (5 clusters), and a three-region setup like this one is squarely a Pro deployment. The meter is the gate, not the feature — Federation isn’t dangled as enterprise-only; it’s the cluster count that scales with the tier.

And one promise we’re not making in this post: no latency numbers, no “Nx faster than tab-hopping.” The win here isn’t a benchmark — it’s that the correlation exists as a single artifact you can read, share in the incident channel, and re-run, instead of a fragile assembly held together in one tired engineer’s short-term memory at 2 AM.

The Morning After

The thing that actually changes isn’t the time you saved at 2 AM, though you’ll take it. It’s that the query you wrote during the incident is durable. You paste it into the postmortem. You save it to the team’s incident-query folder next to the single-cluster triage queries. The next person paged for a multi-region event doesn’t reconstruct the correlation from scratch in eleven tabs — they open the folder and run it.

Cross-cluster correlation stops being a heroic act of manual assembly and becomes what it should have been all along: a query.

Resources

• Helm chart (Federation): softclient4es-helm — see the three-region example
• GitHub: SoftClient4ES Repository
• SQL Reference: JOINs in SoftClient4ES SQL — including the JOIN cardinality note
• Discussions: Ask Questions
• LinkedIn: SoftNetwork

How many browser tabs does your team open during a multi-region incident? If the answer is “too many,” there’s a query that replaces them.

P.S. — Cross-cluster JOIN runs through the Federation coordinator, deploys via the public Helm chart, and turns a three-cluster correlation into one re-runnable SQL statement. Community includes Federation at one cluster; multi-region is Pro.