Arrow Flight SQL

Zero-copy columnar access to Elasticsearch over gRPC — for DuckDB, Python, Apache Superset, Grafana, and any Arrow Flight SQL client.

Sidecar vs federation

There are two Flight SQL servers, and these quickstarts cover only the first:

Single-cluster sidecar (port 32010) — one server in front of one ES cluster. Cross-index JOINs run in the sidecar’s embedded DuckDB engine. This is what this page covers, and it is free in Community.
Multi-cluster federation server — a coordinator that fans out across multiple ES clusters to JOIN data living in different clusters. This is the Pro+ path; see the federation operator guide.

Features

gRPC Protocol — High-performance columnar data access over HTTP/2
BI Tool Compatible — Works with DBeaver, Superset, Grafana, DataGrip, and any Arrow Flight SQL client
Docker Ready — Pre-built Docker images for ES 6, 7, 8, and 9
Lazy Streaming — Memory-efficient; only the current batch resides in memory
Configurable Batch Size — Via arrow.flight.batch-size
Full SQL — DDL + DML + DQL, not just SELECT

Quick Start with Docker

docker run -p 32010:32010 \
  -e ELASTIC_HOST=elasticsearch \
  -e ELASTIC_PORT=9200 \
  -e ELASTIC_CREDENTIALS_USERNAME=elastic \
  -e ELASTIC_CREDENTIALS_PASSWORD=changeme \
  softnetwork/softclient4es8-arrow-flight-sql:latest

Available Docker Images

Elasticsearch	Docker Image
ES 6.x	`softnetwork/softclient4es6-arrow-flight-sql:latest`
ES 7.x	`softnetwork/softclient4es7-arrow-flight-sql:latest`
ES 8.x	`softnetwork/softclient4es8-arrow-flight-sql:latest`
ES 9.x	`softnetwork/softclient4es9-arrow-flight-sql:latest`

Pinning a release, and why there is no JAR

The images above track latest. Pin the tag for anything you need to reproduce — the current release is 0.3.4:

docker run -p 32010:32010 \
  -e ELASTIC_HOST=elasticsearch \
  softnetwork/softclient4es8-arrow-flight-sql:0.3.4

The sidecar ships as a Docker image only. The softclient4es{6,7,8,9}-arrow-flight-sql artifacts published to JFrog are small configuration jars — a manifest and a logback.xml, with no application code — so java -jar on one will not start a server. For Kubernetes, see the federation operator guide.

Python + DuckDB

import adbc_driver_flightsql.dbapi as flight_sql
import duckdb

conn = flight_sql.connect("grpc://localhost:32010")
cursor = conn.cursor()
cursor.execute("SELECT * FROM ecommerce")
table = cursor.fetch_arrow_table()   # zero-copy Arrow table

duckdb.sql("""
  SELECT category, SUM(total_price) AS revenue
  FROM table
  GROUP BY category
  ORDER BY revenue DESC
""")

Your first query

Connect with any Arrow Flight SQL client (here, the Arrow project’s stock adbc_driver_flightsql) and run a plain SELECT:

import adbc_driver_flightsql.dbapi as flight_sql

conn = flight_sql.connect("grpc://localhost:32010")
cursor = conn.cursor()
cursor.execute("SELECT * FROM my_index LIMIT 10")
table = cursor.fetch_arrow_table()   # zero-copy Arrow table

Your first JOIN

Elasticsearch SQL can’t JOIN across indices — the Flight SQL sidecar does. Free in Community: up to 2 cross-index JOINs per query (a 3-table JOIN). Both indices live in the one ES cluster the sidecar fronts; the JOIN runs in the sidecar’s embedded DuckDB engine:

cursor.execute("""
  SELECT e.name, e.salary, d.dept_name
  FROM jdbc_join_emp e
  JOIN jdbc_join_dept d ON e.dept_id = d.dept_id
""")
table = cursor.fetch_arrow_table()
# 5 rows (the orphan employee with dept_id = 99 is dropped by the INNER JOIN)

To JOIN across separate ES clusters, deploy the federation server instead — see Going further.

Configuration

arrow.flight {
  host       = "0.0.0.0"   # env: ARROW_HOST
  port       = 32010        # env: ARROW_PORT
  batch-size = 1000         # env: ARROW_BATCH_SIZE
}

elastic.credentials {
  host     = "localhost"    # env: ELASTIC_HOST (ELASTIC_IP wins when both are set)
  port     = 9200           # env: ELASTIC_PORT
  username = "elastic"      # env: ELASTIC_CREDENTIALS_USERNAME
  password = "changeme"     # env: ELASTIC_CREDENTIALS_PASSWORD
}

Arrow Type Mapping

SQL Type	ES Type	Arrow Type
`TINYINT`	`byte`	`Int(32)`
`SMALLINT`	`short`	`Int(32)`
`INT`	`integer`	`Int(32)`
`BIGINT`	`long`	`Int(64)`
`REAL`	`float`	`Float(SINGLE)`
`DOUBLE`	`double`	`Float(DOUBLE)`
`BOOLEAN`	`boolean`	`Bool`
`DATE`	`datetime`	`Date(MILLISECOND)`
`TIMESTAMP`	`datetime`	`Timestamp(MS)`
`VARCHAR`	`text`	`Utf8`
`KEYWORD`	`keyword`	`Utf8`
`VARBINARY`	`binary`	`Binary`
`STRUCT`	`object`	`Struct`
`GEO_POINT`	`geo_point`	`Struct{Float64, Float64}`
`ARRAY<*>`	—	`List`
`ARRAY<STRUCT>`	`nested`	`List<Struct>`

The table is the declared mapping. A plain query’s Arrow schema is inferred from the values of its rows (a BIGINT column holding small whole numbers arrives as Int(32)); a set operation’s branches are typed from the declared types above when the statement is planned, and a JOIN result carries DuckDB’s types. A column the reader cannot answer — a dictionary-encoded column, an Arrow type it has no reader for — fails naming the column instead of reading NULL (since arrow-extensions 0.3.4).

In-process alternative: the ADBC driver

If you need columnar Arrow access without running a separate server, the in-process ADBC driver delivers the same SQL (including cross-index JOIN) inside your JVM — see the ADBC Driver page. ADBC is Java/JVM in-process; polyglot clients (Python, Go, DuckDB, C++) connect to this Flight SQL server.

Version compatibility

Driver	Scala	ES versions	Clients	Process model
Arrow Flight SQL	cross-built Scala 2.12 + 2.13 (server)	ES 6.x / 7.x / 8.x / 9.x	Any ADBC/Flight SQL client — Python, Go, DuckDB, C++, Grafana, Superset	Separate server (gRPC)

The fat JARs are Scala-version-independent for consumers — you almost never need to think about the Scala axis.

Live Demos

The three demos below are profiles of one docker-compose.yml, in the demo/ directory of the SoftClient4ES repository. Run them from there:

git clone https://github.com/SOFTNETWORK-APP/SoftClient4ES.git
cd SoftClient4ES/demo

Start one profile at a time — the Superset and Grafana profiles both publish host port 32010, so the second one to start fails to bind it.

DuckDB + Python Pipeline

docker compose --profile duckdb up

Apache Superset BI Dashboards

docker compose --profile superset-flight up

Grafana BI Dashboards

docker compose --profile grafana up

Licensing & self-selection

All client drivers (JDBC, ADBC, and the REPL) plus the Arrow Flight SQL sidecar are free in Community, including up to 2 cross-index JOINs per query (a 3-table JOIN). A 4-table JOIN (a 3rd cross-index JOIN in one query) is rejected by the planner with a message ending … Upgrade to Pro … See: https://portal.softclient4es.com/pricing.

The single-cluster sidecar on this page is the free shape. Multi-cluster federation — joining across separate ES clusters — is Pro+ (maxClusters 1 / 5 / ∞). See the pricing & licensing page for the full quota matrix.

What does NOT work yet

Since engine 0.24.0, subqueries (IN (SELECT …) / NOT IN, EXISTS / NOT EXISTS, scalar and quantified comparisons) and derived tables (FROM (SELECT …), JOIN (SELECT …)) are supported, correlated or not. Non-recursive CTEs (WITH name AS (SELECT …)) are supported too, at the top of a SELECT; WITH RECURSIVE and CTE column lists are refused by name. Set operators — UNION ALL, UNION / UNION DISTINCT, INTERSECT / INTERSECT ALL, EXCEPT / EXCEPT ALL — are supported as well; everything but UNION ALL runs on the relational engine. See Known Limitations & Roadmap for the forms that are still refused.

Going further

ADBC Driver — the in-process columnar alternative (no separate server).
Cross-Index JOIN walkthrough — the full JOIN matrix (rows 1/2/3) with worked examples.
Multi-cluster federation operator guide — the Pro+ path: JOIN across separate ES clusters.
Known Limitations & Roadmap — what works in the current release vs what’s coming.

Telemetry

The Arrow Flight SQL sidecar sends one anonymous usage ping per day (no IP, no SQL, no PII). Opt out with softclient4es.telemetry.enabled = false in the server HOCON config, the SOFTCLIENT4ES_TELEMETRY_ENABLED=false environment variable, or -Dsoftclient4es.telemetry.enabled=false. The sidecar has no connection-string opt-out — the grpc:// URI carries no telemetry option. See Telemetry & Privacy for the full field list and per-surface details.

Known limitations

Subqueries, derived tables, non-recursive CTEs — which some BI tools auto-generate — and the UNION / INTERSECT / EXCEPT set operators all work since engine 0.24.0. See Known Limitations & Roadmap for exactly what works today and what is still refused.

License

Arrow Flight SQL is licensed under the Elastic License 2.0 — free to use, not open source.