Architecture Overview

SoftClient4ES is a SQL gateway that translates standard SQL into Elasticsearch operations. It provides a unified GatewayApi that accepts any SQL statement and routes it to the appropriate executor.

High-Level Architecture

Statement Routing

The GatewayApi exposes a single entry point:

def run(sql: String): Future[ElasticResult[QueryResult]]

SQL statements are automatically classified and routed:

SQL Type	Executor	Examples
DQL	DqlExecutor	SELECT
DML	DmlExecutor	INSERT, UPDATE, DELETE, COPY INTO
DDL	DdlRouterExecutor	CREATE/ALTER/DROP TABLE, PIPELINE, WATCHER

The API automatically normalizes SQL (removes comments, trims whitespace), splits multiple statements separated by ; (quote-aware — a ; inside a string literal or a -- comment is not a boundary), parses each statement into AST nodes (each must be complete on its own; leftover input is a parse error, not silently discarded), and dispatches to the correct executor. Statements run sequentially, stopping at the first failure; the last statement’s result is returned.

Result Types

DQL Results

Type	Description
QueryRows	Materialized rows (maps)
QueryStream	Streaming rows via scroll
QueryStructured	Raw Elasticsearch response

DML Results

DmlResult(inserted = N, updated = N, deleted = N, rejected = N)

DDL Results

Type	Returned by
DdlResult(success)	CREATE, ALTER, DROP, TRUNCATE
TableResult(table)	SHOW TABLE
PipelineResult(pipeline)	SHOW PIPELINE
SQLResult(sql)	SHOW CREATE TABLE/PIPELINE
QueryRows	DESCRIBE TABLE/PIPELINE

Module Structure

The project is organized into a multi-module ecosystem:

Module	Purpose
Core (Apache 2.0)	SQL parser, AST, query translation, GatewayApi, REPL
Extensions (Elastic v2)	Materialized Views, SPI extensions
JDBC (Elastic v2)	JDBC Type 4 driver
Arrow Core (Elastic v2)	Arrow vector conversion, shared abstractions
ADBC Driver (Elastic v2)	In-process columnar access via ADBC API
Arrow Flight SQL (Elastic v2)	gRPC-based columnar server

Client Access Methods

Method	Process Model	Data Format	Protocol	Use Case
REPL	Standalone	Text (ASCII/JSON/CSV)	Direct	Ad-hoc queries
JDBC	In-process	Row-based (ResultSet)	JDBC API	Java apps, BI tools
ADBC	In-process	Columnar (Arrow)	ADBC API	Analytics, data engineering
Flight SQL	Separate server	Columnar (Arrow)	gRPC (HTTP/2)	Multi-client, networked

Version Isolation

Each Elasticsearch major version has its own subproject tree (es6/, es7/, es8/, es9/). Version-agnostic code in core/ and sql/ never imports from version-specific modules. A bridge template pattern ensures code sharing across versions while maintaining isolation.

SPI Pattern

Client implementations are discovered via Java’s ServiceLoader mechanism (ElasticClientFactory), allowing clean separation between the SQL engine and Elasticsearch client libraries.

Classloader resolution

Every SPI lookup in the library (ElasticClientSpi, ExtensionSpi and the licensing LicenseManagerSpi) resolves its providers against the classloader that loaded the SPI interface — the one holding softclient4es-core and softclient4es-licensing — never the thread context classloader. The client factory’s built-in elastic.* defaults (ElasticConfig, read from softnetwork-elastic.conf) are resolved the same way. This is what lets the library run inside hosts that own the context classloader (Tableau, plugin containers, application servers) without a provider silently going missing.

Consequences:

• Ship the client, extension and licensing jars on the same classpath as the core jar — a flat classpath (-cp core.jar:lib/*) or a single shaded jar both qualify.
• A provider visible only through the thread context classloader (for example an extension jar placed on a child classloader with the context classloader pointing at it) is no longer discovered.
• An empty provider list is reported with a WARN naming the SPI interface at the two sites that used to degrade silently (ExtensionRegistry, LicenseRefreshStrategyFactory); ElasticClientFactory fails loudly instead, with No ElasticClientSpi implementation found through <classloader>: the client jar must be on the same classpath as softclient4es-core ....