SQL for Elasticsearch · Part 1

Stop Rewriting Your Elasticsearch Code Every Version Upgrade

One client API across Elasticsearch 6, 7, 8 and 9 — so a major-version upgrade stops meaning a rewrite.

November 18, 2025 ·8 min read

The Version Upgrade Nightmare 😤

Picture this: It’s Monday morning. Your DevOps team just announced that production Elasticsearch clusters are being upgraded from version 7 to 8. Great news for security and performance, right?

Not so fast.

You open your IDE and realize that 40% of your codebase needs to be rewritten. The RestHighLevelClient you’ve been using? Deprecated. Your carefully crafted queries? Breaking changes. Your bulk operations? Different API.

Welcome to the Elasticsearch version upgrade treadmill — where every major version means days (or weeks) of refactoring, testing, and praying nothing breaks in production.

Sound familiar?

You’re not alone.

The True Cost of Version Lock-In 💸

Let’s talk numbers. When your codebase is tightly coupled to a specific Elasticsearch version, here’s what you’re really paying:

1. Developer Time = Money

Average refactoring time per service: 3–5 days
Number of microservices using ES: 10–20+
Total cost: Weeks of engineering time that could be spent building features

2. Technical Debt Accumulation

Teams delay upgrades because “it’s too much work”
Security patches get skipped
Performance improvements remain out of reach
The gap between your version and the latest grows wider

3. Production Risk

Rushed migrations lead to bugs
Incomplete testing causes downtime
Rollback complexity increases exponentially
Customer trust erodes with every incident

4. Team Morale

Let’s be honest — no engineer joined your company to rewrite the same Elasticsearch queries every 18 months. They want to solve real problems, not fight with API changes.

Why Does This Keep Happening? 🤔

Elasticsearch’s official clients are version-specific by design:

// Elasticsearch 6 & 7: RestHighLevelClient
val client = new RestHighLevelClient(
  RestClient.builder(new HttpHost("localhost", 9200, "http"))
)

// Elasticsearch 8+: New Java API Client
val client = new ElasticsearchClient(
  new RestClientTransport(restClient, new JacksonJsonpMapper())
)

Different classes. Different APIs. Different mental models.

Every major version brings:

✅ Better features
✅ Performance improvements
❌ Breaking changes that cascade through your entire codebase

The Elasticsearch team has valid reasons for these changes — but you shouldn’t have to pay the migration tax every time.

The Solution: Write Once, Run Everywhere ✨

What if your Elasticsearch code looked like this instead?

import app.softnetwork.elastic.client._

// Works on ES 6, 7, 8, AND 9
val client: ElasticClientApi = ElasticClientFactory.create()
// Index a document
client.index(
  index = "users",
  id = "user-123",
  document = """{"name":"Alice","email":"alice@example.com","age":30}""",
  wait = true
)
// Search using SQL (yes, SQL!)
val results = client.search(
  SQLQuery("SELECT * FROM users WHERE age > 25 ORDER BY name")
)
// Bulk operations with backpressure
implicit val bulkOptions = BulkOptions(defaultIndex = "products")
client.bulkSource(
  items = productStream,
  toDocument = product => product.toJson
).runWith(Sink.foreach {
  case Right(success) => logger.info(s"✅ Indexed: ${success.id}")
  case Left(failure) => logger.error(s"❌ Failed: ${failure.error}")
})

Same code. Four Elasticsearch versions. Zero refactoring.

Meet SoftClient4ES: Your Version-Agnostic Elasticsearch Client 🚀

SoftClient4ES is a production-ready Scala client that abstracts away version differences behind a unified, stable API.

Core Philosophy

Version Resilience: Write your code once, deploy it against any supported ES version
Type Safety: Leverage Scala’s type system for compile-time guarantees
Production Ready: Built-in error handling, validation, and monitoring
Developer Experience: Intuitive APIs that feel natural to Scala developers

How It Works: The Magic Behind Version Agnosticism 🎩

SoftClient4ES uses a trait-based abstraction layer with multiple backend implementations:

// Unified API trait
trait ElasticClientApi
  extends SearchApi
  with IndexApi
  with BulkApi
  with MappingApi
  // ... 12+ composable APIs

// Backend implementations
class JavaClientApi       // For ES 8 & 9
class RestHighLevelClientApi  // For ES 6 & 7
class JestClientApi       // For ES 6 (alternative)

At runtime, the Service Provider Interface (SPI) automatically loads the client available in your classpath:

// Create a unified client using Service Provider Interface (SPI)
val client = ElasticClientFactory.create()

Your code never changes. Only the underlying implementation adapts.

Real-World Example: Before & After 📊

Before: Version-Specific Nightmare

// Elasticsearch 7 code
import org.elasticsearch.client.RestHighLevelClient
import org.elasticsearch.action.search.SearchRequest
import org.elasticsearch.index.query.QueryBuilders

val request = new SearchRequest("users")
request.source().query(
  QueryBuilders.boolQuery()
    .must(QueryBuilders.rangeQuery("age").gt(25))
)
val response = client.search(request, RequestOptions.DEFAULT)
// ❌ Breaks on ES 8 - RestHighLevelClient deprecated
// ❌ Verbose boilerplate
// ❌ No type safety
// ❌ Manual error handling

After: Version-Agnostic Bliss

import app.softnetwork.elastic.client._

val results = client.searchAs[User](
  "SELECT name, email FROM users WHERE age > 25"
)
// ✅ Works on ES 6, 7, 8, 9
// ✅ Clean, readable SQL
// ✅ Type-safe results
// ✅ Built-in error handling

Same functionality. 90% less code. Infinite version compatibility.

Beyond Version Agnosticism: Killer Features 🔥

1. SQL to Elasticsearch Translation

// Complex query in plain SQL
val query = """
  SELECT
    category,
    AVG(price) as avg_price,
    COUNT(*) as product_count
  FROM products
  WHERE stock > 0
    AND price BETWEEN 10 AND 100
    AND description LIKE '%laptop%'
  GROUP BY category
  HAVING avg_price > 50
  ORDER BY product_count DESC
  LIMIT 10
"""

client.search(SQLQuery(query))

Automatically translated to optimized Elasticsearch DSL with aggregations.

2. Zero-Downtime Mapping Migrations

val newMapping = """{
  "properties": {
    "email": {"type": "keyword"},  // Changed from "text"
    "age": {"type": "integer"},
    "verified": {"type": "boolean"}  // New field
  }
}"""

// Automatic migration with rollback on failure
client.updateMapping("users", newMapping) match {
  case ElasticSuccess(_) =>
    logger.info("✅ Migration completed successfully")
  case ElasticFailure(error) =>
    logger.error(s"❌ Migration failed: ${error.message}")
    // Original mapping automatically restored
}

Behind the scenes:

Backup current mapping
Create temporary index with new mapping
Reindex all data
Atomic swap via aliases
Automatic rollback if any step fails

3. Reactive Streams for Bulk Operations

// Process millions of documents without memory issues
val obsoleteProducts: Source[(Product, ScrollMetrics), NotUsed] =
  client.scrollAs[Product](
    "SELECT id, name FROM products WHERE obsolete = true",
    client.defaultScrollConfig
  )

implicit val bulkOptions = BulkOptions(
  defaultIndex = "products",
  maxBulkSize = 1000,
  balance = 4  // Parallelism
)
obsoleteProducts
  .map { case (product, _) => s"""{"id": "${product.id}"}""" }
  .via(client.bulkFlow(delete = true))
  .runWith(Sink.foreach {
    case Right(success) => metrics.increment("deleted")
    case Left(failure) => logger.error(s"Failed: ${failure.error}")
  })

Backpressure-aware, memory-efficient, production-tested.

Production-Ready from Day One 🛡️

Built-In Monitoring

val client = ElasticClientFactory.createWithMonitoring()

// Automatic metrics every 30 seconds:
// === Elasticsearch Metrics ===
// Total Operations: 1,247
// Success Rate: 98.5%
// Average Latency: 45ms
// P95 Latency: 120ms
// =============================
// Automatic alerts:
// ⚠️ HIGH FAILURE RATE: 15.0%
// ⚠️ HIGH LATENCY: 1200ms

Comprehensive Error Handling

sealed trait ElasticResult[+T] {
  def isSuccess: Boolean
  def get: T
  def getOrElse[B >: T](default: => B): B
  def error: Option[ElasticError]
  def map[B](f: T => B): ElasticResult[B]
  def flatMap[B](f: T => ElasticResult[B]): ElasticResult[B]
  def toOption: Option[T]
  def toEither: Either[ElasticError, T]
  def fold[B](onFailure: ElasticError => B, onSuccess: T => B): B
  def foreach(f: T => Unit): Unit
}

case class ElasticSuccess[T](value: T) extends ElasticResult[T]
case class ElasticFailure[T](error: ElasticError) extends ElasticResult[T]
case class ElasticError(
  message: String,
  cause: Option[Throwable] = None,
  statusCode: Option[Int] = None,
  index: Option[String] = None,
  operation: Option[String] = None
)

Functional error handling. No exceptions in production.

Getting Started in 5 Minutes ⏱️

1. Add Dependency

ThisBuild / resolvers +=
  "Softnetwork Server" at "https://softnetwork.jfrog.io/artifactory/releases/"

// Choose your Elasticsearch version
libraryDependencies += "app.softnetwork.elastic" %% "softclient4es8-java-client" % "0.21.0"

2. Initialize Client

import app.softnetwork.elastic.client._

val client = ElasticClientFactory.create()

3. Start Building

// Create index
val mapping = """{
  "properties": {
    "name": {"type": "text"},
    "email": {"type": "keyword"}
  }
}"""

client.createIndex("users", mapping)
// Index documents
client.index(
   "users",
   "user-1",
   """{"name":"Alice","email":"alice@example.com"}""",
   wait = true
)
// Search
val results = client.searchAs[User](
  "SELECT name, email FROM users WHERE name = 'Alice'"
)

That’s it. No version-specific configuration. No complex setup.

The Migration Path: From Official Clients to SoftClient4ES 🛤️

Phase 1: Parallel Adoption (Week 1)

Add SoftClient4ES alongside existing client
Migrate read-only operations first (searches, gets)
Run both clients in parallel, compare results

Phase 2: Write Operations (Week 2)

Migrate indexing and updates
Test bulk operations thoroughly
Monitor error rates and latency

Phase 3: Advanced Features (Week 3–4)

Implement SQL queries where appropriate
Set up automatic mapping migrations
Enable monitoring and alerting

Phase 4: Cleanup (Week 5)

Remove old client dependencies
Delete version-specific code
Document the new patterns

Total migration time: 4–5 weeks for a typical microservice.

Who Should Use SoftClient4ES? 🎯

SoftClient4ES is designed for teams and projects that face specific Elasticsearch challenges. Here’s who benefits most: