It's 3 AM. Production Is Down. Your Only Tool Is curl.

This is Part 3 of the SoftClient4ES series. Previously: Elasticsearch Queries That Never Break in Production and Stop Rewriting Your Elasticsearch Code Every Version Upgrade.

The On-Call Nightmare

Picture this.

It’s 3:47 AM. PagerDuty goes off. Someone stumbles out of bed, laptop in hand, heart racing.

“Elevated error rate in production. Payment service returning 500s.”

SSH into the bastion host. Now: query Elasticsearch logs to find out what’s happening.

The options:

Option 1: curl + JSON DSL

curl -X GET "https://es-prod:9200/logs/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "bool": {
        "filter": [
          {
            "range": {
              "timestamp": {
                "gte": "now-15m"
              }
            }
          },
          {
            "term": {
              "level": {
                "value": "ERROR"
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "by_service": {
        "terms": {
          "field": "service"
        }
      }
    }
  }'

At 3 AM. Half asleep. With a typo somewhere in that JSON.

Option 2: Open Kibana

“What’s the URL again? kibana.internal? kibana-prod? Let me check the wiki…”

5 minutes later, still trying to remember the SSO password.

Option 3: Wake up another engineer

“Hey sorry to bother you at 4 AM but can you write me an ES query…”

There has to be a better way.

TL;DR

The SoftClient4ES REPL is an interactive SQL terminal for Elasticsearch — the kind of tool every on-call engineer wishes they had at 3 AM.

Key Benefits:

• SQL instead of JSON DSL — query with syntax everyone already knows
• Auto-completion — Tab through field names (brain fog friendly)
• Syntax highlighting — spot typos instantly
• Command history — arrow-up to repeat that query from 5 minutes ago
• Works over SSH — no GUI needed, perfect for bastion hosts
• Persistent history — resume where things left off across sessions

Install in 10 seconds:

curl -fsSL https://raw.githubusercontent.com/SOFTNETWORK-APP/softclient4es/main/install.sh | bash

Read time: 10 minutes | Skill level: Beginner to Intermediate

What If Incident Response Looked Like This?

$ softclient4es --host es-prod.internal --port 9200
sql> SELECT service, level, COUNT(*) as error_count
  -> FROM logs
  -> WHERE timestamp >= NOW() - INTERVAL 15 MINUTE
  -> GROUP BY service, level
  -> ORDER BY error_count DESC;
| service        | level | error_count |
|----------------|-------|-------|
| payment-api    | ERROR | 847   |
| payment-api    | WARN  | 1203  |
| user-service   | INFO  | 5621  |
| order-service  | INFO  | 3892  |
4 rows (23ms)

Found the problem in 30 seconds. No JSON. No Kibana. No waking up teammates.

Now drill down:

sql> SELECT timestamp, message, stack_trace
  -> FROM logs
  -> WHERE service = 'payment-api'
  ->   AND level = 'ERROR'
  ->   AND timestamp >= NOW() - INTERVAL 15 MINUTE
  -> ORDER BY timestamp DESC
  -> LIMIT 10;

That’s incident response in 2026.

The curl + JSON Problem at 3 AM

Let’s be honest about what debugging with curl actually looks like in the middle of the night.

1. JSON Syntax Errors (Every. Single. Time.)

# Attempt 1
curl -X GET "localhost:9200/logs/_search" -d '{"query":{"match":{"level":"ERROR"}}'
# ERROR: Unexpected end of JSON
# Attempt 2
curl -X GET "localhost:9200/logs/_search" -d '{"query":{"match":{"level":"ERROR"}}}'
# ERROR: Content-Type header missing
# Attempt 3
curl -X GET "localhost:9200/logs/_search" -H "Content-Type: application/json" \
  -d '{"query":{"match":{"level":"ERROR"}}}'
# Finally works... 10 minutes later

At 3 AM, nobody does nested brackets well.

2. No Auto-Completion

“What’s the field name? Is it serviceName, service_name, or service*?”*

There’s no Tab-complete in curl. So the options are:

• Guess (and get it wrong)
• Run a separate query to check the mapping
• Check documentation (good luck finding it at 3 AM)

3. No History

That perfect query crafted 10 minutes ago? Gone.

“Wait, what was the filter? Let me scroll up… no, the terminal buffer is full…”

4. No Syntax Highlighting

curl -X GET "localhost:9200/logs/_search" -d '{"query":{"bool":{"must":[{"range":{"timestamp":{"gte":"now-1h"}}},{"term":{"level":{"value":"EROR"}}}]}}}'

Spot the typo. Go ahead.

(It’s EROR instead of ERROR. The kind of thing that costs 20 minutes of debugging.)

5. Unreadable Output

{"took":15,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":{"value":10000,"relation":"gte"},"max_score":null,"hits":[{"_index":"logs","_type":"_doc","_id":"abc123","_score":null,"_source":{"timestamp":"2026-02-11T03:45:00Z","service":"payment-api","level":"ERROR","message":"Connection refused to downstream service","host":"prod-payment-01"},"sort":[1739245500000]}]}}

Good luck parsing that at 3 AM.

The REPL Difference

Feature 1: SQL Keyword Auto-Completion

sql> SEL[TAB]
SELECT
sql> SELECT * FROM logs WHERE timestamp >= NOW() - INT[TAB]
INTERVAL
sql> CREATE MAT[TAB]
MATERIALIZED

SQL keywords auto-complete with Tab — no need to remember whether it’s INTERVAL or INTERVALL, MATERIALIZED or MATERIALISED. When the brain is foggy at 3 AM, every keystroke saved matters.

Want field and index name completion too? The REPL and core engine are Apache 2.0 open source — contributions are welcome.

Feature 2: Syntax Highlighting

Keywords in blue. Strings in green. Numbers in yellow. Errors in red.

When someone types EROR instead of ERROR, it doesn’t highlight as a keyword. The mistake is visible immediately. Not 20 minutes later.

Feature 3: Persistent Command History

sql> [arrow-up]  -- Previous command appears
SELECT service, COUNT(*) FROM logs WHERE level = 'ERROR' GROUP BY service;
sql> [arrow-up]  -- Command before that
SELECT message, stack_trace FROM logs WHERE service = 'payment-api' LIMIT 5;
sql> [Ctrl+R]  -- Search history
(reverse-i-search)`payment`: SELECT * FROM logs WHERE service = 'payment-api'

History persists across sessions. Close the terminal, SSH back in tomorrow — the queries are still there.

Feature 4: Multi-Line Queries

sql> SELECT
  ->   service,
  ->   host,
  ->   COUNT(*) as error_count,
  ->   MIN(timestamp) as first_error,
  ->   MAX(timestamp) as last_error
  -> FROM logs
  -> WHERE level = 'ERROR'
  ->   AND timestamp >= NOW() - INTERVAL 1 HOUR
  -> GROUP BY service, host
  -> HAVING COUNT(*) > 10
  -> ORDER BY error_count DESC;

Readable queries. Not single-line JSON nightmares.

Feature 5: Readable Output

| service        | host          | error_count | first_error              | last_error               |
|----------------|---------------|-------------|--------------------------|--------------------------|
| payment-api    | prod-pay-01   | 423         | 2026-02-11T03:32:15.123Z | 2026-02-11T03:47:22.456Z |
| payment-api    | prod-pay-02   | 398         | 2026-02-11T03:33:01.789Z | 2026-02-11T03:47:19.012Z |
| order-service  | prod-order-03 | 45          | 2026-02-11T03:40:55.234Z | 2026-02-11T03:46:12.567Z |
3 rows (127ms)

Formatted tables. Human-readable timestamps. Query duration.

And when the data needs to go somewhere else:

sql> format json
Current format: Json
sql> format csv
Current format: Csv

Three output formats — ASCII tables (default), JSON, and CSV — switchable on the fly.

Incident Response Playbook

Here’s a step-by-step guide for the next 3 AM wake-up call.

Step 1: Connect (30 seconds)

# SSH to bastion
ssh bastion.prod.internal
# Connect to Elasticsearch
softclient4es -s https -h es-prod.internal -p 9200 -u admin -W
Enter password:
sql>

Multiple authentication methods are supported: basic auth (-u/-P), API key (-k), or bearer token (-b).

Step 2: Initial Triage (2 minutes)

What’s happening right now?

-- Error distribution by service (last 15 minutes)
SELECT service, level, COUNT(*) as error_count
FROM logs
WHERE timestamp >= NOW() - INTERVAL 15 MINUTE
GROUP BY service, level
ORDER BY error_count DESC
LIMIT 20;

Output tells us: payment-api has 847 errors. Everything else looks normal.

Step 3: Identify the Pattern (2 minutes)

What kind of errors?

SELECT
  service,
  SUBSTRING(message, 1, 100) as error_pattern,
  COUNT(*) as occurrences,
  MIN(timestamp) as first_seen,
  MAX(timestamp) as last_seen
FROM logs
WHERE service = 'payment-api'
  AND level = 'ERROR'
  AND timestamp >= NOW() - INTERVAL 15 MINUTE
GROUP BY service, SUBSTRING(message, 1, 100)
ORDER BY occurrences DESC
LIMIT 10;

Output tells us: “Connection refused to payment-gateway.internal:443” — 812 occurrences.

Step 4: Check Scope (1 minute)

Is it all hosts or just one?

SELECT host, COUNT(*) as errors
FROM logs
WHERE service = 'payment-api'
  AND level = 'ERROR'
  AND timestamp >= NOW() - INTERVAL 15 MINUTE
GROUP BY host
ORDER BY errors DESC;

Output tells us: All 4 payment-api hosts affected. Not a single bad instance.

Step 5: Timeline Analysis (1 minute)

When did it start?

SELECT
  DATE_TRUNC(timestamp, MINUTE) as minute,
  COUNT(*) as error_count
FROM logs
WHERE service = 'payment-api'
  AND level = 'ERROR'
  AND timestamp >= NOW() - INTERVAL 1 HOUR
GROUP BY DATE_TRUNC(timestamp, MINUTE)
ORDER BY minute DESC
LIMIT 30;

Output tells us: Errors spiked at 03:32. Something changed at that time.

Step 6: Correlate Events (2 minutes)

What happened at 03:32?

SELECT timestamp, service, level, message
FROM logs
WHERE timestamp BETWEEN '2026-02-11T03:30:00Z' AND '2026-02-11T03:35:00Z'
  AND (
    message LIKE '%deploy%'
    OR message LIKE '%restart%'
    OR message LIKE '%config%'
    OR message LIKE '%gateway%'
    OR service = 'payment-gateway'
  )
ORDER BY timestamp;

Output tells us: payment-gateway restarted at 03:31:45 and hasn’t come back up.

Step 7: Root Cause Found

Total time: ~8 minutes.

The picture is clear:

• What’s broken: payment-api can’t reach payment-gateway
• When it started: 03:32
• Why: payment-gateway service is down after a restart
• Scope: All payment-api instances affected

Next action: Check why payment-gateway didn’t come back up.

Comparison: REPL vs. Alternatives

Beyond Incidents: Daily Use Cases

The REPL isn’t just for 3 AM emergencies. It’s a daily tool.

Quick Data Exploration

-- "How many users signed up today?"
SELECT COUNT(*) FROM users WHERE created_at >= CURRENT_DATE;
-- "What's the most popular product category this week?"
SELECT category, COUNT(*) as sales
FROM orders
WHERE order_date >= CURRENT_DATE - INTERVAL 7 DAY
GROUP BY category
ORDER BY sales DESC;

Schema Inspection

-- List all indices
sql> SHOW TABLES;

| name               | type    | pk  | partitioned |
|--------------------|---------|-----|-------------|
| logs-2026.02.11    | REGULAR |     |             |
| users              | REGULAR | id  |             |
| orders             | REGULAR | id  |             |

-- Filter by pattern
sql> SHOW TABLES LIKE 'log%';

| name               | type    | pk  | partitioned |
|--------------------|---------|-----|-------------|
| logs-2026.02.11    | REGULAR |     |             |

-- Describe a table's schema
sql> DESCRIBE TABLE users;

| Field     | Type    | Null | Key | Default | Comment | Script | Extra |
|-----------|---------|------|-----|---------|---------|--------|-------|
| id        | INT     | no   | PRI | NULL    |         |        | ()    |
| name      | VARCHAR | yes  |     | NULL    |         |        | ()    |
| email     | KEYWORD | yes  |     | NULL    |         |        | ()    |

-- Reverse-engineer the CREATE TABLE statement
sql> SHOW CREATE TABLE users;
CREATE OR REPLACE TABLE users (
  id INT NOT NULL,
  name VARCHAR,
  email KEYWORD,
  PRIMARY KEY (id)
)

Non-Interactive Mode (CI/CD, Scripts)

The REPL also runs in non-interactive mode — perfect for automation:

# Execute a single command
softclient4es -h es-prod -c "SELECT COUNT(*) FROM users"

# Execute SQL from a file
softclient4es -h es-prod -f /path/to/migrations.sql

Meta-Commands for Power Users

The REPL includes shortcut commands for common operations:

Installation Guide

Linux / macOS (One-Liner)

curl -fsSL https://raw.githubusercontent.com/SOFTNETWORK-APP/softclient4es/main/install.sh | bash

This will:

• Download the latest version
• Install to ~/softclient4es/
• Create launcher scripts and default configuration

Windows (PowerShell)

irm https://raw.githubusercontent.com/SOFTNETWORK-APP/softclient4es/main/install.ps1 | iex

Installation Options

# Default installation (latest version for ES8)
./install.sh

# Install for a specific Elasticsearch version
./install.sh --es-version 9

# List available versions
./install.sh --list-versions --es-version 8

# Custom installation directory
./install.sh --target /opt/softclient4es

Java Requirements

Add to PATH

# Add to ~/.bashrc or ~/.zshrc
export PATH="$PATH:$HOME/softclient4es/bin"

Verify Installation

softclient4es --help

Pro Tips for the On-Call Engineer

Tip 1: Create Aliases for Common Connections

# In .bashrc or .zshrc
alias es-prod='softclient4es -s https -h es-prod.internal'
alias es-staging='softclient4es -h es-staging.internal'
alias es-dev='softclient4es -h localhost'

Now just type:

$ es-prod

Tip 2: Use Environment Variables for Credentials

export ELASTIC_USERNAME="admin"
export ELASTIC_PASSWORD="your-secure-password"
softclient4es -h es-prod.internal
# Credentials loaded automatically from environment

Tip 3: Save Queries to Files

# Create a queries directory
mkdir -p ~/es-queries
# Save incident triage queries
cat > ~/es-queries/triage.sql << 'EOF'
SELECT service, level, COUNT(*) as error_count
FROM logs
WHERE timestamp >= NOW() - INTERVAL 15 MINUTE
GROUP BY service, level
ORDER BY error_count DESC;
EOF
# Run them when needed
softclient4es -h es-prod -f ~/es-queries/triage.sql

Tip 4: Deploy to Bastion Hosts Before You Need It

# On the bastion host
sudo ./install.sh --target /opt/softclient4es
sudo ln -s /opt/softclient4es/bin/softclient4es /usr/local/bin/

Now every engineer on the team can:

ssh bastion
softclient4es -h es-prod

Common Incident Queries (Cheat Sheet)

Save these for the next 3 AM adventure.

Error Rate by Service

SELECT service, COUNT(*) as errors
FROM logs
WHERE level = 'ERROR'
  AND timestamp >= NOW() - INTERVAL 15 MINUTE
GROUP BY service ORDER BY errors DESC;

Error Timeline (1-Minute Buckets)

SELECT DATE_TRUNC(timestamp, MINUTE) as minute, COUNT(*) as error_count
FROM logs
WHERE level = 'ERROR'
  AND timestamp >= NOW() - INTERVAL 1 HOUR
GROUP BY DATE_TRUNC(timestamp, MINUTE)
ORDER BY minute;

Top Error Messages

SELECT message, COUNT(*) as occurrences
FROM logs
WHERE level = 'ERROR'
  AND timestamp >= NOW() - INTERVAL 30 MINUTE
GROUP BY message ORDER BY occurrences DESC LIMIT 20;

Errors by Host (Find Bad Instance)

SELECT host, COUNT(*) as errors
FROM logs
WHERE service = 'my-service'
  AND level = 'ERROR'
  AND timestamp >= NOW() - INTERVAL 15 MINUTE
GROUP BY host ORDER BY errors DESC;

Trace a Request

SELECT timestamp, service, level, message
FROM logs WHERE trace_id = 'abc123-xyz789'
ORDER BY timestamp;

Find Slow Requests

SELECT service, endpoint,
  AVG(duration_ms) as avg_duration,
  MAX(duration_ms) as max_duration,
  COUNT(*) as request_count
FROM logs
WHERE timestamp >= NOW() - INTERVAL 1 HOUR
  AND duration_ms > 1000
GROUP BY service, endpoint ORDER BY avg_duration DESC;

Check for Deployment Correlation

SELECT timestamp, service, message
FROM logs
WHERE timestamp >= NOW() - INTERVAL 2 HOUR
  AND (
    message LIKE '%deploy%'
    OR message LIKE '%restart%'
    OR message LIKE '%version%'
    OR level = 'FATAL'
  )
ORDER BY timestamp;

User Impact Assessment

SELECT COUNT(DISTINCT user_id) as affected_users,
  COUNT(*) as failed_requests
FROM logs
WHERE level = 'ERROR'
  AND timestamp >= NOW() - INTERVAL 30 MINUTE
  AND user_id IS NOT NULL;

The Impact: Before & After

Before the REPL

03:47 - Alert fires
03:52 - SSH into bastion
03:55 - Start crafting curl command
04:05 - First query works (after 3 JSON syntax errors)
04:15 - Identify affected service
04:25 - Find error pattern
04:35 - Correlate with timeline
04:45 - Root cause identified
Total time: 58 minutes
Queries attempted: 12
Queries with syntax errors: 7

After the REPL

03:47 - Alert fires
03:48 - SSH + connect to ES
03:50 - Initial triage complete
03:52 - Error pattern identified
03:54 - Timeline analysis done
03:55 - Root cause identified
Total time: 8 minutes
Queries run: 6
Queries with errors: 0

MTTR improvement: 86%

The Bottom Line

Every minute spent debugging at 3 AM is a minute of sleep lost.

The SoftClient4ES REPL is not a nice-to-have. It’s a critical tool for incident response:

• Query Elasticsearch in seconds, not minutes
• Use SQL that everyone already knows (no JSON DSL at 3 AM)
• Auto-completion for the foggy brain
• Persistent history so queries are never lost
• Works over SSH on bastion hosts
• Install before the next incident — it takes 30 seconds

Don’t wait for the next outage to install it.

Ready for the Next 3 AM?

Install Now (30 Seconds)

# Linux / macOS
curl -fsSL https://raw.githubusercontent.com/SOFTNETWORK-APP/softclient4es/main/install.sh | bash
# Windows
irm https://raw.githubusercontent.com/SOFTNETWORK-APP/softclient4es/main/install.ps1 | iex

Test It Today

softclient4es --host localhost --port 9200
sql> SHOW TABLES;
sql> SELECT COUNT(*) FROM your_index;
sql> exit
Goodbye!

Deploy to Bastion Hosts

Make it available for the entire team before the next incident hits.

Resources

• REPL Documentation: Full REPL Guide
• SQL Reference: SQL Documentation
• GitHub: SoftClient4ES Repository
• Discussions: Ask Questions

What tools does your team rely on to debug Elasticsearch during incidents? Share them in GitHub Discussions — the on-call community could always use more ideas.

P.S. — The REPL works on Elasticsearch 6, 7, 8, and 9. No version lock-in. Install the bundle for your ES major, and it keeps working across upgrades within it.

Next in the series: Elasticsearch Schema Management Was Hell. Then Someone Typed SQL. — how DDL brings zero-downtime migrations to Elasticsearch.