PAVINDAS / Blue Team Engineering
Loading... reads All Articles

ELK Stack Setup & Configuration Guide

Complete engineering guide to deploying Elasticsearch, Logstash, and Kibana. Master real-time pipeline ingestion, custom Grok parsing filters, and interactive security dashboards.

ELK Stack Pipeline Diagram Click to expand
ELK Stack Pipeline Diagram
Figure: ELK Stack Pipeline Diagram
Section 01 // Baseline

System Requirements & Prerequisites

Minimum Hardware & Network Specification

• Operating System: Ubuntu 20.04 LTS (or compatible Linux distribution)
• Java Runtime: OpenJDK 11 or 17 (mandatory for Elasticsearch & Logstash)
• Hardware Specs: Minimum 4 GB RAM, 2 vCPU cores, 20 GB disk space
• Network Ports: 9200 (Elasticsearch REST API), 5044 (Logstash Beats), 5601 (Kibana Web UI)

Section 02 // Storage Engine

Installing & Configuring Elasticsearch

Elasticsearch indexes and stores incoming log streams for near real-time querying. First, install the OpenJDK dependency:

bash — install openjdk
$sudo apt update && sudo apt install openjdk-11-jdk -y
$java -version

Import the official Elastic GPG key and APT repository:

bash — add elastic repository
$wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
$echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
$sudo apt update && sudo apt install elasticsearch -y

Edit /etc/elasticsearch/elasticsearch.yml to configure the node:

yaml — /etc/elasticsearch/elasticsearch.yml
cluster.name: elk-cluster
node.name: node-1
network.host: 127.0.0.1
http.port: 9200
bash — service management & health verification
$sudo systemctl enable --now elasticsearch
$curl -X GET "http://localhost:9200"
Section 03 // Pipeline Engine

Logstash Installation & Pipeline Configuration

bash — install logstash
$sudo apt install logstash -y

Create a sample pipeline at /etc/logstash/conf.d/logstash-sample.conf:

conf — /etc/logstash/conf.d/logstash-sample.conf
input {
    file {
        path => "/var/log/app.log"
        start_position => "beginning"
    }
}

filter {
    grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}" }
    }
}

output {
    elasticsearch {
        hosts => ["http://localhost:9200"]
        index => "logs-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
}
bash — test & launch logstash
$sudo /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/logstash-sample.conf
$sudo systemctl enable --now logstash
Section 04 // Visualization

Installing & Accessing Kibana

bash — install & configure kibana
$sudo apt install kibana -y

Configure /etc/kibana/kibana.yml:

yaml — /etc/kibana/kibana.yml
server.port: 5601
server.host: "127.0.0.1"
elasticsearch.hosts: ["http://localhost:9200"]
bash — enable & start kibana
$sudo systemctl enable --now kibana

Navigate to http://localhost:5601 to launch the Kibana dashboard interface.

Section 05 // Telemetry

Data Ingestion, Grok Parsing & Visualizations

Create test log entries inside /var/log/app.log:

log — /var/log/app.log
2025-05-31T13:45:00+05:30 INFO Application started successfully 2025-05-31T13:45:01+05:30 ERROR Failed to connect to database

Setting Up Kibana Visualizations:

  • Index Pattern: Under Management → Stack Management → Index Patterns, create a pattern for logs-* with @timestamp as time field.
  • Visualize: Create a Vertical Bar or Metric chart, bucket by hour, and aggregate by log level (INFO, WARN, ERROR).
  • Dashboard: Add visualizations into an executive security overview board.
Section 06 // Diagnostics

Troubleshooting & Next Steps

Common Operational Issues

• Elasticsearch unresponsive: Verify heap memory with sudo systemctl status elasticsearch.
• Logstash Grok failures: Inspect logs at /var/log/logstash/logstash-plain.log.
• Kibana connection refused: Check server.host and port 5601 bindings.

Next Steps for Production Hardening:

  • Enable Security: Enable TLS/SSL certificates and native user authentication with X-Pack.
  • Scale: Configure dedicated master and data nodes in an Elasticsearch cluster.
  • Ingest Beats: Install Filebeat and Metricbeat across endpoint fleets for system-wide auditing.
01. Prerequisites • 0%
Guide Sections