System Requirements & Prerequisites
• Operating System: Ubuntu 20.04 LTS (or compatible Linux distribution)
• Java Runtime: OpenJDK 11 or 17 (mandatory for Elasticsearch & Logstash)
• Hardware Specs: Minimum 4 GB RAM, 2 vCPU cores, 20 GB disk space
• Network Ports: 9200 (Elasticsearch REST API), 5044 (Logstash Beats), 5601 (Kibana Web UI)
Installing & Configuring Elasticsearch
Elasticsearch indexes and stores incoming log streams for near real-time querying. First, install the OpenJDK dependency:
Import the official Elastic GPG key and APT repository:
Edit /etc/elasticsearch/elasticsearch.yml to configure the node:
cluster.name: elk-cluster node.name: node-1 network.host: 127.0.0.1 http.port: 9200
Logstash Installation & Pipeline Configuration
Create a sample pipeline at /etc/logstash/conf.d/logstash-sample.conf:
input {
file {
path => "/var/log/app.log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}" }
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "logs-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
Installing & Accessing Kibana
Configure /etc/kibana/kibana.yml:
server.port: 5601 server.host: "127.0.0.1" elasticsearch.hosts: ["http://localhost:9200"]
Navigate to http://localhost:5601 to launch the Kibana dashboard interface.
Data Ingestion, Grok Parsing & Visualizations
Create test log entries inside /var/log/app.log:
Setting Up Kibana Visualizations:
- Index Pattern: Under Management → Stack Management → Index Patterns, create a pattern for
logs-*with@timestampas time field. - Visualize: Create a Vertical Bar or Metric chart, bucket by hour, and aggregate by log level (INFO, WARN, ERROR).
- Dashboard: Add visualizations into an executive security overview board.
Troubleshooting & Next Steps
• Elasticsearch unresponsive: Verify heap memory with sudo systemctl status elasticsearch.
• Logstash Grok failures: Inspect logs at /var/log/logstash/logstash-plain.log.
• Kibana connection refused: Check server.host and port 5601 bindings.
Next Steps for Production Hardening:
- Enable Security: Enable TLS/SSL certificates and native user authentication with X-Pack.
- Scale: Configure dedicated master and data nodes in an Elasticsearch cluster.
- Ingest Beats: Install Filebeat and Metricbeat across endpoint fleets for system-wide auditing.