Linux

How to Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 20.04

How to Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 20.04

ELK Stack is a full-featured data analytics platform, consisting of three open-source Elasticsearch, Logstash, and Kibana tools. This stack helps you store and manage logs centrally and provides the ability to analyze them.

In this post, we will see how to install the ELK stack on Ubuntu 20.04.

Install ELK Stack

1

Log Monitoring With ELK Stack

Beats – Installed on the client machine, and it collects and sends logs to Logstash.

Logstash – Processing of logs sent by beats (installed on client machines).

Elasticsearch – Stores logs and events from Logstash and offers the ability to search logs in real time

Kibana – Provides visualization of events and logs.

Install Java

Elasticsearch requires OpenJDK or Oracle JDK which is available on your machine.

Here, for this demo, I use OpenJDK. Install Java using the commands below along with the wget and HTTPS support packages for APT.

sudo apt update

sudo apt install -y openjdk-11-jdk wget apt-transport-https curl

Check the Java version.

java -version

Output:

openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-3ubuntu1)
OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-3ubuntu1, mixed mode, sharing)

Add ELK repository

The ELK stack package is available at the official Elastic repository.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Install & Configure Elasticsearch

Elasticsearch is an open-source search engine that provides real-time, multitenant distributed full-text search engines with a web interface (HTTP) and schema-free JSON documents.

Install the latest version of Elasticsearch using the apt command.

sudo apt update

sudo apt install -y elasticsearch-oss

Start and activate the Elasticsearch service.

sudo systemctl start elasticsearch

sudo systemctl enable elasticsearch

Wait for one or two minutes and then run the command below to see the Elasticsearch status.

curl -X GET http://localhost:9200

Output:

{
  "name" : "ubuntu.itzgeek.local",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AB9giOoWQo2nReENAICKig",
  "version" : {
    "number" : "7.7.1",
    "build_flavor" : "oss",
    "build_type" : "deb",
    "build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",
    "build_date" : "2020-05-28T16:30:01.040088Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

The above output confirms that Elasticsearch is active and functioning properly.

Install & Configure Logstash

Logstash is an open source log parsing software that collects logs, parses them, and stores them in Elasticsearch for future use. With the help of available plugins, it can process various types of events without extra work.

sudo apt install -y logstash-oss

Logstash configuration consists of three plugins, namely input, filter, and output. You can put all plugin details in one file or separate files for each section, ending with .conf.

Here, we will use one file to place all three plugins.

Create a configuration file under the /etc/logstash/conf.d/ directory.

 sudo nano /etc/logstash/conf.d/logstash.conf

In the input plugin, we will configure Logstash to listen on port 5044 for logins from agents (Beats) running on client machines.

input {
  beats {
    port => 5044
  }
}

For the filter plugin, we will use Groc to parse syslog messages before sending them to Elasticsearch for safekeeping.

filter {
  if [type] == "syslog" {
     grok {
        match => { "message" => "%{SYSLOGLINE}" }
  }
     date {
        match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
     }
  }
}

In the plugin output, we will determine where the logs are to be stored, explained the Elasticsearch example.

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  }
}

Now start and activate the Logstash service.

sudo systemctl starts logstash

sudo systemctl activates logstash

Logstash log:

sudo cat /var/log/logstash/logstash-plain.log

Install and Configure Kibana

Kibana provides visualization of data stored in Elasticsearch instances. Install Kibana using the apt command.

sudo apt install -y kibana-oss

By default, Kibana listens on localhost, which means you can’t access the Kibana web interface from an external machine. To access Kibana from an external machine, you need to set server.host to the system IP address in the /etc/kibana/kibana.yml file.

sudo nano /etc/kibana/kibana.yml

Make a change like below.

server.host: "192.168.0.10"

Also, in some cases, Elasticsearch and Kibana may run on different machines. In that case, update the below line with the IP address of the Elasticsearch server.

elasticsearch.hosts: ["http://localhost:9200"]

Start and enable Kibana on machine startup.

sudo systemctl start kibana

sudo systemctl enable kibana

Install Filebeat

Filebeat is client software that runs on the client machine to send logs to the Logstash server for rendering (in our case) or directly to Elasticsearch for safekeeping.

We will use the Logstash server host name in the configuration file. So, add DNS records or host entries for the Logstash server on the client machine.

sudo nano /etc/hosts

Make an entry something like below.

192.168.0.10 server.itzgeek.local

Install HTTPS support for apt.

sudo apt update

sudo apt install -y apt-transport-https

Set up the Elastic repository on your system for Filebeat installation.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Install Filebeat using the following command.

sudo apt update

sudo apt install -y filebeat

Edit the filebeat configuration file /etc/filebeat/filebeat.yml to send logs to the Logstash server.

sudo nano /etc/filebeat/filebeat.yml

The below configuration in the inputs section is to send system logs (/var/log/syslog) to the Logstash server.

For this demo, I have commented out /var/log/*.log to avoid sending all logs to the Logstash server.

.    .    .

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/syslog
    #- /var/log/*.log

.    .    .

Since we are sending logs to the Logstash for parsing, comment out the section output.elasticsearch: and uncomment output.logstash: in the output section.

.    .    .

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["server.itzgeek.local:5044"]

.    .    .

Start the Filebeat service.

sudo systemctl start filebeat

Filebeat’s log:

sudo cat /var/log/syslog

Access ELK Dashboard

Access the Kibana web interface by going to the following URL.

http://your-ip-address:5601/

http://your-serve-name:5601

You will get the Kibana homepage.

2

Kibana Start Page

On your first access, you need to create a filebeat index. Go to Management »Index Patterns» Create Index Patterns.

3

Index Patterns

Type the following in the Index pattern box.

filebeat-*

You will see the filebeat index, as below. Click Next step.

4

Create Index Pattern

Select @timestamp, then click Create index pattern.

@timestamp

5

Time Filter Field Name

Check the fields in the index pattern.

6

Fields

Click Find in the left navigation to see the logins from the client machine.

7

Conclusion

That is all. I hope you have learned how to install the ELK stack on Ubuntu 20.04.

Related posts

How to Install OpenCV on Ubuntu 20.04

Linux

How to Install Ubuntu on a Raspberry Pi

Linux

How to make an on-the-fly flash drive with a bootable Linux distribution

Linux

How to Install Android Studio on Ubuntu 20.04

Linux

How to Install and Use a Linux Firewall

Linux

How to Install Flatpaks on Ubuntu 20.04

Linux

How to install Grafana Enterprise Edition on Ubuntu Server 20.04

Linux

How to Install and Configure Redis on Ubuntu 20.04

Linux

How to Change Hostname on Ubuntu 20.04

Linux