Never Ending Security

It starts all here

Tag Archives: Elasticsearch

Open Source Log Visualization with Elasticsearch, Fluentd and Kibana

Open Source Log Visualization

Elasticsearch, Fluentd, and Kibana (EFK) allow you to collect, index, search, and visualize log data. This is a great alternative to the proprietary software Splunk, which lets you get started for free, but requires a paid license once the data volume increases.

This tutorial shows you how to build a log solution using three open source software components: ElasticsearchFluentd and Kibana.

Getting Elasticsearch

Next, download and install Elasticsearch’s deb package as follows.

sudo wget
sudo dpkg -i elasticsearch-1.2.2.deb

Securing Elasticsearch

Up to version 1.2, Elasticsearch’s dynamic scripting capability was enabled by default. Since this tutorial sets up the Kibana dashboard to be accessed from the public Internet, let’s disable dynamic scripting by appending the following line at the end of/etc/elasticsearch/elasticsearch.yml:

script.disable_dynamic: true

Starting Elasticsearch

Start running Elasticsearch with the following command.

sudo service elasticsearch start

Installing and Configuring Kibana

Getting Kibana

Move to your home directory:

cd ~

We will download Kibana as follows:

curl -L | tar xzf -
sudo cp -r kibana-3.1.0 /usr/share/

Configuring Kibana

Since Kibana will use port 80 to talk to Elasticsearch as opposed to the default port 9200, Kibana’s config.js must be updated.

Open /usr/share/kibana-3.1.0/config.js and look for the following line:

elasticsearch: "http://"+window.location.hostname+":9200",

and replace it with the following line:

elasticsearch: "http://"+window.location.hostname+":80",

Installing and Configuring Nginx (Proxy Server)

We will use Nginx as a proxy server to allow access to the dashboard from the Public Internet (with basic authentication).

Install Nginx as follows:

sudo apt-get install nginx --yes

Kibana provides a good default nginx.conf, from this Kibana GitHub repository.

Then, edit /etc/nginx/sites-available/default as follows (changes marked in red):

# Nginx proxy for Elasticsearch + Kibana
# In this setup, we are password protecting the saving of dashboards. You may
# wish to extend the password protection to all paths.
# Even though these paths are being called as the result of an ajax request, the
# browser will prompt for a username/password on the first request
# If you use this, you'll want to point config.js at http://FQDN:80/ instead of
# http://FQDN:9200
server {
 listen                *:80 ;
 server_name           localhost;
 access_log            /var/log/nginx/kibana.log;
 location / {
   root  /usr/share/kibana-3.1.0;
   index  index.html  index.htm;

Finally, restart nginx as follows:

$ sudo service nginx restart

Now, you should be able to see the generic Kibana dashboard at your server’s IP address or domain, using your favorite browser.


Installing and Configuring Fluentd

Finally, let’s install Fluentd. We will use td-agent, the packaged version of Fluentd, built and maintained by Treasure Data.

Installing Fluentd via the td-agent package

Install Fluentd with the following commands:

sudo dpkg -i td-agent_2.0.4-0_amd64.deb

Installing Plugins

We need a couple of plugins:

  1. out_elasticsearch: this plugin lets Fluentd to stream data to Elasticsearch.
  2. outrecordreformer: this plugin lets us process data into a more useful format.

The following commands install both plugins (the first apt-get is for out_elasticsearch: it requires make and libcurl)

sudo apt-get install make libcurl4-gnutls-dev --yes
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-record-reformer

Next, we configure Fluentd to listen to syslog messages and send them to Elasticsearch. Open /etc/td-agent/td-agent.conf and add the following lines at the top of the file:

 type syslog
 port 5140
 tag  system
<match system.*.*>
 type record_reformer
 tag elasticsearch
 facility ${tag_parts[1]}
 severity ${tag_parts[2]}
<match elasticsearch>
 type copy
   type stdout
 type elasticsearch
 logstash_format true
 flush_interval 5s #debug

Starting Fluentd

Start Fluentd with the following command:

sudo service td-agent start

Forwarding rsyslog Traffic to Fluentd

Ubuntu 14.04 ships with rsyslogd. It needs to be reconfigured to forward syslog events to the port Fluentd listens to (port 5140 in this example).

Open /etc/rsyslog.conf (you need to sudo) and add the following line at the top

*.* @

After saving and exiting the editor, restart rsyslogd as follows:

sudo service rsyslog restart

Setting Up Kibana Dashboard Panels

Kibana’s default panels are very generic, so it’s recommended to customize them.

Go to your server’s IP address or domain to view the Kibana dashboard.


There are a couple of starter templates, but let’s choose the blank one called Blank Dashboard: I’m comfortable configuring on my own, shown at the bottom of the welcome text.


Next, click on the + ADD A ROW button on the right side of the dashboard. A configuration screen for a new row (a row consists of one or more panels) should show up. Enter a title, press the Create Row button, followed by Save. This creates a row.


When an empty row is created, Kibana shows the prompt Add panel to empty row on the left. Click this button. It takes you to the configuration screen to add a new panel. Choose histogram from the dropdown menu. A histogram is a time chart; for more information, see Kibana’s documentation.


There are many parameters to configure for a new histogram, but you can just scroll down and press the Save button. This creates a new panel.


Further Information

For further information about configuring Kibana, please see the Kibana documentation page.