Here’s Why You Need Logstash In Linux Servers

Here’s the reason why you need Logstsash and where it stores logs. Precisely, it is a central-repository log. Below, you’ll find two things. One is the purpose of Logstash, and the other one is an Example.


What is Logstash

Logstash is an open-source tool developed by Jordan Sissel. Who currently works for Elastic. Logstash acts as a data pipeline through which it processes the data from multiple servers and systems.

Logstash can take inputs from TCP/UDP protocols, files, and log management systems, such as many more tools in the field that server administrators install to analyze server events. Like Puppet and CFEngine, or with monitoring systems like Nagios, Graphite & Zabbix.

For example, let’s say that you have a few servers running at different locations, and these are clusters or load-balancing servers for your web app.

The best use case is a master machine wherein all of its nodes can send log data, and you can see and search the logs from this master machine.  

Location of Logstash where it stores logs

The Logstash stores all the logs in the path /var/log/logstash by default.

Features of Logstash

Following are some of Logstash’s key features:

  • Logstash is open source and free to use.
  • Logstash is lightweight.
  • Logstash is highly customizable.
  • Logstash is easy to configure.
  • Input and output plug-ins are readily available for Logstash.

Logstash, Elastic Search, Kibana

Logstash, Elastic Search, Kibana setup
Logstash, Elastic Search, Kibana

The relation between the main machine and its nodes is that the servers running Logstash agents are called shippers. They send log events of your applications and services to the Logstash server.

The central Logstash server running services such as brokers, indexers, and storage interface with Kibana, a visualization tool.

Logstash how it works

Logstash collects data from the different sources defined by using the configuration file. Logstash can process any type of logs that are being maintained, including Apache logs, MySQL logs, firewall logs, and error logs.

The best part is that one can store logs from different nodes and services in a centralized place and analyze them there.

Elasticsearch does real-time data analysis from different data sources. It is scalable, and it does a full-text search.

Kibana is a web application designed to visualize data in Elasticsearch. It allows you to search data (in our case, logs) and visualize it in various ways (such as bar graphs, pie charts, and other graph types). Visualizations can be organized in dashboards for quick access to information.


Best reference books

Author: Srini

Experienced software developer. Skills in Development, Coding, Testing and Debugging. Good Data analytic skills (Data Warehousing and BI). Also skills in Mainframe.