ELK Stack with Beats
The Elastic Stack is a three-layered stack of open-source tools that, when fully assembled, allows users to analyze data from any type of source and in any format. Compared to other log analysis tools, the Elastic Stack is known for dashboards and visualizations that present the data in a simple yet detailed manner.
Introduction: ‘ELK Stack’ was an abbreviation of its three main components:
Elasticsearch, Logstash and Kibana. The name of the stack was updated when Elastic's Beats was introduced as the fourth component.
Elasticsearch, Logstash and Kibana form the skeleton of the Stack.
· Elasticsearch is a JSON-based, RESTful, distributed search engine that can search and index files in multiple formats. Elasticsearch runs on port 9200.
What is it? Elasticsearch is a NoSQL database that is based on the Lucence search engine which helps us to store inputs/logs.
· Logstash is used as an ingest pipeline to collate, normalize and distribute logs. Logstash runs on port 5044.
What is it? Logstash is a log pipeline tool that accepts inputs/logs from various sources & exports the data to various targets.
· Kibana is used for data visualization, analysis and reporting. Kibana runs on port 5601.
What is it? Kibana is a visualization UI layer, which helps developers and operators to monitor application and system logs.
· Beats are data shippers that complement the Stack, allowing for multiple indices and sources of data. Beats output data to either Logstash or Elasticsearch.
What is it? Beats are light weight agents that are installed on the log source with a configuration to ship logs to Elasticsearch and Logstash. There are different types of Beats (File, Audit, Metric, Packet, Heart, Winlog etc…)
Note: It is important to note that the Elastic Stack is dynamic and, for the most part, many of the components will work independently of each other; for example, when shipping logs, Logstash is arguably optional since Beats can use Elasticsearch as direct output to Kibana. Therefore, it is not required to configure Logstash to ingest logs into Kibana if the indices of each Beat are configured to Elasticsearch; however, you may wish to manually insert data, in which case Logstash is still beneficial.
Security Information and Event Management (SIEM) is a key component of a security operations center. Elastic offers an SIEM tool designed to make alerting and reporting easier. The Elastic SIEM provides network and host data integrations, shareable analytics based on the Elastic Common Schema (ECS), and the ability to explore your security data with the SIEM app in Kibana. Security teams can use the SIEM tool to monitor threats and gather evidence to help in an investigation.
Kibana Query Language (KQL)
In comparison to many Search Processing Languages, Kibana Query Language is similar.
It has a Index (The main database you want to check) Available Fields (Fields extracted from the log files)
With the available fields, its user friendly to add the fields search query (selected fields) and pinpoint on the detailed data.
All Beats operates as a service, So long as the service is running, data is parsed through the Elastic Stack to be analyzed in Kibana.
Adding to the Elastic Stack, Filebeat's purpose is to forward and files, usually in either .log centralise logs and or .json format. These logs are shipped to either Logstash or Elasticsearch directly.
Filebeat is often confused with Logstash due to its nature of shipping logs and the fact that it can be configured to ship logs directly to Elasticsearch. The main difference is that Filebeat ships logs, whereas Logstash stores and parses them to Elasticsearch, controlling the flow through the pipeline.
Adding to the Elastic Stack, Auditbeat's purpose is to collect Linux audit framework data and monitor file integrity, before being shipped to either Logstash or Elasticsearch directly.
Metricbeat collects metrics from systems and services including CPU, memory usage, and load, as well as other data statistics from network data and process data, before being shipped to either Logstash or Elasticsearch directly.
Like Filebeat, Metricbeat comes with many different modules, ‘System’ being only one. It is capable of gathering metrics from services including Apache, NGINX, MongoDB, MySQL, PostgreSQL, Prometheus, Docker, and Kubernetes.
You can use Metricbeat in an investigation to identify malware activity by observing peaks in CPU usage and network traffic. You can also find out which processes are associated with it.
Packetbeat was the first of the Beats to be developed after the Elastic stack was formed. Packetbeat's primary function is to ship network data to either Logstash, to be transformed or enriched, or to Elasticsearch, to adjust analytics, before visualizing in Kibana.
However, Packetbeat is also capable of capturing the network traffic, decoding network protocols, correlating requests with responses, extracting fields like response times and status, and grouping to JSON to send to Elasticsearch.
Unlike most of the other Beats, Packetbeat does not use modules but instead supports a collection of network protocols from the application and lower-level protocols, databases and key-value stores, including HTTP, DNS, Flows, DHCPv4, MySQL and TLS.
You can use Packetbeat in an investigation to identify suspicious network activity, information (including domain names and IP addresses which may have been deny listed by security applications), or IOCs associated with a specific strain of malware.
Heartbeat allows for active probing to determine whether services are available. This is one of the most simplistic of the Beats family, as its purpose is to determine whether a service is alive. Heartbeat is sometimes confused with Metricbeat, which assesses system functionality, for example CPU usage, memory usage, and load times.
Heartbeat has limited support and less documentation than the other Beats. The dashboards, for example, are not automatically configured when the index is created, and there is only one dashboard that Elastic provides by default, which is available in the uptime-contrib repository on GitHub. This dashboard, which specifically looks at the HTTP protocol.
Heartbeat is useful in an investigation to determine whether an adversary has denied access to a specific service, by means of a Denial of Service (DOS) attack.
Winlogbeat is unique to the Beats family, as it is intended to manage Windows event logs. Winlogbeat was developed to increase the scope of support, such that data from Windows systems can be monitored as well as Linux and Mac OS X devices.
Winlogbeat allows constant monitoring even if network issues occur. It is possible to spool Windows event logs onto disk, ensuring the pipeline doesn’t miss a data point. Data shipping is resumed when the network comes back online.
In an investigation, Winlogbeat can be used to assess suspicious activity in Windows systems.
Windows Sysinternals tools operate in a similar fashion; however, the advantage of Winlogbeat is that Kibana can visualize the data in a manner allowing for easier analysis.