Logstash

Logstash is an open source, server-side data pipeline from Elastic. It takes data from many sources, transforms it and sends it to a destination, which is usually a search store.

Logstash reads events from inputs such as logs, metrics, web applications, data stores and AWS services. Filters then parse each event, pull out named fields and convert them to a common format. Grok can turn unstructured text into structured fields, geographic data can be derived from IP addresses, and sensitive fields can be removed or anonymised. Outputs send the processed data to the chosen destination, and Elasticsearch is the most common choice.

• Input plugins for logs, metrics and cloud services
• Grok parsing of unstructured text
• Geo data from IP addresses
• Removal or anonymising of sensitive fields
• Output plugins for many destinations

Data engineers and operations teams that receive data in many formats and need to clean it before analysis. It is a common choice for teams already using the Elastic Stack.

The site describes Logstash as open source and offers it as a free download. Because it is a pipeline rather than a storage system, it works best alongside a search or storage backend. Complex filter chains can use a fair amount of memory and CPU, so they should be tested under realistic load.