2

I've been having the hardest time understanding the difference between AWS Kinesis and Logstash.

From the description of Logstash it says

Logstash is a light-weight, open-source, server-side data processing pipeline that allows you to collect data from a variety of sources, transform it on the fly, and send it to your desired destination.

and from the description of Kinesis it says

With Amazon Kinesis, you can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications. Amazon Kinesis enables you to process and analyze data as it arrives and respond instantly instead of having to wait until all your data is collected before the processing can begin.

It's not obvious to me what the difference between them is as they both seem to be able to ingest data from many sources at once and can, in turn, process it.

Could somebody clarify here?

Laiv
  • 14,598
devinm
  • 123

1 Answers1

2

To my understanding, AWS Logtash is a sort of ETL, capable of intaking unstructured data from different sources for the purpose mentioned: Extract, transform and load the transformed data somewhere else. It's a collector that makes some changes to the incoming data, probably it normalises different data structures into a canonical one that fits our needs for mining or monitoring. But the data collecting and data mining happen at different times.

On the other hand, Kinesis, unlike Logtash, doesn't collect and transform the incoming data into something different. It's a support to perform analysis in real-time. It generates an additional stream of data based on the analysis and the results. In other words, data is not sent to Kinesis to be transformed and collected, it's sent to be analyzed as soon as possible for us to make decisions as soon as possible too. Probably, Kinesis allows you to send signals to other services based on the results of the analysis. With Logtash, we would have to wait for it to load the transformed data and then perform calculations; with Kinesis these things happen as the streams go.

Logtash

(input) ----> (Logtash : ETL)  ----> (Store)  <--- (Mining & monitoring)

Kinesis

(input) ----> (Kinesis : Analysis)  ---> (output) 
                    |
                    |
                    +-> (Metrics) <--    (Monitoring)
                    +-> (Metdata) <--    (ML/BI)
                    +-> (Statistics) <-- (BI/BI)
Laiv
  • 14,598
  • 1
    I think it worth to mention that while Logstash is a very specific tool for data ingestion, allowing transformation and aggregation of data, Kinesis is composed by a set of tools. What it makes sense is to compare Logstash with Kinesis Data Firehose, because both serves the same purpose. – silmarx Dec 21 '22 at 10:40
  • @silmarx would be right comparing Kinesis and (for example) kibana? – Laiv Dec 21 '22 at 11:13