4

Can anybody explain Apache Flume for me in a plain language? I'd appreciate an explanation with a practical example instead of abstract theoretical definitions, then I can understand better.

What is it used for? At which stage of a BigData analysis is it used?

And what are prerequisites for learning it?

Please

As you would explain for a non-technical person

DanielWelke
  • 163
  • 1
  • 10

2 Answers2

5

What is it used for?

Data ingestion into a distributed datastore (e.g. HDFS). See image (I did not make the image and am only including the image for visual aid). There are other tools that will help you with the ingestion of data as well (Storm and Sqoop are mentioned).

At which stage of a BigData analysis is it used?

It is used for data ingestion into your distributed datastore (e.g. HDFS). So for example a webserver is running logging information into /var/logs/webserver.log. Apache Flume can look at that file, grab what it needs out of it and send it to HDFS. Once the data gets put into your datastore you can then utilize other tools to analyze the imported data (e.g. Hive, Pig, MR, etc.)

And what are prerequisites for learning it?

Understanding of how to write scripts, edit configuration settings, and your way around Linux would be the absolute minimum to get started. This set of instructions are old but a starting point would be to look at the hortwonworks tutorial on flume. http://hortonworks.com/hadoop-tutorial/how-to-refine-and-visualize-server-log-data/

If you would like me to elaborate more I would be glad to but I wanted to try and meet your requirement of a simple, short explanation.

HDP Data Lake Solution Architecture

Community
  • 1
Kevin Vasko
  • 301
  • 1
  • 5
  • Thank you for your nice answer Kevin! I'll wait 1-2 more days for other answers and in case I do not get better ones I'll accept yours. Cheers :) – DanielWelke Jan 12 '16 at 09:52
  • Nice explanation :) is it conceptually similar to Spark Streaming? – Rami Jan 13 '16 at 20:12
  • Not quite. Spark Streaming would be analogous to Apache Storm or Apache Flink. While there might be some overlap in the same concept of ingesting data into HDFS, Spark Streaming, Storm, Flink all are more related to streaming datasets (think Twitter streams). Flume/Sqoop would be more for "static" type datasets (e.g. extracting data from log files or SQL databases). This is a good article that discusses the performance comparison of SS, Flink, Storm. Might give you a better idea of possible utilities. – Kevin Vasko Jan 13 '16 at 20:31
4

What is Apache Flume?

  • Apache Flume is a tool for designed for streaming data ingestion in HDFS. Objective: The main objective of Flume is to capture streaming data from various web servers to HDFS.

Applications of Flume

  • The applications of flume are,

    • Flume is used in e-commerce Company to analyze the customer behavior of different regions.

    • It is used to feed huge log data generated by application servers into HDF5 at a higher speed.

What are prerequisites for learning it?

  • Basics of Hadoop, Big-data is must.
  • Basics of Linux and scripting
  • Main thing is interest towards technology.

For more information refer Apache Flume

Aditya
  • 2,470
  • 2
  • 16
  • 35
kavya
  • 41
  • 1