how to setup Apache Flume with HTTP source and save data in local using File_Roll Sink

Question

The objective of this Question is How to create an Apache Flume setup in which we will get data from HTTP Flume Source and using File-Roll Flume Sink, we will save that data. Here we have taken input data from the user. After receiving the data from the user, we will save that data in a text file.

Answer 1

This configuration file is used to configure the Flume service. Using this file the Flume service runs with HTTP and saves the information in the files.

Http_Source.conf

# Base Config
a1.sources=r1
a1.sinks=k1
a1.channels=c1

# Configure the source
a1.sources.r1.type=http
#a1.sources.r1.bind=localhost
a1.sources.r1.port=8888

# Sink Configuration
a1.sinks.k1.type=file_roll
a1.sinks.k1.sink.rollInterval=60
a1.sinks.k1.sink.directory=/home/flumedata/

# Channel configuration
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000

# Link stuff together
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

Now using this command to run the Flume service.

./bin/flume-ng agent --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/nc_source.conf --name a1 -Dflume.root.logger=INFO,console

Now after the flume service starting, the client sends data to the flume...

curl --location --request POST 'http://localhost:8888' \
--header 'Content-Type: application/json' \
--data-raw '[{"body": "type here data to send flume"}]'

Creates the data file at the location mentioned in the config file (/home/flumedata/).