Flume agent does not contain any valid channels

Question

I am new to Flume. Im trying to pull data from Twitter, but I am not being successful. (I am using Cloudera Quickstart)

My conf file looks like this:

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel 

I have added all these values which are taken from Twitter account consumerKey, consumerSecret, accessToken ,accessTokenSecret,keywords and path also

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.fileType =  DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollsize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

The command I am using to execute the conf file is:

flume-ng agent --conf conf --conf-file flume.conf -Dflume.root.logger=DEBUG,console -name TwitterAgent

The error I am getting is:

18/06/27 12:17:18 WARN conf.FlumeConfiguration: Agent configuration for 'TwitterAgent' does not contain any valid channels. Marking it as invalid.
18/06/27 12:17:18 WARN conf.FlumeConfiguration: Agent configuration invalid for agent 'TwitterAgent'. It will be removed.
18/06/27 12:17:18 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: []
18/06/27 12:17:18 WARN node.AbstractConfigurationProvider: No configuration found for this host:TwitterAgent
18/06/27 12:17:18 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

Please advice me.

Answer 1

I think you have an issue in your execution command, the error is about finding the configuration file.

Command should be

flume-ng agent -c conf -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent

You have to specify your configuration file path. You can try -f conf/flume.conf instead of -f flume.conf