Memory usage of Flume MemoryChannel

145 Views Asked by At

I'm troubleshooting some memory issues I'm encountering while sending messages to Flume from some Java code.

The code runs two EmbeddedAgents, each with a memory channel and some sinks pointing to a remote server.

I read in Flume documentation about memory channel configurations:

byteCapacity

Maximum total bytes of memory allowed as a sum of all events in this channel. The implementation only counts the Event body, (...). Defaults to a computed value equal to 80% of the maximum memory available to the JVM (i.e. 80% of the -Xmx value passed on the command line)

From what I understand this means that in worst case scenario each channel will use 80% of Xmx so 160% in my case (2 agents) which would obviously lead to OOM errors.

Am I right?

If so, I guess it means I have to setup a proper value for this configuration (not set in my current configuration) knowing that my code is actually running two memory channels?

Edit: my question assumes that capacity (maximum number of events in the channel) settings is quite high and the limiting factor would be the size rather than number of events.

0

There are 0 best solutions below