HBase BufferedMutator vs PutList performance

Question

I recently came across BufferedMutator class of HBase which can be used for batch inserts and deletes. I was previously using a List to put data as hTable.put(putList) to do the same. Benchmarking my code didn't seem to show much difference too, where I was instead doing mutator.mutate(putList);. Is there a significant performance improvement of using BufferedMutator over PutList?

Answer 1

Short Answer

BufferedMutator generally provides better throughput than just using Table#put(List<Put>) but needs proper tuning of hbase.client.write.buffer, hbase.client.max.total.tasks, hbase.client.max.perserver.tasks and hbase.client.max.perregion.tasks for good performance.

Explanation

When you pass a list of puts to the HBase client, it groups the puts by destination regions and batches these groups by destination region server. A single rpc request is sent for each batch. This cuts down the rpc overhead, especially in cases when the Puts are very small thus making rpc overhead per request significant.

The Table client sends all the Puts to the region servers immediately and waits for response. This means that any batching that can happen is limited to the number of Puts in the single API call and the api calls are synchronous from the caller's perspective. However, the BufferedMutator keeps buffering the Puts in a buffer and decides to flush the buffered puts based on current buffered size in background threads wrapped around by a class called AsyncProcess. From the caller's perspective, each API call is still synchronous, but the whole buffering strategy gives much better batching. The background flush model also allows a continuous flow of requests, which combined with better batching means ability to support more client threads. However, due to this buffering strategy, the larger the buffer, the worse the per operation latency as seen by the caller, but higher throughput can be sustained by having a much larger number of client threads.

Some of the configs that control BufferedMutator throughput are:

hbase.client.write.buffer: Size (bytes) of the buffer (Higher gives better peak throughput, consumes more memory)

hbase.client.max.total.tasks: Number of pending requests across the cluster before AsyncProcess starts blocking requests (Higher is better, but can starve CPU on client, or cause overload on servers)

hbase.client.max.perserver.tasks: Number of pending requests for one region server before AsyncProcess starts blocking requests.

hbase.client.max.perregion.tasks: Number of pending requests per region.

Also, for the sake of completeness, it should go without saying that if the bottleneck is on the server side instead of client side, you won't see much performance gains by using BufferedMutator over Table on the client.