entity content is too long [658121477] for the configured buffer limit [104857600]

Question

Exception occurred when performing Search operation on Elastic indexes with really large data. Using RestHighLevelClient to perform Search with Http method as RequestOptions.DEFAULT

Error Logs: error is : java.io.IOException: entity content is too long [658121477] for the configured buffer limit [104857600]\n\tat org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:886)

my restclient is:

RestHighLevelClient restHighLevelClient = null; final CredentialsProvider credentialsProvider = new BasicCredentialsProvider(); credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(userName, passWord)); RestClientBuilder builder = RestClient.builder( HttpHost.create(host)) .setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder .setDefaultCredentialsProvider(credentialsProvider)); restHighLevelClient = new RestHighLevelClient(builder);

Answer 1

The maximum number of bytes you can send in an HTTP query is configured in http.max_content_length to be 104857600 (i.e. 100mb).

As you are sending six times as many bytes (i.e. 658121477), you're getting an exception. You now have two options:

A. you increase the http.max_content_length setting, but it's risky because you run the risk of overloading your cluster

B. you split your payload into 6 or 7 smaller chunks that fit into 100mb.