Java-Apache BufferedHttpEntity loads whole file when sending the request

99 Views Asked by At

I want to make a Multipart request to my server from java using the apache CloseableHttpAsyncClient, this is what i currently have:

File file = new File("pathname");
MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create();
multipartEntityBuilder.addBinaryBody("file", new FileInputStream(file), ContentType.DEFAULT_BINARY, "file");
multipartEntityBuilder.addTextBody("filename", "File one");
HttpEntity multipartEntity = multipartEntityBuilder.build();
BufferedHttpEntity buffEntity = new BufferedHttpEntity(multipartEntity);

But when uploading a large file it throws a java.lang.OutOfMemoryError and i see it occupy all my machine memory ( Process goes from 100Mb memory to 3000+Mb ).
I have also tried to replace

multipartEntityBuilder.addBinaryBody("file", new FileInputStream(file), ContentType.DEFAULT_BINARY, "file");

with

multipartEntityBuilder.addBinaryBody("file", file);

But this has limited content length and gives the error "Content length is too long".
So how do I upload a large file (50gb+) using apache CloseableHttpAsyncClient?

EDIT

As suggested, i removed the BufferedHttpEntity but now i get the error "Content Length unknown". If i add a header with content length it tells me that content length header is already present.

EDIT 2

If i change the line

multipartEntityBuilder.addBinaryBody("file", new FileInputStream(file), ContentType.DEFAULT_BINARY, "file");

to

multipartEntityBuilder.addPart("file", fileContentBody);

I get the error that Content length is too long

1

There are 1 best solutions below

0
Alessandro Valentino On BEST ANSWER

After some trial and error i ended up switching to the CloseableHttpClient with RequestBuilder and HttpUriRequest instead of HttpPost to execute the request in a blocking way on a thread.

MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create();
multipartEntityBuilder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
FileBody fileContentBody = new FileBody(file, ContentType.DEFAULT_BINARY, file.getName());
multipartEntityBuilder.addPart("file", fileContentBody);

multipartEntityBuilder.addTextBody("filename", "File number one");
HttpEntity multipartEntity = multipartEntityBuilder.build();


HttpClientContext context = new HttpClientContext();
RequestBuilder requestBuilder = RequestBuilder.post("Myurl.com");
requestBuilder.setEntity(multiPartEntity);
HttpUriRequest multipartRequest = requestBuilder.build();

HttpResponse response = httpClient.execute(multipartRequest, context);