This time, I will explain how to upload a huge file using RestTemplate of Spring Framework. The point is to use RequestCallback.
For the server-side implementation of huge file upload, please refer to "How to realize huge file upload with TERASOLUNA 5.x (= Spring MVC)".
(point)
RequestCallback interface and write the upload data to the request BODY while buffering it.RestTemplate and add a method that allows you to specify RequestCallback using the default ResponseExtractorRequestFactoryLargeFileRequestCallback.java
package todo.app.largefile;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.client.ClientHttpRequest;
import org.springframework.web.client.RequestCallback;
//★ Point 1
public class LargeFileRequestCallback implements RequestCallback {
/**
* LOGGER
*/
private static final Logger LOGGER = LoggerFactory
.getLogger(LargeFileRequestCallback.class);
/**
*★ Point 2
* buffer size 1MB
*/
private static final int BUFFER_SIZE = 1 * 1024 * 1024;
/**
*★ Point 3
*File information
*/
private final FileInfo fileInfo;
/**
*★ Point 3
*constructor
* @param fileInfo File information
*/
public LargeFileRequestCallback(FileInfo fileInfo) {
this.fileInfo = fileInfo;
}
/**
*★ Point 4
* @see org.springframework.web.client.RequestCallback#doWithRequest(org.springframework.http.client.ClientHttpRequest)
*/
@Override
public void doWithRequest(ClientHttpRequest request) throws IOException {
LOGGER.debug("doWithRequest : start");
//★ Point 5
// 1. set Content-type
request.getHeaders().add("Content-type", fileInfo.getContentType());
request.getHeaders().add("Content-Length",
Long.toString(fileInfo.getContentLength()));
request.getHeaders().add("X-SHA1-CheckSum", fileInfo.getCheckSum());
request.getHeaders().add("X-FILE-NAME", fileInfo.getDownloadFileName());
//★ Point 6
// 2. copy contents with buffering
try (InputStream input = new BufferedInputStream(
new FileInputStream(fileInfo.getFilePath()));
OutputStream out = request.getBody();) {
byte[] buffer = new byte[BUFFER_SIZE];
long total = 0;
int len = 0;
while ((len = input.read(buffer)) != -1) {
out.write(buffer, 0, len);
out.flush();
total = total + len;
// LOGGER.debug("writed : " + total);
}
}
LOGGER.debug("doWithRequest : end");
}
}
** ★ Point 1 **
ʻDefine aRequestCallback class that implements the ʻorg.springframework.web.client.RequestCallbackinterface. This class can manipulate the request header when issuing an HTTP request forRestTemplate` and write data to the request BODY. This time, it is used to write the file data to the request BODY.
This time, I defined it as a separate class so that it can be used by others, but it is also possible to write it as (1) arrow operator (lambda expression) and (2) inner class (anonymous class) in the place to be used.
** ★ Point 2 ** Defines the buffer size when writing file data. This time it was 1MB.
** ★ Point 3 **
The file information to be uploaded is linked to the RequestCallback class as a constructor argument.
** ★ Point 4 **
This is the point of this article. Override the doWithRequest method to implement the process of writing the data of the upload file to the HTTP request.
The ClientHttpRequest object passed as an argument is the HTTP request issued by RestTemplate. Manipulate this directly to set the request header or write data to the request BODY.
** ★ Point 5 **
When setting the request header, get the HttpHeaders object with the getHeaders method and add it with the ʻadd` method.
** ★ Point 6 **
Write to the request BODY while buffering the file data.
The request BODY is accessed as an output stream by calling the getBody method of the ClientHttpRequest object.
RestTemplateCallbackExtend.java
package todo.app.largefile;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RequestCallback;
import org.springframework.web.client.ResponseExtractor;
import org.springframework.web.client.RestTemplate;
//★ Point 7
public class RestTemplateCallbackExtend extends RestTemplate {
public RestTemplateCallbackExtend() {
super();
}
/**
*★ Point 8
* Extend the postForEntity method so that callback can be used
* @param url the URL
* @param responseType the type of the return value
* @param requestCallback object that prepares the request
* @return the converted object
*/
public <T> ResponseEntity<T> postForEntityWithCallback(String url,
Class<T> responseType, RequestCallback requestCallback) {
// 1. create ResponseExtractor object
ResponseExtractor<ResponseEntity<T>> responseExtractor = responseEntityExtractor(
responseType);
// 2. send request using RequestCallback and default ResponseExtractor
return super.execute(url, HttpMethod.POST, requestCallback,
responseExtractor);
}
}
** ★ Point 7 **
By default, RestTemplate has only three methods that use RequestCallback, and in that case, you need to specify ResponseExtractor as well.
I've implemented RequestCallback, but ResponseExtractor wants to use the default functionality, so I'll extend RestTemplate a bit.
(reference)
Since it was troublesome to create ResponseExtractor <ResponseEntity <String >> of ResponseEntity <String>, this time I extended RestTemplate.
If you can easily create this ResponseExtractor object, you should call the execute method of RestTemplate` directly.
List of methods that take RequestCallback as an argument in RestTemplate
public <T> T execute(String url,
HttpMethod method,
RequestCallback requestCallback,
ResponseExtractor<T> responseExtractor,
Object... urlVariables)
throws RestClientException
public <T> T execute(String url,
HttpMethod method,
RequestCallback requestCallback,
ResponseExtractor<T> responseExtractor,
Map<String,?> urlVariables)
throws RestClientException
public <T> T execute(URI url,
HttpMethod method,
RequestCallback requestCallback,
ResponseExtractor<T> responseExtractor)
throws RestClientException
** ★ Point 8 **
This time, I wanted a postForEntity method that can specify RequestCallback as an argument in POST, so I added the postForEntityWithCallback method.
Call the protected`` responseEntityExtractor method to use the default ResponseExtractor, and RequestCallback uses the one specified by the argument to execute the ʻexecute` method.
FileUploadControllerTest.java
package todo.app.largefile;
import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.*;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.web.client.HttpClientErrorException;
public class FileUploadControllerTest {
/**
* LOGGER
*/
private static final Logger LOGGER = LoggerFactory
.getLogger(FileUploadControllerTest.class);
@Test
public void test01() {
//★ Point 9
// 1. prepared data
String targetUrl = "http://localhost:8090/todo-rest/upload/chunked";
FileInfo fileInfo = getFileInfo();
//★ Point 10
// 2. create RestTemplate object
RestTemplateCallbackExtend restTemplate = new RestTemplateCallbackExtend();
//★ Point 11
// 3. setting SimpleClientHttpRequestFactory for timeout, streaming
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
requestFactory.setConnectTimeout(1000 * 120);
requestFactory.setReadTimeout(1000 * 120);
requestFactory.setChunkSize(1 * 1024 * 1024);
restTemplate.setRequestFactory(requestFactory);
try {
//★ Point 12
// 4. send request using RequestCallback
ResponseEntity<String> responseEntity = restTemplate
.postForEntityWithCallback(targetUrl, String.class,
new LargeFileRequestCallback(fileInfo));
//★ Point 13
// 5. assert
assertThat(responseEntity.getStatusCode(), is(HttpStatus.CREATED));
assertThat(responseEntity.getBody(), is("success!"));
} catch (HttpClientErrorException e) {
LOGGER.error(e.getMessage()); //Only "403 Forbidden" is included.
LOGGER.error(e.getResponseBodyAsString());
throw e;
}
}
/**
*★ Point 9
*Get file information for testing
* @return file information
*/
private FileInfo getFileInfo() {
FileInfo fileInfo = new FileInfo();
fileInfo.setFilePath("C:/tmp/spring-tool-suite-3.8.2.RELEASE-e4.6.1-win32-x86_64.zip");
fileInfo.setContentType("application/zip");
fileInfo.setCheckSum("ff1a62872fb62edb52c1e040fdf9c720e051f9e8");
fileInfo.setContentLength(418945672);
fileInfo.setDownloadFileName("STS3.8.2.zip");
return fileInfo;
}
}
** ★ Point 9 ** Prepare the data to send. This time, I decided to upload the STS Zip file (a little less than 400MB) that I had. The checksum (hash value) was calculated in "How to obtain the hash value (checksum) of a huge file in Python".
** ★ Point 10 **
Instantiate the RestTemplateCallbackExtend class defined at point 7 instead of RestTemplate.
** ★ Point 11 **
When uploading a huge file, it is necessary to consider the effects of timeout, buffer data size, etc.
ʻConfig in the org.springframework.http.client.SimpleClientHttpRequestFactory class and enable it in the setRequestFactory method of RestTemplate.
setConnectTimeout: Set the connection timeout (unit is milliseconds) for the URL connection.setReadTimeout: Set the read timeout (unit is milliseconds) of the URL connection.setChunkSize: Set the chunk size (in bytes) when writing data to the HTTP request BODY.setBufferRequestBody: Set whether the request factory buffers internally (true: default or false)As explained below, this time we will send a request with a large data size, so set false.
Default is true. When sending large amounts of data via POST or PUT, it is recommended to change this property to false, so as not to run out of memory.
** ★ Point 12 **
★ Call the postForEntityWithCallback method implemented in point 8. At that time, specify the LargeFileRequestCallback object as an argument.
Now the file data is written to the request BODY using the RequestCallback implemented this time.
** ★ Point 13 **
After executing the HTTP request, it will be the same as the normal RestTemplate.
Access the HTTP response status code with the getStatusCode method and access the HTTP response BODY with the getBody method.
If HttpClientErrorException occurs, the getMessage method gets a summary of the exception. If you need more details, check the response BODY on exception with the getResponseBodyAsString method.
This time, I explained how to upload a huge file using RestTemplate and RequestCallback.
I tried this time and found that there is no method that can specify RequestCallback independently in RestTemplate (there is a set with ResponseExtractor), which was a little inconvenient.
Recommended Posts