How to realize huge file upload with TERASOLUNA 5.x (= Spring MVC)

1.First of all

This time, I will explain how to upload a file with a large data size with TERASOLUNA 5.x (= Spring MVC). If the data size is small, you don't have to worry about it, but if it is large, you need to be careful about server resources.

(point)

Please refer to "How to realize huge file download with TERASOLUNA 5.x (= Spring MVC)" for download.

2. Source code

2.1. Poor code

(Not good code 1) When taking an HTTP request as an argument


@RequestMapping(path = "chunked", method = RequestMethod.POST)
public ResponseEntity<String> streamUpload(HttpServletRequest httpRequest) {
            
        // You should be getted 5 parameters from HttpServletRequest and validated it
            
        // omitted
}

This is the old-fashioned method familiar with ʻAction of ServletandStruts. It receives HttpServletRequestas an argument of the handler method and reads the uploaded file data from BODY. The disadvantage of this method is that it handles rawHttpServletRequest` and is not testable.

(Not good code 2) When parameters are defined individually


@RequestMapping(path = "chunked", method = RequestMethod.POST)
public ResponseEntity<String> streamUpload(InputStream input,
        @RequestHeader(name = "Content-Type", required = false) String contentType,
        @RequestHeader(name = "Content-Length", required = false) Long contentLength,
        @RequestHeader(name = "X-SHA1-CheckSum", required = false) String checkSum,
        @RequestHeader(name = "X-FILE-NAME", required = false) String fileName) {
            
        // You should be validated 5 parameters
            
        // omitted
}

Compared to the above code, it has come to use the function of spring-mvc such as using @ RequestHeader, but it seems complicated because there are many parameters. Also, with this method, input checking using Bean Validation and BindingResult cannot be used.

**(Caution) If you specify ʻInputStream` as an argument of the handler method, you can receive the BODY data of the HTTP request as an input stream. In other words, it will be the uploaded binary file data. ** **

2.2. Introductory code

** I would like to use ModelAttributeMethodProcessor to receive the class that stores the required data as an argument of the handler method. There is a HandlerMethodArgumentResolver interface as a similar function, but this usesModelAttributeMethodProcessor because input checking is not possible. ** **

StreamFile.java


public class StreamFile implements Serializable {

    private static final long serialVersionUID = 1L;

    @NotNull
    private InputStream inputStream;

    @Size(max = 200)
    @NotEmpty
    private String contentType;

    @Min(1)
    private long contentLength;

    @Size(max = 40)
    @NotEmpty
    private String checkSum;

    @Size(max = 200)
    @NotEmpty
    private String fileName;

    // omitted setter, getter
}

Annotate the input check of Bean Validation as well as the input check of Form class(form backing bean).

StreamFileModelAttributeMethodProcessor.java


package todo.app.largefile;

import java.io.IOException;
import javax.servlet.http.HttpServletRequest;

import org.springframework.core.MethodParameter;
import org.springframework.web.bind.ServletRequestParameterPropertyValues;
import org.springframework.web.bind.WebDataBinder;
import org.springframework.web.context.request.NativeWebRequest;
import org.springframework.web.method.annotation.ModelAttributeMethodProcessor;

//★ Point 1
public class StreamFileModelAttributeMethodProcessor extends
                                                           ModelAttributeMethodProcessor {

    //★ Point 2
    public StreamFileModelAttributeMethodProcessor() {
        super(false);
    }

    //★ Point 3
    @Override
    public boolean supportsParameter(MethodParameter parameter) {
        return StreamFile.class.equals(parameter.getParameterType());
    }

    //★ Point 4
    @Override
    protected void bindRequestParameters(WebDataBinder binder,
            NativeWebRequest request) {

        //★ Point 5
        HttpServletRequest httpRequest = request
                .getNativeRequest(HttpServletRequest.class);
        ServletRequestParameterPropertyValues pvs = new ServletRequestParameterPropertyValues(
                httpRequest);

        //★ Point 6
        pvs.add("contentType", httpRequest.getContentType());
        pvs.add("contentLength", httpRequest.getContentLengthLong());
        pvs.add("checkSum", httpRequest.getHeader("X-SHA1-CheckSum"));
        pvs.add("fileName", httpRequest.getHeader("X-FILE-NAME"));

        //★ Point 7
        try {
            pvs.add("inputStream", httpRequest.getInputStream());
        } catch (IOException e) {
            pvs.add("inputStream", null);
        }

        //★ Point 8
        binder.bind(pvs);
    }
}

** ★ Point 1 ** Defines a class that extends ʻorg.springframework.web.method.annotation.ModelAttributeMethodProcessor. ModelAttributeMethodProcessor is a class that implements the ʻorg.springframework.web.method.support.HandlerMethodArgumentResolver interface.

** ★ Point 2 ** [Constructor of ModelAttributeMethodProcessor](https://docs.spring.io/autorepo/docs/spring-framework/4.3.5.RELEASE/javadoc-api/org/springframework/web/method/annotation/ModelAttributeMethodProcessor.html#ModelAttributeMethodProcessor- Call boolean-). This time we will create an object with the default false.

** ★ Point 3 ** ʻOrg.springframework.web.method.support.HandlerMethodArgumentResolver's supportsParametermethod. Process when the data type of the argument of the handler method isStreamFile.class`.

** ★ Point 4 ** This is the point of this method. Implement the process to bind (set) the data obtained from the HTTP request to the argument object (StreamFile this time) of the handler method in the bindRequestParameters method. That is, it gets the BODY input stream of the HTTP request and the value of the HTTP request header and binds (sets) it to WebDataBinder.

** ★ Point 5 ** org.springframework.beans.PropertyValues Implementation class [org.springframework.web.bind.ServletRequestParameterPropertyValues](https://docs.spring.io/autorepo/docs/spring-framework/4.3.5.RELEASE/javadoc-api/org/springframework/web/ bind / ServletRequestParameterPropertyValues.html) Create an object of class.

** ★ Point 6 ** Set the value obtained from the HTTP request header with the ʻadd method of ServletRequestParameterPropertyValues`.

** ★ Point 7 ** ★ Set the BODY input stream of the HTTP request in the same way as point 6. I decided to set null when ʻIOException` occurs.

** ★ Point 8 ** Bind (set) the PropertyValues generated at point 5 with the bind method of WebDataBinder. With this, the value of ★ points 6 and 7 will be set in the argument object (StreamFile this time) of the handler method.

FileUploadController.java


package todo.app.largefile;

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.validation.BindingResult;
import org.springframework.validation.annotation.Validated;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;

@Controller
@RequestMapping("upload")
public class FileUploadController {

    /**
     * LOGGER
     */
    private static final Logger LOGGER = LoggerFactory
            .getLogger(FileUploadController.class);

    /**
     *★ Point 9
     * define max upload file size
     */
    private static final long MAX_FILE_SIZE = 500 * 1024 * 1024;

    /**
     *★ Point 9
     * buffer size 1MB
     */
    private static final int BUFFER_SIZE = 1 * 1024 * 1024;

    //★ Point 10
    @RequestMapping(path = "chunked", method = RequestMethod.POST)
    public ResponseEntity<String> streamUpload(
            @Validated StreamFile streamFile,
            BindingResult result) {

        //★ Point 11
        if (result.hasErrors()) {
            LOGGER.debug("validated error = {}", result.getAllErrors());
            return new ResponseEntity<String>("validated error!",
                    HttpStatus.BAD_REQUEST);
        }

        //★ Point 12
        if (MAX_FILE_SIZE < streamFile.getContentLength()) {
            return fileSizeOverEntity();
        }

        //★ Point 13
        try {
            File uploadFile = File.createTempFile("upload", null);
            InputStream input = streamFile.getInputStream();
            try (OutputStream output = new BufferedOutputStream(
                    new FileOutputStream(uploadFile))) {
                byte[] buffer = new byte[BUFFER_SIZE];
                long total = 0;
                int len = 0;
                while ((len = input.read(buffer)) != -1) {
                    output.write(buffer, 0, len);
                    output.flush();
                    total = total + len;
                    LOGGER.debug("writed : " + total);
                    //★ Point 12
                    if (MAX_FILE_SIZE < total) {
                        return fileSizeOverEntity();
                    }
                }
            }
            LOGGER.debug(uploadFile.getAbsolutePath());
            //★ Point 14
            return new ResponseEntity<String>("success!", HttpStatus.CREATED);
        } catch (IOException e) {
            LOGGER.error(e.getMessage());
            //★ Point 15
            return new ResponseEntity<String>("error!",
                    HttpStatus.INTERNAL_SERVER_ERROR);
        }
    }

    /**
     *★ Point 12
     * @return ResponseEntity when file size excess error occurs
     */
    private ResponseEntity<String> fileSizeOverEntity() {
        return new ResponseEntity<String>(
                "file size is too large. " + MAX_FILE_SIZE + "(byte) or less",
                HttpStatus.BAD_REQUEST);
    }

    /**
     *Display the upload form screen
     * @return upload form screen
     */
    @RequestMapping(path = "form", method = RequestMethod.GET)
    public String form() {
        return "upload/form";
    }
}

** ★ Point 9 ** Define the upper limit of the data size of the upload file and the buffer size used when saving to the file.

** ★ Point 10 ** Specify @Validated and BindingResult so that input checking is enabled as an argument of the handler method. ★ By processing StreamFileModelAttributeMethodProcessor defined in point 1, it is possible to take StreamFile that has been input checked as an argument.

** ★ Point 11 ** As with normal input check, check if there is an input check error with the hasErrors method of BindingResult. This time, in the case of an input error, we will return a response with HttpStatus.BAD_REQUEST, that is, the HTTP response status code 400.

** ★ Point 12 ** Check if the data size of the uploaded file exceeds the upper limit defined in ★ Point 9. Check the data size in two places: (1) the value of the Content-Length header and (2) the actually read data size. If the limit is exceeded, we will return a response with HttpStatus.BAD_REQUEST, that is, the HTTP response status code 400. ,

** ★ Point 13 ** Read the data from the input stream of BODY of the HTTP request with the buffer size defined in point 9, and save the uploaded file as a file. This time I decided to save it in a temporary directory. Please modify here according to your business requirements.

** ★ Point 14 ** Since the uploaded file could be saved as a file on the server, we will return a response with HttpStatus.CREATED, that is, the HTTP response status code 201.

(Caution) This time, the metadata of the uploaded file (file name, content type, data size, checksum (hash value), etc.) is not saved. In the actual system, when accessing the uploaded file (downloading, opening with the application, etc.), metadata is required, so the metadata is saved in the database etc.

** ★ Point 15 ** If ʻIOException occurs during processing, this time we will return a response with HttpStatus.INTERNAL_SERVER_ERROR`, that is, HTTP response status code 500.

Spring Bean definition file (spring-mvc.xml)


<!-- omitted -->
<mvc:annotation-driven>
    <mvc:argument-resolvers>
        <!-- omitted -->
        <bean class="todo.app.largefile.StreamFileModelAttributeMethodProcessor"/>
    </mvc:argument-resolvers>
</mvc:annotation-driven>

In order to enable the StreamFileModelAttributeMethodProcessor implemented this time, add the Bean of ★ point 1 to<mvc: argument-resolvers>. For TERASOLUNA 5.x, define it in the spring-mvc.xml file.

Application server settings (eg tomcat server.xml)


<Connector connectionTimeout="200000" port="8090" 
    protocol="HTTP/1.1" redirectPort="8443"
    maxPostSize="-1" maxParameterCount="-1"
    maxSavePostSize="-1" maxSwallowSize="-1"
    socket.appReadBufSize="40960"/>

When uploading a huge file, you also need to check the application server settings.

For Tomcat8.0, please refer to https://tomcat.apache.org/tomcat-8.0-doc/config/http.html.

3. Finally

This time, I explained how to upload a file with a large data size with TERASOLUNA 5.x (= Spring MVC). The point is the setting of arguments using ModelAttributeMethodProcessor and the necessity of tuning the application server due to the large data size. ** In the test implemented by the client "How to upload a file with ajax without using multipart", a file of less than 400MB is uploaded in 6 seconds. I was able to do. ** **

Recommended Posts

How to realize huge file upload with TERASOLUNA 5.x (= Spring MVC)
How to realize huge file upload with Rest Template of Spring
I tried to implement file upload with Spring MVC
How to achieve file upload with Feign
File upload with Spring Boot
How to test file upload screen in Spring + Selenium
How to create an Excel form using a template file with Spring MVC
Implement file download with Spring MVC
How to load a Spring upload file and view its contents
How to achieve file download with Feign
[Spring MVC] How to pass path variables
How to split Spring Boot message file
How to mock each case with Mockito 1x
How to use MyBatis2 (iBatis) with Spring Boot 1.4 (Spring 4)
How to use built-in h2db with spring boot
How to bind to property file in Spring Boot
[Spring Boot] How to refer to the property file
[Java] How to omit spring constructor injection with Lombok
How to remove Tiles from TERASOLUNA 5.x blank project
How to boot by environment with Spring Boot of Maven
File upload with Spring Boot (do not use Multipart File)
How to change a TERASOLUNA 5.x blank project to support PostgreSQL
How to request a CSV file as JSON with jMeter
How to decompile apk file to java source code with MAC
Extract SQL to property file with jdbcTemplate of spring boot
[Easy] How to automatically format Ruby erb file with vsCode
How to get started with Gatsby (TypeScript) x Netlify x Docker
To receive an empty request with Spring Web MVC @RequestBody
Enable WebJars for blank projects in TERASOLUNA 5.x (= Spring MVC)
How to change the file name with Xcode (Refactor Rename)
Java Config with Spring MVC
iOS: File upload with SFTP
How to number (number) with html.erb
How to update with activerecord-import
How to deal with the event that Committee :: InvalidRequest occurs in committee during Rspec file upload test
Introduction to Spring Boot x OpenAPI ~ OpenAPI made with Generation gap pattern ~
How to make a jar file with no dependencies in Maven
I tried connecting to MySQL using JDBC Template with Spring MVC
How to create your own Controller corresponding to / error with Spring Boot
How to open a script file from Ubuntu with VS code
Automatic file upload with old Ruby gem What to do with Watir
How to use Lombok in Spring
How to scroll horizontally with ScrollView
How to unit test Spring AOP
How to get started with slim
How to use Spring Data JDBC
How to enclose any character with "~"
[How to install Spring Data Jpa]
How to set Spring Boot + PostgreSQL
How to convert erb file to haml
How to use mssql-tools with alpine
How to use ModelMapper (Spring boot)
How to get along with Rails
[Beginner] How to delete NO FILE
How to start Camunda with Docker
How to apply thymeleaf changes to the browser immediately with #Spring Boot + maven
How to read Body of Request multiple times with Spring Boot + Spring Security
How to realize hybrid search using morphological analysis and Ngram with Solr
How to access Socket directly with the TCP function of Spring Integration
How to build a Ruby on Rails development environment with Docker (Rails 6.x)
How to create a server executable JAR and WAR with Spring gradle