The CSV file that I was able to download suddenly started to appear on the page.

In the rush work, we released a web application that downloads data entered by multiple people in CSV format. I used the following OSS.

There was no particular problem on the day of the start of operation, and when I left the office and drank with my friends, my smartphone was noisy. .. .. When I answered the incoming call, I received a message saying "Download is not possible". When I checked the details, he said, "Until 3 hours ago, when I downloaded the file, the CSV file was output, but suddenly the CSV was displayed on the browser."

I made a mistake with a friend, stopped drinking, and started investigating the cause. For the time being, I knew that it would work if I fixed it like this, so I released it with service priority first. However, the principle remains unclear, so we conducted a continuous investigation.

Details of the correction

Below is the simplified code for the download process that caused this problem and the modified code.

  @PostMapping
  fun export(response: HttpServletResponse) {
    /**
     *CSV response.Write to outputStream.
     */

    response.contentType = "text/csv"
    response.setHeader("Content-Disposition", "attachment; filename=export.csv")
    response.outputStream.flush()
  }
  @PostMapping
  fun export(response: HttpServletResponse) {
    response.contentType = "text/csv"
    response.setHeader("Content-Disposition", "attachment; filename=export.csv")

    /**
     *CSV response.Write to outputStream.
     */

    response.outputStream.flush()
  }

The point is whether Content-Type and Content-Disposition are after or before writing to the outputStream. Subsequent investigations have shown that if Content-Disposition is in front, the file will be downloaded.

The reason why I set Content-Disposition after writing to outputStream is that the article I referred to happened to be like this, and I think that setting the header and writing to the body (stream) are independent processes. That's why.

Research of cause

However, even if Content-Disposition is set after writing to outputStream, the file can be downloaded normally for a while, so it can be inferred that the size of the CSV output has an effect. So I entered the data size on the screen and created a reproduction program that outputs CSV of that size.

  @PostMapping
  fun export(@RequestParam size: Int, response: HttpServletResponse) {
    (1..size).forEach {
      response.outputStream.print("a")
    }
    response.contentType = "text/csv"
    response.setHeader("Content-Disposition", "attachment; filename=export.csv")
    response.outputStream.flush()
  }

Then, the response header confirmed by Chrome's developer tools that the size is 8KByte

Connection: keep-alive
Content-Disposition: attachment; filename=export.csv
Content-Type: text/csv
Date: Sat, 08 Feb 2020 10:26:07 GMT
Keep-Alive: timeout=60
Transfer-Encoding: chunked

It was 9KByte

Connection: keep-alive
Date: Sat, 08 Feb 2020 10:27:48 GMT
Keep-Alive: timeout=60
Transfer-Encoding: chunked

Doesn't Content-Disposition disappear like this? In the end, 8191 bytes is the former, and 8192 bytes is the latter. I think it's because 8192, which is a power of 2, exceeds something set.

Consideration

I think that the following is probably happening in Tomcat.

Cause investigation 2

So, while quickly monitoring the packet with tcpdump, I tried step-by-step writing from 8190 bytes to 1 byte with the debugger, and it was confirmed that the HTTP header and body were sent to the browser when 8192 bytes were written. ..

By the way, when I stepped in when writing 8192 bytes, I found the following code.

java:org.apache.catalina.connector.OutputBuffer


    public void append(byte src[], int off, int len) throws IOException {
        if (bb.remaining() == 0) {
            appendByteArray(src, off, len);
        } else {
            int n = transfer(src, off, len, bb);
            len = len - n;
            off = off + n;
            if (isFull(bb)) {
                flushByteBuffer();
                appendByteArray(src, off, len);
            }
        }
    }

bb is a member variable of ByteBuffer type, and the transmission data is stored in bb in the appendByteArray and transfer methods. In the isFull method, it becomes true when the size accumulated in bb reaches the allowable amount, and it is sent to the browser by the flushByteBuffer method.

If you follow the flushByteBuffer method further,

java:org.apache.coyote.http11.Http11OutputBuffer


    @Override
    public int doWrite(ByteBuffer chunk) throws IOException {

        if (!response.isCommitted()) {
            // Send the connector a request for commit. The connector should
            // then validate the headers, send them (using sendHeaders) and
            // set the filters accordingly.
            response.action(ActionCode.COMMIT, null);
        }

        if (lastActiveFilter == -1) {
            return outputStreamOutputBuffer.doWrite(chunk);
        } else {
            return activeFilters[lastActiveFilter].doWrite(chunk);
        }
    }

If it is not committed, it sends an HTTP header during the commit process to set the committed flag. After that, the HTTP body is written.

Consideration 2

So, the code that I made easily was released without noticing it because it happened to behave as intended, but considering the difficulty on the side of Tomcat that processes various sizes, it is a natural implementation, there It was a story that I went all the way to get hooked.


(2020/2/8) Posted. (2020/2/9) Fixed an error in the size of the event switching before and after in the cause investigation. Added cause investigation 2 and consideration.

Recommended Posts

The CSV file that I was able to download suddenly started to appear on the page.
I was addicted to looping the Update statement on MyBatis
About the matter that I was addicted to how to use hashmap
I made a tool to output the difference of CSV file
I was able to deploy the Docker + laravel + MySQL app to Heroku!
By checking the operation of Java on linux, I was able to understand compilation and hierarchical understanding.
A story that I was addicted to twice with the automatic startup setting of Tomcat 8 on CentOS 8
I want to download a file on the Internet using Ruby and save it locally (with caution)
A story that was embarrassing to give anison file to the production environment
[Ruby] I tried to summarize the methods that frequently appear in paiza
[Ruby] I tried to summarize the methods that frequently appear in paiza ②
I was addicted to the roll method
I was addicted to the Spring-Batch test
I was addicted to using RXTX on Sierra
I was addicted to installing Ruby/Tk on MacOS
Double-click to open the jar file on Windows
[Ruby On Rails] Description that allows only specific users to transition to the edit page
The story that did not disappear when I tried to delete mysql on ubuntu
A site that was easy to understand when I was a beginner when I started learning Spring Boot
A story I was addicted to when getting a key that was automatically tried on MyBatis
I was addicted to the NoSuchMethodError in Cloud Endpoints
[Ruby] Misunderstanding that I was using the module [Beginner]
Completely delete the migration file that you failed to delete
I want to simplify the log output on Android
I had to figure out where the eclipse plugins folder was on my Mac. (Memo)
[Java] I tried to make a rock-paper-scissors game that beginners can run on the console.