A story that was embarrassing to give anison file to the production environment

Overview

This year is almost over. Since "this year's reflection will be within this year," I would like to post a story that I did in production at the end of the year **.

Two days back, I was an intern who is currently an engineer, fixing a ** file-uploading system bug **. Coding proceeded smoothly, and deployment was completed successfully. I did a light test to see if I could upload properly in production, and left the company that day.

The next day, as soon as I went to work, my boss called me.

Boss * "○○ -kun, did you touch the file upload system? Something strange file is up ..." * I * "Weird file ...?" *

When I checked the specified url, the sound source of ** My Idol Declaration sung by a friend was uploaded there. ** ** (It's a really good song, please listen)

Fortunately, the file was set to private, and the upload was done with my own test account, so the rumor that "I am Kimoota" only flowed throughout the company. ~~ No, I want to die normally. ~~ But if the file is public and has been seen by the user ... just thinking about it makes me chill.

(Digression) What kind of implementation were you trying to implement?

As a background, I will briefly touch on ** what implementation I was trying to implement **. (It's a little connected to the story of why I had to test in a production environment, but it's okay to skip it.)

It all started with the problem "** For some reason I can't upload large files for production **". In the first place, we have adopted the mechanism of "passing the file itself to the API and uploading it to s3" as shown in the figure below, and in the actual production, the API Gateway was installed in the previous stage. As a result of the investigation, it was discovered that due to the specifications of ** API Gateway, requests for files larger than 10MB are rejected. ** ** Therefore, as a workaround, we decided to issue a signed-url for uploading from the ** API => upload the file directly from the front **.

For those who want a more detailed explanation, this article etc. will be helpful.

By the way, in the ** staging environment, the same situation did not occur because the API Gateway in the previous stage was omitted. ** (That was part of the cause ...)

What the hell was the cause

Why did such a mistake happen? It's easy to put it all together because I was incompetent, but I'll try to analyze it properly so that I don't make the same mistakes again.

I gave a file that I did not understand well at the time of development

Regarding the file, it appeared at the top when I searched for a video in Finder. Although I knew the contents, during development, I was conscious of ** "because it is a development environment anyway" **, and I was testing the upload with the same file without thinking about anything. As a result, it led to a situation where the same file was unknowingly given during the test upload in production. I learned the lesson that ** do not do things that should not be done in the production environment during development, such as giving an appropriate file and giving an appropriate name (name it ●).

The consciousness of production was clear

In the first place, the service under development was before the official release, there were few users, and even if a program with some strange data or bugs came up in the production, there was almost no damage. From here, the spoiling of "** It's a production, but it's appropriate to some extent **" arose, and as a result, it led to this time itself. (Thanks to that, there was almost no damage to the failure ...) However, with this awareness, we may have made similar mistakes even after the official release and made irreparable mistakes. ** I learned the lesson that the awareness that it is a production environment even before the release is proper **.

Staging wasn't working well

Although the so-called "staging environment" existed at our company, it was just an environment for "checking for inconsistencies with the same data as the production", and there was a considerable difference from the production configuration. ** (In this case, API Gateway is not installed before API) In the first place, if we had the same configuration for production and staging, we would not have had to test in production. I learned that ** staging should be made with exactly the same composition as the actual production as a copy of the actual production **.

Be careful next year

Based on the above, I would like to pay attention to the following points when developing in the future.

-** What you don't do in production isn't done in development. ** ** -** Have a proper awareness that it is a production environment. ** ** -** Make staging work (create with the same configuration as the production environment) **

Next year, I will do my best not to make such a small mistake.