It is the 13th day of Ateam cyma Advent Calendar 2019. Today, Ateam Co., Ltd. EC Business Headquarters cyma engineer @ihsiek will be in charge.
"People who have done it in a production environment Advent Calendar 2019" is very popular. Recently, the CI / CD environment has been enhanced and the chances of shell access to the production environment are decreasing, and the failure stories of others (other companies) will be a cane that will not fall down so as not to follow the same rut. I would like the engineers of the company to read such failure stories.
So, I would like to write a story of failure in a production environment due to an operation that I was completely unaware of the risks I took in my previous job.
--force option of the Linux command, it is better to check the behavior of the version to be used.
cp -rf [backup] [destination] to verify that the backup works properly
cp -rf / tmp / hoge / tmp / hoge
cp -rf is bad, so I checked the specifications of the
-f and --force options with
man cp and found the cause.
The following is the result of
man cp in the environment of occurrence.
If you try to copy the file to itself(Except for issuing an error message)do nothing.
When copying to an existing file, the destination file is ‘open(path,O_WRONLY|O_TRUNC)’Opened at.
When copying to a new file, the new file is ‘open(path,O_WRONLY|O_CREAT,mode)’Created with.
The copy destination file exists and-If opening a file in this format fails when the f option is given,
cp deletes existing files(Or unlink)Try.
Then, if the deletion is successful, the instruction is continued as a copy to a new file.
Although it is on the second line from the bottom, the destination of the symbolic link is looking at the same file in the backup and release destination, so
That seems to have been the end of the matter. By the way, the behavior of the cp command seems to be different for each distribution, but I have confirmed that such a problem did not occur even at that time if the version was upgraded properly.
I think that the trouble was caused by the fact that I was careful that the cp command was just a copy command and that it would be okay, that I had prepared a way to verify only in production, and that I was not able to thoroughly update the OS. .. Here are my lessons learned:
-f, --force options as much as possible, and check the specifications if necessary.
How was the 13th day of Ateam cyma Advent Calendar 2019? Since it was a story of my previous job, I looked back while thinking that it would not be possible in my current workplace, but I pray that everyone will not follow the same rut. We apologize for the inconvenience caused to all the people involved at that time: bow_tone1:
Also, in the sense that I want you to feed on the mistakes of others, "Advent Calendar 2019" is really Since it is recommended, I would like various people to read it, including those who have never experienced production operation. Tomorrow is @namedpython, a new graduate engineer who is a little messy returnee. I'm looking forward to posting newcomers who are expected to build a solid data analysis platform.
Ateam Co., Ltd. is looking for colleagues with a strong spirit of challenge to work with.
If you are interested in working as an engineer, please see cyma's Qiita Jobs.
For other occupations, see Ateam Group Recruitment Site.