An easy way to measure the processing speed of a disk recognized by Linux

TL;DR Even when dealing with various layers such as OS and middleware, the read / write speed of the disk always exists as a factor that can be a bottleneck. Introducing a simple method for measuring disk throughput when you suspect that a disk is the bottleneck.

--Disk throughput can be easily measured by using the dd command and pv command.

Use scene

As an example, I think it is useful to investigate by the method introduced when the compression process is slow for some reason when performing the file compression process.

Measure Disk Read Throughput

You can measure Read throughput by using the dd command to set the disk to be measured to ʻif and specifying / dev / null for ʻof. By the way, note that it is a dd command, so it is important to be aware that it will be a sequential access.

--Example of measuring Read throughput of / dev / sda1

dd if=/dev/sda1 of=/dev/null bs=16k status=progress

The output will be as follows.

584122368 bytes (584 MB) copied, 4.61756 s, 127 MB/s

Measure the processing speed of each command

The method introduced above results in near pure disc speed. However, as in the usage scene example, for example, when the compression process is slow, you may want to distinguish whether it is pure disk speed or the compression process is the bottleneck and slow. In such a case, the pv command, which can measure the processing speed of each command, is effective.

What is the pv command?

The pv command is a command that measures the amount of data that crosses a pipe when performing pipe processing. For example, if you have the following example, you can check the amount of data passed from the tar command to the lz4 command, that is, how fast the tar command is being processed. By doing this, you can find out whether the tar command is the bottleneck or the lz4 command is the bottleneck when the execution speed of the following command is slow for some reason.

tar -b 256 -cvf - /data/ | lz4 -c

How to use

It is possible to measure by inserting the pv command between the pipes you want to measure as shown below.

--Example of measuring the speed of the tar command

tar -b 256 -cvf - /data/ | pv | lz4 -c > /dev/null

--Example of measuring the speed of the lz4 command

tar -b 256 -cvf - /data/mysql/ | lz4 -c | pv > /dev/null

The output will be as follows.

8.66GiB 0:00:04 [2.19GiB/s] [  <=>           ]

in conclusion

I think there are many ways to measure the performance of a disk, but this time I introduced the above method as an easy way to check. Since there are various types of disk access such as Read`` Write, Random IO Sequential IO, and speed index ʻIOPS`` Throughput, "What is this time?" If you are conscious of "Is it a process that causes an access pattern?", I think that it will be easier to find a bottleneck.

Recommended Posts

An easy way to measure the processing speed of a disk recognized by Linux
Python: I want to measure the processing time of a function neatly
How to access the contents of a Linux disk on a Mac (but read-only)
Easy way to check the source of Python modules
An easy way to pad the number with zeros depending on the number of digits [Python]
python3 Measure the processing speed.
How to increase the processing speed of vertex position acquisition
Create a 2D array by adding a row to the end of an empty array with numpy
An easy way to view the time taken in Python and a smarter way to improve it
An easy way to hit the Amazon Product API in Python
Consider the speed of processing to shift the image buffer with numpy.ndarray
An easy way to re-execute a previously executed command in ipython
How to output the output result of the Linux man command to a file
[Python3] Define a decorator to measure the execution time of a function
A command to easily check the speed of the network on the console
Investigate an efficient procedure to increase the sensitivity of a female knight by 3.5 billion times by dynamic programming
[Golang Error handling] The best way to separate processing according to the type of error
[Linux] Command to get a list of commands executed in the past
I tried to verify the result of A / B test by chi-square test
Measure the relevance strength of a crosstab
A quick overview of the Linux kernel
The story of blackjack A processing (python)
[python] How to sort by the Nth Mth element of a multidimensional array
I made an appdo command to execute a command in the context of the app
Save an array of numpy to a wav file using the wave module
Paste a link to the data point of the graph created by jupyterlab & matplotlib
How to calculate the volatility of a brand
About the processing speed of SVM (SVC) of scikit-learn
A clever way to time processing in Python
Easy way to load CPU / memory on Linux
How to measure line speed from the terminal
I checked the processing speed of numpy one-dimensionalization
An easy way to call Java from Python
The story of Linux that I want to teach myself half a year ago
[Python] You can save an object to a file by using the pickle module.
Usage to call a method of an instance before it is returned by __new__
[Ruby] How to replace only a part of the string matched by the regular expression?
I tried to compare the processing speed with dplyr of R and pandas of Python
Don't take an instance of a Python exception class directly as an argument to the exception class!
Add a function to tell the weather of today to slack bot (made by python)
A memo to visually understand the axis of pandas.Panel
Take a peek at the processing of LightGBM Tuner
Save the graph drawn by pyqtgraph to an image
An easy way to create an import module with jupyter
Steps to calculate the likelihood of a normal distribution
Create a shape on the trajectory of an object
Python Note: The mystery of assigning a variable to a variable
Even beginners can do it! An easy way to write a Sankey Diagram on Plotly
[Linux] [C / C ++] How to get the return address value of a function and the function name of the caller
An example of a mechanism that returns a prediction by HTTP from the result of machine learning
How to plot a lot of legends by changing the color of the graph continuously with matplotlib
A story about porting the code of "Try and understand how Linux works" to Rust
How to make an interactive LINE BOT 004 (answer the closing date of a listed company)