Set the access load that can be changed graphically with JMeter (Part 2)

Introduction

Last time, I installed ThroughputShapingTimer (plug-in) in JMeter, I explained how to adjust the load amount graphically. Set the access load that can be changed graphically with JMeter (Part 1) ↑ Access load control image Official page

It's been a long story to introduce a simple usage, so this time I will explain practical usage.

Last synopsis

Mr. A, who was entrusted with the performance test of the website by his boss. In response to the unreasonable swing from the boss saying "I want to test the surge in access when buzzing on Twitter" By using ThroughputShapingTimer, Mr. A can now perform the test that meets his expectations with JMeter.

However, Mr. A has only verified the test on a single page. Based on the website access record, he struggled to create more complex test scenarios. .. ..

Try using Throughput Shaping Timer (advanced edition)

With this small story, I will explain the test pattern and JMeter's test scenario creation immediately.

In this article, we will introduce the following three test patterns in the flow of hop, step, and jump.

--Sequentially execute one screen transition pattern --Execute one screen transition pattern in parallel --Execute multiple screen transition patterns in parallel

Environmental information

• Mac(Catalina)
• Apache httpd server(2.4.x)
• Java8(Oracle java 8u202)
• JMeter(5.3)

Environmental recovery

If Apache is stopped, the startup procedure is different from the previous one, so leave it as a reminder.

Communication confirmation

You can also check by sending a request from JMeter, but here we will check with the CURL command.

$ curl http://localhost:8080/index.html
curl: (7) Failed to connect to localhost port 8080: Connection refused

If it says Failed, you know that you have not been able to communicate with the Apache you used last time.

Checking the status of the container

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS                  PORTS               NAMES
e2dc7c3714ee        httpd:2.4           "httpd-foreground"   6 days ago          Exited (0) 5 days ago                       my-apache-app

Since Status is ʻExited`, you can see that the container is stopped (normal).

If it is ʻUp`, it means that it is in a hung state, so stop the container once.

Stop container (reference)

$ docker stop e2dc7c3714ee

Specify the container ID and stop. After executing the command, check the container status again and confirm that it is ʻExited`. (If it doesn't work, restart the PC)

Start container

$ docker start e2dc7c3714ee
e2dc7c3714ee

$ docker ps -a             
CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS              PORTS                                           NAMES
e2dc7c3714ee        httpd:2.4           "httpd-foreground"   6 days ago          Up 3 seconds        0.0.0.0:8080->80/tcp, 0.0.0.0:32772->8080/tcp   my-apache-app

$ curl http://localhost:8080/index.html
hello

You are now ready.

Serially execute one screen transition pattern

Taking Amazon as an example, most people use the site according to the following flow.

1. Display the top page
2. Select a product
3. Buy

In this pattern, we will explain how to adjust the load on the assumption that one user repeats the same page transition endlessly.

Situation settings

--One user --Page 01-> Page 02-> Page 03 ――Usually 15RPS (RequestPerSecond), but it rises to 75RPS momentarily

Express the flow of access with JMeter

Use the thread group created last time Since we have not prepared the content for the test this time, we will access three types of URLs by accessing the same HTML with different query characters.

• Page 01 *

• Page 02 *

• Page 03 *

Set ThroughputShapingTimer

The load amount is being changed.

Set up a thread group

Set the number of threads (degree of parallelism) to 1.

Run JMeter

You were able to access the three URLs almost evenly. (Although it is not completely even, ThroughputShapingTimer is playful, so please forgive me)

Apache log analysis

 $ docker logs e2dc7c3714ee | tail -1094 | awk '{print $4}' | sed 's_\[18/Oct/2020:__' | uniq -c
〜〜〜
  16 13:20:00
  14 13:20:01
  16 13:20:02
  14 13:20:03
  16 13:20:04
  14 13:20:05
  15 13:20:06
  26 13:20:07
  38 13:20:08
  50 13:20:09
  63 13:20:10
  75 13:20:11
  63 13:20:12
  51 13:20:13
  39 13:20:14
  27 13:20:15
  16 13:20:16
  14 13:20:17
  15 13:20:18
  14 13:20:19
〜〜〜

It went up from 15RPS to 75RPS and returned to 15RPS.

• Since we will test several times from now on, we are checking the log of 1094 lines, which is the total number of accesses this time.
• You can specify a delimiter other than a slash for the sed command! It became easier to see with the advice

Apache log analysis (confirm that it is serial)

 $ docker logs e2dc7c3714ee | tail -10 | awk '{print $4, $7}'
[18/Oct/2020:13:20:36 /index.html?test02
[18/Oct/2020:13:20:36 /index.html?test03
[18/Oct/2020:13:20:36 /index.html?test01
[18/Oct/2020:13:20:36 /index.html?test02
[18/Oct/2020:13:20:36 /index.html?test03
[18/Oct/2020:13:20:36 /index.html?test01
[18/Oct/2020:13:20:36 /index.html?test02
[18/Oct/2020:13:20:36 /index.html?test03
[18/Oct/2020:13:20:36 /index.html?test01
[18/Oct/2020:13:20:36 /index.html?test02

You can see that the access is coming in the order of page 01-> page 02-> page 03.

Execute one screen transition pattern in parallel

Last time, I adjusted the load on the assumption that one user repeats the same page transition endlessly, but it can not be an actual website.

I will explain how to adjust the load amount on the assumption that three users will make the same screen transition as last time at the same time.

Situation settings

--Three users --Page 01-> Page 02-> Page 03 ――Usually 15RPS (RequestPerSecond), but it rises to 75RPS momentarily

Set up a thread group

Set the number of threads (degree of parallelism) to 3.

• Other settings are the same as last time

Run JMeter

You were able to access the three URLs almost evenly this time.

Apache log analysis

 $ docker logs e2dc7c3714ee | tail -1082 | awk '{print $4}' | sed 's_\[18/Oct/2020:__' | uniq -c
〜〜〜
  14 14:44:50
  15 14:44:51
  14 14:44:52
  15 14:44:53
  15 14:44:54
  14 14:44:55
  16 14:44:56
  14 14:44:57
  16 14:44:58
  13 14:44:59
  28 14:45:00
  38 14:45:01
  50 14:45:02
  65 14:45:03
  75 14:45:04
  63 14:45:05
  51 14:45:06
  37 14:45:07
  28 14:45:08
  15 14:45:09
  15 14:45:10
  15 14:45:11
  15 14:45:12
  15 14:45:13
〜〜〜

It went up from 15RPS to 75RPS and returned to 15RPS.

Apache log analysis (confirm that it is parallel)

 $ docker logs e2dc7c3714ee | tail -10 | awk '{print $4, $7}'
[18/Oct/2020:14:45:29 /index.html?test03
[18/Oct/2020:14:45:29 /index.html?test02
[18/Oct/2020:14:45:29 /index.html?test01
[18/Oct/2020:14:45:29 /index.html?test02
[18/Oct/2020:14:45:29 /index.html?test03
[18/Oct/2020:14:45:29 /index.html?test02
[18/Oct/2020:14:45:29 /index.html?test03
[18/Oct/2020:14:45:29 /index.html?test01
[18/Oct/2020:14:45:29 /index.html?test03
[18/Oct/2020:14:45:29 /index.html?test01

Compared to the previous time, the access order is no longer in the order of page 01-> page 02-> page 03. You can see that three users accessed at the same time and the order was changed.

Execute multiple screen transition patterns in parallel

Until the last time, I tried the following patterns. (Pattern 1)

1. Display the top page
2. Select a product
3. Buy

This time, add the following screen transitions as another thread group. (Pattern 2)

1. Display the top page
2. Select a product
3. Select other recommended products

This time (Pattern 1) Three users make screen transitions at the same time (Pattern 2) Five users make screen transitions at the same time I will explain how to adjust the load amount on the premise.

Situation settings

(Pattern 1) * Same as last time

--Three users --Page 01-> Page 02-> Page 03 ――Usually 15RPS (RequestPerSecond), but it rises to 75RPS momentarily

(Pattern 2)

--5 users --Page 11-> Page 12-> Page 13 --50RPS (RequestPerSecond) occurs continuously

Express the flow of access with JMeter

The thread group created last time will be used.

Duplicate and place the thread group as shown in the picture below. Place the statistics report in the same hierarchy as the thread group.

Change the duplicated settings as follows.

Thread group: Number of threads: 5 HTTP request (test11): Path: /index.html?test11 HTTP request (test12): Path: /index.html?test12 HTTP request (test13): Path: /index.html?test13

Set ThroughputShapingTimer

Make it uniform at 50 RPS.

• JMeter cannot suddenly reach full power, so let's run up gradually until the specified load is reached.

Run JMeter

You could keep a uniform load on each of thread groups 1 and 2.

Create a JMeter report

Since the test content is complicated this time, it is difficult to confirm that it is working as expected by analyzing the Apache log. Let's output a report with the standard function of JMeter and check the result.

1. Adjust the graph drawing interval of JMeter report Suddenly, this is a procedure that is not normally performed. .. .. The report draws a graph in minutes by default. Since the test time is very short this time, the graph drawing interval is also shortened.
   Open <the directory where you extracted JMeter> /apache-jmeter-5.3/bin/user.properties. Since there is a setting value of the graph drawing interval called jmeter.reportgenerator.overall_granularity around the 76th line, uncomment it and change it from 60000 milliseconds to 1000 milliseconds.

2. Open the report generation menu Select Tools-> Generate HTML report from the menu bar.

3. Enter the information required to generate the report Result file (csv or jtl): Output file name specified in the statistical report user.properties file: Under bin (file rewritten earlier) Output directory: Anywhere is OK

4. Generate a report Click Generate report at the bottom and check mark to complete the generation.

Check JMeter report

1. Open the generated report in your browser Since the report has been generated in the location specified in the Output directory earlier, open ʻindex.html`.

2. Transition to the target graph Select Throughput.

3. Transition to the target graph Select Transactions Per Seconds.

4. Impressed by the beautiful graph You were able to draw the waveform exactly as set in ThroughputShapingTimer! !!
   ** Congratulations, you are also a JMeter Master! ** **

Later talk

Mr. A "Finally, the setting of JMter is completed!" Boss "Thank you, I'll leave this performance test to Mr. A. Good luck!" Mr. A "Yes!"

Mr. A is ready to measure the performance of the web application under test. But this was the beginning of the real hell. Mr. A, who is still inexperienced, could not yet predict the many problems that would be in the future. .. ..

We don't plan on a sequel, but if you dare to give it a title next time ** Next time: "There are some performance failures and production failures." **

Performance testing is only a means of finding problems. The shortcut to a happy ending is for your boss to guide A's exam plan to a more appropriate one, and for A to involve the people involved in various problems, large and small.

A aside, I hope this article will increase the number of people who can lead a richer JMeter life.

Appendix

While writing the article, I will summarize the points that I thought "I may stumble on this point ..." in QA format. If you have any other concerns, please comment.

Q1 The set load amount is not reached

Mr. A "The setting is correct, but the expected load is not reached ... It is slightly below the set value ..." Boss "Specifications?"

In this verification, it was not a problem because only a small amount of load was applied, but the measured value is slightly below the setting from around several hundred RPS.

Since it is OSS, let's overlook it and set a slightly larger load on the assumption that it will be lower.

Q2 The set load is not reached at all. .. ..

Mr. A "I set the load amount equivalent to the actual production, but the measured value is significantly lower ... JMeter is also throwing an error ..." Boss "I can't say without seeing the error, but JMeter may have reached its limit."

JMeter runs on Java, so it consumes a lot of resources. First, try increasing the memory allocated to JMeter.

reference: What to do if OutOfMemory occurs in JMeter

If there is enough memory but the expected load is not reached, it is possible that the maximum number of OS connections has been reached. Let's increase the maximum number of file descriptors. (If you're not sure, ask the person in charge of the infrastructure to do it!)

reference: [Edit /etc/security/limits.conf instead of ulimit -n if you want to keep the settings after reboot / shutdown ("Too many open files" "Too many open files" error countermeasures)](https:: //qiita.com/neko_the_shadow/items/841bf59c4f80588baad7)

Q3 No matter how hard I try, the expected load does not come out. .. ..

Mr. A "I reviewed various settings, but the load equivalent to the actual production does not come out ..." Boss "Maybe it's the limit of the JMeter server ..."

In a large-scale system, there is a limit to the amount of load that can be output by one JMeter so that processing is distributed among multiple servers. (I can't say it unconditionally ...) Let's distribute JMeter's processing by one of the following methods.

Example: When processing is distributed to 10 JMeters

--Set the setting value of ThroughputShapingTimer to 1/10 and place it in 10 JMeters and execute it. --In addition to the above, make JMeter a Master-Slave configuration

reference: Building JMeter Master / Slave environment on AWS (CentOS 7) FireWall and KeyStore are miso

The advantage of the Master-Slave configuration is that the execution logs are aggregated in the Master, making it easier to create reports. However, the cost is a disadvantage because the number of server instances is increased by one.

at the end

It's been a long time, but this article is complete.

I realized that writing an article is quite difficult.

Until now, Qiita and others have helped me many times, but somehow I just skimmed it. From now on, I'd like to read the article while biting a little more.

This time I posted an article about performance testing, but my main business is customizing the Java framework. (Spring Framework, etc.) The Java framework has a lot of business-related content and is difficult to write articles, so I think it may be possible to take this opportunity to challenge fields that have not been touched up until now.

I'm thinking of making a web page with GitHub Pages and GitHub Actions. Foundation-> Java framework I have never touched the screen, so I will do my best not to disappear.

reference

ThrouputShapingTimer JMeter generating-dashboard