I did a load test on the smartphone game app, so I would like to share how it was done. This time, we conducted a load test assuming 100,000 DAUs.

Tools used

Load test client
- locust
Performance monitoring
New Relic (In addition to normal monitoring, you should include the MySQL Plugin)
AWS standard monitor

Load test with a simple configuration

First of all, the load test is performed with a simple configuration, including the meaning of communication confirmation. This time, I made an API that just echoes the passed json and conducted a load test using locust. If you do it with multiple units suddenly, you will not know where to adjust when performance is not achieved. First, let's see how well the Django framework can perform on the app server itself. (Originally, it's better to do it with plain Django, but this time it is in the form of adding API to Django of the application server we are developing)

As a result, Django's App server handled about 600 RPS with about 70% CPU usage. I think it's a good idea to compare it with the benchmark of Django itself and review whether it is extremely slow.

Another benefit of doing this test is to see if the load client locust itself can perform well.

If the simple configuration works, then create a test scenario.

Defining the user to test and creating a test scenario

When performing a load test, determine the expected user definitions and goals to be achieved.

I think it depends on the characteristics of the app, but this time we will define the users as ** heavy users **, ** general users **, ** new users ** as follows.

Heavy users: Users who are absorbed in the game
General users: Existing users who have played games but who play lightly
New user: A user who will install and play a new game on that day

Even if we assume 100,000 DAUs, the number and types of APIs that can be hit will differ depending on whether there are many new users or heavy users, and the load will be completely different. If there are many heavy users, the core part of the game, such as quests and PvP, will be played a lot. On the other hand, if there are many new users, the API of the registration process after signing up will be hit a lot.

Since we divided the users into distinctive segments, we defined what APIs will be hit next as follows.

Define the API that each segment hits
Define how much each segment hits its API
Calculate the number of APIs that one user in each segment hits per day (340 for heavy users in the figure)
Multiply 3 by the DAU of each segment, add them together, and get the number of daily requests that come to the system (13,510,000 in the figure). Average RPS can be calculated by 5.4
It is necessary for the system to issue 469 RPS (Request per Seconds), which is estimated to be about 3 times the peak value based on the experience values so far.

I actually created a scenario with locust and did a load test on what was defined above.

Load test on one server using a scenario

Suddenly, without load testing using multiple App servers, we will do it with only one. The aim is to confirm the communication of test scenarios and reduce costs. Suddenly setting up multiple App servers is not good because it costs money even during various trials and errors due to errors. Test the test scenario including MySQL and Redis with the following configuration.

Load test to withstand the expected load

Finally, set up the server as follows and perform a load test until the target RPS is achieved. If the load on the app server is high (CPU is 70% or more), increase the number of app servers or improve the specifications. The same applies to MySQL and Redis.

Summary

I wrote about how I did load testing in my project, but the points were as follows.

Increase from the minimum configuration
Create a test scenario from an assumed user